Singing recognition methods, devices, equipment and storage media

Abstract

This invention relates to a singing recognition method, apparatus, device, and storage medium. The method includes: acquiring a media stream; extracting audio information and image information from the media stream; obtaining one or more of the following probabilities based on the audio information: a speaking probability, a singing probability, and a background music probability; obtaining one or more of the following probabilities based on the image information: a face in the image, a mouth opening probability, and an instrument in the image; and obtaining a singing recognition result based on the audio information and the image information, including obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, face in the image, mouth opening probability, and instrument in the image. The singing recognition method of this invention utilizes multimodal information for real-time recognition, which can improve the recognition effect.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a singing recognition method, apparatus, device, and storage medium based on multimodal information. Background Technology

[0002] The statements herein are provided only as background information in relation to the present invention and do not necessarily constitute prior art.

[0003] In live streaming platforms, the content of a streamer's broadcast is highly variable. Streamers often play background music from singers in their broadcasts, and some streamers may even use external devices to input background music. Therefore, existing methods that rely on voice to identify whether a streamer is singing in real time are very difficult and inaccurate, especially in situations such as when background music is input from external devices, where the accuracy of real-time voice recognition is extremely low.

[0004] Currently, singing live streaming platforms mainly adopt the following solutions: 1. Tags filled in by the streamer; 2. Singing solely based on voice recognition. These solutions typically face the following problems: Problem 1: The tags filled in by the streamer are not easily changed, therefore they cannot reflect whether the streamer's current live content is singing, making real-time recognition and recommendation difficult; Problem 2: Relying solely on voice recognition for singing can lead to recognition failures due to external device access. Summary of the Invention

[0005] The purpose of this invention is to provide a new singing recognition method, apparatus, device, and storage medium to improve recognition performance by using multimodal information.

[0006] The objective of this invention is achieved using the following technical solution. A singing recognition method according to this invention includes the following steps: acquiring a media stream; extracting audio information and image information from the media stream; obtaining one or more of the following probabilities based on the audio information: speaking probability, singing probability, and background music probability; obtaining one or more of the following probabilities based on the image information: probability of a face in the image, probability of an open mouth, and probability of an instrument in the image; and obtaining a singing recognition result based on the audio information and the image information, including obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, and probability of an instrument in the image.

[0007] The objective of this invention is also achieved using the following technical solution. A singing recognition device according to this disclosure includes: an extraction module for acquiring a media stream and extracting audio information and image information from the media stream; a first probability determination module for obtaining one or more of the following probabilities based on the audio information: a speaking probability, a singing probability, and a background music probability; a second probability determination module for obtaining one or more of the following probabilities based on the image information: a face in the image, a mouth opening probability, and a musical instrument in the image; and a recognition module for obtaining a singing recognition result based on the audio information and the image information, specifically for obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, face in the image, mouth opening probability, and musical instrument in the image.

[0008] The objective of this invention is also achieved through the following technical solution. A singing recognition device according to this disclosure includes: a memory for storing non-transitory computer-readable instructions; and a processor for executing the computer-readable instructions, such that the processor, when executed, implements the aforementioned singing recognition method.

[0009] The objective of this invention is also achieved by the following technical solution. A computer-readable storage medium according to this disclosure is used to store non-transitory computer-readable instructions, which, when executed by a computer, cause the computer to perform the aforementioned singing recognition method.

[0010] Compared with the prior art, the present invention has significant advantages and beneficial effects. Through the above technical solution, the singing recognition method, apparatus, device, and storage medium proposed in this invention have at least the following advantages and beneficial effects:

[0011] 1. This invention integrates audio models, face detection models, mouth opening detection models, and musical instrument recognition models by employing a strategy that combines voice and image information, effectively improving accuracy and recall. Accuracy is the ratio of the number of correctly identified songs to the total number of samples, and recall is the ratio of the number of correctly identified songs to the total number of songs sung.

[0012] 2. The audio model, face detection model, lip feature point detection model, and musical instrument recognition model of the present invention can all be inferred in real time, thereby realizing real-time detection of live singing.

[0013] 3. This invention uses features based on the probability of a face in the image, the probability of an open mouth, and the probability of the presence or absence of musical instruments in the image to perform singing recognition, which can effectively perform recognition in complex situations such as when external devices are connected.

[0014] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of a singing recognition method according to an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of a system provided in one embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of an audio model according to an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of a face detection model according to an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of a lip feature point detection model according to an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of a musical instrument recognition model according to an embodiment of the present invention;

[0021] Figure 7 This is a schematic diagram of a singing recognition device according to an embodiment of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description, in conjunction with the accompanying drawings and preferred embodiments, describes the specific implementation methods, structures, features, and effects of the singing recognition method, apparatus, device, and storage medium proposed according to the present invention.

[0023] It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] The singing live stream mentioned in this article refers to a live stream where the host sings or sings along, and the host's face appears in the live stream camera.

[0025] Figure 1 This is a schematic flowchart of an embodiment of the singing recognition method of the present invention. Figure 2 This is a schematic diagram of a system provided according to one embodiment of the present invention. Please refer to other embodiments of the present invention. Figure 1 , Figure 2 The singing recognition method of this invention mainly includes the following steps:

[0026] Step S11: Obtain the media stream, extract audio and image information from the media stream, and use the audio and image information for singing recognition. Optionally, the media stream is a user-generated audio and video stream. Optionally, obtain the TS (Transport Stream) bitstream from the client and decode it into audio and screenshot information.

[0027] Step S12: Obtain one or more of the following probabilities based on the audio information: speaking probability, singing probability, and background music probability. The speaking probability can represent the probability that the audio in the media stream is a speaking voice or a speaking voice; the singing probability can represent the probability that the audio in the media stream is a singing voice or a singing voice; and the background music probability can represent the probability that the audio in the media stream is background music or background noise.

[0028] Step S13: Based on the image information, obtain one or more of the following probabilities: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image.

[0029] Step S14: Obtain the singing recognition result based on audio and image information, including obtaining the singing recognition result based on one or more of the following: speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, and probability of an instrument in the image. Optionally, a singing recognition score is determined as the singing recognition result based on at least one of the speaking probability, singing probability, and background music probability obtained from audio information, and at least one of the probabilities of a face in the image, probability of an open mouth, and probability of an instrument in the image obtained from image information. Optionally, the singing recognition result is obtained simultaneously based on the speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, and probability of an instrument in the image.

[0030] The singing recognition method proposed in this invention can improve the recognition effect by performing real-time recognition based on multimodal information.

[0031] In some embodiments of the present invention, the aforementioned step S12 specifically includes: inputting audio information into a pre-trained audio recognition model to obtain one or more of the following in real time: speaking score, singing score, and background music score, as the aforementioned speaking probability, singing probability, and background music probability. Optionally, the score is a decimal between 0 and 1, with a higher score indicating a higher probability.

[0032] In some embodiments of the present invention, step S13 specifically includes: inputting image information into one or more of a pre-trained face detection model, lip feature point detection model (also known as a mouth opening detection model), and musical instrument recognition model, and obtaining one or more corresponding scores for the presence of a face, a mouth opening, and a musical instrument in the image in real time, which are used as the probabilities of a face, a mouth opening, and a musical instrument in the image. Optionally, the score is a decimal between 0 and 1, with a higher score indicating a higher probability.

[0033] In some embodiments of the present invention, step S14 specifically includes: using one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the scene, probability of an open mouth, and probability of an instrument in the scene as recognition features; inputting the recognition features into a pre-trained decision tree model in real time to obtain a singing recognition score, which is the result of singing recognition. Optionally, if the singing recognition score is higher than a threshold, it indicates that singing is taking place; otherwise, it indicates that singing is not taking place.

[0034] Optionally, the aggregated features x = [probability of speaking, probability of singing, probability of background music, probability of a face in the scene, probability of an open mouth, probability of musical instruments in the scene] are used as input features into the decision tree model.

[0035] It should be noted that during the training process of the decision tree model, it also utilizes recognition features obtained or composed of one or more of the following: speaking probability, singing probability, background music probability, probability of a face in the scene, probability of an open mouth, and probability of musical instruments in the scene. These features are then trained on the training set.

[0036] In some embodiments of the present invention, during the training of the decision tree model, the model parameters are optimized by solving the following optimization problem: minimizing the sum of squares between the predicted singing probability of a sample based on its features and the actual singing situation recorded by the sample label. Specifically, the actual singing situation recorded by the sample label can be represented by the sample label being 0 or 1, indicating whether the sample is not singing or is singing.

[0037] In a specific embodiment of the present invention, during the training process of the decision tree model, the above optimization problem is solved using the following formula:

[0038]

[0039]

[0040] Among them, X i = (x1, x2, ..., x6) represents the recognition features of the i-th sample; x1 is the probability of speaking, x2 is the probability of singing, x3 is the probability of background music, x4 is the probability of a face in the image, x5 is the probability of an open mouth, and x6 is the probability of a musical instrument in the image; y i Let y be the label of the i-th sample. i =1 indicates that the label records the actual situation of the corresponding sample as singing, y i =0 indicates that the label records the actual situation of the corresponding sample as not singing; The probability of whether singing is taking place, predicted by the model, is typically a decimal between 0 and 1; V = (X 1 X 2 , ..., X m ), where m is the number of samples.

[0041] It should be noted that each time slot of the media stream constitutes a sample. A single broadcaster can correspond to multiple time slots of the media stream, thus corresponding to multiple samples. The same broadcaster may be singing or not singing in different time slots.

[0042] In some embodiments of the present invention, the aforementioned step S14, which involves inputting the recognition features into a pre-trained decision tree model in real time to obtain the singing recognition result, specifically includes:

[0043] The identification features are input into a pre-trained decision tree model in real time to obtain the probability of singing at the current moment. For example, the model outputs a score indicating that the singer is singing at the current moment. The higher the score, the greater the probability of singing at the current moment. Then, it is determined whether the probability of singing at the current moment is greater than a first threshold.

[0044] Take a time period before the current moment as the current time period (e.g., within 10 minutes before the current moment). The current time period contains multiple moments. Determine whether the probability of singing at multiple moments within the current time period exceeds the second threshold. The ratio of the number of moments exceeding the second threshold to the total number of moments within the current time period is taken as the singing percentage of the current time period.

[0045] If the probability of singing at the current moment is greater than the first threshold, and / or the proportion of singing in the current time period is greater than the third threshold, then it is determined that singing is taking place.

[0046] Optionally, if it is determined that the streamer is singing, the recommendation will be given higher priority and the distribution will be increased. For example, a separate tag will be created for the streamer who is singing to increase the streamer's exposure.

[0047] It should be noted that, optionally, the second threshold can be equal to the first threshold, or the second threshold can be different from the first threshold, for example, slightly smaller than the first threshold.

[0048] As a specific example of the aforementioned singing percentage: select 6 moments within a one-minute time period and determine the corresponding 6 scores. Count the number of times a score is greater than 0.66, s. If s > 2, then the one minute is considered to be during singing; where, the singing percentage is equivalent to s / 6 > 2 / 6.

[0049] In some embodiments of the present invention, the probability of a musical instrument appearing in the scene includes the probability of a person in the scene holding a musical instrument and / or a microphone, that is, the probability of operating the instrument and the probability of opening their mouth to the microphone. The step S13 above, which involves determining the probability of a musical instrument appearing in the scene based on image information, includes determining the relative position of the musical instrument and / or microphone in the scene to the person in the scene based on the image information, and determining the probability of the person holding the musical instrument and / or microphone based on the relative position. Specifically, the positions of the musical instrument and the person can be determined separately first, and then the relative position of the musical instrument and the person can be determined, or the relative position of the musical instrument and the person can be determined directly. The present invention improves the accuracy of singing recognition by utilizing the probability of a person holding a musical instrument in the scene.

[0050] In some embodiments of the present invention, the singing recognition method of the present invention further includes: before obtaining the singing recognition result based on audio information and image information, determining the types of instruments in the background music based on the audio information, and determining the types of instruments in the image based on the image information. The aforementioned step S14, obtaining the singing recognition result based on audio information and image information, further includes: matching the types of instruments in the background music with the types of instruments in the image to obtain an instrument matching result, and obtaining the singing recognition result based on one or more of the following: speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, probability of an instrument in the image, and instrument matching result.

[0051] Optionally, the aforementioned determination of instrument types in background music based on audio information and the aforementioned determination of instrument types in images based on image information include: using pre-trained audio- and image-based instrument classification models to obtain audio- and image-based instrument classification results in real time. The aforementioned matching of instrument types in background music with instrument types in images to obtain instrument matching results specifically includes: inputting the audio- and image-based instrument classification results into a pre-trained machine learning model (e.g., a neural network model) to obtain an instrument matching score, which serves as the instrument matching result. This instrument matching score indicates the degree of matching between the instrument types in the background music and the instrument types in the images; for example, a higher score indicates a higher degree of matching.

[0052] In some embodiments of the present invention, the singing recognition method further includes: after acquiring the media stream, identifying multiple individuals in the media stream based on audio information and / or image information; and performing the aforementioned steps S12 to S14 on each identified individual to obtain a singing recognition result corresponding to each individual. Specifically, based on the audio information, one or more of the following probabilities are obtained: the probability of speaking, the probability of singing, and the probability of background music corresponding to each identified individual; based on the image information, one or more of the following probabilities are obtained: the probability of a face in the image, the probability of an open mouth, and the probability of an instrument in the image corresponding to each identified individual; and the singing recognition result corresponding to each identified individual is obtained using the aforementioned step S14.

[0053] The singing recognition method of this invention improves recognition performance by performing real-time recognition based on multimodal information. Specifically, this invention provides a real-time singing live stream detection system and method based on multimodal information: it utilizes an audio recognition model, a face detection model, a lip feature point detection model (also known as a mouth opening detection model), and an instrument recognition model to detect in real-time the probability of a user speaking, singing, background music, the presence of a face in the frame, the probability of a mouth opening, and the probability of the presence or absence of musical instruments in the frame. These probabilities are then used as features to train a decision tree model on a training set to detect singing live streams in real-time, thereby improving recognition performance.

[0054] It should be noted that the singing recognition method of this invention can be used for tasks such as real-time recognition of singing anchors and real-time recognition of entertainment anchors.

[0055] The singing recognition method of this invention can be used to identify whether a streamer is singing in a live video broadcast, thereby creating a unique tag for the singing streamer to increase their exposure. In this case, the corresponding product may require both high recall and high precision, for example, a precision of over 80% and a recall of over 60%. Existing methods cannot simultaneously meet business needs in terms of recall and precision, while this invention, by using multimodal information for singing recognition, can achieve higher precision and recall. For example, compared to relying solely on a speech model, the recall rate is increased from 52% to 85% while maintaining an accuracy of 84.7%.

[0056] In some embodiments of the present invention, the singing recognition method and corresponding system of the present invention utilize one audio model, three image models, and one decision tree model, as detailed below.

[0057] I. Audio Model:

[0058] An audio data segment spanning the current moment and the preceding 10 seconds is sliced ​​into 400 frames, each lasting 25 milliseconds. 128-dimensional Mel-frequency cepstral coefficients are extracted from each frame as features, resulting in a 400*128 dimensional matrix corresponding to this audio data segment. This matrix is ​​then input into the first neural network, net1, to obtain the probabilities of speaking, singing, and background music. It should be noted that this invention does not limit the duration of the audio data, the number of slices, the dimension of the extracted Mel-frequency cepstral coefficients, or the dimension of the corresponding matrix. For example, the specific values ​​mentioned above, such as extracting 400 frames from an audio data segment and extracting 128-dimensional Mel-frequency cepstral coefficients from each slice, are merely empirical values ​​and can be varied.

[0059] In some examples of the present invention, such as Figure 3 As shown, the audio model uses the structure of a first neural network net1 containing three binary classifications.

[0060] In a specific example, if the input of the current neural network block (i.e., the output of the previous layer's neural network block) is w*h*c dimensional (specifically, in the aforementioned example, the input of the first layer's block is 400*128*1 dimensional), then the output obtained by the current neural network block is... Dimensionality. Optionally, the processing of each neural network block may include performing a 1*1*4c convolution, followed by a 3*3 convolution with stride=2 (i.e., a stride of 2), and then a 1*1*2c convolution, thereby halving w and h and doubling c for each block in the first neural network net1. It should be noted that this invention does not limit the specific values ​​of w, h, and c.

[0061] It should be noted that this invention does not limit the specific structure of the audio model; for example, it can be as follows: Figure 3 The example shown uses one model to simultaneously output the probability of speaking, singing, and background music. Alternatively, three models can be used to output the probability of speaking, singing, and background music separately.

[0062] II. Image Model:

[0063] The image screenshot is scaled to 256*128*3 pixels, and then processed through a face detection model, a mouth opening detection model, and a musical instrument recognition model to obtain the probabilities of a face, a mouth opening, and a musical instrument in the image. It should be noted that this invention does not limit the size of the scaled image; the specific values ​​mentioned above are merely illustrative. In reality, mobile phones occupy a very large portion of the screen during live streaming, and for convenience, images are usually adjusted to 256*128*3 pixels.

[0064] In some examples of the present invention, such as Figure 4 , Figure 5 , Figure 6 As shown, the image model includes three binary classification neural network structures: the second neural network net2 used by the face detection model, the third neural network net3 used by the mouth opening detection model, and the fourth neural network net4 used by the musical instrument recognition model.

[0065] It should be noted that this invention does not limit the specific structure of the audio model and the image model; for example, it can use a more complex design. Figures 3-6 The diagram shows a more complex neural network structure.

[0066] III. Decision Tree Model:

[0067] The integrated modeling and solution process for the features of speaking probability, singing probability, background music probability, probability of a face in the scene, probability of an open mouth, and probability of musical instruments in the scene is as follows:

[0068] x1: Probability of speaking; x2: Probability of singing; x3: Probability of background music; x4: Probability of a face in the scene; x5: Probability of an open mouth; x6: Probability of no musical instrument in the scene; X = (x1, x2, ..., x6), y i Let y be the tag of the i-th streamer. i =1 indicates that the streamer is currently live-streaming singing, y i =0 indicates that the streamer is currently doing a non-singing live stream, V=(X 1 X 2 , ..., X m ), where m is the number of samples.

[0069] The optimization problem model corresponding to this problem is as follows:

[0070]

[0071]

[0072] We use a decision tree model to solve this optimization problem.

[0073] It should be noted that this invention does not limit the solution method for multi-feature models. In addition to the optimization problem model in this example, other algorithms such as xgb (extreme gradient boosting) can also be used.

[0074] Embodiments of the present invention also provide a singing recognition device, which mainly includes: an extraction module, a first probability determination module, a second probability determination module, and a recognition module.

[0075] The extraction module is used to: acquire media streams and extract audio and image information from the media streams.

[0076] The first probability determination module is used to: obtain one or more of the following probabilities based on audio information: speaking probability, singing probability, and background music probability.

[0077] The second probability determination module is used to: obtain one or more of the following probabilities based on image information: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image.

[0078] This recognition module is used to obtain singing recognition results based on audio and image information, and specifically to obtain singing recognition results based on one or more of the following: speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, and probability of an instrument in the image.

[0079] In some embodiments of the present invention, the first probability determination module is specifically used to: input audio information into a pre-trained audio recognition model to obtain one or more of the following probabilities in real time: speaking probability, singing probability, and background music probability. The second probability determination module is specifically used to: input image information into one or more of a pre-trained face detection model, lip feature point detection model, and musical instrument recognition model to obtain one or more of the following probabilities in real time: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image. The recognition module is specifically used to: use one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, and probability of a musical instrument in the image as recognition features, and input the recognition features in real time into a pre-trained decision tree model to obtain the singing recognition result.

[0080] Optionally, the second probability determination module includes a face probability determination submodule, a mouth opening probability determination submodule, and an instrument probability determination submodule, which are used to obtain the probability of a face in the image, the probability of a mouth opening, and the probability of an instrument in the image in real time, respectively.

[0081] In some embodiments of the present invention, the recognition module is specifically used to: input the recognition features into a pre-trained decision tree model in real time to obtain the probability of singing at the current moment, and determine whether the probability of singing at the current moment is greater than a first threshold; take a time period before the current moment as the current time period, determine whether the probability of singing at multiple moments within the current time period exceeds a second threshold, and take the ratio of the number of moments exceeding the second threshold within the current time period to the total number of moments within the current time period as the proportion of singing in the current time period; if the probability of singing at the current moment is greater than the first threshold and / or the proportion of singing in the current time period is greater than a third threshold, then it is determined that singing is taking place.

[0082] In some embodiments of the present invention, the singing recognition device further includes a model training module. This model training module is used to optimize the model parameters during the training of the decision tree model by solving the following optimization problem: minimizing the sum of squares between the predicted singing probability of a sample based on its features and the actual singing situation recorded by the sample label.

[0083] In some embodiments of the present invention, the probability of a musical instrument in the image includes the probability of a person in the image holding a musical instrument and / or a microphone. Furthermore, the second probability determination module is specifically used to: determine the relative position of the musical instrument and / or microphone in the image to the person in the image based on image information, and determine the probability that the person is holding a musical instrument and / or a microphone based on the relative position.

[0084] In some embodiments of the present invention, the first probability determination module is further configured to: determine the type of musical instrument in the background music based on audio information. The second probability determination module is further configured to: determine the type of musical instrument in the image based on image information. The singing recognition device of the present invention also includes an instrument matching module, configured to match the type of musical instrument in the background music with the type of musical instrument in the image to obtain an instrument matching result. Specifically, this recognition module is configured to: obtain the singing recognition result based on one or more of the following: speaking probability, singing probability, background music probability, probability of a face in the image, probability of an open mouth, probability of musical instruments in the image, and instrument matching result.

[0085] In some embodiments of the present invention, the singing recognition device of the present invention further includes a person determination module, used to identify multiple persons in a media stream based on audio information and / or image information. The first probability determination module is specifically used to: obtain one or more of the following probabilities corresponding to each identified person: speaking probability, singing probability, and background music probability, based on the audio information. The second probability determination module is specifically used to: obtain one or more of the following probabilities corresponding to each identified person: a face in the image, an open mouth probability, and an instrument in the image, based on the image information. The recognition module is specifically used to: obtain the singing recognition result corresponding to each identified person.

[0086] Furthermore, the various singing recognition devices shown in the embodiments of the present invention include modules and units for performing the methods described in the foregoing embodiments, and their detailed descriptions and technical effects can be found in the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0087] Figure 7 This is a schematic block diagram illustrating a singing recognition device according to an embodiment of the present invention. Figure 7 As shown, the singing recognition device 100 according to an embodiment of the present disclosure includes a memory 101 and a processor 102.

[0088] The memory 101 is used to store non-transitory computer-readable instructions. Specifically, the memory 101 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0089] The processor 102 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the singing recognition device 100 to perform desired functions. In one embodiment of this disclosure, the processor 102 is used to execute the computer-readable instructions stored in the memory 101, causing the singing recognition device 100 to perform all or part of the steps of the singing recognition methods of the foregoing embodiments of this disclosure.

[0090] For detailed descriptions and technical effects of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0091] Embodiments of the present invention also provide a computer storage medium storing computer instructions. When the computer instructions are executed on a device, the device performs the aforementioned method steps to implement the singing recognition method in the above embodiments.

[0092] Embodiments of the present invention also provide a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the singing recognition method in the above embodiments.

[0093] In addition, embodiments of the present invention also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the singing recognition method in the above-described method embodiments.

[0094] The apparatus, computer storage medium, computer program product or chip provided by the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A singing recognition method, characterized in that, The method includes the following steps: Acquire a media stream, and extract audio and image information from the media stream; Based on the audio information, one or more of the following probabilities are obtained: speaking probability, singing probability, and background music probability. Based on the image information, obtain one or more of the following probabilities: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image; The singing recognition result is obtained based on the audio information and the image information, including obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, and one or more of the probability of a face in the picture, the probability of an open mouth, and the probability of an instrument in the picture. Specifically, obtaining one or more of the speaking probability, singing probability, and background music probability based on the audio information includes: inputting the audio information into a pre-trained audio recognition model to obtain one or more of the speaking probability, singing probability, and background music probability in real time. The method of obtaining one or more of the probabilities of a face in the image, an open mouth, and a musical instrument in the image based on the image information specifically includes: inputting the image information into one or more of a pre-trained face detection model, lip feature point detection model, and musical instrument recognition model, and obtaining one or more of the corresponding probabilities of a face in the image, an open mouth, and a musical instrument in the image in real time. The method of obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, and one or more of the probability of a face in the scene, the probability of an open mouth, and the probability of an instrument in the scene specifically includes: using one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the scene, the probability of an open mouth, and the probability of an instrument in the scene as recognition features, and inputting the recognition features into a pre-trained decision tree model in real time to obtain the singing recognition result.

2. The singing recognition method according to claim 1, characterized in that, The process of inputting the recognition features into a pre-trained decision tree model in real time to obtain the singing recognition result specifically includes: The identification features are input into a pre-trained decision tree model in real time to obtain the probability of singing at the current moment, and to determine whether the probability of singing at the current moment is greater than the first threshold. Take the time period before the current moment as the current time period, determine whether the probability of singing at multiple moments within the current time period exceeds a second threshold, and take the ratio of the number of moments exceeding the second threshold within the current time period to the total number of moments within the current time period as the singing percentage of the current time period. If the probability of singing at the current moment is greater than the first threshold, and / or the proportion of singing in the current time period is greater than the third threshold, then it is determined that singing is taking place.

3. The singing recognition method according to claim 1, characterized in that, During the training process of the decision tree model, the model parameters are optimized by solving the following optimization problem: This minimizes the sum of squares between the predicted singing probability of the sample based on its features and the actual singing situation recorded by the sample label. The optimization problem is solved using the following formula: in, Let i be the identification feature of the i-th sample; x 1 represents the speaking probability. x 2 represents the probability of the singing voice. x 3 represents the probability of the background music. x 4 represents the probability that there is a face in the image. x 5 represents the probability of opening the mouth. x 6 represents the probability that the image contains musical instruments; Let be the label of the i-th sample. This indicates that the sample corresponding to the label is actually singing. This indicates that the sample corresponding to the label is actually performing something other than singing. The probability of whether or not singing is being predicted by the model; , where m is the number of samples.

4. The singing recognition method according to claim 1, characterized in that: The probability that there is a musical instrument in the scene includes the probability that a person in the scene is holding a musical instrument and / or a microphone. The step of obtaining the probability of a musical instrument in the image based on the image information includes determining the relative position of the musical instrument and / or microphone in the image to the person in the image based on the image information, and determining the probability that the person is holding the musical instrument and / or microphone based on the relative position.

5. The singing recognition method according to claim 1, characterized in that: The method further includes: determining the types of musical instruments in the background music based on the audio information, and determining the types of musical instruments in the image based on the image information; The method of obtaining the singing recognition result based on the audio information and the image information further includes: matching the types of instruments in the background music with the types of instruments in the image to obtain an instrument matching result, and obtaining the singing recognition result based on one or more of the speaking probability, the singing probability, the background music probability, the probability of a face in the image, the probability of an open mouth, the probability of an instrument in the image, and the instrument matching result.

6. The singing recognition method according to claim 1, characterized in that, The method further includes: after the step of acquiring the media stream, identifying multiple people in the media stream based on the audio information and / or the image information; The step of obtaining one or more of the speaking probability, singing probability, and background music probability based on the audio information specifically includes: obtaining one or more of the speaking probability, singing probability, and background music probability corresponding to each identified person based on the audio information. The step of obtaining one or more of the following probabilities based on the image information: the probability of a face in the image, the probability of an open mouth, and the probability of musical instruments in the image; specifically includes obtaining one or more of the following probabilities based on the image information: the probability of a face in the image, the probability of an open mouth, and the probability of musical instruments in the image corresponding to each identified person. The process of obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the picture, probability of an open mouth, and probability of an instrument in the picture specifically includes obtaining the singing recognition result corresponding to each identified person.

7. A singing recognition device, characterized in that, The device includes: The extraction module is used to acquire a media stream and extract audio and image information from the media stream; The first probability determination module is used to obtain one or more of the speaking probability, singing probability, and background music probability based on the audio information. The second probability determination module is used to determine one or more of the following probabilities based on the image information: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image. The recognition module is used to obtain the singing recognition result based on the audio information and the image information. Specifically, it is used to obtain the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, and one or more of the probability of a face in the image, the probability of an open mouth, and the probability of an instrument in the image. Specifically, the first probability determination module is used to obtain one or more of the speaking probability, singing probability, and background music probability based on the audio information, including: inputting the audio information into a pre-trained audio recognition model to obtain one or more of the speaking probability, singing probability, and background music probability in real time. The second probability determination module is used to obtain one or more of the following probabilities based on the image information: the probability of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image. Specifically, it includes: inputting the image information into one or more of a pre-trained face detection model, lip feature point detection model, and musical instrument recognition model, and obtaining one or more of the corresponding probabilities of a face in the image, the probability of an open mouth, and the probability of a musical instrument in the image in real time. The method of obtaining the singing recognition result based on one or more of the obtained speaking probability, singing probability, background music probability, and one or more of the probability of a face in the scene, the probability of an open mouth, and the probability of an instrument in the scene specifically includes: using one or more of the obtained speaking probability, singing probability, background music probability, probability of a face in the scene, the probability of an open mouth, and the probability of an instrument in the scene as recognition features, and inputting the recognition features into a pre-trained decision tree model in real time to obtain the singing recognition result.

8. A singing recognition device, comprising: Memory is used to store non-transitory computer-readable instructions; as well as A processor for executing the computer-readable instructions such that, when executed by the processor, the computer-readable instructions implement the singing recognition method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that, Includes computer instructions that, when executed on the device, cause the device to perform the singing recognition method as described in any one of claims 1 to 6.

Images