Music generation method and device, electronic equipment and storage medium
By analyzing user behavior data and using music tag matching models and generation models to automatically match and generate music, the problem of music recommendation in the existing technology relying on user tag selection is solved, and a more accurate and automated music recommendation process is achieved.
Patent Information
- Application Number
- CN202411213571.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing secondary audio and video creation platforms, music recommendations rely on users to choose tags themselves, and accuracy depends on the quality of tags selected by users, and lack automation and accuracy.
By determining candidate image or video data and user behavior data, using music tag matching models and generation models, music data that matches users' preferences are automatically matched and generated without the need for the user to manually select tags.
It realizes that music data meets user preferences automatically generates music data without the need for user selection, improving the accuracy and automation of music recommendations.
Smart Images

Figure CN118945441A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio and video processing, and in particular to a method and device for generating music, an electronic device, and a storage medium. Background Art
[0002] Currently, social platforms can provide users with some audio and video secondary creation functions. For example, users can enter tags corresponding to the type of music they want to add when recreating audio and video, and the social platform can recommend suitable music to users based on the tags entered by users. However, this type of music recommendation method also has defects, such as requiring users to select tags themselves, and the accuracy of the recommendation depends on the tags selected by users. Summary of the invention
[0003] In view of the above problems, a method and device for music generation, an electronic device, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems.
[0004] A first aspect of the present invention provides a method for generating music, the method comprising:
[0005] Determining candidate image data or candidate video data;
[0006] Acquire user behavior data associated with the candidate image data or candidate video data;
[0007] Determining target image data or target video data based on the user behavior data;
[0008] Inputting the target picture data or the target video data into a music tag matching model, wherein the music tag matching model determines and outputs target tag information matching the target picture data or the target video data from a plurality of preset music tag information;
[0009] The target tag information is input into a music generation model to obtain music data output by the music generation model that matches the target tag information, as music data that matches the target picture data or the target video data.
[0010] A second aspect of the present invention provides a device for music generation, the device comprising:
[0011] A candidate data determination module, used to determine candidate image data or candidate video data;
[0012] A user behavior data acquisition module, used to acquire user behavior data associated with the candidate image data or candidate video data;
[0013] A target data screening module, used to determine target image data or target video data based on the user behavior data;
[0014] A label information determination module, used for inputting the target picture data or the target video data into a music label matching model, wherein the music label matching model determines and outputs target label information matching the target picture data or the target video data from a plurality of preset music label information;
[0015] The music data determination module is used to input the target tag information into the music generation model, and obtain the music data output by the music generation model that matches the target tag information, so as to serve as the music data that matches the target picture data or the target video data.
[0016] A third aspect of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method for generating music as described above when executed by the processor.
[0017] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for generating music as described above is implemented.
[0018] A fifth aspect of the present invention provides a computer product, comprising a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the method for generating music as described in any of the above items.
[0019] The embodiments of the present invention have the following advantages:
[0020] In an embodiment of the present invention, candidate picture data or candidate video data can be determined; user behavior data associated with the candidate picture data or candidate video data can be obtained; based on the user behavior data, target picture data or target video data can be determined; the target picture data or target video data can be input into a music tag matching model, and the music tag matching model determines and outputs target tag information that matches the target picture data or target video data from a plurality of preset music tag information; the target tag information is input into a music generation model, and music data that matches the target tag information output by the music generation model is obtained as music data that matches the target picture data or target video data, thereby automatically generating music data that meets the user's preferences without the user selecting a music tag, thereby achieving accurate music recommendations. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0022] Figure 1a is a flowchart of a method for generating music provided by an embodiment of the present invention;
[0023] Figure 1b is a schematic diagram of training steps of a music tag matching model provided by an embodiment of the present invention;
[0024] Figure 1c is a flow chart of steps for determining target picture data or target video data provided by one embodiment of the present invention;
[0025] Figure 2 is a flowchart of another method for generating music provided by an embodiment of the present invention;
[0026] Figure 3 A music generation system is provided by an embodiment of the present invention;
[0027] Figures 4a to 4f It is a structural diagram of a music generation device and substructures provided by an embodiment of the present invention;
[0028] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] Reference Figure 1a , shows a flowchart of a method for generating music provided by an embodiment of the present invention, which may specifically include the following steps:
[0031] Step 101, determining candidate picture data or candidate video data;
[0032] Among them, the candidate image data or candidate video data can be the image data or video data browsed or played by the users of the video website in the recent preset time period. The preset time period can be set according to the actual scenario, such as the recent day, the recent week, or the recent month. There is no limitation on this in the embodiment of the present invention.
[0033] In practical applications, candidate image data or candidate video data may be collected on a video website.
[0034] In one example, when a user has a demand for music recommendation, the music generation method in an embodiment of the present invention can be triggered to generate music data that can be recommended to the user.
[0035] For example, when it is detected that a user is editing a status on a video website, it is determined that the user has a need for audio recommendations.
[0036] For another example, when it is detected that the number of times a user inputs a music tag within a preset time period is greater than a preset number, the music generation method in the embodiment of the present invention can be triggered to generate music data that can be recommended to the user.
[0037] Step 102, obtaining user behavior data associated with the candidate image data or the candidate video data;
[0038] User behavior data is data associated with pictures or videos. For example, user behavior data may include but is not limited to video or picture browsing data, video click data, viewing time data, viewing category data (variety shows, movies, TV series, short videos, etc.).
[0039] Step 103, determining target image data or target video data based on the user behavior data;
[0040] After obtaining a plurality of user behavior data, analysis may be performed based on the user behavior data to determine one or more target image data or target video data that can be used to represent the user's behavior habit preference.
[0041] Step 104, inputting the target picture data or the target video data into the music tag matching model, and the music tag matching model determines and outputs the target tag information matching the target picture data or the target video data from among the preset multiple music tag information;
[0042] In an embodiment of the present invention, different music tags can be pre-set, such as emotion tags: happy, sad, melancholy, etc., style tags: pop music, rock, folk, jazz, classical music, electronic music, blues, country music, etc.; theme tags: life, freedom, dream, inspiration, love, friendship, family, etc.
[0043] After determining the target image data or the target video data, the music tag matching the target image data or the target video data may be determined. It should be noted that the target tag information in the embodiment of the present invention may include one or more music tags.
[0044] Specifically, the image type or video type corresponding to the target image data or the target video data can be determined, and then the image type or video type is matched with the set multiple music tags to obtain the target tag information. The video types of the video data can include but are not limited to sweet love dramas, historical dramas, suspense dramas, martial arts dramas, life dramas, etc.
[0045] For example, when the video data is of the sweet love drama type, the emotion tag that can be matched is the happiness tag, and the theme tag that can be matched is the love tag.
[0046] In the embodiment of the present invention, refer to Figure 1b As shown in Figure 1, the training process of the music tag matching model includes the following steps:
[0047] Step S11, obtaining sample picture data or sample video data for training a music label matching model, and annotating the sample picture data or sample video data with music labels to obtain sample music labels corresponding to the sample picture data or sample video data;
[0048] In practical applications, sample image data and sample video data can be collected from various websites, and sample music tags can be annotated for the collected sample image data and sample video data.
[0049] Step S12, inputting the sample picture data or sample video data into the initial music label matching model, and outputting the predicted music label corresponding to the sample picture data or sample video data;
[0050] After obtaining sample image data or sample video data, an initial music label matching model may be input, so that the initial music label matching model performs prediction based on the input data to obtain a predicted music label.
[0051] Among them, the initial music tag matching model can adopt a convolutional neural network (CNN) model.
[0052] Step S13, adjusting the target model parameters of the initial music label matching model based on the sample music label and the predicted music label;
[0053] The predicted music label is compared with the actual sample music label, and then the target model parameters of the initial music label matching model are determined. The target model parameters are such that the predicted music label predicted by the adjusted music label matching model is infinitely close to the sample music label.
[0054] Step S14, when the current model training meets the preset model training termination condition, output the music label matching model corresponding to the target model parameters;
[0055] The preset model training termination condition may be the number of iterations or the loss value of each iteration.
[0056] For example, when the number of model iterations is greater than or equal to a preset number, it is determined that the current model training meets the preset model training terminal conditions, and the music label matching model is output according to the target model parameters obtained through training. For another example, when the loss value is less than a preset threshold, it is determined that the current model converges, that is, the current model training meets the preset model training terminal conditions, and the music label matching model is output according to the target model parameters obtained through training.
[0057] Step S15, when the current model training does not meet the preset model training termination conditions, training is performed based on the music tag matching model corresponding to the target model parameters.
[0058] When the current model training does not meet the preset model training termination conditions, the model can continue to be iteratively trained.
[0059] Step 105, input the target tag information into the music generation model, and obtain the music data output by the music generation model that matches the target tag information as the music data that matches the target picture data or the target video data.
[0060] The input data of the music generation model is label information, and the output data is music data. Through the training of the music generation model, the model can learn the relationship between the label information of video or image data and music.
[0061] After determining the target tag information, the preset interface is called to convert the target tag information into a target instruction, which is input into the music generation model corresponding to the preset interface to generate music data matching the target tag information, wherein the preset interface is used to convert the target tag data into instructions that can be recognized and executed by the music generation model, and then imported into the music generation model. In the music generation model, music data matching the target tag information can be generated, wherein the music data can be existing music, secondary music generated based on existing music, or completely original music.
[0062] In an embodiment of the present invention, the music generation model can select or combine music clips from an existing music library according to the target tag information to generate music data that matches the tag. The music generation model can also recreate based on existing music to generate new music works while maintaining matching with the target tag information. The music generation model can generate a completely new music work from scratch according to the target tag information without relying on any existing music.
[0063] For example, when the target tag information is pop music, a piece of music in the pop music style is randomly generated; when the target tag information is a happy tag, a piece of music expressing a happy emotion is randomly generated.
[0064] In an embodiment of the present invention, the music generation model may be a pre-trained model, and may generate music data based on target tag information input by a preset interface. For example, the music generation model may be a music generation model in OpenAI (Open Artificial Intelligence), and the preset interface is an OpenAI interface, specifically a v1 / completions OpenAI interface.
[0065] In one embodiment of the present invention, referring to Figure 1c As shown, step 102 includes the following sub-steps:
[0066] Sub-step S21, determining multiple candidate picture data or multiple candidate video data associated with multiple user behavior data;
[0067] The candidate image data or candidate video data associated with the user behavior data is the image data or video data targeted by the user behavior.
[0068] For example, the candidate image data or candidate video data associated with the user's browsing behavior is the video data or image data browsed by the user; the candidate video data associated with the user's playing behavior is the video played by the user.
[0069] Sub-step S22, obtaining first weight information of each user behavior data;
[0070] Different weights can be pre-set for different types of user behavior data, that is, the first weight information of the user behavior data. The first weight is used to reflect the degree of influence of the user behavior data on the picture or video. It should be noted that in an embodiment of the present invention, the weight of each user behavior data can be set according to actual needs. When the user behavior data has a great influence on the picture or video, a larger weight can be set for the user behavior data, and when the user behavior data has a small influence on the picture or video, a smaller weight can be set for the user behavior data. For example, the weight of the user's browsing behavior can be smaller than the weight of the user's playing behavior.
[0071] Sub-step S23, determining second weight information of each candidate picture data or each candidate video data based on the first weight information;
[0072] In an embodiment of the present invention, each candidate image data may simultaneously involve one or more user behavior data. For example, user A may browse comments related to video 1 and play video 1. Thus, video 1 may simultaneously involve user browsing behavior and user playing behavior.
[0073] In the case where the first weight of each user behavior data has been determined, the second weight information of each candidate picture data or candidate video data can be determined based on the user behavior data involved in the candidate picture data or candidate video data. The second weight information is used to reflect the user's attention to different picture data or video data. The higher the user's second weight information for the picture data or video data, the higher the user's attention to the picture data or video data, and the more the picture data or video data can reflect the user's preference habits.
[0074] In one embodiment of the present invention, sub-step S23 may specifically be: weighting the user behavior data of each candidate image data or each candidate video data according to the first weight information corresponding to the user behavior data to obtain the second weight information of each candidate image data or each candidate video data.
[0075] That is, the second weight information of the candidate picture data or the candidate video data=user behavior data 1^first weight 1+...user behavior data n^first weight n, (n is a positive integer greater than 1).
[0076] Among them, user behavior data 1 to user behavior data n are one or more user behavior data associated with candidate image data or candidate video data, the first weight 1 is the first weight information corresponding to user behavior data 1, and the first weight n is the first weight information corresponding to user behavior data n.
[0077] For example, user A clicks on video 2 5 times in one day, and the playback time of video 2 is 120 minutes. If the first weight information of the user's click behavior data is set to 0.1, the first weight information of the playback time data is set to 0.2, and the playback time data is calculated in units of 60 minutes, then the second weight information of video 2 = 0.1*5+(120 / 60*0.2) = 0.9.
[0078] Sub-step S24, determining target picture data or target video data from a plurality of candidate picture data or a plurality of candidate video data according to the second weight information.
[0079] After the second weight information is determined, the candidate picture data or the candidate video data may be screened according to the second weight data to determine the target picture data or the target video data.
[0080] Specifically, sub-step S14 may specifically be: taking a preset number of candidate picture data or candidate video data from high to low according to the second weight information as target picture data or target video data.
[0081] In one example, the candidate picture data or candidate video data corresponding to the maximum value of the second weight information can be taken as the target picture data or target video data. The higher the second weight information of the user for the picture data or video data, the higher the user's attention to the picture data or video data. The target picture data or target video data is filtered according to the maximum value of the second weight information, so that the obtained target picture data or target video data is the most popular picture data or video data at present, and the music tag generated for the picture data or video data can fully reflect the user's preference.
[0082] In another example, a plurality of candidate picture data or candidate video data may be selected in order from high to low according to the second weight data as the target picture data or target video data.
[0083] It should be noted that, in the embodiment of the present invention, the number of target image data or target video data to be selected can be set according to actual needs.
[0084] In another example, a weight threshold may be pre-set, and after determining the second weight data, the candidate image data or candidate video data whose second weight data is greater than the weight threshold may be used as the target video data. It should be noted that the weight threshold may be set according to actual needs, and in the embodiments of the present invention, there are no excessive restrictions on this.
[0085] In another example, the picture data and the video data may be comprehensively analyzed for weight, and all the picture data and the video data may be sorted from high to low according to the second weight data, and the target picture data and / or the target video data may be obtained from the sorting.
[0086] In the embodiment of the present invention, after the user triggers the function option for automatically generating music on the music generation front-end interface, the user behavior data can be collected according to steps 101 to 105 to automatically generate music.
[0087] In a feasible implementation scheme, the user can also input any picture or any video on the music generation front-end interface, and then determine the target tag information matching the picture or video among the preset multiple music tag information, and call the preset interface to input the target tag information into the music generation model corresponding to the preset interface to generate music data matching the target tag information.
[0088] In an embodiment of the present invention, multiple user behavior data associated with images or videos collected on video websites can be obtained; then target image data or target video data can be determined based on the multiple user behavior data; target tag information matching the target image data or target video data can be determined among multiple preset music tag information; thereby, a preset interface can be called, and the target tag information can be input into a music generation model corresponding to the preset interface to generate music data matching the target tag information, and music data that meets the user's preferences can be automatically generated without the user selecting a music tag, thereby achieving accurate music recommendation.
[0089] Reference Figure 2 , shows a flowchart of another method for generating music provided by an embodiment of the present invention, which may specifically include the following steps:
[0090] Step 201, determining candidate picture data or candidate video data;
[0091] Step 202, obtaining user behavior data associated with the candidate image data or the candidate video data;
[0092] Step 203, determining target image data or target video data based on the user behavior data;
[0093] Step 203, inputting the target picture data or the target video data into a first model for extracting feature vectors, and obtaining a feature vector output by the first model that represents the target picture data or the target video data;
[0094] The model input of the first model is the target image data or the target video data, and the model output is the feature vector. The first model can be generated based on convolutional neural network training. Convolutional neural network (CNN) is a deep learning model that learns features from images by stacking convolutional layers, pooling layers, and fully connected layers. When a picture is input, CNN passes a series of convolution and pooling operations on the original pixel data of the picture, and finally generates an abstract feature representation, which can be called a feature vector.
[0095] Specifically, the process of generating feature vectors based on input image data or video data in the convolutional neural network model is as follows:
[0096] Each frame in the image or video is input into the convolutional neural network model. The image or video frame is represented in the form of a three-dimensional tensor with dimensions (height, width, channels), where height and width are the height and width of the image, respectively, and channels is the number of channels of the image. For example, for color images, channels is usually 3, corresponding to the red, green, and blue channels respectively.
[0097] Convolution and pooling in convolutional neural network models: The first part of a CNN is usually a series of convolutional and pooling layers. Convolutional layers learn local features in an image through convolution operations, while pooling layers reduce the dimensionality of feature maps through downsampling. These operations allow the network to gradually understand the abstract features of an image.
[0098] Fully connected layer: After a series of convolution and pooling layers, the dimension of the feature map is reduced, and then these features are mapped to the output layer through the fully connected layer to form the final feature vector.
[0099] Output layer: The final fully connected layer is usually connected to an output layer for classification, regression or other tasks.
[0100] It can be understood that the image data and video data are processed by the first model, and high-dimensional images can be converted into low-dimensional images to obtain feature vectors for easy classification.
[0101] In one embodiment of the present invention, the first model training can be performed in advance based on the image data or video data. Specifically, the first feature vector that matches the image data or video data can be determined in the vector database, and the image data or video data can be input into the initially constructed convolutional neural network model to obtain the second feature vector. The first feature vector and the second feature vector are then compared, and the comparison result is transmitted back to the initial convolutional neural network model, and the model parameters of the convolutional neural network model are adjusted to implement iterative model updates. After multiple rounds of iterative updates, when the prediction accuracy of the convolutional neural network model reaches a preset value, it can be determined that the convolutional neural network model training is completed.
[0102] Step 204: input the feature vector into a second model for classifying the feature vector into music tags, and obtain target tag information corresponding to the feature vector output by the second model.
[0103] After the feature vector is obtained, the feature vector may be input into the second model, and the target label information corresponding to the feature vector may be output by processing the feature vector in the second model.
[0104] The second model is used to determine the music label corresponding to each feature vector. The second model can be generated based on the training of the convolutional layer network model. Specifically, the model training of the second model can be performed based on the feature vector and the annotated music label as sample data. The training goal of the second model is to learn how to map features to corresponding labels.
[0105] The training process of the second model is as follows: the collected feature vector samples are input into the second model, the predicted music label is output, the predicted music label is compared with the sample label annotated by the feature vector, and the model parameters of the second model are adjusted in reverse order to make the label predicted by the second model infinitely close to the label annotated by the feature vector.
[0106] Step 205, calling a preset interface, inputting the target tag information into a music generation model corresponding to the preset interface, and obtaining text melody information corresponding to the target tag information output by the music generation model;
[0107] In the embodiment of the present invention, the text melody information is specifically music score information, five-line staff information, etc. For example, the text melody information can be specifically composed of multiple digital notes of 1, 2, 3, 4, 5, 6, and 7.
[0108] The input data of the music generation model is the target label information, and the output is the text melody information.
[0109] In one embodiment of the present invention, music data stored locally in the user terminal may also be collected, and the text melody information may be adjusted based on the music data stored locally in the user terminal, so that the generated text melody information may be more in line with the user's music preference.
[0110] Among them, the adjustment strategies may include melody similarity matching, rhythm adaptive adjustment, harmony coordination optimization, style consistency optimization, etc.
[0111] For example, when the adjustment strategy is melody similarity matching, the melody of the text melody information can be adjusted according to the melody characteristics of the user's local music data, so that the generated music melody is more similar to the melody liked by the user.
[0112] For another example, when the adjustment strategy is rhythm adaptive adjustment, the rhythm of the text melody information is adjusted according to the rhythm characteristics of the user's local music data, so that the generated music rhythm is more consistent with the rhythm preferred by the user.
[0113] For another example, when the adjustment strategy is harmony coordination optimization, the harmony of the text melody information is adjusted according to the harmony characteristics of the user's local music data, so that the generated harmony is more coordinated with the harmony liked by the user.
[0114] For another example, when the adjustment strategy is style consistency adjustment, the music style of the text melody information is adjusted according to the style characteristics of the user's local music data, so that the generated music style is more consistent with the style preferred by the user.
[0115] Step 206: Generate music data according to the text melody information.
[0116] After the text melody information is obtained, playable music data can be compiled based on the text melody information.
[0117] In one embodiment of the present invention, music data is generated according to text melody information, including: converting the text melody information into a musical instrument digitization interface file; converting the musical instrument digitization interface file according to a preset audio format to generate music data.
[0118] Musical Instrument Digital Interface (MIDI file) is a command sequence file for describing the process of music performance, which has the advantages of unified standard format and high flexibility. After obtaining the text melody information, the notes and symbols in the text melody information can be extracted, and corresponding formatting can be performed to extract the music features such as pitch, rhythm, and intensity from the text melody. In the process of extracting music features, a music feature extraction algorithm or tool can be preset, and the music feature extraction algorithm or tool can be set according to actual needs. In the embodiment of the present invention, there is no excessive restriction on this. Then the extracted music features are learned and converted into a form that can be understood by the machine. In the conversion process, algorithms such as deep learning can be used. According to the learned music representation, a MIDI generation algorithm or tool is used to generate a MIDI file. Among them, the algorithm or tool for converting the text melody into a MIDI file may include but is not limited to music information retrieval, natural language processing, music representation learning, MIDI generator, etc.
[0119] After the MIDI file is generated, the MIDI text is converted into a format, wherein the preset audio format may be any one of WAV and MP3.
[0120] In an embodiment of the present invention, after obtaining the text melody information, the MIDIUtil library in Python can be used to generate a MIDI file based on the text melody information, and then a locally downloaded and installed music synthesizer (such as FluidSynth) can be used to convert the MIDI file into a preset audio format to obtain music data that can be played locally.
[0121] In an embodiment of the present invention, multiple user behavior data associated with a picture or video collected on a video website can be obtained; then a first model for extracting a feature vector is used to convert the target picture data or the target video data into a feature vector for characterizing the target picture data or the target video data; a second model for classifying the feature vector into a music tag is used to determine the target tag information corresponding to the feature vector from a plurality of preset music tag information; thereby calling a preset interface, inputting the target tag information into a music generation model corresponding to the preset interface, and outputting text melody information corresponding to the target tag information; and generating music data according to the text melody information, thereby realizing that by analyzing the user behavior associated with the picture or video, the music tag can be accurately matched for the target picture data or the target video data based on the first model and the second model, and then music is automatically generated based on the music tag, and music that conforms to the user's behavior habits can be generated without the user setting tags.
[0122] In an embodiment of the present invention, a music generation system can also be provided, referring to Figure 3 As shown, it is a schematic diagram of a music generation system of the present invention. The music generation system may include a basic layer, a model layer, a business layer, and a presentation layer.
[0123] The basic layer can include vector databases, distributed training institutions, public clouds, MySQL and other basic settings to provide technical support for the model layer and business layer.
[0124] The model layer may include the training and application process of the first model and the second model. Specifically, the general data domain, user data, music data, etc. may be collected through the data collector. Then, based on the convolutional neural network model (i.e., the first model), the feature vector is extracted (i.e., feature extraction), and the feature vector of the image is input into the constructed second model for training (i.e., training model). The goal of the model is to learn how to map features to corresponding labels, and the model is evaluated and fine-tuned by comparing the music labels preset by the model with the actual music labels.
[0125] Based on the trained model, the label corresponding to the feature vector is determined. Then, in the business layer, the label can be input into the music generation model (such as OpenAI) through the preset interface to obtain the text melody, and then the MIDI file is generated based on the MIDIUtil library in the Python library. The music synthesizer FluidSynth is downloaded and installed locally to convert the MIDI file into common audio formats, such as WAV, MP3, etc.
[0126] The generated music data is sent to the AIBGM generation front end of the presentation layer for playback and presentation.
[0127] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0128] Reference Figure 4a , shows a schematic diagram of the structure of a music generation device provided by an embodiment of the present invention, which may specifically include the following modules:
[0129] A candidate data determination module 401 is used to determine candidate picture data or candidate video data;
[0130] A user behavior data acquisition module 402 is used to acquire user behavior data associated with the candidate image data or the candidate video data;
[0131] A target data screening module 403 is used to determine target image data or target video data based on the user behavior data;
[0132] The tag information determination module 404 is used to input the target image data or the target video data into a music tag matching model, and the music tag matching model determines and outputs target tag information matching the target image data or the target video data from a plurality of preset music tag information;
[0133] The music data determination module 405 is used to input the target tag information into the music generation model to obtain the music data output by the music generation model that matches the target tag information as the music data that matches the target picture data or the target video data.
[0134] In one embodiment of the present invention, referring to Figure 4b As shown, the tag information determination module 404 may include the following submodules:
[0135] A feature vector generating submodule 11 is used to input the target picture data or the target video data into a first model for extracting feature vectors, and obtain a feature vector output by the first model that represents the target picture data or the target video data;
[0136] The label information generating submodule 12 is used to input the feature vector into a second model for classifying the feature vector into music labels, and obtain the target label information corresponding to the feature vector output by the second model.
[0137] In one embodiment of the present invention, referring to Figure 4c As shown, the device also includes:
[0138] The sample data acquisition module 406 is used to acquire sample picture data or sample video data for training the music label matching model, and to annotate the sample picture data or the sample video data with music labels to obtain sample music labels corresponding to the sample picture data or the sample video data;
[0139] A music label prediction module 407, configured to input the sample picture data or the sample video data into an initial music label matching model, and output a predicted music label corresponding to the sample picture data or the sample video data;
[0140] A target model parameter adjustment module 408, configured to adjust target model parameters of the initial music label matching model based on the sample music label and the predicted music label;
[0141] The training termination module 409 is used to output the music label matching model corresponding to the target model parameters when the current model training meets the preset model training termination condition;
[0142] The iterative training module 410 is used to perform iterative training based on the music tag matching model corresponding to the target model parameters when the current model training does not meet the preset model training termination condition.
[0143] In one embodiment of the present invention, referring to Figure 4d As shown, the music data determination module 405 may include:
[0144] The text melody information determination submodule 13 is used to call a preset interface, input the target tag information into a music generation model corresponding to the preset interface, and obtain the text melody information corresponding to the target tag information output by the music generation model;
[0145] The music data generating submodule 14 is used to generate music data according to the text melody information.
[0146] In one embodiment of the present invention, referring to Figure 4e As shown, the music data generation submodule 14 may include the following units:
[0147] The musical instrument digitization interface file unit 141 is used to convert the text melody information into a musical instrument digitization interface file;
[0148] The music data generating submodule 142 is used to convert the musical instrument digitization interface file according to a preset audio format to generate music data.
[0149] In one embodiment of the present invention, referring to Figure 4f As shown, the target data screening module 403 may include:
[0150] A candidate picture or video determination submodule 15 is used to determine a plurality of candidate picture data or a plurality of candidate video data associated with the plurality of user behavior data;
[0151] A first weight acquisition module 16, used to acquire first weight information of each user behavior data;
[0152] A second weight calculation submodule 17, configured to determine second weight information of each candidate picture data or each candidate video data based on the first weight information;
[0153] The picture or video screening submodule 18 is used to determine target picture data or target video data from the multiple candidate picture data or the multiple candidate video data according to the second weight information.
[0154] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.
[0155] Memory 503, used for storing computer programs;
[0156] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:
[0157] Determining candidate image data or candidate video data;
[0158] Acquire user behavior data associated with the candidate image data or candidate video data;
[0159] Determining target image data or target video data based on the user behavior data;
[0160] Inputting the target picture data or the target video data into a music tag matching model, wherein the music tag matching model determines and outputs target tag information matching the target picture data or the target video data from a plurality of preset music tag information;
[0161] The target tag information is input into a music generation model to obtain music data output by the music generation model that matches the target tag information, as music data that matches the target picture data or the target video data.
[0162] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0163] The communication interface is used for communication between the above terminal and other devices.
[0164] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0165] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0166] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned method for generating music is implemented.
[0167] An embodiment of the present invention further provides a computer product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method for generating music as described above is implemented.
[0168] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0169] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0170] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0171] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0172] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0174] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0175] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0176] The above provides a detailed introduction to a music generation method and device, electronic device, and storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for generating music, characterized in that: The method comprises: Determining candidate image data or candidate video data; Acquire user behavior data associated with the candidate image data or candidate video data; Determining target image data or target video data based on the user behavior data; Inputting the target picture data or the target video data into a music tag matching model, wherein the music tag matching model determines and outputs target tag information matching the target picture data or the target video data from a plurality of preset music tag information; The target tag information is input into a music generation model to obtain music data output by the music generation model that matches the target tag information, as music data that matches the target picture data or the target video data.
2. The method according to claim 1, characterized in that The music tag matching model determines and outputs target tag information matching the target picture data or the target video data from among a plurality of preset music tag information, including: Inputting the target picture data or the target video data into a first model for extracting feature vectors, and obtaining a feature vector output by the first model that represents the target picture data or the target video data; The feature vector is input into a second model for classifying the feature vector into music tags, and the target tag information corresponding to the feature vector output by the second model is obtained.
3. The method according to claim 1, characterized in that The training process of the music tag matching model includes the following steps: Obtaining sample picture data or sample video data for training a music label matching model, and annotating the sample picture data or the sample video data with music labels to obtain sample music labels corresponding to the sample picture data or the sample video data; Inputting the sample picture data or the sample video data into an initial music label matching model, and outputting a predicted music label corresponding to the sample picture data or the sample video data; Adjusting target model parameters of the initial music tag matching model based on the sample music tag and the predicted music tag; When the current model training meets the preset model training termination condition, output the music label matching model corresponding to the target model parameters; When the current model training does not meet the preset model training termination conditions, iterative training is performed based on the music tag matching model corresponding to the target model parameters.
4. The method according to claim 1, characterized in that: The step of inputting the target tag information into a music generation model to obtain music data output by the music generation model and matching the target tag information includes: Calling a preset interface, inputting the target tag information into a music generation model corresponding to the preset interface, and obtaining text melody information corresponding to the target tag information output by the music generation model; Music data is generated according to the text melody information.
5. The method according to claim 4, characterized in that The step of generating music data according to the text melody information comprises: Converting the text melody information into a musical instrument digital interface file; The musical instrument digitization interface file is converted according to a preset audio format to generate music data.
6. The method according to claim 1, characterized in that The determining target image data or target video data based on the user behavior data includes: Obtain the first weight information of each user's behavior data; Determining second weight information for each candidate picture data or each candidate video data based on the first weight information, wherein the second weight information is used to reflect the user's attention to different picture data or video data; According to the second weight information, target picture data or target video data is determined from the plurality of candidate picture data or the plurality of candidate video data.
7. A device for music generation, characterized in that: The device comprises: A candidate data determination module, used to determine candidate image data or candidate video data; A user behavior data acquisition module, used to acquire user behavior data associated with the candidate image data or candidate video data; A target data screening module, used to determine target image data or target video data based on the user behavior data; A label information determination module, used for inputting the target picture data or the target video data into a music label matching model, wherein the music label matching model determines and outputs target label information matching the target picture data or the target video data from a plurality of preset music label information; The music data determination module is used to input the target tag information into the music generation model to obtain the music data output by the music generation model that matches the target tag information, as the music data that matches the target picture data or the target video data.
8. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method for generating music as claimed in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating music as claimed in any one of claims 1 to 6 is implemented.
10. A computer product, characterized in that: It comprises a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, it implements the method for generating music according to any one of claims 1 to 6.
