Real-time generation method and device for dance score

By capturing dance pictures in real time and analyzing visual information, generating music attribute information, and sampling with music and or pictures, the problem that the existing technology cannot generate music in real time is solved, and the effect of instantly creating music during the dance process is achieved.

CN119007695AActive Publication Date: 2024-11-22BEIJING INSTITUTE FOR GENERAL ARTIFICIAL INTELLIGENCE

Patent Information

Application Number
CN202411480895.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-22
Estimated Expiration
2044-10-23

AI Technical Summary

Technical Problem

The existing dance music generation method cannot capture images and generate music in real time. The deep learning network parameters are numerous and complex, so real-time music generation cannot be achieved.

Method used

By capturing the human body's motion picture in real time, using machine vision to analyze dance visual information and scene information, extracting human posture characteristic parameters, generating music attribute information, and using music and or pictures to sample dance music in real time.

Benefits of technology

It realizes the real-time generation of soundtracks during the dance process, avoids the dependence on completed dance videos, and has the advantage of dual copyrights for creating dance and music.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007695A_ABST
    Figure CN119007695A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a real-time generation method and device for dance score. The method comprises the following steps: capturing a real-time picture of a human body action, and analyzing dance visual information and scene information in the real-time picture of the human body action; performing human body action information extraction and facial expression analysis on the dance visual information to obtain human body posture characteristic parameters, and generating real-time music attribute information according to the human body posture characteristic parameters and scene information; and taking the music attribute information as node features in the music and / or graph, sampling the music and / or graph, and generating real-time dance score. According to the real-time generation method and device of the dance score, the score does not need to be generated according to a completed dance video, and music creation can be completed while a user dance, so that the user has double copyrights of the dance and music works.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of audio data processing, and in particular to a real-time generation method and device for dance music. Background Art

[0002] At present, the generation schemes of dance music are all based on the deep learning neural network architecture, and the implementation stage includes two parts: encoding and decoding. The purpose of encoding is to extract features from input information and compress it; the purpose of decoding is to reconstruct the information (features) into music. In the encoding stage, deep learning methods are used, including CNN (convolutional neural network) or Transformer method; in the decoding stage, Transformer method is used.

[0003] However, the above-mentioned music generation scheme requires a large amount of data for learning, whether in the encoding stage or the decoding stage. In addition, due to the characteristics of the deep learning network itself, that is, the large number of parameters, such methods cannot perform real-time generation, that is, it is impossible to capture real-time images to generate real-time music. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for real-time generation of dance music, so as to solve the problem that the existing technical solution cannot capture real-time images to generate real-time music.

[0005] In order to solve the above technical problems, a specific embodiment of the present invention provides a real-time generation method of dance music, including: capturing real-time images of human body movements, and parsing dance visual information and scene information in the real-time images of human body movements; extracting human body movement information and facial expression analysis from the dance visual information to obtain human body posture feature parameters, and generating real-time music attribute information based on the human body posture feature parameters and scene information; using the music attribute information as node features in a music AND-OR graph, sampling the music AND-OR graph, and generating real-time dance music.

[0006] Furthermore, in one embodiment, the real-time picture of human body movements is captured, and dance visual information and scene information in the real-time picture of human body movements are parsed, including: the real-time picture of human body movements is a two-dimensional RGB picture of human body movements, and the dance visual information and scene information in the two-dimensional RGB picture of human body movements are parsed through machine vision; wherein, the dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by a human posture estimation algorithm; and the scene information includes the background, objects in the picture, and dancers' costumes.

[0007] Furthermore, in one embodiment, the human body motion information is extracted and facial expression is analyzed from the dance visual information to obtain human body posture feature parameters, including: determining the human body posture feature parameters based on the three-dimensional position coordinates of the 33 human body joints; wherein the human body posture feature parameters include: the distance of the limbs relative to the center of the human body, joint angles, body contour width and height, joint speed, joint acceleration, joint pause point detection, motion amplitude, and facial expression recognition.

[0008] Furthermore, in one embodiment, the real-time music attribute information is generated according to the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label according to the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0009] Furthermore, in one embodiment, the point with the highest coincidence with the action beat is selected as the beat, and the initial music speed bpm is calculated to generate the rhythm attribute information.

[0010] Furthermore, in one embodiment, the music attribute information is used as a node feature in a music AND OR graph, and the music AND OR graph is sampled to generate a real-time dance soundtrack, including: corresponding the music attribute information to different nodes of the music AND OR graph according to the time dimension, sampling the music attribute information of each node in the music AND OR graph, and then generating a real-time dance soundtrack; wherein, generating a real-time dance soundtrack also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, combining with a rhythmic pattern library under chord constraints to generate a chord texture, and generating a percussion accompaniment segment under beat constraints.

[0011] Furthermore, in one embodiment, the method further includes: further generating corresponding music scores from the generated real-time dance music.

[0012] In order to solve the above technical problems, the specific implementation of the present invention also provides a real-time generation device for dance music, including: a real-time image analysis module, used to capture real-time images of human body movements, and parse out dance visual information and scene information in the real-time images of human body movements; a music attribute information generation module, used to extract human body movement information and facial expression analysis from the dance visual information, obtain human body posture feature parameters, and generate real-time music attribute information based on the human body posture feature parameters and scene information; a dance music real-time generation module, used to use the music attribute information as a node feature in a music AND-OR graph, sample the music AND-OR graph, and generate real-time dance music.

[0013] Furthermore, in one embodiment, the music attribute information generation module generates real-time music attribute information based on the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label based on the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0014] Furthermore, in one embodiment, the music attribute information generation module selects the point with the highest coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

[0015] Furthermore, in one embodiment, the dance music real-time generation module uses the music attribute information as a node feature in a music AND OR graph, samples the music AND OR graph, and generates a real-time dance music, including: corresponding the music attribute information to different nodes of the music AND OR graph according to the time dimension, sampling the music attribute information of each node in the music AND OR graph, and then generating a real-time dance music; wherein, generating a real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, combining with the rhythmic pattern library under chord constraints to generate a chord texture, and generating a percussion accompaniment segment under beat constraints.

[0016] Furthermore, in one embodiment, the real-time generation device of dance music further includes: a music score generation module, which is used to further generate corresponding music scores from the generated real-time dance music.

[0017] Another aspect of an embodiment of the present invention provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the method of the embodiment of the present invention is implemented.

[0018] Another aspect of an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method of the embodiment of the present invention when executed by a processor.

[0019] Another aspect of an embodiment of the present invention provides a computer program, wherein the computer program includes computer executable instructions, and the instructions are used to implement the method of the embodiment of the present invention when being executed.

[0020] The real-time generation method and device of dance music proposed in the embodiment of the present invention are different from the existing method of generating music using ready-made dance videos. Instead, the method captures the dance screen in real time, parses the dance action information and scene information in the RGB screen through machine vision, and then maps this information into music attributes as the characteristics of the nodes in the music and OR graph, and uses the music and OR graph for sampling to generate dance music in real time and generate corresponding music scores at the same time. Through the method of this embodiment, there is no need to generate music based on the completed dance video, but music creation can be completed while dancing, thereby owning the dual copyright of creating dance and music works. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings below are part of the specification of the present invention, and illustrate exemplary embodiments of the present invention. Together with the description of the specification, the accompanying drawings are used to explain the principles of the present invention.

[0022] Figure 1 is a processing flow chart of a real-time generation method of dance music according to an embodiment of the present invention; Figure 2 A schematic diagram of the three-dimensional position coordinates of 33 human joints predicted by the human posture estimation algorithm according to an embodiment of the present invention; Figure 3 This is a simple example diagram of multimodal music generation based on music and / or graph according to an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a real-time generation device of dance music according to an embodiment of the present invention; Figure 5 It is a schematic structural diagram of a real-time generation device for dance music according to another embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0024] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0025] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0026] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0027] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0028] As used herein, "and / or" includes any or all combinations of the items described.

[0029] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality of groups” includes “two groups” and “more than two groups”.

[0030] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0033] The real-time generation method of dance music in the embodiment of the present invention is different from the existing method of generating music using ready-made dance videos. Instead, it captures the dance screen in real time, parses the dance action information and scene information in the RGB screen through machine vision, and then maps this information into music attributes as the features of the nodes in the music AND / OR graph, and uses the music AND / OR graph for sampling to generate dance music in real time and generate corresponding music scores at the same time. Through the method of this embodiment, there is no need to generate music based on the completed dance video, but music creation can be completed while dancing, thereby owning the dual copyrights of the creation of dance and music works.

[0034] Figure 1 The present invention is a flowchart of a method for real-time generation of dance music according to an embodiment of the present invention. As shown in the figure, the method of the present embodiment includes: step S101, capturing a real-time picture of human body movements, and parsing dance visual information and scene information in the real-time picture of human body movements; step S102, extracting human body movement information and facial expression analysis from the dance visual information, obtaining human body posture feature parameters, and generating real-time music attribute information based on the human body posture feature parameters and scene information; step S103, using the music attribute information as a node feature in a music AND-OR graph, sampling the music AND-OR graph, and generating real-time dance music.

[0035] In step S101 of this embodiment, the human body motion picture captured in real time may be a two-dimensional RGB picture of human body motion, for example, a picture of a dancer's motion on the stage captured in real time by a camera.

[0036] In this embodiment, the extraction of dance visual information by machine vision includes two aspects: human motion information extraction and facial expression analysis. Human motion information extraction mainly relies on the extraction of three-dimensional information of human skeleton; facial expression analysis can extract facial feature points and perform regression or classification based on facial feature points; or regression or classification can be performed directly based on facial images.

[0037] like Figure 2 As shown in the figure, the dance movement information includes the three-dimensional position coordinates of 33 human joints predicted by the human posture estimation algorithm. In specific implementation, the two-dimensional RGB image of human movement can be input into the BlazePose lightweight convolutional neural network structure to obtain the coordinate information of each joint in the image coordinate system. Each joint information is represented as (x, y, z, v); where x and y represent the normalized coordinate position of the joint in the image coordinate system; z represents the depth of the joint with the midpoint depth of the hip as the origin; v represents the possibility value of the joint being visible.

[0038] In this embodiment, the scene information parsed from the captured two-dimensional RGB image of human body movements includes background information, object information in the image, dancer's clothing information, etc. This information is involved in the style prediction in the following steps, for example, the style label can be obtained based on the dancer's clothing information and the stage background information.

[0039] In step S102, human body motion information is extracted and facial expression is analyzed from the dance visual information to obtain human body posture characteristic parameters, including: determining the human body posture characteristic parameters according to the three-dimensional position coordinates of the 33 human body joints.

[0040] In specific implementation, after obtaining the original information of the joints, a series of features can be calculated, that is, the position information of the human joints is converted into feature vectors. For example, the center point of the left and right hips of the human body is regarded as the center point of the human body, and the distance from the center point of the human body to the center point of the left shoulder and the right shoulder is regarded as the size of the human body trunk; then the relative position of each joint point is initialized based on the center point of the human body, and the relative distance between each joint point and the center point of the human body is taken as part of the feature, and the pairwise distance between joints is extracted as another part of the feature according to the human joint association rule.

[0041] In an embodiment of the present invention, the human body posture characteristic parameters obtained according to the above method include: the distance between the limbs and the center of the human body, joint angles, body contour width and height, joint speed, joint acceleration, joint pause point detection, movement amplitude, facial expression recognition, etc.

[0042] In step S102, after obtaining a series of human posture feature parameters, a style label is predicted and generated using a multi-layer perceptron based on the human posture feature parameters and scene information to generate corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0043] In the specific implementation, a multi-layer perceptron model is used to predict the style label, thereby generating matching instrument combinations, scales, and chords, etc. At the same time, the point with the highest overlap with the action beat is selected as the beat, and the initial music speed (bpm, Beat Per Minute) is calculated based on the beat.

[0044] In the embodiment of the present invention, the multi-layer perceptron (MLP) is a small neural network. Using the multi-layer perceptron to predict the style is to use the neural network to classify the style. This part includes two processes: training and use. The training process is to collect data of different styles, such as human joint information of folk dance, human joint information of modern dance, etc., and use this information to train a small neural network. The use process is to send the human posture feature parameters obtained in step S102 into the network and output which dance type it belongs to.

[0045] In the embodiment of the present invention, the music attribute information includes some specific features in the music, such as "music style", "chords" composed of notes, "intervals" between notes, "music rhythm", etc. In the embodiment of the present invention, it can be divided into accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information. For example, the accompaniment attribute information includes the combination of soundtracks, the melody attribute information includes scales, interval relationships, etc., the harmony attribute information includes chord composition, etc., and the rhythm attribute information includes beats, bpm and rhythm arrangement, etc.

[0046] In this embodiment, for the attribute of "music rhythm", for example, the point where the action and the beat overlap the most can be selected as the beat. When the action changes suddenly, that is, within a certain period of time, the moment when the action changes the most, that is, the time point when the acceleration of the human dance action is the largest, can be regarded as a beat point. However, the generation method of the music rhythm attribute of the present invention is not limited to this. For example, the expression corresponding to the joint point of the face can also be used to control the rhythm of the music, and the mapping relationship can be modified. Similarly, the generation of other music attribute information can be modified or improved according to the mapping relationship of the multi-layer perceptron.

[0047] After obtaining the music attribute information, it is used as the node feature in the music AND-OR graph, and the music AND-OR graph is used for sampling to generate real-time dance background music.

[0048] In the embodiment of the present invention, a piece of music is subdivided spatially into: melody, accompaniment, harmony, rhythm and other parts, and is also subdivided temporally into music segments, phrases, measures, notes and the like. Such subdivision is purposeful and reasonable: many notes constitute a measure node, many measures constitute a phrase node, many phrases constitute a segment node, and many segments constitute a piece of music; at the same time, the segments and phrases directly reflect the melody, supplemented by different accompaniments, harmonies, rhythms, etc. Many nodes are interwoven together, and music is the result of the specific instantiation of each node: many nodes coexist, and each node has different choices, thus forming a music and / or graph.

[0049] The learning of the music AND-OR graph is to use an algorithm to obtain the music attribute information of each node, such as the interval relationship, chord composition, rhythm arrangement, etc. in the music corresponding to the node; and music generation is the reverse process of learning, that is, using an algorithm to sample the music attribute information from each node. Therefore, as long as the music attribute information can be matched with the nodes in the music AND-OR graph, only sampling is required to generate music. The music attribute information can come from the statistical laws of many kinds of music, or it can be obtained artificially through different technical means, which is the core of multimodal music generation based on the music AND-OR graph. Of course, in the embodiment of the present invention, the music attribute information is the music attribute information generated by extracting and analyzing the real-time captured human body action picture and mapping the attribute features through steps S101 and S102.

[0050] In the specific implementation, the music attribute information is used as a global constant to participate in the music generation control at each moment. For the music attribute information corresponding to each frame of the image, it is mapped to different nodes of the music AND / OR graph according to the time dimension, and the music AND / OR graph is used to generate music, that is, the music attributes of each node are sampled. In addition, for each frame, the Hastings-Metropolis algorithm is used to predict the next melody note under the constraints of music theory, and the chord texture is generated by combining it with the rhythmic library under the constraints of chords, and the percussion accompaniment fragment is generated under the constraints of beats, thereby generating real-time dance music. Among them, "music theory constraints", "chord constraints", "rhythmic library" and "beat constraints" are all restrictions on the generated music, the purpose of which is to make the generated music more in line with people's needs / expectations. The Metropolis Hastings algorithm is a key part of the Markov chain Monte Carlo (MCMC) sampling technology. The algorithm is used to enhance music generation. At the same time, by adjusting and adopting Metropolis Hastings sampling, the controlled generation of music can be achieved, allowing fine control of the modification of the existing music structure, so that users can finely control the work.

[0051] Figure 3This is a simple example diagram of multimodal music generation based on a music AND-OR graph according to an embodiment of the present invention. In this example, a phrase is divided into four parts: melody, accompaniment, harmony, and rhythm. Each part is corresponding to a different node in the music AND-OR graph according to the time dimension (the top white dot: "solid white dot" is the selected AND-OR graph node, meaning the phrase to be generated / changed; "hollow dot" is an unselected node). These four parts correspond to different music attributes, and their characteristics can be assigned values ​​through different modes. As for the use of music AND-OR graphs for music generation, it is nothing more than sampling the music attributes of each node.

[0052] Figure 4 The present invention is a schematic diagram of the structure of the real-time generation device of dance music according to the embodiment of the present invention. As shown in the figure, it includes: a real-time picture analysis module 401, which is used to capture the real-time picture of human body movements and parse out the dance visual information and scene information in the real-time picture of human body movements; a music attribute information generation module 402, which is used to extract human body movement information and facial expression analysis from the dance visual information, obtain human body posture feature parameters, and generate real-time music attribute information according to the human body posture feature parameters and scene information; a dance music real-time generation module 403, which is used to use the music attribute information as the node feature in the music AND-OR graph, sample the music AND-OR graph, and generate real-time dance music.

[0053] In this embodiment, the real-time image analysis module 401 captures real-time images of human body movements, and analyzes dance visual information and scene information in the real-time images of human body movements, specifically including: the real-time images of human body movements are two-dimensional RGB images of human body movements, and the dance visual information and scene information in the two-dimensional RGB images of human body movements are analyzed by machine vision; wherein, the dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by the human posture estimation algorithm; the scene information includes the background, objects in the image, and dancers' costumes.

[0054] In this embodiment, the music attribute information generation module 402 extracts human motion information and analyzes facial expressions on the dance visual information to obtain human posture feature parameters, including: determining the human posture feature parameters based on the three-dimensional position coordinates of the 33 human joints; wherein the human posture feature parameters include: the distance of the limbs relative to the center of the human body, the joint angle, the width and height of the body contour, the joint speed, the joint acceleration, the joint pause point detection, the movement amplitude, and the facial expression recognition.

[0055] In this embodiment, the music attribute information generation module 402 generates real-time music attribute information based on the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label based on the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0056] In this embodiment, the music attribute information generating module 402 selects the point with the highest coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

[0057] In this embodiment, the dance music real-time generation module 403 uses the music attribute information as the node feature in the music AND OR graph, samples the music AND OR graph, and generates real-time dance music, including: corresponding the music attribute information to different nodes of the music AND OR graph according to the time dimension, sampling the music attribute information of each node in the music AND OR graph, and then generating real-time dance music; the dance music real-time generation module 403 is also used to use the music attribute information as a global constant to participate in the music generation control at each moment, use the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, combine with the rhythmic pattern library under chord constraints to generate chord textures, and generate percussion accompaniment fragments under beat constraints.

[0058] In this embodiment, Figure 5 As shown, the real-time generation device of dance music further includes a music score generation module 404, which is used to further generate corresponding music scores from the generated real-time dance music.

[0059] The real-time generation method and device of dance music disclosed in the above embodiment is different from the existing method of generating music using ready-made dance videos. Instead, it captures the dance screen in real time, parses the dance action information and scene information in the RGB screen through machine vision, and then maps this information into music attributes as the characteristics of the nodes in the music and OR graph, and uses the music and OR graph for sampling to generate dance music in real time and generate corresponding music scores at the same time. Through the method of this embodiment, there is no need to generate music based on the completed dance video, but music creation can be completed while dancing, thereby owning the dual copyright of creating dance and music works.

[0060] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic device, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0061] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0062] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device.

[0063] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0064] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0065] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A real-time generation method for dance music, characterized in that: include: Capturing real-time images of human body movements, and parsing dance visual information and scene information in the real-time images of human body movements; Extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters, and generating real-time music attribute information based on the human posture feature parameters and scene information; The music attribute information is used as a node feature in a music AND-OR graph, and the music AND-OR graph is sampled to generate a real-time dance soundtrack.

2. The real-time generation method of dance music according to claim 1, characterized in that: The capturing of the real-time picture of human body movements and parsing out dance visual information and scene information in the real-time picture of human body movements include: The real-time human body movement picture is a two-dimensional RGB picture of human body movement, and dance visual information and scene information in the two-dimensional RGB picture of human body movement are parsed by machine vision; The dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by a human body posture estimation algorithm; The scene information includes the background, objects in the picture, and dancers' costumes.

3. The real-time generation method of dance music according to claim 2, characterized in that: The step of extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters includes: Determine the human body posture characteristic parameters according to the three-dimensional position coordinates of the 33 human body joints; Among them, the human body posture characteristic parameters include: the distance between the limbs and the center of the human body, joint angles, body outline width and height, joint speed, joint acceleration, joint pause point detection, movement amplitude, and facial expression recognition.

4. The real-time generation method of dance music according to claim 3, characterized in that: The generating of real-time music attribute information according to the human body posture characteristic parameters and scene information comprises: Based on the human body posture feature parameters and scene information, a multi-layer perceptron is used to predict and generate style labels to generate corresponding music attribute information; The music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style tag.

5. The real-time generation method of dance music according to claim 4, characterized in that: The point with the highest coincidence with the action beat is selected as the beat, and the initial music speed bpm is calculated to generate the rhythm attribute information.

6. The real-time generation method of dance music according to claim 4, characterized in that: The music attribute information is used as a node feature in a music AND-OR graph, and the music AND-OR graph is sampled to generate a real-time dance soundtrack, including: The music attribute information is mapped to different nodes of the music AND / OR graph according to the time dimension, and the music attribute information of each node in the music AND / OR graph is sampled to generate a real-time dance soundtrack; Among them, generating real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm under music theory constraints to predict the next melody note, combining with the rhythmic pattern library under chord restrictions to generate chord textures, and generating percussion accompaniment clips under beat restrictions.

7. The real-time generation method of dance music according to any one of claims 1 to 6, characterized in that: Also includes: The generated real-time dance music is further used to generate corresponding music scores.

8. A real-time generation device for dance music, characterized in that: include: A real-time picture analysis module is used to capture the real-time picture of human body movements and analyze the dance visual information and scene information in the real-time picture of human body movements; A music attribute information generation module is used to extract human motion information and analyze facial expressions from the dance visual information to obtain human posture feature parameters, and generate real-time music attribute information based on the human posture feature parameters and scene information; The dance music real-time generation module is used to use the music attribute information as the node feature in the music AND-OR graph, sample the music AND-OR graph, and generate real-time dance music.

9. The real-time generation device for dance music according to claim 8, characterized in that: The music attribute information generation module generates real-time music attribute information according to the human body posture feature parameters and scene information, including: Based on the human body posture feature parameters and scene information, a multi-layer perceptron is used to predict and generate style labels to generate corresponding music attribute information; The music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style tag.

10. The real-time generation device for dance music according to claim 9, characterized in that: The music attribute information generation module selects the point with the highest coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

11. The real-time generation device for dance music according to claim 8, characterized in that: The dance music real-time generation module uses the music attribute information as a node feature in a music AND / OR graph, samples the music AND / OR graph, and generates a real-time dance music, including: The music attribute information is mapped to different nodes of the music AND / OR graph according to the time dimension, and the music attribute information of each node in the music AND / OR graph is sampled to generate a real-time dance soundtrack; Among them, generating real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm under music theory constraints to predict the next melody note, combining with the rhythmic pattern library under chord restrictions to generate chord textures, and generating percussion accompaniment clips under beat restrictions.

12. The real-time generation device for dance music according to claim 8, characterized in that: Also includes: The music score generation module is used to further generate corresponding music scores from the generated real-time dance music.

13. An electronic device comprising one or more processors and a storage device, characterized in that: The storage device is used to store executable instructions, and when the executable instructions are executed by the above-mentioned processor, the real-time generation method of dance music as described in any one of claims 1-7 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, enable the processor to execute the real-time generation method of dance music according to any one of claims 1 to 7.

15. A computer program, characterized in that The computer program includes computer executable instructions, which, when executed, are used to implement the real-time generation method of dance music as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Music reconstruction method base on movement image analysis

    CN101409070A

  • Method, device and apparatus to control dance movement of robot, and medium

    CN108527376A

  • Method and device for generating sports video games

    CN112685592A

  • Music dance posture generation method based on multi-feature fusion strategy

    CN114998984A

  • Multi-musical-instrument music generation method based on dance

    CN116758881A

Cited By

  • Dance performance method and device of mechanical arm, electronic equipment and storage medium

    CN119839862A

  • A dance performance method and device of a mechanical arm, an electronic device and a storage medium

    CN119839862B