A real-time generation method and device for dance music

Abstract

An embodiment of the present invention provides a method and device for real-time generation of dance music. The method includes: capturing real-time images of human body movements, and parsing dance visual information and scene information in the real-time images of human body movements; extracting human body movement information and facial expression analysis from the dance visual information to obtain human body posture feature parameters, and generating real-time music attribute information based on the human body posture feature parameters and scene information; using the music attribute information as node features in a music AND / OR graph, sampling the music AND / OR graph, and generating real-time dance music. Through the real-time generation method and device for dance music of this embodiment, it is not necessary to generate music based on a completed dance video, but music creation can be completed while dancing, thereby owning dual copyrights of creating dance and music works.

Description

Technical Field

[0001] The present invention relates to the technical field of audio data processing, and in particular to a method and device for generating dance music in real time. Background Art

[0002] Currently, dance music generation solutions are all based on deep learning neural network architectures, and the implementation phase consists of two parts: encoding and decoding. The purpose of encoding is to extract features from the input information and compress it; the purpose of decoding is to reconstruct this information (features) into music. In the encoding phase, deep learning methods, including CNN (convolutional neural network) or Transformer methods, are used; in the decoding phase, Transformer methods are used.

[0003] However, the aforementioned music generation schemes require a large amount of data for learning, both in the encoding and decoding stages. Furthermore, due to the inherent characteristics of deep learning networks, namely the large number of parameters, such methods cannot perform real-time generation, that is, they cannot capture real-time images to generate real-time music. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for real-time generation of dance music, so as to solve the problem that existing technical solutions cannot capture real-time images to generate real-time music.

[0005] In order to solve the above technical problems, a specific embodiment of the present invention provides a real-time generation method of dance music, including: capturing real-time images of human body movements, and parsing dance visual information and scene information in the real-time images of human body movements; extracting human body movement information and analyzing facial expressions from the dance visual information to obtain human body posture feature parameters, and generating real-time music attribute information based on the human body posture feature parameters and scene information; using the music attribute information as node features in a music AND / OR graph, sampling the music AND / OR graph, and generating real-time dance music.

[0006] Furthermore, in one embodiment, the capturing of real-time images of human body movements and parsing of dance visual information and scene information in the real-time images of human body movements include: the real-time images of human body movements are two-dimensional RGB images of human body movements, and the dance visual information and scene information in the two-dimensional RGB images of human body movements are parsed through machine vision; wherein, the dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by a human posture estimation algorithm; and the scene information includes the background, objects in the image, and dancers' costumes.

[0007] Furthermore, in one embodiment, the human body movement information is extracted and facial expression is analyzed from the dance visual information to obtain human body posture feature parameters, including: determining the human body posture feature parameters based on the three-dimensional position coordinates of the 33 human body joints; wherein the human body posture feature parameters include: the distance between the limbs and the center of the human body, the joint angle, the width and height of the body contour, the joint speed, the joint acceleration, the joint pause point detection, the movement amplitude, and the facial expression recognition.

[0008] Furthermore, in one embodiment, the real-time music attribute information is generated based on the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label based on the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0009] Furthermore, in one embodiment, the point with the highest degree of coincidence with the action beat is selected as the beat, and the initial music speed bpm is calculated to generate the rhythm attribute information.

[0010] Furthermore, in one embodiment, the music attribute information is used as a node feature in a music AND / OR graph, and the music AND / OR graph is sampled to generate a real-time dance soundtrack, including: mapping the music attribute information to different nodes of the music AND / OR graph according to the time dimension, sampling the music attribute information of each node in the music AND / OR graph, and then generating a real-time dance soundtrack; wherein, generating a real-time dance soundtrack also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, generating a chord texture by combining it with a rhythmic pattern library under chord constraints, and generating a percussion accompaniment segment under beat constraints.

[0011] Furthermore, in one embodiment, the method further includes: generating corresponding music scores for the generated real-time dance music.

[0012] In order to solve the above technical problems, a specific embodiment of the present invention also provides a real-time generation device for dance music, including: a real-time image analysis module, used to capture real-time images of human body movements, and parse out dance visual information and scene information in the real-time images of human body movements; a music attribute information generation module, used to extract human body movement information and facial expression analysis from the dance visual information, obtain human body posture feature parameters, and generate real-time music attribute information based on the human body posture feature parameters and scene information; a real-time dance music generation module, used to use the music attribute information as node features in a music AND / OR graph, sample the music AND / OR graph, and generate real-time dance music.

[0013] Furthermore, in one embodiment, the music attribute information generation module generates real-time music attribute information based on the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label based on the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0014] Furthermore, in one embodiment, the music attribute information generation module selects the point with the highest degree of coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

[0015] Furthermore, in one embodiment, the dance music real-time generation module uses the music attribute information as a node feature in a music AND / OR graph, samples the music AND / OR graph, and generates real-time dance music, including: mapping the music attribute information to different nodes of the music AND / OR graph according to the time dimension, sampling the music attribute information of each node in the music AND / OR graph, and then generating real-time dance music; wherein, generating real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, generating a chord texture in combination with a rhythmic pattern library under chord constraints, and generating a percussion accompaniment segment under beat constraints.

[0016] Furthermore, in one embodiment, the apparatus for generating dance music in real time further includes: a music score generating module, configured to further generate a corresponding music score from the generated real-time dance music.

[0017] Another aspect of an embodiment of the present invention provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the method of the embodiment of the present invention is implemented.

[0018] Another aspect of an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method of the embodiment of the present invention when executed by a processor.

[0019] Another aspect of an embodiment of the present invention provides a computer program, wherein the computer program includes computer-executable instructions, and the instructions are used to implement the method of the embodiment of the present invention when executed.

[0020] The real-time dance music generation method and device proposed in the embodiments of the present invention differ from existing methods that generate music using existing dance videos. Instead, the method captures dance footage in real time, uses machine vision to parse dance movement and scene information from the RGB images, and then maps this information into musical attributes, which serve as node features in a music AND / OR graph. The music AND / OR graph is then used for sampling, generating the dance music in real time and the corresponding musical score simultaneously. The method of this embodiment eliminates the need to generate music based on a completed dance video; instead, music creation can be completed while the dancer is dancing, thereby securing dual copyrights for both the dance and the musical work. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are part of the specification of the present invention, illustrate exemplary embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0022] Figure 1 4 is a process flow chart of a method for real-time generation of dance music according to an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of the three-dimensional position coordinates of 33 human joints predicted by the human posture estimation algorithm according to an embodiment of the present invention;

[0024] Figure 3 This is an example diagram of a simple multimodal music generation based on music and / or graph according to an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of the structure of a real-time generation device for dance music according to an embodiment of the present invention;

[0026] Figure 5 2 is a schematic structural diagram of a device for real-time generation of dance music according to another embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed in the present invention will be clearly illustrated with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the contents of the present invention based on the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0028] The exemplary embodiments of the present invention and their description are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0029] The terms “first,” “second,” etc. used herein do not particularly refer to an order or sequence, nor are they intended to limit the present invention. They are merely used to distinguish elements or operations described with the same technical terms.

[0030] The directional terms used herein, such as up, down, left, right, front, or back, are only used to refer to the directions in the accompanying drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0031] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0032] As used herein, "and / or" includes any and all combinations of the items mentioned.

[0033] Regarding "plurality" in this document, "plurality" includes "two" and "more than two"; regarding "plurality groups" in this document, "plurality groups" includes "two groups" and "more than two groups".

[0034] As used herein, the terms "substantially" and "approximately" are used to modify any quantity or error that may vary slightly, but such variations or errors do not alter the essence of the quantity. Generally speaking, the range of such variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art will appreciate that the aforementioned values ​​may be adjusted based on actual needs and are not intended to be limiting.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0036] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). When expressions such as "at least one of A, B, or C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases indicating two or more optional items, whether in the specification, claims, or drawings, should be understood to provide the possibility of including one, either, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0037] The real-time dance music generation method of the embodiment of the present invention differs from existing methods that generate music using existing dance videos. Instead, it captures dance footage in real time, uses machine vision to parse dance movement and scene information from the RGB images, and then maps this information into musical attributes, which serve as node features in a music AND / OR graph. The music AND / OR graph is then used for sampling, generating the dance music in real time and simultaneously generating the corresponding musical score. The method of this embodiment eliminates the need to generate music based on a completed dance video, allowing the user to create music while dancing, thereby securing dual copyrights for both the dance and the music.

[0038] Figure 1 This is a process flow chart for a real-time dance music generation method according to an embodiment of the present invention. As shown, the method includes: Step S101, capturing real-time human motion footage and parsing the dance visual information and scene information from the real-time human motion footage; Step S102, extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters, and generating real-time music attribute information based on the human posture feature parameters and scene information; Step S103, using the music attribute information as node features in a music AND / OR graph, sampling the music AND / OR graph, and generating real-time dance music.

[0039] In step S101 of this embodiment, the human body movement picture captured in real time may be a two-dimensional RGB picture of human body movement, for example, a picture of a dancer's movement on the stage captured in real time by a camera.

[0040] In this embodiment, the extraction of dance visual information through machine vision involves two aspects: human motion information extraction and facial expression analysis. Human motion information extraction primarily relies on the extraction of three-dimensional information about the human skeleton; facial expression analysis can extract facial feature points and perform regression or classification based on these features, or directly perform regression or classification based on facial images.

[0041] like Figure 2 As shown in the figure, dance motion information includes the 3D position coordinates of 33 human joints predicted by the human pose estimation algorithm. In specific implementations, a 2D RGB image of human motion can be input into the BlazePose lightweight convolutional neural network architecture to obtain the coordinate information of each joint in the image coordinate system. Each joint is represented as (x, y, z, v), where x and y represent the normalized coordinate position of the joint in the image coordinate system; z represents the depth of the joint with the midpoint of the hip as the origin; and v represents the probability of the joint being visible.

[0042] In this embodiment, the scene information parsed from the captured 2D RGB images of human motion includes background information, object information in the image, dancer's clothing information, etc. This information is used in the subsequent style prediction steps. For example, a style label can be obtained based on the dancer's clothing information and the stage background information.

[0043] In step S102, human motion information is extracted and facial expression is analyzed from the dance visual information to obtain human posture characteristic parameters, including: determining the human posture characteristic parameters according to the three-dimensional position coordinates of the 33 human joints.

[0044] In practice, after obtaining the raw joint information, a series of features can be calculated, converting the positional information of the human joints into feature vectors. For example, the center point between the left and right hips is considered the human center, and the distance from the center point to the center point of the left and right shoulders is considered the human torso size. The relative position of each joint is then initialized based on the human center point, and the relative distance between each joint and the center point is used as one part of the feature. Pairwise distances between joints are extracted as another part of the feature according to the joint association rules.

[0045] In an embodiment of the present invention, the human posture characteristic parameters obtained according to the above method include: the distance between the limbs and the center of the human body, joint angles, body outline width and height, joint speed, joint acceleration, joint pause point detection, movement amplitude, facial expression recognition, etc.

[0046] In step S102, after obtaining a series of human posture feature parameters, a multi-layer perceptron is used to predict and generate a style label based on the human posture feature parameters and scene information to generate corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0047] In practice, a multi-layer perceptron model is used to predict style labels, thereby generating matching instrument combinations, scales, and chords. The beats are then selected as the points with the highest degree of overlap with the action's re-beats, and the initial music tempo (BPM) is calculated based on these beats.

[0048] In this embodiment of the present invention, a multi-layer perceptron (MLP) is a small neural network. Using an MLP for style prediction is equivalent to using a neural network for style classification. This involves two steps: training and utilization. The training process involves collecting data on different styles, such as human joint information for folk dance and modern dance, and using this information to train a small neural network. The utilization process involves feeding the human posture feature parameters obtained in step S102 into the network to output the dance genre.

[0049] In embodiments of the present invention, music attribute information includes specific features of music, such as "musical style," "chords" composed of notes, "intervals" between notes, and "musical rhythm." In embodiments of the present invention, this information can be categorized into accompaniment attribute information, melody attribute information, harmony attribute information, and rhythm attribute information. For example, accompaniment attribute information includes the soundtrack composition, melody attribute information includes scales and interval relationships, harmony attribute information includes chord composition, and rhythm attribute information includes tempo, BPM, and rhythmic arrangement.

[0050] In this embodiment, for the attribute "music rhythm," for example, the point where the action has the highest overlap with the beat can be selected as the beat. The moment of sudden change in the action, that is, the moment when the action changes the most within a certain period of time, or the point when the human dance movement has the greatest acceleration, can be considered a beat point. However, the present invention is not limited to this method for generating music rhythm attributes. For example, the expressions corresponding to facial joints can also be used to control the rhythm of the music by modifying the mapping relationship. Similarly, the generation of other music attribute information can be modified or improved based on the mapping relationship of the multi-layer perceptron.

[0051] After obtaining the music attribute information, it is used as the node feature in the music AND-OR graph, and the music AND-OR graph is used for sampling to generate real-time dance background music.

[0052] In this embodiment of the present invention, a musical work is spatially subdivided into components such as melody, accompaniment, harmony, and rhythm, and temporally subdivided into sections, phrases, measures, and notes. This subdivision is purposeful and rational: numerous notes form a measure node, numerous measures form a phrase node, numerous phrases form a section node, and numerous sections constitute a piece of music. Furthermore, sections and phrases intuitively reflect the melody, supplemented by various accompaniments, harmonies, and rhythms. These numerous nodes interweave together, and music is the result of the specific instantiation of these nodes: numerous nodes coexist, each with different options, thus forming a musical AND / OR graph.

[0053] The learning of the music AND / OR graph is to use an algorithm to obtain the music attribute information of each node, such as the interval relationship, chord composition, rhythm arrangement, etc. in the music corresponding to the node; and music generation is the reverse process of learning, that is, using an algorithm to sample the music attribute information from each node. Therefore, as long as the music attribute information can be matched with the nodes in the music AND / OR graph, only sampling is required to generate music. The music attribute information can come from the statistical laws of a large number of music, or it can be obtained artificially through different technical means. This is also the core of multimodal music generation based on the music AND / OR graph. Of course, in the embodiment of the present invention, the music attribute information is the music attribute information generated by extracting and analyzing the real-time captured human body movement images and mapping the attribute features through steps S101 and S102.

[0054] In specific implementations, music attribute information acts as a global constant and participates in the music generation control at each moment. The music attribute information corresponding to each image frame is mapped to different nodes in a music AND / OR graph based on the time dimension. Music generation is performed using the music AND / OR graph by sampling the music attributes of each node. Furthermore, for each frame, the Hastings-Metropolis algorithm is used to predict the next melody note under musical constraints. This is then combined with a rhythmic pattern library to generate a chord texture under chord constraints, and percussion accompaniment is generated under tempo constraints, ultimately generating a real-time dance score. The "musical constraints," "chord constraints," "rhythmic pattern library," and "tempo constraints" all impose constraints on the generated music, aiming to ensure that the generated music better meets user needs and expectations. The Metropolis Hastings algorithm, a key component of Markov Chain Monte Carlo (MCMC) sampling technology, is employed to enhance music generation. By adjusting and employing Metropolis Hastings sampling, controlled music generation can be achieved, allowing for fine-grained control over the modification of existing musical structures and enabling users to fine-tune their work.

[0055] Figure 3This is a simple example diagram of multimodal music generation based on a music AND OR graph according to an embodiment of the present invention. In this example, a musical phrase is divided into four parts: melody, accompaniment, harmony, and rhythm. Each part is mapped to a different node in the music AND OR graph according to the time dimension (the top white dot: the "solid white dot" is the selected AND OR graph node, which means the musical phrase to be generated / changed; the "hollow dot" is an unselected node). These four parts correspond to different musical attributes, and their characteristics can be assigned values ​​through different modes. As for the use of music AND OR graphs for music generation, it is nothing more than sampling the musical attributes of each node.

[0056] Figure 4 This is a schematic diagram of the structure of a real-time dance music generation device according to an embodiment of the present invention. As shown in the figure, the device comprises: a real-time image analysis module 401 for capturing real-time human motion images and parsing the images to extract dance visual information and scene information; a music attribute information generation module 402 for extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters, and then generating real-time music attribute information based on the human posture feature parameters and scene information; and a real-time dance music generation module 403 for using the music attribute information as node features in a music AND / OR graph, sampling the music AND / OR graph, and generating real-time dance music.

[0057] In this embodiment, the real-time image parsing module 401 captures real-time images of human body movements and parses out dance visual information and scene information from the real-time images of human body movements, specifically including: the real-time images of human body movements are two-dimensional RGB images of human body movements, and the dance visual information and scene information from the two-dimensional RGB images of human body movements are parsed through machine vision; wherein, the dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by the human posture estimation algorithm; the scene information includes the background, objects in the image, and dancers' costumes.

[0058] In this embodiment, the music attribute information generation module 402 extracts human motion information and analyzes facial expressions on the dance visual information to obtain human posture feature parameters, including: determining the human posture feature parameters based on the three-dimensional position coordinates of the 33 human joints; wherein, the human posture feature parameters include: the distance between the limbs and the center of the human body, the joint angle, the width and height of the body contour, the joint speed, the joint acceleration, the joint pause point detection, the movement amplitude, and the facial expression recognition.

[0059] In this embodiment, the music attribute information generation module 402 generates real-time music attribute information based on the human posture feature parameters and scene information, including: using a multi-layer perceptron to predict and generate a style label based on the human posture feature parameters and scene information, and generating corresponding music attribute information; the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information and rhythm attribute information that match the style label.

[0060] In this embodiment, the music attribute information generating module 402 selects the point with the highest degree of coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

[0061] In this embodiment, the dance music real-time generation module 403 uses the music attribute information as the node feature in the music AND OR graph, samples the music AND OR graph, and generates real-time dance music, including: mapping the music attribute information to different nodes of the music AND OR graph according to the time dimension, sampling the music attribute information of each node in the music AND OR graph, and then generating real-time dance music; the dance music real-time generation module 403 is also used to use the music attribute information as a global constant to participate in the music generation control at each moment, use the Hastings-Metropolis algorithm to predict the next melody note under music theory constraints, combine it with the rhythm pattern library under chord constraints to generate chord texture, and generate percussion accompaniment fragments under beat constraints.

[0062] In this embodiment, Figure 5 As shown, the real-time generation apparatus for dance music further includes a music score generation module 404 for further generating corresponding music scores from the generated real-time dance music.

[0063] The method and apparatus for real-time generation of dance music disclosed in the above-mentioned embodiments differ from existing methods that generate music using existing dance videos. Instead, the method captures dance footage in real time, uses machine vision to parse dance movement and scene information from the RGB images, and then maps this information into musical attributes, which serve as node features in a music AND / OR graph. The music AND / OR graph is then used for sampling, generating the dance music in real time and simultaneously generating the corresponding musical score. The method of this embodiment eliminates the need to generate music based on a completed dance video, allowing the user to create music while dancing, thereby securing dual copyrights for both the dance and the music.

[0064] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic devices, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0065] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0066] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0068] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of the present invention may be made, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of the present invention may be made, without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0069] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which are intended to fall within the scope of the present invention.

Claims

1. A real-time generation method for dance music, characterized in that: include: Capturing real-time images of human body movements, and parsing dance visual information and scene information in the real-time images of human body movements, wherein the scene information includes background, objects in the images, and dancers' costumes; Extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters, and using a multi-layer perceptron to predict and generate style labels based on the human posture feature parameters and scene information to generate corresponding music attribute information, wherein the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information, and rhythm attribute information that match the style label; The music attribute information is mapped to different nodes of the music AND-OR graph according to the time dimension, and the music attribute information of each node in the music AND-OR graph is sampled to generate real-time dance music, wherein the music AND-OR graph includes measure nodes composed of a large number of notes, phrase nodes composed of a large number of measures, and segment nodes composed of a large number of phrases, and a large number of nodes are interwoven together to form the music AND-OR graph; Among them, generating real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm under music theory constraints to predict the next melody note, combining with the rhythmic pattern library under chord restrictions to generate chord textures, and generating percussion accompaniment clips under beat restrictions.

2. The real-time generation method of dance music according to claim 1, characterized in that: The capturing of the real-time picture of human body movements and parsing out dance visual information and scene information in the real-time picture of human body movements include: The real-time human body movement picture is a two-dimensional RGB picture of human body movement, and dance visual information and scene information in the two-dimensional RGB picture of human body movement are parsed by machine vision; The dance visual information includes the three-dimensional position coordinates of 33 human body joints predicted by a human body posture estimation algorithm.

3. The real-time generation method of dance music according to claim 2, characterized in that: The step of extracting human motion information and analyzing facial expressions from the dance visual information to obtain human posture feature parameters includes: Determine the human body posture characteristic parameters according to the three-dimensional position coordinates of the 33 human body joints; Among them, the human body posture characteristic parameters include: the distance between the limbs and the center of the human body, joint angles, body outline width and height, joint speed, joint acceleration, joint pause point detection, movement amplitude, and facial expression recognition.

4. The real-time generation method of dance music according to claim 1, characterized in that: The point with the highest coincidence with the action beat is selected as the beat, and the initial music speed bpm is calculated to generate the rhythm attribute information.

5. The real-time generation method of dance music according to any one of claims 1 to 4, characterized in that: Also includes: The generated real-time dance music is further used to generate corresponding music scores.

6. A real-time generation device for dance music, characterized in that: include: A real-time picture analysis module is used to capture the real-time picture of human body movements and analyze the dance visual information and scene information in the real-time picture of human body movements, wherein the scene information includes the background, objects in the picture, and dancers' costumes; A music attribute information generation module is used to extract human motion information and analyze facial expressions from the dance visual information to obtain human posture feature parameters, and to generate style labels using a multi-layer perceptron based on the human posture feature parameters and scene information to generate corresponding music attribute information, wherein the music attribute information includes accompaniment attribute information, melody attribute information, harmony attribute information, and rhythm attribute information that match the style label; A dance music real-time generation module is used to correspond the music attribute information to different nodes of the music AND-OR graph according to the time dimension, sample the music attribute information of each node in the music AND-OR graph, and then generate real-time dance music, wherein the music AND-OR graph includes a bar node composed of a large number of notes, a phrase node composed of a large number of bars, and a music segment node composed of a large number of phrases, and a large number of nodes are interwoven together to form the music AND-OR graph; Among them, generating real-time dance music also includes: using the music attribute information as a global constant to participate in the music generation control at each moment, using the Hastings-Metropolis algorithm under music theory constraints to predict the next melody note, combining with the rhythmic pattern library under chord restrictions to generate chord textures, and generating percussion accompaniment clips under beat restrictions.

7. The real-time generation device for dance music according to claim 6, characterized in that: The music attribute information generation module selects the point with the highest coincidence with the action beat as the beat, and calculates the initial music speed bpm to generate the rhythm attribute information.

8. The real-time generation device for dance music according to claim 6, characterized in that: Also includes: The music score generation module is used to further generate corresponding music scores from the generated real-time dance music.

9. An electronic device comprising one or more processors and a storage device, characterized in that: The storage device is used to store executable instructions, and when the executable instructions are executed by the above-mentioned processor, the real-time generation method of dance music as described in any one of claims 1-5 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, enable the processor to execute the real-time generation method of dance music according to any one of claims 1 to 5.

11. A computer program, characterized in that The computer program includes computer executable instructions, which, when executed, are used to implement the real-time generation method of dance music as described in any one of claims 1-5.

Images