Dance music matching method and apparatus, and entertainment device

Abstract

The application relates to a dance music matching method and device and entertainment equipment, the method comprising the following steps: capturing action images of a user dance segment, and identifying bone frame data of each frame of action images; wherein the bone frame data comprises each bone point and corresponding bone coordinates; calculating an action frequency parameter according to the bone frame data, and combining the bone frame data into several groups of real-time action data according to the action frequency parameter; comparing the real-time action data with reference action data pre-stored in an action library to determine an action type; and matching target music from a music library according to the action type and the real-time action data; according to the technical scheme, in an impromptu dancing scene, suitable music can be automatically matched and played according to dance actions randomly made by a user, so that the music played by the entertainment equipment can be best matched to the dance actions, and the application experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a dance music matching method, device, entertainment equipment, computer equipment and computer storage medium. Background Art

[0002] At present, with the improvement of people's living standards, dancing has become an extremely popular form of entertainment. In some impromptu dance scenes, users will make some random movements, and the backstage operators will select suitable music for playback based on the classification and frequency of the movements. In the process of selecting music, the professional requirements for the operators are high, and the selected music sometimes cannot best match the user's dance movements. Summary of the Invention

[0003] In order to solve one of the above technical defects, the present application provides a dance music matching method, device, entertainment equipment, computer equipment and computer storage medium, which can automatically match music for users and optimally match dance movements.

[0004] A music matching method for dance, comprising:

[0005] Capturing motion images of a user's dance clips and identifying skeletal frame data of each frame of the motion images; wherein the skeletal frame data includes each skeletal point and its corresponding skeletal coordinates;

[0006] Calculating motion frequency parameters according to the skeleton frame data, and combining the skeleton frame data into a plurality of groups of real-time motion data according to the motion frequency parameters;

[0007] Comparing the real-time motion data with reference motion data pre-stored in a motion library to determine the motion type;

[0008] Target music is matched from a music library according to the action type and the real-time action data.

[0009] In one embodiment, the action library includes multiple action types and corresponding actions, and each action includes skeletal coordinate data of at least one key frame.

[0010] In one embodiment, the music library includes categories corresponding to the action types, each category corresponds to a number of music, and each music includes an audio resource and music parameters matching the action data.

[0011] In one embodiment, calculating motion frequency parameters based on the skeleton frame data, and combining the skeleton frame data into several groups of real-time motion data based on the motion frequency parameters, includes:

[0012] Calculating the bone motion amplitude parameter according to each of the bone frame data;

[0013] Calculating a set number of skeleton frame data to obtain a set of skeleton motion amplitude parameters of action image frames;

[0014] Determining the low peak position of the skeletal motion amplitude from each skeletal motion amplitude parameter, and dividing the motion into a plurality of groups using the low peak position as the motion segmentation point;

[0015] The corresponding action BPM and action amplitude parameters are calculated based on the skeletal motion amplitude parameters of each group of actions; wherein the action BPM is defined as the number of times the action is completed within one minute.

[0016] In one embodiment, calculating the skeletal motion amplitude parameter according to each of the skeletal frame data includes:

[0017] Read two adjacent skeleton frame data in sequence;

[0018] Calculate the offset vector of each bone point in the two bone frame data;

[0019] Adding the lengths of the offset vectors of the various skeleton points to obtain the lengths and values ​​of the offset vectors of the various skeleton points, and using the lengths and values ​​as the skeleton motion amplitude parameters of the next frame of skeleton frame data;

[0020] In this way, the bone motion amplitude parameters of each frame of bone frame data are calculated.

[0021] In one embodiment, the corresponding action BPM and action amplitude parameters are calculated based on the skeletal motion amplitude parameters of each group of actions, including:

[0022] Calculate the period of the motion curve of each group of movements in real time to determine its minimum beat interval; calculate the valley point of the motion curve using a moving window; calculate the movement BPM based on the minimum beat interval and the valley point;

[0023] The motion amplitude parameters are calculated based on the skeletal motion amplitude parameters of each group of actions.

[0024] In one embodiment, comparing the real-time motion data with reference motion data pre-stored in a motion library to determine the motion type includes:

[0025] The skeleton frame data of each action data is compared with the skeleton coordinate data of each action key frame in the action library; the reference action data with the highest similarity is determined, and its type is used as the action type.

[0026] In one embodiment, the music parameters include audio BPM and sound intensity range;

[0027] Matching target music from a music library according to the action type and the real-time action data includes:

[0028] Matching the action type to a music category corresponding to the music library;

[0029] Matching a number of music pieces with the closest audio BPM in the music category according to the action BPM;

[0030] Target music with a matching sound intensity range is selected from the music according to the motion amplitude parameter.

[0031] A music matching device for dance, comprising:

[0032] A skeleton processing module is used to capture the motion image of the user's dance clip and identify the skeleton frame data of each frame of the motion image; wherein the skeleton frame data includes each skeleton point and its corresponding skeleton coordinates;

[0033] an action combination module, configured to calculate action frequency parameters based on the skeleton frame data, and combine the skeleton frame data into a plurality of groups of real-time action data based on the action frequency parameters;

[0034] An action comparison module, configured to compare the real-time action data with reference action data pre-stored in an action library to determine the action type;

[0035] The music matching module is used to match target music from a music library according to the action type and the real-time action data.

[0036] An entertainment device includes: a camera and a music playing device; wherein the camera is used to capture the motion image of the user's dance segment; and the host device is used to execute the steps of the above-mentioned dance music matching method.

[0037] The above technical solution captures the motion images of the user's dance clips and identifies the skeleton frame data, calculates the motion frequency parameters and combines the real-time motion data based on the skeleton frame data; uses the real-time motion data to compare and determine the motion type from the motion library, and then matches the target music from the music library; this technical solution can automatically match the appropriate music for playback based on the dance movements that the user may randomly perform in an impromptu dance scene, so that the music played by the entertainment device can be best matched to the dance movements, thereby improving the user's application experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 This is a schematic diagram of an example hardware environment;

[0040] Figure 2 This is a diagram of the action library data structure of an example;

[0041] Figure 3 This is a diagram of an example music library data structure;

[0042] Figure 4 is a flow chart of a method for matching dance music according to an embodiment;

[0043] Figure 5 This is an example offset vector diagram;

[0044] Figure 6 This is a schematic diagram of an example of dividing several groups of actions;

[0045] Figure 7 The figure is a schematic structural diagram of a dance music matching device according to an embodiment. DETAILED DESCRIPTION

[0046] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0047] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "the," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of the stated features, integers, steps, and operations, but does not preclude the presence or addition of one or more other features, integers, steps, and operations.

[0048] The technical solution of this application first builds an action library and a music library. Then, when performing music matching, the user dances in front of the camera, and the camera captures the user's dance action image. By processing the action image, the user's action data is obtained. The action data is used to compare the action library and the matching music library to obtain the target music corresponding to the dance clip. For the real-time hardware environment of the technical solution of this application, please refer to Figure 1 , Figure 1 This is a schematic diagram of an example hardware environment. The music playback device connected to the camera can be connected to the cloud server through the network. The action library and music library can be built on the server. When the user dances in front of the camera, the music playback device can process the action image captured by the camera, compare it with the action library, and match the target music from the music library. It should be noted that the action library and music library can also be built locally on the music playback device.

[0049] Based on this, in one embodiment, an action library can be pre-generated. For the action library data structure, refer to Figure 2 As shown, Figure 2 This is an example diagram of the action library data structure, which can include multiple action types, type 1 to type n in the figure, such as Tai Chi, Latin dance, street dance, etc.; each type corresponds to several actions, and each action type includes several actions, action 1 to action m in the figure, such as White Crane Spreads Wings, Single Whip, etc.; an action can include one or more action key frame skeleton coordinate data, such as key frame 1 skeleton frame data - key frame k skeleton frame data in the figure.

[0050] Correspondingly, you can pre-generate a music library. For the music library data structure, refer to Figure 3 As shown, Figure 3 This is a diagram of the data structure of a music library. The music library includes types corresponding to action types, such as Type 1 to Type P in the figure, such as Tai Chi, Latin dance, and street dance. Each type corresponds to several pieces of music, such as Music 1 to Music Q in the figure. Each piece of music includes audio resources and music parameters that match the action data. The music parameters can be audio BPM (Beat Per Minute, the unit of beats per minute) and sound intensity range, etc.

[0051] refer to Figure 4 As shown, Figure 4 The flowchart of a method for matching dance music according to an embodiment includes the following steps:

[0052] S10, capturing motion images of a user's dance clip, and identifying skeleton frame data of each frame of the motion image; wherein the skeleton frame data includes each skeleton point and its corresponding skeleton coordinates.

[0053] In this step, the camera can be used to capture the user's dance motion image in real time, identify the skeleton points of each frame of the motion image, such as 14 skeleton points, obtain the skeleton coordinates of each skeleton point, and obtain skeleton frame data.

[0054] S20, calculating motion frequency parameters according to the skeleton frame data, and combining the skeleton frame data into several groups of real-time motion data according to the motion frequency parameters.

[0055] In this step, the motion frequency parameters of each frame of skeleton frame data are calculated, and each action is identified by the motion frequency parameters. Then, the skeleton frame data is divided into several actions, and the skeleton frame data of each frame is combined into real-time action data.

[0056] In one embodiment, the process of step S20 may specifically include the following:

[0057] S201, calculating the skeleton motion amplitude parameter according to each skeleton frame data; as an embodiment, step S201 specifically includes the following:

[0058] a. Read two adjacent skeleton frame data in sequence; for example, take out the skeleton frame data of two adjacent action images, assuming they are named frame A skeleton frame data and frame B skeleton frame data.

[0059] b. Calculate the offset vector of each bone point in the two bone frame data; specifically, calculate the offset vector corresponding to each bone point in the A frame bone frame data and the B frame bone frame data.

[0060] c. Add the lengths of the offset vectors of each bone point to obtain the length and value of the offset vector of each bone point, and use the length and value as the bone motion amplitude parameters of the next frame of bone frame data.

[0061] Specifically, the lengths of the offset vectors of the various skeleton points are added together to obtain the lengths and values ​​of the offset vectors of the various skeleton points of the skeleton frame data of each frame, and the lengths and values ​​of the offset vectors are used as the motion amplitude parameters of the B-frame skeleton frame data.

[0062] By analogy, the above steps ac are repeated to calculate the bone motion amplitude parameters of each frame of bone frame data.

[0063] For the above calculation process, take the coordinates of the bone points as two-dimensional coordinates as an example, refer to Figure 5 As shown, Figure 5 This is an example of an offset vector diagram. The motion amplitude parameter can be calculated as follows:

[0064] Hypothetical design offset vector O 1 For bone points A 1 (1,1) to B 1 The offset vector of (2, 2) is O 1 The calculation is (1,1), O 1 The length is ; Assuming that there are 14 key points for bone sampling, there are A 1 … A 14 and B 1 … B 14 Two sets of bone points, the offset vector of each bone point can be calculated as O 1 … O 14, can be calculated according to the above formula O 1 … O 14 The length of the B action image frame is O 1 … O 14 Length and value .

[0065] S202, calculating a set number of skeleton frame data to obtain a set of skeleton motion amplitude parameters of action image frames; specifically, after accumulating a certain number of skeleton frame data, a set of skeleton motion amplitude parameters of action image frames can be obtained.

[0066] S203 , determining the low peak position of the skeletal motion amplitude from each skeletal motion amplitude parameter, and dividing the motion into several groups using the low peak position as the motion segmentation point.

[0067] refer to Figure 6 As shown, Figure 6 This is a schematic diagram of an example of dividing several groups of actions. It can be seen from the figure that in the motion curve of each action, there is a low peak position of the bone motion amplitude in each bone motion amplitude parameter, and the action is divided into several groups of actions and their corresponding bone motion amplitude parameters based on the low peak position. The bone motion amplitude parameters of each frame in the group of actions are combined to obtain the motion amplitude data of the group of actions; as shown in the figure, action 1, action 2 and action 3 are respectively combined into corresponding motion amplitude data.

[0068] S204 , calculating the corresponding action BPM and action amplitude parameters according to the skeletal motion amplitude parameters of each action group; wherein the action BPM is defined as the number of times the action is completed within one minute.

[0069] As an embodiment, step S204 specifically includes:

[0070] (1) Calculate the BPM of an action. The specific method is as follows:

[0071] (I) Calculate the period of the motion curve of each group of actions in real time to determine its minimum beat interval R. Specifically, calculate the autocorrelation of the motion curve and its period. Based on the strong periodicity of the beat points, the period of the motion data can be calculated in real time to determine its minimum beat interval R. The calculation formula of R is as follows:

[0072]

[0073] In the formula, we get R(t) The maximum value maxR is the beat interval, t For the range of movement.

[0074] (II) Calculate the valley points of the motion curve using a moving window. Specifically, the valley points are calculated using a moving window and a first-order difference method. The calculation method is as follows:

[0075] 1) Initialize the window length to N , the window moves to 1 / 4N , real-time calculation cycle R , N>2R .

[0076] 2) Calculate the first-order difference within the window length. If and , then the point may be a valley point P. If the interval between this point and the previous valley point P0 is , then the point is considered to be a valley point, in order to prevent the local valley point caused by the pulling of the action curve. Threshould is the critical point.

[0077] (III) Calculate the BPM of the action based on the minimum beat interval and the valley point; specifically, the duration of each action can be calculated based on the segmented actions t , calculate the action BPM using the action duration, such as action BPM = 60s / t .

[0078] (2) Calculating the motion amplitude parameter based on the skeletal motion amplitude parameters of each group of motions; specifically, the average value of the skeletal motion amplitude parameters of each group of motions can be calculated as the motion amplitude parameter, or the highest value among the skeletal motion amplitude parameters of each group of motions can be selected as the motion amplitude parameter.

[0079] The solution of the above embodiment defines the action BPM and calculates the corresponding action BPM through the skeletal movement amplitude parameter, so that the user can be matched with music with the same audio BPM, thereby improving the accuracy of music matching.

[0080] S30: Compare the real-time motion data with the reference motion data pre-stored in the motion library to determine the motion type.

[0081] In this step, the pre-generated action library is used to compare the action types, such as Figure 1 In the process, a large amount of reference action data is stored in the action library. Combined with the skeleton frame data of the key frames under each action in the data structure, the action type corresponding to the current dance clip can be determined.

[0082] In one embodiment, step S30 may specifically include the following:

[0083] First, the skeleton frame data of each action data is compared with the skeleton coordinate data of each action key frame in the action library; through the comparison, the reference action data with the highest similarity can be determined, and its type is used as the action type.

[0084] S40, matching target music from a music library according to the action type and the real-time action data.

[0085] In one embodiment, step S40 may specifically include the following:

[0086] S401, matching the action type to the corresponding music category in the music library; specifically, the music can be classified according to the action type in the music library. Of course, a music library can also be established for each action type.

[0087] S402, matching several pieces of music with the closest audio BPM in the music classification according to the action BPM; specifically, combining the action BPM defined and calculated above, matching music with similar beats, and obtaining multiple candidate pieces of music.

[0088] S403, select target music with a sound intensity range that matches the movement amplitude parameter from the music; specifically, after determining multiple pieces of music, select target music based on the sound intensity ranges of different music that match the movement amplitude parameter, so as to match the user with music with a sound intensity range that is suitable for the movement amplitude, thereby improving the matching degree between the movement and the music.

[0089] Based on the technical solutions of the above embodiments, it is possible to automatically match appropriate music for playback in an impromptu dance scene based on the dance movements that the user may randomly perform. It is possible to match music with a consistent beat to the dance movements, and to match music with appropriate frequency bands to dance movements with different amplitudes, thereby greatly improving the user's application experience.

[0090] An embodiment of a music matching device for dance is described below.

[0091] refer to Figure 7 , Figure 7 This is a schematic structural diagram of a dance music matching device according to an embodiment, comprising:

[0092] The skeleton processing module 10 is used to capture the motion image of the user's dance clip and identify the skeleton frame data of each frame of the motion image; wherein the skeleton frame data includes each skeleton point and its corresponding skeleton coordinates;

[0093] an action combining module 20 for calculating action frequency parameters according to the skeleton frame data, and combining the skeleton frame data into a plurality of groups of real-time action data according to the action frequency parameters;

[0094] An action comparison module 30 is used to compare the real-time action data with reference action data pre-stored in an action library to determine the action type;

[0095] The music matching module 40 is used to match target music from a music library according to the action type and the real-time action data.

[0096] The dance music matching device of this embodiment can execute a dance music matching method provided in the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the dance music matching device in each embodiment of the present application correspond to the steps in the dance music matching method in each embodiment of the present application. For the detailed functional description of each module of the dance music matching device, please refer to the description of the corresponding dance music matching method shown in the previous text, and will not be repeated here.

[0097] The following describes embodiments of a computer device and computer-readable storage medium of the present application. The computer device includes one or more processors and a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the dance music matching method according to any of the above-described embodiments. The storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded by the processor and executes the dance music matching method according to any of the above-described embodiments.

[0098] The following describes an embodiment of the entertainment device of the present application.

[0099] like Figure 1 As shown, the entertainment device provided by the present application includes: a camera and a music playback device; wherein the camera is used to capture the motion images of the user's dance clips, and the music playback device is used to execute the steps of the music matching method for the dance of any of the above embodiments.

[0100] The computer device, computer-readable storage medium, and entertainment device of the above-mentioned embodiments can automatically match appropriate music for playback based on the dance movements that the user may randomly perform in an impromptu dance scene, so that the music played by the entertainment device can be optimally matched to the dance movements, thereby improving the user's application experience.

[0101] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A dance music matching method, characterized in that: include: Capturing motion images of a user's dance clips and identifying skeletal frame data of each frame of the motion images; wherein the skeletal frame data includes each skeletal point and its corresponding skeletal coordinates; Calculating motion frequency parameters according to the skeleton frame data, and combining the skeleton frame data into a plurality of groups of real-time motion data according to the motion frequency parameters; Comparing the real-time motion data with reference motion data pre-stored in a motion library to determine the motion type; the motion library includes a plurality of motion types and a plurality of corresponding motions, each motion including skeletal coordinate data of at least one key frame; Target music is matched from a music library according to the action type and the real-time action data; the music library includes types corresponding to the action types, each type corresponds to a number of music, and each music includes audio resources and music parameters matching the action data.

2. The dance music matching method according to claim 1, characterized in that: Calculating motion frequency parameters according to the skeleton frame data, and combining the skeleton frame data into several groups of real-time motion data according to the motion frequency parameters, including: Calculating the bone motion amplitude parameter according to each of the bone frame data; Calculating a set number of skeleton frame data to obtain a set of skeleton motion amplitude parameters of action image frames; Determining the low peak position of the skeletal motion amplitude from each skeletal motion amplitude parameter, and dividing the motion into a plurality of groups using the low peak position as the motion segmentation point; The corresponding action BPM and action amplitude parameters are calculated based on the skeletal motion amplitude parameters of each group of actions; wherein the action BPM is defined as the number of times the action is completed within one minute.

3. The dance music matching method according to claim 2, characterized in that: Calculating the bone motion amplitude parameters according to each of the bone frame data includes: Read two adjacent skeleton frame data in sequence; Calculate the offset vector of each bone point in the two bone frame data; Adding the lengths of the offset vectors of the various skeleton points to obtain the lengths and values ​​of the offset vectors of the various skeleton points, and using the lengths and values ​​as the skeleton motion amplitude parameters of the next frame of skeleton frame data; In this way, the bone motion amplitude parameters of each frame of bone frame data are calculated.

4. The dance music matching method according to claim 3, characterized in that: Calculate the corresponding action BPM and action amplitude parameters based on the skeletal motion amplitude parameters of each group of actions, including: Calculate the period of the motion curve of each group of movements in real time to determine its minimum beat interval; calculate the valley point of the motion curve using a moving window; calculate the movement BPM based on the minimum beat interval and the valley point; The motion amplitude parameters are calculated based on the skeletal motion amplitude parameters of each group of actions.

5. The dance music matching method according to claim 4, characterized in that: Comparing the real-time motion data with reference motion data pre-stored in a motion library to determine the motion type includes: The skeleton frame data of each action data is compared with the skeleton coordinate data of each action key frame in the action library; the reference action data with the highest similarity is determined, and its type is used as the action type.

6. The dance music matching method according to claim 5, characterized in that: The music parameters include audio BPM and sound intensity range; Matching target music from a music library according to the action type and the real-time action data includes: Matching the action type to a music category corresponding to the music library; Matching a number of music pieces with the closest audio BPM in the music category according to the action BPM; Target music with a matching sound intensity range is selected from the music according to the motion amplitude parameter.

7. A dance music matching device, characterized in that: include: A skeleton processing module is used to capture the motion image of the user's dance clip and identify the skeleton frame data of each frame of the motion image; wherein the skeleton frame data includes each skeleton point and its corresponding skeleton coordinates; an action combination module, configured to calculate action frequency parameters based on the skeleton frame data, and combine the skeleton frame data into a plurality of groups of real-time action data based on the action frequency parameters; An action comparison module is used to compare the real-time action data with reference action data pre-stored in an action library to determine the action type; the action library includes multiple action types and their corresponding actions, and each action includes skeletal coordinate data of at least one key frame; A music matching module is used to match target music from a music library according to the action type and the real-time action data; the music library includes types corresponding to the action types, each type corresponds to a number of music, and each music includes audio resources and music parameters matching the action data.

8. An entertainment device, characterized in that: include: A camera and a music playing device; wherein the camera is used to capture the motion images of the user's dance clips; and the music playing device is used to execute the steps of the dance music matching method according to any one of claims 1 to 6.

Images