Key position recognition method and device, footstep editing method and device, dance machine

Abstract

The present application relates to a key position identification method and device, a footstep pattern editing method and device, a computer device, and a computer-readable storage medium. The key position identification method includes: reading a key position pressure value detected by a footstep motion capture device at a beat point position, and determining a predicted key position stepped on by a user on the footstep motion capture device based on the key position pressure value; obtaining a real-time video image of the user stepping on the key position on the footstep motion capture device, and extracting human skeletal motion data corresponding to the beat point position from the real-time video image; calculating the angle parameters between each limb of the user relative to a preset human skeletal basic template based on the human skeletal motion data, and determining a predicted area where the key position stepped on by the user on the footstep motion capture device is located; if the predicted key position coincides with the predicted area, it is determined that the predicted key position is stepped on, and a corresponding key position symbol is generated; this technical solution can reduce invalid key signals, improve the footstep pattern editing effect, and enhance the user application experience.

Description

Technical Field

[0001] The present application relates to the technical field of entertainment equipment, and in particular to a key position recognition method and device, a footstep editing method and device, a dance machine, a computer device, and a computer-readable storage medium. Background Art

[0002] With the popularity of dance machines, people's lifestyles and entertainment have been greatly enriched. More and more players are starting to get involved in game dances. Even if they have no dance foundation, they can still move along with the rhythm of the music.

[0003] The advent of dance machines has spawned a host of footstep editing features, requiring motion capture devices such as footstep capture devices and dance mats to capture footstep movements. Early dance machines relied on the dancer's proficiency in the dance to record footstep positions based on the dancer's movements. Depending on the level of difficulty, footstep capture devices could be divided into five-key, seven-key, nine-key, or five-key plus gestures. Because creating a complete footstep plan for each song requires considerable time and effort, including determining footstep positions, beat timing, and difficulty adjustment, footstep capture devices were required to generate footstep plans to facilitate user creation.

[0004] However, the current recognition process that relies on footstep motion capture devices is prone to misoperation, resulting in a large number of invalid key signals, which seriously affects the editing effect of the footsteps. Summary of the Invention

[0005] In response to one of the above technical deficiencies, the present application provides a key recognition method and device, a footstep editing method and device, a computer device, and a computer-readable storage medium, thereby reducing invalid key signals of a footstep motion capture device and improving the footstep editing effect.

[0006] First aspect:

[0007] The present application provides a key position identification method, comprising:

[0008] Reading a key pressure value detected by a footstep motion capture device at a beat point position, and determining a predicted key position stepped on by a user on the footstep motion capture device according to the key pressure value;

[0009] Acquire a real-time video image of the user stepping on the key position on the footstep motion capture device, and extract human skeletal motion data corresponding to the beat point position from the real-time video image;

[0010] Calculating angle parameters between each limb of the user and a preset human skeleton basic template according to the human skeleton motion data;

[0011] Determining a predicted area where the key position stepped by the user on the footstep motion capture device according to the angle parameter;

[0012] If the predicted key position coincides with the predicted area, it is determined that the predicted key position is stepped on, and a corresponding key position symbol is generated.

[0013] In one embodiment, reading a key pressure value detected by a footstep motion capture device at a beat point position, and determining a predicted key position stepped on by a user on the footstep motion capture device according to the key pressure value, includes:

[0014] Reading detection data output by pressure sensors on various keys of a footstep motion capture device at the beat point;

[0015] Calculating a corresponding measured pressure value according to the detection data;

[0016] Determine whether the measured pressure value corresponding to each key position reaches the set pressure threshold;

[0017] If it is reached, the corresponding key position is determined as the predicted key position stepped on by the user on the footstep motion capture device.

[0018] In one embodiment, obtaining a real-time video image of a user stepping on a key position on the footstep motion capture device, and extracting human skeletal motion data corresponding to the beat point position from the real-time video image includes:

[0019] Using a camera to capture a real-time video image of the user stepping on the keys on the footstep motion capture device;

[0020] Predicting the real-time video image using a pre-trained human skeleton three-dimensional motion data model to obtain the three-dimensional coordinate values ​​of the human skeleton motion data on each frame of the image;

[0021] The three-dimensional coordinate values ​​of the human skeleton motion data of several frames of pictures within a set range before and after the beat point position are extracted.

[0022] In one embodiment, calculating angle parameters between each limb of the user and a preset human skeleton basic template based on the human skeleton motion data includes:

[0023] Calculating the angle between the user's designated skeletal point and the skeletal point corresponding to the preset human skeletal basic template using the three-dimensional coordinate values ​​of the human skeletal motion data of the plurality of frames of images;

[0024] The step of determining the predicted area where the key position stepped by the user on the footstep motion capture device is located according to the angle parameter includes:

[0025] Calculating a cosine value based on the angle;

[0026] Comparing the cosine values ​​according to the correspondence between the cosine value range and the respective areas of the footstep motion capture device panel;

[0027] Determine the predicted area where the key position stepped by the user on the footstep motion capture device according to the comparison result.

[0028] In one embodiment, the key position identification method further includes:

[0029] Capture a basic image of the user standing upright in a T-shape; wherein the T-shape refers to a state where the user's hands are open and the feet are side by side;

[0030] Extract skeleton points from the basic image, and construct a basic human skeleton template of the user according to the skeleton points.

[0031] Second aspect:

[0032] The present application provides a key position recognition device, comprising:

[0033] A key prediction module is used to read the key pressure value detected by the footstep motion capture device at the beat point position, and determine the predicted key position stepped by the user on the footstep motion capture device according to the key pressure value;

[0034] a skeleton recognition module for acquiring a real-time video image of a user stepping on a key on the footstep motion capture device, and extracting human skeleton motion data corresponding to the beat point position from the real-time video image;

[0035] An angle calculation module, used to calculate the angle parameters between each limb of the user and a preset human skeleton basic template based on the human skeleton movement data;

[0036] An area prediction module, configured to determine a predicted area where the key position stepped by the user on the footstep motion capture device is located according to the angle parameter;

[0037] The key position determination module is configured to determine that the predicted key position is stepped on if the predicted key position coincides with the predicted area, and generate a corresponding key position symbol.

[0038] The third aspect:

[0039] This application provides a method for editing footnotes, including:

[0040] Obtaining the beat position of a music file, playing the music file to the user, and displaying beat auxiliary lines;

[0041] Obtaining key symbols of a user's footstep motion capture device at each beat point; wherein the key symbols are obtained by the above-mentioned key recognition method;

[0042] Generate a footnote according to each of the key symbols.

[0043] In one embodiment, the footstep editing method further includes:

[0044] Obtain the initialization characteristic parameters of each frame signal of the music file;

[0045] Calculating the beat feature of each frame of the music signal of the music file using a plurality of beat feature algorithms in parallel according to the initialized feature parameters;

[0046] The optimal beat feature is selected as the beat point position, and the beat point auxiliary line is generated.

[0047] In one embodiment, the parallel use of multiple beat feature algorithms to calculate the beat feature of each frame of the music signal of the music file includes:

[0048] Obtaining the initial eigenvalues ​​of each frame of the music signal, and performing autocorrelation calculation on the initial eigenvalues ​​to obtain an autocorrelation curve;

[0049] Calculating a beat period for beat point determination of a Markov chain according to a product of the autocorrelation curve and the Viterbi path;

[0050] The Gaussian model of Markov's beat period is used to estimate the autocorrelated information of each time period;

[0051] Obtaining rhythm points of corresponding time periods according to the autocorrelated information, and determining each beat feature through a reverse loop method;

[0052] The selecting of the optimal beat feature as the beat point position includes:

[0053] Bayesian estimation and information entropy are used to select the beat point position with the best feature information from each beat point position.

[0054] In one embodiment, the footstep editing method further includes:

[0055] The deviation value between the key symbol of the score file and the beat point position is calculated, and the position of the key symbol of the score file is corrected according to the deviation value so that the key symbol is aligned with the beat point position.

[0056] In one embodiment, the step of calculating the deviation between the key symbols of the score file and the beat point positions, and correcting the positions of the key symbols of the score file according to the deviation, includes:

[0057] Divide the music file into multiple sections according to the beat points, and correspond the beat points to each key symbol on the score file according to the beat points auxiliary lines;

[0058] Set the window length and frame shift, and use the sliding window method to calculate the local deviation value between the beat point auxiliary line and the key position symbol of each segmented score;

[0059] Correcting the position of each key symbol segment by segment according to the local deviation value and the window length;

[0060] Calculate the global deviation mean between the footnote file and the beat point auxiliary line, and modify the footnote file according to the global deviation mean.

[0061] In one embodiment, the footstep editing method further includes:

[0062] In the secondary editing stage, the human skeleton motion data of each frame of real-time video image is recorded to form the motion trajectory of each limb;

[0063] Play the motion trajectory of each frame of real-time video image to the user according to the slow motion, fast play or replay function selected by the user;

[0064] The key positions of the footnote file are manually edited and corrected according to the motion trajectory of the playback.

[0065] In one embodiment, the recording of the human skeleton motion data of each frame of real-time video image to form the motion trajectory of each limb includes:

[0066] Determine the key points of each limb based on the human skeleton data of each frame of real-time video image;

[0067] Calculate the coordinates of the key points of each frame and connect the coordinates of the key points of each frame into a line to get the corresponding limb;

[0068] The motion trajectory of each limb in each frame of real-time video image is recorded with a set window length;

[0069] The manual secondary editing and correction of the key positions of the footnote file according to the played motion trajectory includes:

[0070] When the key positions of the footstep file need to be edited and corrected for a second time, the motion track is paused, and the key positions are stepped on on the footstep motion capture device or the key positions of the footstep file are re-edited on the editing interface.

[0071] Fourth aspect:

[0072] The present application provides a footnote editing device, comprising:

[0073] A beat point acquisition module is used to obtain the beat point position of the music file, play the music file to the user and display the beat point auxiliary line;

[0074] A key symbol determination module is used to obtain the key symbols of the user stepping on the footstep motion capture device at each beat point; wherein the key symbols are obtained by the key position recognition method mentioned above;

[0075] The footer pattern generation module is used to generate footer patterns according to the key symbols.

[0076] Fifth aspect:

[0077] The present application provides a dance machine, comprising: a host, and a display device, a camera, and a footstep motion capture device connected to the host; wherein the host is further connected to a server;

[0078] The host is configured to execute the steps of the key position recognition method or the footstep pattern editing method of any embodiment; the camera is used to capture image data; the footstep motion capture device is used to output the pressure value of the key being stepped on; and the display device is used to display image information.

[0079] Sixth aspect:

[0080] The present application provides a computer device, comprising:

[0081] one or more processors;

[0082] Memory;

[0083] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the above-mentioned key position recognition method or footnote editing method.

[0084] Seventh aspect:

[0085] The present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded by the processor and executes the above-mentioned key position recognition method or footnote editing method.

[0086] The above-mentioned key position recognition method and device, footstep editing method and device, computer equipment and computer-readable storage medium read the key position pressure value detected by the footstep motion capture device to obtain the predicted key position stepped by the user, and extract the human skeleton movement data based on the real-time video image of the user; by calculating the angle parameters between each limb of the user and the preset human skeleton basic template, the predicted area where the stepped key position is located is detected; when the predicted key position coincides with the predicted area, it is determined to be valid and the corresponding key position symbol is generated; this technical solution collects three-dimensional human skeleton movement data to reflect the human posture movement trajectory in real time, combines the pressure value of the footstep motion capture device with the human posture movement trajectory to jointly determine the key position symbol, which can reduce the invalid key position signals generated in the footstep editing process, improve the footstep editing effect, and enhance the user application experience.

[0087] Furthermore, by calculating the beat points of the music file and generating corresponding beat point auxiliary lines, it is convenient for users to refer to and mark the music score, thereby improving the music score editing effect.

[0088] Furthermore, the beat auxiliary lines of the music file are used to calculate the deviation of the key symbols, and the tablature is corrected by combining segmented correction and global correction to ensure the accuracy of the tablature alignment with the beat points and ensure that the tablature generated by the user is at the beat point position.

[0089] Furthermore, after the footsteps file is generated, the automatically edited footsteps can be manually edited and corrected for a second time. By recording the motion trajectory of each limb based on the human skeleton movement data of each frame of real-time image, users can slow down the motion trajectory and combine it with the real-time video image to make a second judgment on the motion trajectory. For unsatisfactory or inaccurate key positions on the footsteps, they can adjust them to reasonable key positions, thereby improving the editing effect of the footsteps.

[0090] Additional aspects and advantages of the present application will be given in part in the following description, which will become apparent from the following description, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0092] Figure 1 This is a block diagram of the structure of a dance machine;

[0093] Figure 2 This is a diagram of the five-button panel of the footstep motion capture device;

[0094] Figure 3 This is a diagram of the nine-button panel of the footstep motion capture device;

[0095] Figure 4is a flow chart of a key position identification method according to an embodiment;

[0096] Figure 5 It is to obtain the predicted key position flow chart;

[0097] Figure 6 It is a diagram of the human skeleton;

[0098] Figure 7 It is a schematic diagram of the basic template of the human skeleton;

[0099] Figure 8 is a structural diagram of a key position recognition device according to an embodiment;

[0100] Figure 9 A flowchart of a method for editing footsteps according to an embodiment of the present invention is shown.

[0101] Figure 10 It is a flow chart of the method for obtaining the beat point position;

[0102] Figure 11 This is an example of a flowchart for automatic correction of footnotes;

[0103] Figure 12 This is an example of the overall flow chart for editing footnotes;

[0104] Figure 13 It is a structural diagram of a key position identification device according to an embodiment. DETAILED DESCRIPTION

[0105] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0106] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "the," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of the stated features, integers, steps, and operations, but does not preclude the presence or addition of one or more other features, integers, steps, and operations.

[0107] The key position recognition solution of this application can be applied to dance machines, such as Figure 1 As shown, Figure 1This is a block diagram of the structure of an example dance machine; the host is connected to the display device, camera and footstep motion capture device, and is connected to the server through the network; generally, the footstep motion capture device can be a dance mat, dance pedal and foot pedal, etc. There are 5-key, 7-key and 9-key types; each key is marked with an arrow indicating the direction, and the arrow will light up when the user is dancing, indicating the key position stepped on; Figure 2 As shown, Figure 2 This is a schematic diagram of the five-button panel of the footstep motion capture device, with five buttons (up, down, left, right, and center). Figure 3 As shown, Figure 3 This is a schematic diagram of the nine-key panel of the footstep motion capture device, with nine keys (up, down, left, right, upper left, lower left, upper right, lower right, and center); the camera is used to capture video images of the user dancing; the host can import music data and play it through the audio system, and the user can choreograph according to the process of playing music. During the choreography process, the host can record the stepped keys by detecting the stepping signals of the footstep motion capture device and finally generate a foot score, which is stored locally or on a server. The display device can display the screen information output by the host, such as arrow icons showing the key positions, music beat lines, animations, etc.

[0108] The following describes an embodiment of the key position identification method, referring to Figure 4 As shown, Figure 4 The flowchart of the key position recognition method of an embodiment mainly includes the following steps:

[0109] Step S110: reading the key pressure value detected by the footstep motion capture device at the beat point position, and determining the predicted key position stepped by the user on the footstep motion capture device according to the key pressure value.

[0110] In this step, during the footstep editing process, the pressure value output by the footstep motion capture device is detected at the beat position of the music file. When the user steps on a key, a large number of pressure sensors are set under each key of the footstep motion capture device. Each step will output a pressure value (the sensitivity of different devices varies, and the output pressure range will also vary). The pressure value is determined according to the set threshold to determine which key the user stepped on, that is, the predicted key.

[0111] In one embodiment, the method for obtaining the predicted key position in step S110 is as follows: Figure 5 As shown, Figure 5 The flowchart for obtaining predicted key positions may include the following steps:

[0112] s11, reading the detection data output by the pressure sensors on each key position of the footstep motion capture device at the beat point position.

[0113] s12, calculating the corresponding measured pressure value based on the detection data.

[0114] s13, determine whether the measured pressure value corresponding to each key position reaches the set pressure threshold; if so, execute s14.

[0115] s14, determining the corresponding key position as the predicted key position stepped on by the user on the footstep motion capture device.

[0116] In the above technical solution, since the footstep motion capture device will continuously output the current pressure value during the period from stepping to releasing, it is predicted that there is a situation where the key position is accidentally stepped on. Accurate key position recognition cannot be achieved by relying on the output of the footstep motion capture device, and thus footstep editing cannot be performed.

[0117] Step S120: obtaining a real-time video image of the user stepping on the key positions on the footstep motion capture device, and extracting human skeleton movement data corresponding to the beat point position from the real-time video image.

[0118] In this step, the human body behavior is photographed by the camera, and the trained human skeleton three-dimensional motion data model is used for prediction, so as to obtain the three-dimensional coordinate value of the human skeleton motion data of each frame of the picture.

[0119] In one embodiment, step S120 may specifically include the following:

[0120] s21, using a camera to capture a real-time video image of the user stepping on the key positions on the footstep motion capture device.

[0121] s22, using a pre-trained human skeleton three-dimensional motion data model to predict the real-time video image, and obtain the three-dimensional coordinate value of the human skeleton motion data on each frame of the image.

[0122] s23, extracting the three-dimensional coordinate values ​​of the human skeleton motion data of several frames of pictures within a set range before and after the beat point position.

[0123] Specifically, a three-dimensional coordinate system can be established with the normal view angle of the footstep motion capture device plane, and a 3D human skeleton map can be constructed with 15 points. The motion trajectory of the human limbs can be clearly known based on the 3D human skeleton map, such as Figure 6 As shown, Figure 6 This is a diagram of the human skeleton; the 15 bone points can include:

[0124] Right shoulder (-60.0, -300.0, -69.0), right elbow (-185.0, -165.0, -80.0), right wrist (-173.0, -74.0, -212.0), left shoulder (43.0, -294.0, -53.0), left elbow (107.0, -134.0, -45.0), left wrist (105.0, -58.0, -173.0), right calf (-47.0, 73.0, 7.0), right knee (- : The following table shows the weight of the body: 161.0, 343.0, -47.0), right ankle (-226.0, 647.0, 25.0), left thigh (50.0, 72.0, 15.0), left knee (164.0, 345.0, -18.0), left ankle (195.0, 638.0, 93.0), head (-1.0, -409.0, -11.0), neck (-3.0, -383.0, -72.0), abdomen (0.0, 0.0, 0.0).

[0125] Step S130: Calculating angle parameters between each limb of the user and a preset human skeleton basic template based on the human skeleton motion data.

[0126] In this step, the measured human skeleton motion data is used to calculate the angle parameters of each limb of the user relative to the human skeleton basic template; then the angle parameters of each limb relative to the human skeleton basic template are calculated.

[0127] In one embodiment, the angle between each limb relative to the human skeleton basic template can be calculated; based on this, the three-dimensional coordinate values ​​of the human skeleton motion data of several frames of pictures can be used to calculate the angle between the user's specified skeleton point and the skeleton point corresponding to the preset human skeleton basic template.

[0128] For the human skeleton basic template, it refers to a reference template, which is a posture of a human body standing in a T shape in a vertical state (ie, with both hands open and both feet standing together) as the human skeleton basic template. As an example, Figure 7 As shown, Figure 7 It is a schematic diagram of the basic template of the human skeleton; it can be achieved by taking a basic image of a user standing vertically in a T-shape; extracting bone points from the basic image, and then constructing the basic template of the human skeleton of the user based on the bone points.

[0129] Step S140: determining the predicted area where the key stepped by the user on the footstep motion capture device is located according to the angle parameter.

[0130] In this step, the angle cosine value can be calculated based on the angle parameter to serve as a basis for determining the key position stepped on the footstep motion capture device; and determining the predicted area where the key position stepped on by the user on the footstep motion capture device is located.

[0131] In one embodiment, the method for determining the prediction area may include the following:

[0132] (I) Calculating the cosine value based on the angle.

[0133] (II) According to the correspondence between the cosine value range and each area of ​​the footstep motion capture device panel, the cosine value is compared.

[0134] (III) determining a predicted area where the user steps on a key on the footstep motion capture device according to the comparison result.

[0135] Specifically, each bone in the human skeleton has a direction vector, and the angle can be calculated based on the direction vector. For example, if the vector of the right arm is measured as a, the calculation formula can be:

[0136]

[0137] That is, the coordinates of the right elbow bone point - the coordinates of the right shoulder bone point; correspondingly, the vector of the right arm of the human skeleton basic template is b, and its calculation formula can be:

[0138]

[0139] That is, the coordinates of the right elbow bone point of the human skeleton basic template minus the coordinates of the right shoulder bone point; the formula for calculating the cosine value through the included angle can be:

[0140]

[0141] The cosine value obtained by calculation can be used to determine the key position stepped on by the user at different values; in the specific implementation process, the cosine value range corresponding to each key position can be calculated in advance, and the key position where the human skeleton falls can be calculated in combination with the coordinate y value; if the y value is greater than the threshold 50, the y value will deviate from the threshold obtained according to different 3D human skeleton algorithms, and the key position can be considered to be above the footstep motion capture device. If the y value is less than -50, the key position can be considered to be below the footstep motion capture device; the range it falls into can be judged based on the direction determined by the y value and the cosine value, so as to determine which key position area it is stepping on, that is, the predicted area.

[0142] For example: According to the three-dimensional information (x, y, z) values in the human body bone movement data, based on the cosine value and the magnitude of the y value corresponding to the coordinate, if it exceeds the upper threshold (i.e., y > 50), it is considered that the key position is above the foot movement capture device; if it is lower than the lower threshold (i.e., y < -50), it is considered that the key position is below the foot movement capture device; if it is within the middle threshold (-50 < y < 50), it is considered that the key position is at the exact center position of the foot movement capture device; thus, the predicted area where the key position is located can be obtained.

[0143] Step S150: If the predicted key position coincides with the predicted area, it is determined that the predicted key position is stepped on, and a corresponding key symbol is generated.

[0144] In this step, it is judged whether the predicted key position detected by the foot movement capture device coincides with the predicted area calculated from the real-time video image. If it coincides, it means that the detected key position is the truly stepped-on key position, and a corresponding key symbol can be generated; if it does not coincide, it means that the foot movement capture device has misdetected, and the key symbol is discarded.

[0145] For example, taking a 5-key foot movement capture device as an example, assuming that after determining the human body bone movement data, the predicted area of the key position where the right foot is located is above, and the predicted area of the key position where the left foot is located is in the middle. At this time, through the pressure value detection of the foot movement capture device, the predicted key positions are detected at the middle key position and the upper right key position, and a predicted key position is also detected at the upper left key position due to the detected pressure value. However, since this key position does not coincide with the predicted area, it can be determined that the key positions stepped on by the user are the upper right key position and the middle key position, and the value of the upper left key position is an error value and is discarded.

[0146] The technical solution of the above embodiment uses the motion trajectory of each frame and the motion direction of the two legs in each frame, calculates the similarity degree with the basic template using the cosine value to judge the motion orientation of the user's limbs, and then judges by obtaining the pressure value of each key position output by the pressure sensors of each key position. The key position direction where the pressure is greater than the threshold and is consistent with the direction of the two legs in the human body bone is the currently stepped-on key position direction. This technical solution can collect the three-dimensional motion data of the human body bone to reflect the human body posture motion trajectory in real time, combine the pressure value of the foot movement capture device and the human body posture motion trajectory to jointly determine the key symbol, which can reduce the invalid key position signals generated during the foot spectrum editing process, improve the foot spectrum editing effect, and enhance the user application experience.

[0147] The embodiments of the key position recognition device are described below.

[0148] Refer to Figure 8 as shown in Figure 8 is a structural schematic diagram of a key position recognition device according to an embodiment, including:

[0149] The key prediction module 110 is configured to read the key pressure value detected by the footstep motion capture device at the beat point position, and determine the predicted key position stepped on by the user on the footstep motion capture device according to the key pressure value;

[0150] The skeleton recognition module 120 is used to obtain a real-time video image of the user stepping on the key position on the footstep motion capture device, and extract human skeleton movement data corresponding to the beat point position from the real-time video image;

[0151] An angle calculation module 130, configured to calculate angle parameters between each limb of the user and a preset human skeleton basic template based on the human skeleton motion data;

[0152] An area prediction module 140 is configured to determine a predicted area where the key position stepped by the user on the footstep motion capture device is located according to the angle parameter;

[0153] The key position determination module 150 is configured to determine that the predicted key position is stepped on if the predicted key position coincides with the predicted area, and generate a corresponding key position symbol.

[0154] The key position recognition device of this embodiment can execute a key position recognition method provided by the embodiment of the present disclosure, and its implementation principle is similar. The actions performed by each module in the key position recognition device in each embodiment of the present disclosure correspond to the steps in the key position recognition method in each embodiment of the present disclosure. For the detailed functional description of each module of the key position recognition device, please refer to the description in the corresponding key position recognition method shown in the previous text, and will not be repeated here.

[0155] An embodiment of the method for editing footsteps is described below.

[0156] refer to Figure 9 As shown, Figure 9 The flowchart of a method for editing footnotes according to an embodiment includes:

[0157] Step S210: Obtain the beat point position of the music file, play the music file to the user and display the beat point auxiliary line.

[0158] In this step, the beat point position of the music file is first obtained and the beat point auxiliary line is generated, and then the music file is played as dance music for footnote editing. For the beat point assistance, an auxiliary function can be provided to the user to accurately place the key symbols at the beat point position.

[0159] Since different types of songs have different degrees of beat feature prominence, some may have obvious pitch but no sudden changes, such as Pitched non-percussive (PNP) with obvious pitch but no sudden changes, such as Non-pitched Percussive (NPP) with no pitch, such as drums and cymbals, and Pitched Percussive (PP) with pitch and sudden changes, such as Complex Mixtures (Mix) of bowed strings and wind instruments, such as rock and pop songs. In order to better determine the rhythm information of a music file, the embodiment of the present application uses multiple feature information to calculate the beat feature to select the best beat feature to represent the beat point position of the music file.

[0160] Accordingly, in one embodiment, Figure 10 As shown, Figure 10 The flowchart of the method for obtaining the beat point position may include the following:

[0161] (1) Obtain the initialization characteristic parameters of each frame signal of the music file.

[0162] (2) Calculating the beat features of each frame of the music signal of the music file using a plurality of beat feature algorithms in parallel according to the initialized feature parameters.

[0163] Specifically, five beat feature calculation methods can be used for synchronous and parallel calculation, namely, infogain, beats, mel flux, effective level (RMS), and complex domain. The beat feature uses subband separation technology to enhance feature information, thereby strengthening signals with unclear beat features. The mel spectrum characteristics of the mel flux information stream are similar to the human ear's auditory characteristics, so it can well simulate the characteristics of human hearing beats, and has higher recognition efficiency for music with more obvious beats. As for the effective level, since a significant sudden change in the music file will produce a strong instantaneous level, the beat point position of the music file can be found through the effective level. The complex domain mainly considers phase information. When a beat point appears, the phase will have an instantaneous change, which can be detected through the complex domain.

[0164] In one embodiment, the beat feature calculation process may include the following:

[0165] (I) Obtaining the initial eigenvalues ​​of each frame of music signal, and performing autocorrelation calculation on the initial eigenvalues ​​to obtain an autocorrelation curve.

[0166] (II) Calculating the beat period for beat point determination of the Markov chain based on the product of the autocorrelation curve and the Viterbi path.

[0167] (III) The Gaussian model of the Markov beat period in each time period is used to estimate the autocorrelated information of each time period.

[0168] (IV) obtaining rhythm points of corresponding time periods according to the autocorrelated information, and determining each beat feature through a reverse loop method.

[0169] (3) Select the optimal beat feature as the beat point position and generate the beat point auxiliary line.

[0170] Specifically, Bayesian estimation and information entropy are used to jointly determine the most accurate beat feature. Based on the deviation information between the five detection methods, the more accurate beat feature is determined. The information entropy is used to calculate the discrete degree of the deviation information, and the beat feature with the smallest discrete degree is selected as the beat point position. Then, beat point auxiliary lines are generated at each beat point position to provide users with a reference for editing the score.

[0171] Step S220: obtaining key symbols of the user's footsteps at each beat point on the motion capture device; wherein the key symbols are obtained by the key recognition method of any of the above embodiments.

[0172] The user steps on the keys according to the rhythm of the music file and refers to the appearance of each beat point auxiliary line. After the actual stepped keys are determined through the solution provided by the above embodiment, key position symbols are generated.

[0173] Step S230: Generate a footnote according to each of the key symbols.

[0174] Specifically, the tablature is composed of a music file and key symbols at each beat point. The tablature file can be generated through the key symbols to complete the tablature editing process.

[0175] The footnote editing method of the above embodiment, based on the combined determination of key symbols by combining the pressure value of the footstep motion capture device with the human body posture motion trajectory, can reduce the generation of invalid key signals during the footnote editing process; before footnote editing, by calculating the beat points of the music file and generating corresponding beat point auxiliary lines, it is convenient for users to refer to and mark the footnotes, thereby improving the footnote editing effect.

[0176] Due to the significant differences in dance proficiency and musical sensitivity between players, the timing of each user's footsteps on the motion capture device and the beat position are not completely consistent. This often leads to situations where the user steps too fast or too slow during footstep editing, and even with the help of a beat line, there will still be deviations. Therefore, to ensure that the keys on the footsteps accurately fall on the beat positions of the music, the embodiments of the present application also provide an automated beat correction technology solution, which uses a deviation correction method to perform corrections.

[0177] Accordingly, in one embodiment, the footnote editing method of the present application can further calculate the deviation value between the key symbols of the footnote file and the beat point position, and correct the position of the key symbols of the footnote file according to the deviation value so that the key symbols are aligned with the beat point position.

[0178] Preferably, the method for correcting the position of the key symbols in the script file may include the following steps:

[0179] a. Divide the music file into multiple sections according to the beat points, and correspond the beat points to each key symbol on the score file according to the auxiliary lines.

[0180] b. Set the window length and frame shift, and use a sliding window method to calculate the local deviation value between the beat point auxiliary line and the key symbols of each segmented score.

[0181] c. Correct the position of each key symbol segment by segment according to the local deviation value and the window length.

[0182] d. Calculate the global deviation mean between the footnote file and the beat point auxiliary line, and modify the footnote file according to the global deviation mean.

[0183] Specifically, refer to Figure 11 As shown, Figure 11 This is an example of an automatic tablature correction flow chart. First, the music file is divided into multiple segments based on the beat point position, and the user-generated tablature file is mapped to the same beat point auxiliary line. In each tablature key symbol corresponding to the beat point, a sliding window method is used with a partitioning window length of 0.5s and a frame shift of 0.25s to calculate the deviation degree d between the beat point auxiliary line and the tablature key symbol within the window range. The formula is:

[0184] d=x1-x2

[0185] Where x1 is the position of the beat point auxiliary line, and x2 is the position of the key symbol in the footer. Then, according to the window length, the position x' of each key symbol in the footer is corrected section by section. The formula is:

[0186] x'=x+d

[0187] In the formula, x is the position of each key symbol in the script; when correcting, each section is moved up or down, and finally the global deviation mean is calculated and a secondary correction is performed to obtain the required script file.

[0188] The technical solution of the above embodiment uses the beat auxiliary lines of the music file to calculate the deviation of the key symbols, and adopts a combination of segmented correction and global correction to correct the tablature, thereby ensuring the accuracy of the tablature alignment with the beat points and ensuring that the tablature generated by the user is at the beat point position.

[0189] refer to Figure 12 As shown, Figure 12 This is an example of the overall flow chart for editing tablature. The music file is imported, edited, and then analyzed through STFT (Short-Time Fourier Transform) to obtain the music spectrum. The beat point position is calculated from the music spectrum and the beat point auxiliary line is generated. By playing the music file and displaying the beat point auxiliary line, the user can edit the tablature.

[0190] During the editing process, because each player's pedaling force and orientation vary, the pressure values ​​of the footstep motion capture device and real-time images captured by the camera are analyzed. Accurate key symbols are determined through multiple joint judgments, including analysis of human skeletal motion data, pressure value determination, and limb motion trajectories. The human motion trajectory is calculated, normalized and regularized, and the three-dimensional coordinate system is calibrated. Relative distances, such as Euclidean distance, and relative angles, such as trigonometric cosine, are calculated. Finally, combined with the pressure threshold on the pedal, the user's final key symbol is determined. This improves the accuracy of key position determination and completely solves the problem of multiple misjudged keys. Furthermore, based on the beat of the music, a fixed window length is used to slide and correct the time point corresponding to each foot key symbol. The footsteps are corrected using the musical beat line to ensure that the user's generated footsteps are always on beat.

[0191] In one embodiment, in order to ensure that users obtain accurate and satisfactory footsteps, the embodiment of the present application also provides a multiple footsteps correction method, providing users with a technical solution for secondary editing and correction.

[0192] Accordingly, the footstep editing method of the present application may further include:

[0193] (1) In the secondary editing stage, the human skeleton motion data of each frame of real-time video image is recorded to form the motion trajectory of each limb.

[0194] Specifically, the key points of each limb are determined based on the human skeleton data of each frame of real-time video image; the coordinates of the key points of each frame are calculated, and the coordinates of the key points of each frame are connected into lines to obtain the corresponding limbs; and the movement trajectory of each limb in each frame of real-time video image is recorded with a set window length.

[0195] (2) Play the motion trajectory of each frame of real-time video image to the user according to the slow motion, fast play or replay function selected by the user.

[0196] Specifically, the user can choose slow motion and judge whether the movements are consistent with the key positions of the footsteps by observing the slow motion of the footsteps; the user can choose fast motion or replay, and the user can re-interpret the edited footsteps file to determine the accuracy of the footsteps key positions.

[0197] (3) Manually edit and correct the key positions of the footnote file according to the motion trajectory of the playback.

[0198] Specifically, when the key positions of the footstep file need to be edited and corrected for the second time, the motion trajectory is paused, and the key positions are stepped on the footstep motion capture device or the key positions of the footstep file are re-edited on the editing interface.

[0199] For example, in the secondary editing mode, the system automatically records each limb (left hand, right hand, left arm, right arm, left foot, right foot, left leg, right leg, etc.) in each frame. Each limb records a certain time window length. For example, with a time window of 0.5s, the movement trajectory trend of 0.5 seconds is recorded. The direction of the trajectory is calculated with the front of the human body as the positive direction, and the movement direction of the limb is determined to determine the direction of the foot pattern key position. If the determined direction is consistent with the direction determined by the first edit, it means that the direction is accurately determined. Otherwise, it is automatically corrected and the user is informed that there is a deviation between the first edit and the second edit at the current time node, reminding the user to pay attention. According to the movement trajectory, the front is the positive direction.

[0200] During the second edit, the user will be able to play each frame of human skeleton data and the corresponding frame of video simultaneously on the large screen of the display device. The user can select slow motion, fast play, or replay (without changing the speed) according to the playback speed. If the user chooses slow play, it is equivalent to observing the slow motion of the dance steps while checking whether the corresponding movements are consistent with the footstep keys, which acts as a dance decomposition function; if the user chooses fast play or replay (without changing the speed), the user can follow the dance video of the first edit and re-perform it to confirm the accuracy of the footstep keys. If the user needs to make changes during the second edit, they can also directly press the pause button on the modified part, directly re-step on the foot pedal or select the corresponding footstep keys on the screen.

[0201] An embodiment of the footnote editing device is described below.

[0202] refer to Figure 13 As shown, Figure 13 The structure diagram of a key position recognition device according to an embodiment includes:

[0203] A beat point acquisition module 210 is used to obtain the beat point position of a music file, play the music file to the user, and display the beat point auxiliary lines;

[0204] The key symbol determination module 220 is used to obtain the key symbols of the user stepping on the footstep motion capture device at each beat point; wherein the key symbols are obtained by the key recognition method described above;

[0205] The footer pattern generating module 230 is used to generate footer patterns according to the key symbols.

[0206] The footstep editing device of this embodiment can execute a footstep editing method provided by the embodiments of the present disclosure. The implementation principles are similar. The actions executed by each module in the footstep editing device in each embodiment of the present disclosure correspond to the steps in the key position recognition method in each embodiment of the present disclosure. For the detailed functional description of each module in the footstep editing device, please refer to the description of the corresponding footstep editing method shown above, and will not be repeated here.

[0207] An embodiment of a dance machine is described below.

[0208] The dance machine provided in this application includes: a host, and a display device, a camera and a footstep motion capture device connected to the host; wherein the host is also connected to a server.

[0209] During use, the host is configured to execute the steps of the key recognition method or footstep editing method of any embodiment; the camera is used to capture image data; the footstep motion capture device is used to output the pressure value of the key being stepped on; and the display device is used to display image information.

[0210] An embodiment of a computer device of the present application is described below. The computer device includes:

[0211] one or more processors;

[0212] Memory;

[0213] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the key position recognition method according to any of the above embodiments.

[0214] The following describes an embodiment of a computer-readable storage medium of the present application, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded by the processor and executes the key position recognition method of any of the above embodiments.

[0215] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for editing footsteps, characterized in that: include: Obtaining the initial eigenvalues ​​of each frame of the music signal, and performing autocorrelation calculation on the initial eigenvalues ​​to obtain an autocorrelation curve; The method comprises calculating a beat period for beat point determination of a Markov chain based on the product of the autocorrelation curve and the Viterbi path; estimating the autocorrelation information of each time period using a Gaussian model of the Markov beat period; obtaining the rhythm point of the corresponding time period based on the autocorrelation information, and determining each beat feature through a reverse loop; selecting the optimal beat feature as the beat point position, and generating a beat point auxiliary line; obtaining the beat point position of a music file, playing the music file to a user, and displaying the beat point auxiliary line; Using a key position recognition method to obtain the key position symbols of the user's footsteps motion capture device at each beat point; Generate footnotes according to each of the key symbols.

2. The method for editing footsteps according to claim 1, wherein: Before obtaining the initial eigenvalues ​​of each frame of music signal, it also includes: Get the initialization characteristic parameters of each frame signal of the music file.

3. The method for editing footsteps according to claim 2, characterized in that: The selecting of the optimal beat feature as the beat point position includes: Bayesian estimation and information entropy are used to select the beat point position with the best feature information from each beat point position.

4. The method for editing footsteps according to claim 1, wherein: Also includes: The deviation value between the key symbol of the score file and the beat point position is calculated, and the position of the key symbol of the score file is corrected according to the deviation value so that the key symbol is aligned with the beat point position.

5. The method for editing footsteps according to claim 4, characterized in that: The step of calculating the deviation between the key symbols of the score file and the beat point positions, and correcting the positions of the key symbols of the score file according to the deviation, includes: Divide the music file into multiple sections according to the beat points, and correspond the beat points to each key symbol on the score file according to the beat points auxiliary lines; Set the window length and frame shift, and use the sliding window method to calculate the local deviation value between the beat point auxiliary line and the key position symbol of each segmented score; Correcting the position of each key symbol segment by segment according to the local deviation value and the window length; The global deviation mean of the deviation between the footnote file and the beat point auxiliary line is calculated, and the footnote file is corrected according to the global deviation mean.

6. The method for editing footsteps according to claim 5, characterized in that: Also includes: In the secondary editing stage, the human skeleton motion data of each frame of real-time video image is recorded to form the motion trajectory of each limb; Play the motion trajectory of each frame of real-time video image to the user according to the slow motion, fast play or replay function selected by the user; The key positions of the footnote file are manually edited and corrected according to the motion trajectory of the playback.

7. The method for editing footsteps according to claim 6, wherein: The human skeleton motion data recorded in each frame of real-time video image forms the motion trajectory of each limb, including: Determine the key points of each limb based on the human skeleton data of each frame of real-time video image; Calculate the coordinates of the key points of each frame and connect the coordinates of the key points of each frame into a line to get the corresponding limb; The motion trajectory of each limb in each frame of real-time video image is recorded with a set window length; The manual secondary editing and correction of the key positions of the footnote file according to the played motion trajectory includes: When the key positions of the footstep file need to be edited and corrected for a second time, the motion track is paused, and the key positions are stepped on on the footstep motion capture device or the key positions of the footstep file are re-edited on the editing interface.

8. The method for editing footsteps according to any one of claims 1 to 7, characterized in that: The key position identification method comprises: Reading a key pressure value detected by a footstep motion capture device at a beat point position, and determining a predicted key position stepped on by a user on the footstep motion capture device according to the key pressure value; Acquire a real-time video image of the user stepping on the key position on the footstep motion capture device, and extract human skeletal motion data corresponding to the beat point position from the real-time video image; Calculating angle parameters between each limb of the user and a preset human skeleton basic template according to the human skeleton motion data; Determining a predicted area where the key position stepped by the user on the footstep motion capture device according to the angle parameter; If the predicted key position coincides with the predicted area, it is determined that the predicted key position is stepped on, and a corresponding key position symbol is generated.

9. The method for editing footsteps according to claim 8, characterized in that: Reading a key pressure value detected by a footstep motion capture device at a beat point position, and determining a predicted key position stepped on by a user on the footstep motion capture device according to the key pressure value, including: Reading detection data output by pressure sensors on various keys of a footstep motion capture device at the beat point; Calculating a corresponding measured pressure value according to the detection data; Determine whether the measured pressure value corresponding to each key position reaches the set pressure threshold; If it is reached, the corresponding key position is determined as the predicted key position stepped on by the user on the footstep motion capture device.

10. The method for editing footsteps according to claim 9, characterized in that: Acquiring a real-time video image of a user stepping on a key position on the footstep motion capture device, and extracting human skeletal motion data corresponding to the beat point position from the real-time video image, including: Using a camera to capture a real-time video image of the user stepping on the keys on the footstep motion capture device; Predicting the real-time video image using a pre-trained human skeleton three-dimensional motion data model to obtain the three-dimensional coordinate values ​​of the human skeleton motion data on each frame of the image; The three-dimensional coordinate values ​​of the human skeleton motion data of several frames of pictures within a set range before and after the beat point position are extracted.

11. The method for editing footsteps according to claim 10, characterized in that: Calculating angle parameters between each limb of the user and a preset human skeleton basic template according to the human skeleton motion data includes: Calculating the angle between the user's designated skeletal point and the skeletal point corresponding to the preset human skeletal basic template using the three-dimensional coordinate values ​​of the human skeletal motion data of the plurality of frames of images; The step of determining the predicted area where the key position stepped by the user on the footstep motion capture device is located according to the angle parameter includes: Calculating a cosine value based on the angle; Comparing the cosine values ​​according to the correspondence between the cosine value range and the respective areas of the footstep motion capture device panel; The predicted area where the key position stepped by the user on the footstep motion capture device is located is determined according to the comparison result.

12. The method for editing footsteps according to claim 11, characterized in that: Also includes: Capture a basic image of the user standing upright in a T-shape; wherein the T-shape refers to a state where the user's hands are open and the feet are parallel; Extract skeleton points from the basic image, and construct a basic human skeleton template of the user according to the skeleton points.

13. A footnote editing device, characterized in that: include: A beat point acquisition module is used to obtain the initial characteristic value of each frame of music signal and obtain an autocorrelation curve by performing autocorrelation calculation on the initial characteristic value; The beat period for beat point determination of the Markov chain is calculated based on the product of the autocorrelation curve and the Viterbi path; the autocorrelation information of each time period is estimated using the Gaussian model of the Markov beat period; the rhythm point of the corresponding time period is obtained based on the autocorrelation information, and each beat feature is determined through a reverse loop method; the optimal beat feature is selected as the beat point position, and the beat point auxiliary line is generated to obtain the beat point position of the music file, and the music file is played to the user and the beat point auxiliary line is displayed; A key symbol determination module is used to obtain the key symbols of the user's stepping motion capture device at each beat point using a key recognition method; The footer pattern generation module is used to generate footer patterns according to each of the key symbols.

14. A computer device, characterized in that: It includes: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the method for editing footsteps according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the method for editing the footnotes according to any one of claims 1 to 12.

Images