A music generation and gait analysis method and system based on human motion information

Abstract

The application discloses a music generation and gait analysis method and system based on human motion information, and the method comprises the following steps: acquiring human motion information collected by an inertial sensor installed on a human body; processing the human motion information to obtain target human motion information; generating music instructions according to the target human motion information and user input information; sending the music instructions to an electronic music player to control the electronic music player to generate and play music according to the music instructions; and performing gait analysis according to the music played by the electronic music player to obtain gait analysis results. The application collects human motion information through a wearable inertial sensor, generates personalized and instant music according to the collected human motion information, realizes the mapping between human actions and music, reflects the differences in human gait through the generated music, realizes gait analysis, and can be used as a rehabilitation treatment tool to enhance the synchronization of sensory motion and improve the treatment enthusiasm of patients.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and particularly relates to a music generation and gait analysis method and system based on human motion information, a terminal and a computer readable storage medium. BACKGROUND

[0002] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information, on the one hand, it can help patients improve motor coordination, and on the other hand, music can also be used as a positive stimulus and motivation to make patients more actively participate in the rehabilitation process.

[0003] When generating music by using a traditional electronic music generator (electronic musical instrument), such as an electronic piano, a keyboard, a drum machine and the like, the creator needs to deliberately complete the control action of knocking and the like. This requires the creator to have certain training to generate meaningful music, otherwise, the music generated by controlling these electronic musical instruments is often random, which is difficult to be called a musical piece. In addition to the above method, existing artificial intelligence models can also generate music, such as Suno AI music generator and the like, but the intelligent music generator lacks the active participation of people in music creation, and therefore usually lacks emotion, originality and artistic quality.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a music generation and gait analysis method and system based on human motion information, a terminal and a computer readable storage medium, which aims to solve the problem that the existing music generation scheme by using an electronic musical instrument requires experienced creators and high operation requirements, and the music generation scheme by using an artificial intelligence model lacks individuality, thereby resulting in poor effect.

[0006] To achieve the above-mentioned application purposes, the present application provides a music generation and gait analysis method based on human motion information, which comprises:

[0007] acquiring human motion information collected by an inertial sensor installed on a human body;

[0008] processing the human motion information to obtain target human motion information;

[0009] generating a music instruction according to the target human motion information and user input information;

[0010] sending the music instruction to an electronic music player to control the electronic music player to generate music and play according to the music instruction;

[0011] The gait analysis is performed according to music played by the electronic music player, and gait analysis results are obtained.

[0012] Optionally, the human body motion information collected by the inertial sensor mounted on the human body is acquired, and specifically, the human body motion information is:

[0013] The human body motion information collected by the inertial sensor is received through a UDP communication protocol, the human body motion information is acceleration data in a Z-axis direction, and the inertial sensor is mounted on any part of the human body.

[0014] Optionally, the target human body motion information includes an acceleration sum in the Z-axis direction.

[0015] The human body motion information is processed to obtain target human body motion information, and specifically, the processing includes:

[0016] The acceleration data in the Z-axis direction is summed after a preset number of frames of the acceleration data in the Z-axis direction are accumulated, and an acceleration sum in the Z-axis direction is obtained.

[0017] Optionally, the music instruction includes a Note On instruction.

[0018] The music instruction is generated according to the target human body motion information and user input information, and specifically, the generation includes:

[0019] According to a Note On instruction in a MIDI protocol, a string structure is defined, the string structure includes a data frame start mark, a data frame end mark, a data segmentation mark, and a control byte.

[0020] The string is received and data in the string is parsed, if the data is parsed correctly, the user input information and the acceleration sum in the Z-axis direction are assigned to corresponding positions of the control byte of the string, and if the data is parsed incorrectly, the string is received again.

[0021] Control information included in the assigned string is extracted, and the Note On instruction is generated according to the control information.

[0022] Optionally, the control byte of the string includes a first byte, a second byte, and a third byte.

[0023] The user input information and the acceleration sum in the Z-axis direction are assigned to corresponding positions of the control byte of the string, and specifically, the assignment includes:

[0024] The number of playing channels in the user input information is assigned to the first byte position of the string, and is used to indicate that music is played in a playing channel corresponding to the number of playing channels.

[0025] assigning the acceleration sum value in the Z-axis direction to a second byte position of the string, for indicating playing music with the acceleration sum value in the Z-axis direction as a pitch;

[0026] assigning the loudness value in the user input information to a third byte position of the string, for indicating playing music with the loudness value as a volume.

[0027] Optionally, the sending the music instruction to the electronic music player, and controlling the electronic music player to generate music and play according to the music instruction specifically comprises:

[0028] sending the Note On instruction to the electronic music player with a built-in audio codec through serial communication;

[0029] controlling the audio codec of the electronic music player to decode the Note On instruction, and generate a playable music melody;

[0030] controlling the electronic music player to play the music melody through a loudspeaker component of the electronic music player.

[0031] Optionally, the gait analysis according to the music played by the electronic music player, and obtaining a gait analysis result specifically comprises:

[0032] judging the current gait as a normal gait or an abnormal gait according to the music played by the electronic music player, and obtaining a gait analysis result;

[0033] if the gait analysis result indicates that the current gait is a normal gait, not issuing a warning;

[0034] if the gait analysis result indicates that the current gait is an abnormal gait, issuing a warning.

[0035] To achieve the above-mentioned purposes, the present application further provides a music generation and gait analysis system based on human motion information, which comprises:

[0036] a data acquisition module, configured to acquire human motion information collected by an inertial sensor installed on a human body;

[0037] a data processing module, configured to process the human motion information, and obtain target human motion information;

[0038] a music instruction generation module, configured to generate a music instruction according to the target human motion information and user input information;

[0039] The music generation module is configured to send the music instruction to an electronic music player, and control the electronic music player to generate music according to the music instruction and play the music.

[0040] The gait analysis module is configured to perform gait analysis according to the music played by the electronic music player, and obtain a gait analysis result.

[0041] To achieve the above-mentioned object, the present application further provides a terminal, which comprises a memory, a processor, and a music generation and gait analysis program based on human motion information stored in the memory and executable on the processor, wherein the music generation and gait analysis program based on human motion information implements the steps of the music generation and gait analysis method based on human motion information when executed by the processor.

[0042] To achieve the above-mentioned object, the present application further provides a computer readable storage medium, which stores a music generation and gait analysis program based on human motion information, wherein the music generation and gait analysis program based on human motion information implements the steps of the music generation and gait analysis method based on human motion information when executed by a processor.

[0043] In the present application, human motion information collected by an inertial sensor installed on a human body is acquired, the human motion information is processed to obtain target human motion information, music instruction is generated according to the target human motion information and user input information, the music instruction is sent to an electronic music player, the electronic music player is controlled to generate music according to the music instruction and play the music, and gait analysis is performed according to the music played by the electronic music player to obtain a gait analysis result. The present application collects human motion information by a wearable inertial sensor, generates personalized and instant music according to the collected human motion information, realizes the mapping between human action and music, generates music based on human motion, and provides a more interactive and interesting music generation mode without requiring the creator to have music creation experience. The gait analysis is realized by reflecting the gait difference of the human body through the generated music, and the present application can be used as a rehabilitation tool to enhance the sensory motor synchrony, improve the treatment enthusiasm of patients, especially patients with motor dysfunction or gait problems, and promote neural rehabilitation by enhancing the sensory motor synchrony and increasing the participation degree of rehabilitation. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a structural diagram of an existing electronic music generation system;

[0045] Figure 2 is a flowchart of a preferred embodiment of the music generation and gait analysis method based on human motion information of the present application;

[0046] Figure 3 is a flow chart of the working principle of the core controller of the present application;

[0047] Figure 4 is a schematic diagram of the string structure of the present application;

[0048] Figure 5 is a structural diagram of the wearable music generation system of the present application;

[0049] Figure 6 is a waveform diagram of the sound signal obtained in the experiment of the present application;

[0050] Figure 7 is a spectrogram of the sound signal obtained in the experiment of the present application;

[0051] Figure 8 is a spectrogram of the sound signal obtained in the experiment of the present application;

[0052] Figure 9 is a structural diagram of the preferred embodiment of the music generation and gait analysis system based on human motion information of the present application;

[0053] Figure 10 is a structural diagram of the preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present application more clear and definite, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0055] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information, on the one hand, it can help patients improve motor coordination, on the other hand, music can also be used as a positive stimulus and motivation to make patients more actively participate in the rehabilitation process.

[0056] In addition to traditional musical instruments, electronic music has been booming in recent decades. Electronic music is created and played using electronic musical instruments, synthesizers, computers and digital technology. Its characteristic is to use electronic devices and software to synthesize, process and modify sound to create unique musical effects and sound textures. The origin of electronic music can be traced back to the early 20th century, when musicians began to use electronic musical instruments and electronic devices to explore new ways of musical creation. With the continuous development of technology, electronic music gradually became an independent music genre and was widely developed and popularized in the second half of the 20th century and the early 21st century.

[0057] Currently, the production of electronic music requires human creation by musicians to organize different musical elements into melodies or chords; for example, using drum machines, percussion instrument samples or programmed drum beats to define the rhythm of the music; using tools such as synthesizers, samplers and audio effects to adjust and process the sound to create unique timbres; and then using the musician's individual creation to determine the arrangement, transition and structure of the music track. Existing electronic music generation systems, such as Figure 1 (a) shown in (a), first need to generate an "input signal of electronic music" through an electronic music generator 1 (also called an electronic instrument, a MIDI instrument); the generated signal is converted into a MIDI signal (also called a MIDI instruction, MIDI data) by a built-in digital audio encoder; and the MIDI signal is sent to an electronic music player 2 with a built-in digital audio decoder, and finally a playable melody is generated. As shown in Figure 1 (b), some of the steps can be completed by connecting a computer 3 with a built-in digital audio codec, and the computer 3 is connected to the electronic music player 2 through a MIDI interface device 4.

[0058] Musical Instrument Digital Interface (MIDI) is a standard electronic communication protocol for electronic music generation. The MIDI protocol defines various notes or playing codes for electronic instruments and other performance devices (such as synthesizers), allowing electronic instruments, computers, mobile phones or other stage performance devices to be connected, adjusted and synchronized with each other to exchange performance data in real time.

[0059] In the prior art, when generating music using traditional electronic music generators (electronic instruments) such as electronic pianos, keyboards, drum machines and other devices, the creator needs to deliberately perform control actions such as hitting. This requires the creator to undergo certain training to generate meaningful music, otherwise the music generated by controlling these electronic instruments is often random and cannot be called a musical piece. In addition to the above method, existing artificial intelligence models can also generate music, such as Suno AI music generator, etc., but intelligent music generators lack human initiative in music creation, so they often lack emotion, originality and artistic quality.

[0060] In order to solve the above technical problems, the application provides a music generation and gait analysis method based on human motion information, human motion information collected by an inertial sensor installed on a human body is acquired; the human motion information is processed to obtain target human motion information; music instructions are generated according to the target human motion information and user input information; the music instructions are sent to an electronic music player to control the electronic music player to generate music and play according to the music instructions; gait analysis is performed according to the music played by the electronic music player to obtain gait analysis results. The application collects human motion information through a wearable inertial sensor, generates personalized and instant music according to the collected human motion information, realizes the mapping between human actions and music, combines the inertial sensor and the music generation system, generates music based on human motion, provides a more interactive and interesting music generation mode, and does not require the creator to have music creation experience; the generated music reflects the gait difference of the human body, realizes gait analysis, can be used as a rehabilitation tool, enhances the sensory motor synchrony, improves the treatment enthusiasm of patients, especially patients with motor dysfunction or gait problems, can promote neural rehabilitation by enhancing the sensory motor synchrony and increasing the participation degree of rehabilitation.

[0061] The application content will be further described through the description of the embodiments in combination with the drawings.

[0062] The preferred embodiment of the music generation and gait analysis method based on human motion information of the application is specifically shown in the following. Figure 2

[0063] S1, human motion information collected by an inertial sensor installed on a human body is acquired.

[0064] In one implementation mode of the embodiment, the human motion information collected by the inertial sensor installed on the human body is acquired, specifically:

[0065] The human motion information collected by the inertial sensor is received through the UDP communication protocol, the human motion information is acceleration data in the Z-axis direction, and the inertial sensor is installed on any part of the human body.

[0066] Specifically, the core element of the self-developed inertial sensor of the application is a 9-axis inertial sensor chip (BNO055) and a single-chip microcomputer (esp32_wroom). The 9-axis inertial sensor chip (BNO055) is used to collect human motion information, and the application acquires Z-axis direction acceleration data collected by the BNO055 chip, and the sampling rate is 100 hz. The single-chip microcomputer is used to send the data collected by the 9-axis inertial sensor chip to the core controller through the UDP communication protocol.

[0067] ​S2, processing the human motion information to obtain target human motion information.

[0068] In one implementation manner of the embodiment, the target human motion information includes a Z-axis direction acceleration sum value;

[0069] Processing the human motion information to obtain target human motion information, specifically:

[0070] After accumulating the Z-axis direction acceleration data of the set frame number (5 frames), summing is performed to obtain the Z-axis direction acceleration sum value.

[0071] Specifically, the core controller adopted by the present application includes an embedded system based on Raspberry Pi, which is used for receiving inertial sensor data and deploying a mapping algorithm; a digital audio codec based on VS1053b chip, which is used for generating music instructions (MIDI instructions); and a serial connector (CH340), which is used for connecting the core controller with an electronic music player. After the embedded system of the core controller receives the inertial sensor data through the UDP communication protocol, the inertial sensor data is processed through the "action-music mapping algorithm" proposed by the present application, and then the processed inertial sensor data is encoded into MIDI instructions by using the digital audio codec, and finally the MIDI instructions are sent to the electronic music player through serial communication.

[0072] In the present application, the embedded system in the core controller, i.e. the single-chip microcomputer, receives the inertial sensor data, i.e. the human motion information and the Z-axis direction acceleration mentioned above, and after accumulating 5 frames of data, summing is performed to obtain the acceleration sum value (AccZ) in the Z-axis direction, and AccZ is taken as the input value of the mapping algorithm. The present application selects to sum 5 frames of data, which can increase the change degree of adjacent inertial sensor signals, and the change of adjacent AccZ is larger than the original adjacent acceleration change value, which can lead to more obvious sound change after mapping; on the other hand, selecting 5 frames of data will not cause too large delay.

[0073] S3, generating music instructions according to the target human motion information and user input information.

[0074] In one implementation manner of the embodiment, the music instructions include Note On instructions;

[0075] Generating music instructions according to the target human motion information and user input information, specifically including:

[0076] S31, defining a string structure according to the Note On instructions in the MIDI protocol, and the string structure includes a data frame start marker, a data frame end marker, a data segmentation marker and a control byte;

[0077] S32, receiving the string and performing data parsing on the string, if the data parsing is correct, assigning the user input information and the acceleration and value in the Z-axis direction to the corresponding positions of the control bytes of the string; if the data parsing is incorrect, re-receiving the string;

[0078] S33, extracting the control information contained in the assigned string, and generating a Note On instruction according to the control information.

[0079] In one implementation manner of the embodiment, the control bytes of the string include a first byte, a second byte and a third byte;

[0080] Assigning the user input information and the acceleration and value in the Z-axis direction to the corresponding positions of the control bytes of the string, specifically including:

[0081] S321, assigning the number of play channels in the user input information to the first byte position of the string, for indicating playing music in the play channel corresponding to the number of play channels;

[0082] S322, assigning the acceleration and value in the Z-axis direction to the second byte position of the string, for indicating playing music with the acceleration and value in the Z-axis direction as the pitch;

[0083] S323, assigning the loudness value in the user input information to the third byte position of the string, for indicating playing music with the loudness value as the volume.

[0084] Specifically, an embedded code is deployed in the core controller, after the core controller is started, the function implemented by the code is as shown in Figure 3 , that is, the working process of the core controller, specifically including:

[0085] 1) the embedded system in the core controller, that is, the single-chip microcomputer receives the inertial sensor data, accumulates 5 frames of data and then performs summation, obtains the acceleration and value AccZ in the Z-axis direction, and takes AccZ as the input value of the mapping algorithm; here, selecting to sum 5 frames of data can increase the change degree of adjacent inertial sensor signals, the change of adjacent AccZ is larger than the original adjacent acceleration change value, which can cause more obvious sound change after mapping; on the other hand, selecting 5 frames of data will not cause too large delay;

[0086] 2) according to the music instruction in the MIDI protocol, that is, the MIDI instruction-Note On instruction, a new string structure is defined, as shown in Figure 4As shown, the string structure includes a data frame start marker (#), a data frame end marker (*), two data division markers ( / ), and three control bytes (a first byte, a second byte, and a third byte, denoted as data1, data2, and data3, respectively). The three control bytes data1, data2, and data3 correspond to three data bytes of a "Note On instruction", and their controlled functions are playing channel, pitch, and loudness, respectively.

[0087] 3) The string is sent to the embedded system of the core controller, i.e., a single-chip microcomputer, through serial communication for data analysis. Whether the data frame is successfully received is determined by the data frame start marker (#) and the data frame end marker (*). If the markers cannot be recognized, it is determined that "data analysis is incorrect (the number of data does not meet the requirement)", and the data is received again. If the markers are successfully recognized, it is determined that "data analysis is correct".

[0088] 4) When the data analysis is correct, the three control bytes of the string are assigned values, including:

[0089] i. The value "0x90" is assigned to the data1 control byte position of the string, indicating that the Note on instruction (playing a note) is performed on channel 1.

[0090] ii. The value "AccZ" is assigned to the data2 control byte position of the string as a signal for controlling the pitch.

[0091] iii. The value "90" is assigned to the data3 control byte position of the string, indicating the dynamics (volume size) of the note.

[0092] 5) The audio codec of the core controller substitutes the assigned string as three data bytes into the "Note On instruction" to generate a music instruction, i.e., a MIDI instruction, and sends the instruction to the audio codec built in the electronic music player through serial communication. The audio codec of the electronic music player decodes the instruction to generate a music melody, and the electronic music player produces music through an audio line.

[0093] S4, the music instruction is sent to the electronic music player to control the electronic music player to generate music and play according to the music instruction.

[0094] In one implementation of the embodiment, the music instruction is sent to the electronic music player to control the electronic music player to generate music and play according to the music instruction, specifically including:

[0095] S41, the Note On instruction is sent to the electronic music player with a built-in audio codec through serial communication.

[0096] S42, control the audio codec of the electronic music player to decode the Note On instruction to generate a playable music melody;

[0097] S43, control the electronic music player to play the music melody through the speaker component of the electronic music player.

[0098] Specifically, the core controller is connected with the electronic music player through a serial port connector (CH340), and sends the music instruction (i.e. MIDI instruction) Note On instruction to the electronic music player with a built-in audio codec through serial communication, the audio codec of the electronic music player decodes the above music instruction to generate a playable music melody, and the electronic music player generates music through an audio line, and the electronic music player plays music through a speaker component.

[0099] S5, performing gait analysis according to the music played by the electronic music player to obtain a gait analysis result.

[0100] In one implementation of the embodiment, the gait analysis is performed according to the music played by the electronic music player to obtain a gait analysis result, which specifically includes:

[0101] judging whether the current gait is a normal gait or an abnormal gait according to the music played by the electronic music player to obtain a gait analysis result;

[0102] if the gait analysis result indicates that the current gait is a normal gait, no warning is given;

[0103] if the gait analysis result indicates that the current gait is an abnormal gait, a warning is given.

[0104] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information, on the one hand, it can help patients improve motor coordination, on the other hand, music can also be used as a positive stimulus and motivation to make patients more actively participate in the rehabilitation process. The present application realizes the mapping between human motion and music, generates music based on human motion information, so that human motion can be reflected through music, such as reflecting the difference between normal and abnormal gaits through music signals to realize gait analysis.

[0105] In addition, the music generation and gait analysis method based on human motion information further includes:

[0106] collecting historical human motion information collected by the inertial sensor installed on the human body;

[0107] performing data processing and calibration on the historical human motion information to obtain a data set;

[0108] The dataset is divided into a training set and a test set, and the deep learning network model is trained and tested using the training set and the test set to obtain a music generation and gait analysis model;

[0109] Real-time human motion information collected by the inertial sensor is obtained, and the real-time human motion information is input into the music generation and gait analysis model, and the music generation and gait analysis model outputs music generation and gait analysis results.

[0110] The present application can also use artificial intelligence and machine learning algorithms to analyze human motion data and generate music. This method can use a deep learning model to understand the motion pattern and convert it into music output. Through deep learning, complex motion data is automatically analyzed and converted into music, reducing human intervention; it can create new music styles and patterns, with high innovation potential.

[0111] In summary, the present application innovatively proposes a music generation method based on human motion information, realizes a wearable music generation system based on wearable inertial sensors (IMU) using a core controller to generate music signals, and proposes an innovative "action-music mapping algorithm" by combining the above hardware system with music theory, which can generate corresponding sound / music according to different human actions. The design route of the whole system is as shown in Figure 5 The inertial sensor 4 is worn on different parts of the human body 5, the inertial sensor 4 collects human motion information, and the human motion information is sent to the core controller 6 through the UDP communication protocol, the core controller 6 generates corresponding music instructions, i.e. MIDI instructions, according to the human motion information, and then sends the MIDI instructions to the electronic music player 2 through serial communication, and the electronic music player 2 generates music according to the MIDI instructions and plays the music.

[0112] The present application proposes a wearable music generation system based on human motion information (action), which is composed of multiple wearable inertial sensors (IMU) and digital audio codecs. After the user wears the IMU, the IMU can obtain real-time human motion information during the movement, and wirelessly transmit it to the digital audio codec, generate electronic music through the MIDI protocol, and play the music through the external sound box, earphone or host computer with sound card. That is, the present application innovatively combines wearable inertial sensors with music generation systems to create a more interactive and interesting music generation mode, method and system, providing personalized and instant music feedback for humans. At the application level, the present application can provide a new rehabilitation tool, especially for patients with motor dysfunction or gait problems, which can promote neural rehabilitation by enhancing sensory motor synchronization and increasing the participation degree of rehabilitation.

[0113] The method is verified by experiments. The IMU inertial sensor is placed on the vamp by pasting, sensor signals at different walking speeds (including fast walking at a speed of about 3 m / s and slow walking at a speed of about 1.2 m / s) are obtained, and then different music is generated by using the method proposed in the application.

[0114] The experiment compares the differences of the generated music from multiple dimensions such as the waveform graph, the spectrum graph and the frequency spectrum graph of the sound signal, so as to prove that the method proposed in the application can generate different sound signals based on different gaits, and therefore has application value and application prospect in the fields of gait analysis and gait rehabilitation.

[0115] The waveform graph of the sound signal obtained in the experiment is shown in Figure 6 Figure 6 (a) in the figure is the waveform graph of the sound signal corresponding to fast walking, Figure 6 (b) in the figure is the waveform graph of the sound signal corresponding to normal walking. The waveform graph reflects the information of the time characteristics of the generated sound signal, and visualizes the case that the amplitudes (loudnesses) of the two sound signals change with time. It can be seen that the waveform graphs of the sounds generated by people through different walking speeds have obvious differences, which are reflected in the shapes and structures of the sound waveforms, indicating that the sounds generated thereby have specific characteristics and can effectively reflect the gait differences.

[0116] The spectrum graph of the sound signal obtained in the experiment is shown in Figure 7 Figure 7 (a) in the figure is the spectrum graph of the sound signal corresponding to fast walking, Figure 7 (b) in the figure is the spectrum graph of the sound signal corresponding to normal walking. The spectrum graph reveals the frequency distribution in the sound signal. The colors of the spectrum graph represent the intensities or powers of the corresponding frequency components, and reflect how the frequency content of the signal changes with time. There is an obvious periodic change pattern in this spectrum graph, which indicates the existence of harmonics in the sound signal generated by walking. However, the spectrum graphs of the sounds generated by different walking speeds are different, and the periodic change is more obvious during fast walking. Since the interval between adjacent spectrums represents the harmonic relationship, this result may indicate that the sound signal generated during fast walking is more stable.

[0117] The frequency spectrum graph of the sound signal obtained in the experiment is shown in Figure 8 Figure 8 (a) in the figure is the frequency spectrum graph of the sound signal corresponding to fast walking, Figure 8 ​​​(b) in the figure is the spectrogram of the sound signal corresponding to normal walking. The spectrogram provides information about the amplitude or power of each frequency component. The height or intensity of each peak in the spectrum represents the intensity or energy of that particular frequency component in the signal. The spectrogram allows us to gain insight into the relative loudness or intensity of different frequencies in the sound, and which frequencies dominate or stand out in the signal. The results of the spectrogram also illustrate the differences in frequency distribution of the sound produced by different walking speeds. These differences play an important guiding role in the further optimization of the tone quality, timbre or characteristic sound of the generated audio.

[0118] In addition, based on the above-mentioned music generation and gait analysis method based on human body motion information, the present application also correspondingly provides a music generation and gait analysis system based on human body motion information, wherein the preferred embodiment of the music generation and gait analysis system based on human body motion information, as shown in Figure 9 , specifically includes:

[0119] Data acquisition module 01: used for acquiring human body motion information collected by the inertial sensor installed on the human body;

[0120] Data processing module 02: used for processing the human body motion information to obtain target human body motion information;

[0121] Music instruction generation module 03: used for generating music instructions according to the target human body motion information and user input information;

[0122] Music generation module 04: used for sending the music instructions to the electronic music player to control the electronic music player to generate music and play according to the music instructions;

[0123] Gait analysis module 05: used for performing gait analysis according to the music played by the electronic music player to obtain gait analysis results.

[0124] In addition, based on the above-mentioned music generation and gait analysis method and system based on human body motion information, the present application also correspondingly provides a terminal, wherein the preferred embodiment of the terminal, as shown in Figure 10 , specifically includes a processor 10, a memory 20 and a display 30. Figure 10 Only part of the components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0125] The memory 20 can be an internal storage unit of the terminal, such as a hard disk or a memory, in some embodiments. The memory 20 can also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, and the like, in other embodiments. Further, the memory 20 can include both an internal storage unit and an external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes of the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores the music generation and gait analysis program based on human motion information 40, which can be executed by the processor 10 to implement the steps of the music generation and gait analysis method based on human motion information in the present application.

[0126] The processor 10 can be a Central Processing Unit (CPU), a microprocessor, or other data processing chip, used to run program codes stored in the memory 20 or process data, such as the music generation and gait analysis program based on human motion information 40, in some embodiments.

[0127] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, and the like, in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0128] In an embodiment, the steps of the music generation and gait analysis method based on human motion information as described above are implemented when the processor 10 executes the music generation and gait analysis program based on human motion information 40 in the memory 20.

[0129] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a music generation and gait analysis program based on human motion information, which implements the steps of the music generation and gait analysis method based on human motion information as described above when executed by a processor.

[0130] It should be noted that, in the present document, the terms "comprises / comprising" or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more limitations, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0131] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable storage medium readable by a computer. When the program is executed, it can include the processes of the above-mentioned method embodiments. The computer readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0132] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A music generation and gait analysis method based on human motion information, characterized in that, The music generation and gait analysis method based on human motion information comprises: acquiring human motion information collected by an inertial sensor installed on a human body; processing the human motion information to obtain target human motion information; generating music instructions according to the target human motion information and user input information; sending the music instructions to an electronic music player to control the electronic music player to generate and play music according to the music instructions; performing gait analysis according to the music played by the electronic music player to obtain gait analysis results; receiving the human motion information collected by the inertial sensor through a UDP communication protocol; the target human motion information comprises a Z-axis direction acceleration sum; the music instructions comprise Note On instructions; the generation of the music instructions according to the target human motion information and user input information specifically comprises: defining a string structure according to Note On instructions in a MIDI protocol, the string structure comprising a data frame start marker, a data frame end marker, a data division marker and a control byte; receiving a string and performing data analysis on the string, if the data analysis is correct, assigning the user input information and the Z-axis direction acceleration sum to corresponding positions of the control byte of the string, if the data analysis is incorrect, receiving a string again; extracting control information contained in the assigned string and generating Note On instructions according to the control information; the control byte of the string comprises a first byte, a second byte and a third byte; the assignment of the user input information and the Z-axis direction acceleration sum to corresponding positions of the control byte of the string specifically comprises: assigning a playback channel number in the user input information to a first byte position of the string to indicate playing music in a playback channel corresponding to the playback channel number; assigning the Z-axis direction acceleration sum to a second byte position of the string to indicate playing music with the Z-axis direction acceleration sum as a pitch; assigning a loudness value in the user input information to a third byte position of the string to indicate playing music with the loudness value as a volume.

2. The music generation and gait analysis method based on human motion information according to claim 1, characterized in that, the acquisition of the human motion information collected by the inertial sensor installed on the human body specifically comprises: the human motion information is Z-axis direction acceleration data, and the inertial sensor is installed on any part of the human body.

3. The music generation and gait analysis method based on human motion information according to claim 2, characterized in that, the processing of the human motion information to obtain target human motion information specifically comprises: summing up the Z-axis direction acceleration data after accumulating a set number of frames to obtain a Z-axis direction acceleration sum.

4. The music generation and gait analysis method based on human motion information according to claim 1, characterized in that, the sending of the music instructions to the electronic music player to control the electronic music player to generate and play music according to the music instructions specifically comprises: sending the Note On instructions to an electronic music player with a built-in audio codec through serial communication; controlling the audio codec of the electronic music player to decode the Note On instructions to generate a playable music melody; The electronic music player is controlled to play the music melody through a speaker component of the electronic music player.

5. The music generation and gait analysis method based on human motion information according to claim 1, characterized in that, The gait analysis is performed according to the music played by the electronic music player, and a gait analysis result is obtained. The current gait is determined to be a normal gait or an abnormal gait according to the music played by the electronic music player, and a gait analysis result is obtained. If the gait analysis result indicates that the current gait is a normal gait, no warning is given. If the gait analysis result indicates that the current gait is an abnormal gait, a warning is given.

6. A music generation and gait analysis system based on human motion information, the music generation and gait analysis system based on human motion information being applied to the music generation and gait analysis method based on human motion information according to any one of claims 1 to 5, characterized in that, The music generation and gait analysis system based on human motion information comprises: a data acquisition module configured to acquire human motion information collected by an inertial sensor installed on a human body; a data processing module configured to process the human motion information to obtain target human motion information; a music instruction generation module configured to generate music instructions according to the target human motion information and user input information; a music generation module configured to send the music instructions to an electronic music player, and control the electronic music player to generate music according to the music instructions and play the music; a gait analysis module configured to perform gait analysis according to the music played by the electronic music player, and obtain a gait analysis result.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a music generation and gait analysis program based on human motion information stored on the memory and executable on the processor. When the music generation and gait analysis program based on human motion information is executed by the processor, the steps of the music generation and gait analysis method based on human motion information according to any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a music generation and gait analysis program based on human motion information. When the music generation and gait analysis program based on human motion information is executed by the processor, the steps of the music generation and gait analysis method based on human motion information according to any one of claims 1-5 are implemented.

Images