Music generation and gait analysis method and system based on human body motion information

Abstract

The invention discloses a music generation and gait analysis method and system based on human body motion information. The method comprises the steps that human body motion information collected by an inertial sensor installed on a human body is acquired; processing the human body motion information to obtain target human body motion information; generating a music instruction according to the target human body motion information and the user input information; sending the music instruction to an electronic music player, and controlling the electronic music player to generate music according to the music instruction and play the music; and performing gait analysis according to music played by the electronic music player to obtain a gait analysis result. Human body motion information is collected through the wearable inertial sensor, personalized and instant music is generated according to the collected human body motion information, and mapping between human body motions and the music is achieved; the gait difference of the human body is reflected through the generated music, and gait analysis is achieved; the device can be used as a rehabilitation treatment tool to enhance sensory movement synchronism and improve patient treatment enthusiasm.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method, system, terminal and computer-readable storage medium for music generation and gait analysis based on human motion information. Background Art

[0002] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information can help patients improve their motor coordination on the one hand, and on the other hand, music can also be used as a positive stimulus and motivation to make patients more actively participate in the rehabilitation process.

[0003] When using traditional electronic music generators (electronic musical instruments), such as electronic keyboards, keyboards, drum machines and other devices to generate music, the creator needs to deliberately complete control actions such as tapping. This requires the creator to undergo a certain amount of training to generate meaningful music. Otherwise, the music generated by controlling these electronic instruments is often random and can hardly be called music. In addition to the above methods, existing artificial intelligence models can also generate music, such as the Suno AI music generator, but intelligent music generators lack active human participation in music creation, and therefore usually lack emotion, originality and artistic quality.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a music generation and gait analysis method, system, terminal and computer-readable storage medium based on human motion information, aiming to solve the problems that the existing music generation schemes through electronic musical instruments require the creators to have experience and high operation requirements, and the music schemes generated through artificial intelligence models lack individuality, resulting in poor results.

[0006] To achieve the above-mentioned purpose of the invention, the present invention provides a music generation and gait analysis method based on human motion information, and the music generation and gait analysis method based on human motion information comprises:

[0007] Acquire human motion information collected by an inertial sensor installed on the human body;

[0008] Processing the human body motion information to obtain target human body motion information;

[0009] Generate music instructions according to the target human body movement information and user input information;

[0010] Sending the music instruction to an electronic music player, and controlling the electronic music player to generate and play music according to the music instruction;

[0011] Gait analysis is performed according to the music played by the electronic music player to obtain a gait analysis result.

[0012] Optionally, the acquiring of human body motion information collected by an inertial sensor installed on the human body is specifically:

[0013] The human body motion information collected by the inertial sensor is received through the UDP communication protocol, wherein the human body motion information is acceleration data in the Z-axis direction, and the inertial sensor is installed at any part of the human body.

[0014] Optionally, the target human body motion information includes acceleration and value in the Z-axis direction;

[0015] The processing of the human body motion information to obtain target human body motion information is specifically as follows:

[0016] Accumulate the acceleration data in the Z-axis direction for a set number of frames and then sum them to obtain the acceleration sum value in the Z-axis direction.

[0017] Optionally, the music instruction includes a Note On instruction;

[0018] The generating of music instructions according to the target human body motion information and the user input information specifically includes:

[0019] According to the Note On instruction in the MIDI protocol, a string structure is defined, wherein the string structure includes a data frame start mark, a data frame end mark, a data segmentation mark and a control byte;

[0020] Receive a character string and perform data analysis on the character string. If the data analysis is correct, assign the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the character string; if the data analysis is wrong, receive the character string again;

[0021] The control information contained in the assigned character string is extracted, and a Note On instruction is generated according to the control information.

[0022] Optionally, the control bytes of the character string include a first byte, a second byte and a third byte;

[0023] The assigning the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the character string specifically includes:

[0024] Assigning the playback channel number in the user input information to the first byte position of the character string to indicate that music is played on the playback channel corresponding to the playback channel number;

[0025] Assigning the acceleration and value in the Z-axis direction to the second byte position of the character string to indicate that the music is played with the acceleration and value in the Z-axis direction as the pitch;

[0026] The loudness value in the user input information is assigned to the third byte position of the character string to indicate that the music is played with the loudness value as the volume.

[0027] Optionally, sending the music instruction to an electronic music player and controlling the electronic music player to generate and play music according to the music instruction specifically includes:

[0028] Sending the Note On command to an electronic music player with a built-in audio codec via serial communication;

[0029] Controlling the audio codec of the electronic music player to decode the Note On instruction to generate a playable music melody;

[0030] The electronic music player is controlled to play the music melody through a speaker component of the electronic music player.

[0031] Optionally, performing gait analysis according to the music played by the electronic music player to obtain a gait analysis result specifically includes:

[0032] Determine whether the current gait is a normal gait or an abnormal gait according to the music played by the electronic music player, and obtain a gait analysis result;

[0033] If the gait analysis result indicates that the current gait is a normal gait, no warning is issued;

[0034] If the gait analysis result indicates that the current gait is an abnormal gait, a warning is issued.

[0035] To achieve the above-mentioned purpose of the invention, the present invention also provides a music generation and gait analysis system based on human motion information, the music generation and gait analysis system based on human motion information comprises:

[0036] Data acquisition module: used to obtain human motion information collected by the inertial sensor installed on the human body;

[0037] Data processing module: used for processing the human body motion information to obtain target human body motion information;

[0038] Music instruction generation module: used to generate music instructions according to the target human body movement information and user input information;

[0039] Music generation module: used for sending the music instruction to the electronic music player, controlling the electronic music player to generate and play music according to the music instruction;

[0040] Gait analysis module: used to perform gait analysis based on the music played by the electronic music player to obtain gait analysis results.

[0041] In order to achieve the above-mentioned purpose of the invention, the present invention also provides a terminal, which includes: a memory, a processor, and a music generation and gait analysis program based on human motion information stored in the memory and run on the processor, and the music generation and gait analysis program based on human motion information, when executed by the processor, implements the steps of the music generation and gait analysis method based on human motion information as described above.

[0042] In order to achieve the above-mentioned purpose of the invention, the present invention also provides a computer-readable storage medium, which stores a music generation and gait analysis program based on human motion information. When the music generation and gait analysis program based on human motion information is executed by a processor, the steps of the music generation and gait analysis method based on human motion information as described above are implemented.

[0043] In the present invention, human motion information collected by an inertial sensor installed on a human body is obtained; the human motion information is processed to obtain target human motion information; music instructions are generated according to the target human motion information and user input information; the music instructions are sent to an electronic music player, and the electronic music player is controlled to generate and play music according to the music instructions; gait analysis is performed according to the music played by the electronic music player to obtain gait analysis results. The present invention collects human motion information through a wearable inertial sensor, generates personalized and instant music according to the collected human motion information, realizes the mapping between human motion and music, generates music based on human motion, and provides a more interactive and interesting music generation mode, which does not require the creator to have music creation experience; reflects the difference in human gait through the generated music, and realizes gait analysis; can be used as a rehabilitation treatment tool to enhance sensory-motor synchronization and improve the patient's treatment enthusiasm, especially for patients with motor dysfunction or gait problems, can promote neurological rehabilitation by enhancing sensory-motor synchronization and increasing the degree of participation in rehabilitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of the existing electronic music generation system;

[0045] Figure 2 It is a flow chart of a preferred embodiment of the method for music generation and gait analysis based on human motion information of the present invention;

[0046] Figure 3 It is a flow chart of the working principle of the core controller of the present invention;

[0047] Figure 4 is a schematic diagram of the string structure of the present invention;

[0048] Figure 5 is a structural diagram of the wearable music generating system of the present invention;

[0049] Figure 6 is a waveform diagram of a sound signal obtained by the experiment of the present invention;

[0050] Figure 7 is a spectrum diagram of the sound signal obtained by the experiment of the present invention;

[0051] Figure 8 is a spectrum diagram of a sound signal obtained by the experiment of the present invention;

[0052] Fig. 9 It is a structural diagram of a preferred embodiment of the music generation and gait analysis system based on human motion information of the present invention;

[0053] Fig.10 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information can help patients improve their motor coordination on the one hand, and on the other hand, music can also be used as a positive stimulus and motivation to make patients more actively participate in the rehabilitation process.

[0056] In addition to traditional instruments, electronically generated music has flourished in recent decades. Electronic music is the creation and performance of music using electronic instruments, synthesizers, computers and digital technology. It is characterized by the use of electronic devices and software to synthesize, process and modify sounds to create unique musical effects and sound textures. The origins of electronic music can be traced back to the early 20th century, when musicians began to use electronic instruments and electronic devices to explore new ways of creating music. With the continuous development of technology, electronic music has gradually become an independent music genre and has been widely developed and popularized in the second half of the 20th century and the early 21st century.

[0057] At present, the production of electronic music requires the manual creation of musicians to organize different musical elements into melodies or chords; for example, using drum machines, percussion instrument sampling or programmed drum beats to define the rhythm of the music; adjusting and processing the sound to create a unique timbre through tools such as synthesizers, samplers and audio effects; and then determining the arrangement, transition and structure of the track through the personalized creation of musicians. Existing electronic music generation systems, such as Figure 1 As shown in (a), first, an "electronic music input signal" needs to be generated by an electronic music generator 1 (also called an electronic musical instrument, MIDI musical instrument); the generated signal is converted into a MIDI signal (also called a MIDI command, MIDI data) by a built-in digital audio encoder; and then the MIDI signal is sent to an electronic music player 2 with a built-in digital audio decoder, and finally a playable melody is generated. Figure 1 As shown in (b), some of the steps can be completed by connecting to a computer 3 with a built-in digital audio codec, and the computer 3 is externally connected to the electronic music player 2 via a MIDI interface device 4.

[0058] Musical Instrument Digital Interface (MIDI) is a standard electronic communication protocol for the generation of electronic music. The MIDI protocol defines various notes or playing codes for electronic musical instruments and other performance devices (such as synthesizers), allowing electronic musical instruments, computers, mobile phones or other stage performance equipment to connect, adjust and synchronize with each other, and exchange performance data in real time.

[0059] In the prior art, when using traditional electronic music generators (electronic musical instruments), such as electronic keyboards, keyboards, drum machines and other devices to generate music, the creator needs to deliberately complete control actions such as tapping. This requires the creator to undergo certain training to generate meaningful music. Otherwise, the music generated by controlling these electronic instruments is often random and can hardly be called music. In addition to the above methods, existing artificial intelligence models can also generate music, such as the Suno AI music generator, but intelligent music generators lack active human participation in music creation, and therefore usually lack emotion, originality and artistic quality.

[0060] In order to solve the above technical problems, the present invention provides a music generation and gait analysis method based on human motion information, obtaining human motion information collected by an inertial sensor installed on the human body; processing the human motion information to obtain target human motion information; generating music instructions according to the target human motion information and user input information; sending the music instructions to an electronic music player, controlling the electronic music player to generate and play music according to the music instructions; performing gait analysis according to the music played by the electronic music player to obtain gait analysis results. The present invention collects human motion information through a wearable inertial sensor, generates personalized and instant music according to the collected human motion information, and realizes the mapping between human motion and music; by combining the inertial sensor and the music generation system, music is generated based on human motion, providing a more interactive and interesting music generation mode, which does not require the creator to have music creation experience; gait analysis is realized by reflecting the difference in human gait through the generated music; it can be used as a rehabilitation treatment tool to enhance sensory motor synchronization and improve the patient's treatment enthusiasm, especially for patients with motor dysfunction or gait problems, it can promote neurological rehabilitation by enhancing sensory motor synchronization and increasing the degree of participation in rehabilitation.

[0061] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0062] A preferred embodiment of the music generation and gait analysis method based on human motion information of the present invention is as follows: Figure 2 As shown, specifically including:

[0063] S1. Acquire human motion information collected by an inertial sensor installed on a human body.

[0064] In one implementation of this embodiment, the human body motion information collected by an inertial sensor installed on the human body is obtained, specifically:

[0065] The human body motion information collected by the inertial sensor is received through the UDP communication protocol. The human body motion information is the acceleration data in the Z-axis direction. The inertial sensor is installed in any part of the human body.

[0066] Specifically, the core components of the inertial sensor developed by the present invention are a 9-axis inertial sensor chip (BNO055) and a single-chip microcomputer (esp32_wroom). The 9-axis inertial sensor chip (BNO055) is used to collect human motion information. The present invention obtains the Z-axis acceleration data collected by the BNO055 chip, with a sampling rate of 100hz. The single-chip microcomputer is used to send the data collected by the 9-axis inertial sensor chip to the core controller through the UDP communication protocol.

[0067] S2. Processing the human body motion information to obtain target human body motion information.

[0068] In one implementation of this embodiment, the target human body motion information includes the acceleration and value in the Z-axis direction;

[0069] The human body motion information is processed to obtain the target human body motion information, specifically:

[0070] Accumulate the acceleration data in the Z-axis direction for a set number of frames (5 frames) and sum them to obtain the acceleration sum value in the Z-axis direction.

[0071] Specifically, the core controller used in the present invention includes an embedded system based on Raspberry Pi, which is used to receive inertial sensor data and deploy mapping algorithms; a digital audio codec based on the VS1053b chip, which is used to generate music instructions (MIDI instructions); and a serial port connector (CH340), which is used to connect the core controller to the electronic music player. After the embedded system of the core controller receives the inertial sensor data through the UDP communication protocol, the inertial sensor data is processed by the "action-music mapping algorithm" proposed in the present invention, and then the processed inertial sensor data is encoded into MIDI instructions using the digital audio codec, and finally the MIDI instructions are sent to the electronic music player through serial port communication.

[0072] In the present invention, the embedded system in the core controller, i.e., the single-chip microcomputer, receives the inertial sensor data, i.e., the human body motion information, i.e., the acceleration in the Z-axis direction, accumulates 5 frames of data, and then sums them to obtain the acceleration sum value (AccZ) in the Z-axis direction, and uses AccZ as the input value of the mapping algorithm. The present invention chooses to sum 5 frames of data, which can increase the degree of change of adjacent inertial sensor signals. The change of adjacent AccZ is greater than the original adjacent acceleration change value, which can lead to more obvious sound changes after mapping; on the other hand, selecting 5 frames of data will not cause excessive delay.

[0073] S3. Generate music instructions based on the target human body motion information and user input information.

[0074] In one implementation of this embodiment, the music instruction includes a Note On instruction;

[0075] Generate music instructions based on the target human body movement information and user input information, including:

[0076] S31, according to the Note On instruction in the MIDI protocol, define a string structure, the string structure includes a data frame start mark, a data frame end mark, a data segmentation mark and a control byte;

[0077] S32, receiving a character string and performing data analysis on the character string. If the data analysis is correct, assigning the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the character string; if the data analysis is wrong, receiving the character string again;

[0078] S33. Extract the control information contained in the assigned character string, and generate a Note On instruction according to the control information.

[0079] In one implementation of this embodiment, the control bytes of the character string include a first byte, a second byte, and a third byte;

[0080] Assign the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the string, including:

[0081] S321, assigning the playback channel number in the user input information to the first byte position of the character string, so as to indicate that the music is played on the playback channel corresponding to the playback channel number;

[0082] S322, assigning the acceleration and value in the Z-axis direction to the second byte position of the character string, to indicate that the music is played with the acceleration and value in the Z-axis direction as the pitch;

[0083] S323: Assign the loudness value in the user input information to the third byte position of the character string, so as to indicate that the music is played with the loudness value as the volume.

[0084] Specifically, the embedded code is deployed in the core controller. After the core controller is powered on, the functions implemented by the code are as follows: Figure 3 As shown in the figure, the working process of the core controller includes:

[0085] 1) The embedded system in the core controller, i.e., the single-chip microcomputer, receives the inertial sensor data, accumulates 5 frames of data, and then sums them to obtain the acceleration and value AccZ in the Z-axis direction, and uses AccZ as the input value of the mapping algorithm; here, choosing to sum the 5 frames of data can increase the degree of change of adjacent inertial sensor signals. The change of adjacent AccZ is greater than the original adjacent acceleration change value, which can lead to more obvious sound changes after mapping; on the other hand, choosing 5 frames of data will not cause excessive delay;

[0086] 2) According to the music instruction in the MIDI protocol, that is, the MIDI instruction - Note On instruction, define a new string structure, such as Figure 4As shown in the figure, this string structure includes a data frame start mark (#), a data frame end mark (*), two data segmentation marks ( / ) and three control bytes (the first byte, the second byte and the third byte, respectively represented as data1, data2, and data3). The three control bytes data1, data2 and data3 correspond to the three data bytes of the "Note On command" respectively, and their control functions are respectively the playback channel, pitch and loudness;

[0087] 3) This string is sent to the embedded system of the core controller, i.e., the single-chip microcomputer, through the serial communication, for data parsing. The data frame start mark (#) and the data frame end mark (*) are used to determine whether the data frame is received successfully. If the mark cannot be recognized, it will be judged as "data parsing error (data number does not match)" and the data will be received again; if the mark is successfully recognized, it will be judged as "data parsing is correct";

[0088] 4) When the data is parsed correctly, assign values ​​to the three control bytes of the string, including:

[0089] i. Assign the value "0x90" to the data1 control byte position of this string, indicating that the Noteon command (playing a note) is performed on channel 1;

[0090] ii. Assign the value "AccZ" to the data2 control byte position of this string as a signal to control the pitch;

[0091] iii. Assign the value "90" to the data3 control byte position of this string to indicate the intensity (volume) of the note;

[0092] 5) The audio codec of the core controller substitutes the assigned string as three data bytes into the "NoteOn instruction" to generate a music instruction, namely a MIDI instruction, and sends the instruction to the audio codec built into the electronic music player through serial port communication. The audio codec of the electronic music player decodes the instruction to generate a music melody, and enables the electronic music player to produce music through the audio line.

[0093] S4, sending the music instruction to the electronic music player, and controlling the electronic music player to generate and play music according to the music instruction.

[0094] In one implementation of this embodiment, sending a music instruction to an electronic music player and controlling the electronic music player to generate and play music according to the music instruction specifically includes:

[0095] S41, sending the Note On command to the electronic music player with built-in audio codec via serial communication;

[0096] S42, controlling the audio codec of the electronic music player to decode the Note On instruction to generate a playable music melody;

[0097] S43, controlling the electronic music player to play the music melody through the speaker component of the electronic music player.

[0098] Specifically, the core controller is connected to the electronic music player through a serial port connector (CH340), and sends the music instruction (i.e., MIDI instruction) - Note On instruction to the electronic music player with a built-in audio codec through serial port communication. The audio codec of the electronic music player decodes the above music instruction to generate a playable music melody, and the electronic music player produces music through the audio cable, and the electronic music player plays the music through the speaker component.

[0099] S5. Perform gait analysis based on the music played by the electronic music player to obtain a gait analysis result.

[0100] In one implementation of this embodiment, gait analysis is performed based on the music played by the electronic music player to obtain a gait analysis result, which specifically includes:

[0101] According to the music played by the electronic music player, it is determined whether the current gait is a normal gait or an abnormal gait, and a gait analysis result is obtained;

[0102] If the gait analysis result indicates that the current gait is normal, no warning will be issued;

[0103] If the gait analysis result indicates that the current gait is abnormal, an alarm is issued.

[0104] In recent years, music therapy has become an important application in neuroscience and rehabilitation. Using music as feedback information can help patients improve their motor coordination on the one hand, and on the other hand, music can also be used as a positive stimulus and motivation to allow patients to participate more actively in the rehabilitation process. The present invention realizes the mapping between human movement and music, and generates music based on human movement information. Therefore, human movement can be reflected by music, such as reflecting the difference between normal and abnormal gait through music signals, so as to realize gait analysis.

[0105] In addition, the music generation and gait analysis methods based on human motion information also include:

[0106] Collect historical human motion information collected by inertial sensors mounted on the human body;

[0107] Perform data processing and calibration on historical human motion information to obtain a data set;

[0108] The data set is divided into a training set and a test set, and the deep learning network model is trained and tested using the training set and the test set to obtain a music generation and gait analysis model;

[0109] Real-time human motion information collected by inertial sensors is obtained, and the real-time human motion information is input into the music generation and gait analysis model, and the music generation and gait analysis model outputs the music generation and gait analysis results.

[0110] The present invention can also use artificial intelligence and machine learning algorithms to analyze human motion data and generate music. This method can use deep learning models to understand motion patterns and convert them into musical output. Through deep learning, complex motion data can be automatically analyzed and converted into music, reducing human intervention; completely new music styles and patterns can be created, with high innovation potential.

[0111] In summary, the present invention innovatively proposes a music generation method based on human motion information, realizes a wearable music generation system based on a wearable inertial sensor (IMU) that uses a core controller to generate music signals, and proposes an innovative "action-music mapping algorithm" by combining the above hardware system with music theory, which can generate corresponding sounds / music according to different human movements. The design route of the entire system is as follows: Figure 5 As shown, the inertial sensor 4 is worn on different parts of the human body 5, the inertial sensor 4 collects human motion information, and sends the human motion information to the core controller 6 through the UDP communication protocol. The core controller 6 generates corresponding music instructions, namely MIDI instructions, according to the human motion information, and then sends the MIDI instructions to the electronic music player 2 through the serial port communication. The electronic music player 2 generates music according to the MIDI instructions and plays it.

[0112] The present invention proposes a wearable music generation system based on human motion information (action), which is composed of multiple wearable inertial sensors (IMU) and digital audio codecs. After the user wears the IMU, during the exercise, the IMU can obtain human motion information in real time, and transmit it wirelessly to the digital audio codec, generate electronic music through the MIDI protocol, and play the music through an external speaker, headphones or a host with a sound card. In other words, the present invention innovatively combines wearable inertial sensors with a music generation system to create a more interactive and interesting music generation mode, method and system to provide humans with personalized and instant music feedback. At the application level, the present invention can provide a new rehabilitation treatment tool, especially for patients with motor dysfunction or gait problems, which can promote neurological rehabilitation by enhancing sensory-motor synchronization and increasing the degree of participation in rehabilitation.

[0113] The method of the present invention is verified by an experiment. The IMU inertial sensor is placed on the shoe surface by gluing, and the sensor signals at different walking speeds (including fast walking at a speed of about 3m / s and slow walking at a speed of about 1.2m / s) are obtained, and then different music is generated using the method proposed by the present invention.

[0114] The experiment compared the differences in the generated music from multiple dimensions such as the waveform diagram, spectrogram and spectrum diagram of the sound signal, thereby proving that the method proposed in the present invention can generate different sound signals based on different gaits, and therefore has application value and application prospects in the fields of gait analysis and gait rehabilitation.

[0115] The waveform of the sound signal obtained in the experiment is as follows Figure 6 As shown, Figure 6 (a) is the waveform of the sound signal corresponding to fast walking. Figure 6 (b) in the figure is a waveform diagram of the sound signal corresponding to normal walking. The waveform diagram reflects the information of the time characteristics of the generated sound signal, and visualizes the change of the amplitude (loudness) of the two sound signals over time. It can be seen that the waveform diagrams of the sound generated by people at different walking speeds in the present invention are obviously different, which is reflected in the shape and structure of the sound waveform, indicating that the sound generated thereby has specific characteristics and can effectively reflect the gait difference.

[0116] The spectrum of the sound signal obtained in the experiment is as follows: Figure 7 As shown, Figure 7 (a) is the spectrum of the sound signal corresponding to fast walking. Figure 7 (b) in the figure is the spectrogram of the sound signal corresponding to normal walking. The spectrogram reveals the distribution of frequencies in the sound signal. The color of the spectrogram indicates the intensity or power of the corresponding frequency component, and reflects how the frequency content of the signal changes over time. There is an obvious periodic change pattern in this spectrogram, indicating the existence of harmonics in the sound signal generated by walking. However, the sound spectrograms generated by different walking speeds are different in form, and the periodic changes are more obvious during fast walking. Since the spacing between adjacent spectra represents the harmonic relationship, this result may indicate that the sound signal generated during fast walking is more stable.

[0117] The spectrum of the sound signal obtained in the experiment is as follows: Figure 8 As shown, Figure 8 (a) is the spectrum of the sound signal corresponding to fast walking. Figure 8(b) in the figure is the spectrogram of the sound signal corresponding to normal walking. The spectrogram provides information about the amplitude or power of each frequency component. The height or intensity of each peak in the spectrum represents the strength or energy of that particular frequency component in the signal. The spectrogram can give us insight into the relative loudness or intensity of different frequencies in the sound, and which frequencies dominate or stand out in the signal. The results of the spectrogram also illustrate the differences in the frequency distribution of sounds generated by different walking speeds. These differences play an important guiding role in further optimizing the pitch quality, timbre or characteristic sound of the generated audio.

[0118] In addition, based on the above-mentioned music generation and gait analysis method based on human motion information, the present invention also provides a music generation and gait analysis system based on human motion information, wherein a preferred embodiment of the music generation and gait analysis system based on human motion information is as follows: Fig. 9 As shown, specifically including:

[0119] Data acquisition module 01: used to obtain human motion information collected by an inertial sensor installed on the human body;

[0120] Data processing module 02: used to process human body motion information to obtain target human body motion information;

[0121] Music instruction generation module 03: used to generate music instructions according to target human body movement information and user input information;

[0122] Music generation module 04: used to send music instructions to the electronic music player, and control the electronic music player to generate and play music according to the music instructions;

[0123] Gait analysis module 05: used to perform gait analysis based on the music played by the electronic music player to obtain gait analysis results.

[0124] In addition, based on the above-mentioned music generation and gait analysis method and system based on human motion information, the present invention also provides a terminal accordingly, wherein a preferred embodiment of the terminal is as follows: Fig.10 As shown, it specifically includes a processor 10, a memory 20 and a display 30. Fig.10 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0125] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card) equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as program codes of the storage terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a music generation and gait analysis program 40 based on human motion information is stored on the memory 20, and the music generation and gait analysis program 40 based on human motion information can be executed by the processor 10, thereby realizing the steps of the music generation and gait analysis method based on human motion information in the present application.

[0126] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing a music generation and gait analysis program 40 based on human motion information.

[0127] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0128] In one embodiment, when the processor 10 executes the music generation and gait analysis program 40 based on human motion information in the memory 20, the steps of the music generation and gait analysis method based on human motion information as described above are implemented.

[0129] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a music generation and gait analysis program based on human motion information, and when the music generation and gait analysis program based on human motion information is executed by a processor, the steps of the music generation and gait analysis method based on human motion information are implemented as described above.

[0130] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0131] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0132] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A music generation and gait analysis method based on human motion information, characterized in that: The music generation and gait analysis method based on human motion information includes: Acquire human motion information collected by an inertial sensor installed on the human body; Processing the human body motion information to obtain target human body motion information; Generate music instructions according to the target human body movement information and user input information; Sending the music instruction to an electronic music player, and controlling the electronic music player to generate and play music according to the music instruction; Gait analysis is performed according to the music played by the electronic music player to obtain a gait analysis result.

2. The method for music generation and gait analysis based on human motion information according to claim 1, characterized in that: The obtaining of the human body motion information collected by the inertial sensor installed on the human body is specifically as follows: The human body motion information collected by the inertial sensor is received through the UDP communication protocol, wherein the human body motion information is acceleration data in the Z-axis direction, and the inertial sensor is installed at any part of the human body.

3. The method for music generation and gait analysis based on human motion information according to claim 2, characterized in that: The target human body motion information includes the acceleration and value in the Z-axis direction; The processing of the human body motion information to obtain target human body motion information is specifically as follows: Accumulate the acceleration data in the Z-axis direction for a set number of frames and then sum them to obtain the acceleration sum value in the Z-axis direction.

4. The method for music generation and gait analysis based on human motion information according to claim 3, characterized in that: The music instructions include Note On instructions; The generating of music instructions according to the target human body motion information and the user input information specifically includes: According to the Note On instruction in the MIDI protocol, a string structure is defined, wherein the string structure includes a data frame start mark, a data frame end mark, a data segmentation mark and a control byte; Receive a character string and perform data analysis on the character string. If the data analysis is correct, assign the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the character string; if the data analysis is wrong, receive the character string again; The control information contained in the assigned character string is extracted, and a Note On instruction is generated according to the control information.

5. The method for music generation and gait analysis based on human motion information according to claim 4, characterized in that: The control bytes of the character string include a first byte, a second byte and a third byte; The assigning the user input information and the acceleration and value in the Z-axis direction to the corresponding position of the control byte of the character string specifically includes: Assigning the playback channel number in the user input information to the first byte position of the character string to indicate that music is played on the playback channel corresponding to the playback channel number; Assigning the acceleration and value in the Z-axis direction to the second byte position of the character string to indicate that the music is played with the acceleration and value in the Z-axis direction as the pitch; The loudness value in the user input information is assigned to the third byte position of the character string to indicate that the music is played with the loudness value as the volume.

6. The method for music generation and gait analysis based on human motion information according to claim 4 or 5, characterized in that: The sending of the music instruction to the electronic music player and controlling the electronic music player to generate and play music according to the music instruction specifically includes: Sending the Note On command to an electronic music player with a built-in audio codec via serial communication; Controlling the audio codec of the electronic music player to decode the Note On instruction to generate a playable music melody; The electronic music player is controlled to play the music melody through a speaker component of the electronic music player.

7. The method for music generation and gait analysis based on human motion information according to claim 1, characterized in that: The step of performing gait analysis according to the music played by the electronic music player to obtain a gait analysis result specifically includes: Determine whether the current gait is a normal gait or an abnormal gait according to the music played by the electronic music player, and obtain a gait analysis result; If the gait analysis result indicates that the current gait is a normal gait, no warning is issued; If the gait analysis result indicates that the current gait is an abnormal gait, a warning is issued.

8. A music generation and gait analysis system based on human motion information, characterized in that: The music generation and gait analysis system based on human motion information includes: Data acquisition module: used to obtain human motion information collected by the inertial sensor installed on the human body; Data processing module: used for processing the human body motion information to obtain target human body motion information; Music instruction generation module: used to generate music instructions according to the target human body movement information and user input information; Music generation module: used for sending the music instruction to the electronic music player, controlling the electronic music player to generate and play music according to the music instruction; Gait analysis module: used to perform gait analysis based on the music played by the electronic music player to obtain gait analysis results.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a music generation and gait analysis program based on human motion information stored in the memory and executable on the processor. When the music generation and gait analysis program based on human motion information is executed by the processor, the steps of the music generation and gait analysis method based on human motion information as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a music generation and gait analysis program based on human motion information, and when the music generation and gait analysis program based on human motion information is executed by a processor, the steps of the music generation and gait analysis method based on human motion information as described in any one of claims 1-7 are implemented.

Images