Motion prediction model training method, motion prediction method and device thereof

Abstract

The embodiment of the invention provides a motion prediction model training method, a motion prediction method and a motion prediction device. The motion prediction model training method comprises the following steps: acquiring physiological data and image data which are simultaneously acquired in a motion process of an animal; marking the motion state of the animal in each frame of image of the image data as a first motion state and a second motion state; dividing the physiological data collected in the movement process into a plurality of physiological data segments along a time axis; for each physiological data segment in the at least part of physiological data segments, according to the motion state of the animal marked in the image data, marking the motion state of the animal in the physiological data segment; and training the motion prediction model by taking the marked physiological data fragments as training sample data to obtain a trained motion prediction model. According to the scheme, the trained motion prediction model can predict the motion state of the animal in real time according to the physiological data of the animal.

Description

Technical Field

[0001] The present application relates to the field of motion prediction, and more specifically to a motion prediction model training method, a motion prediction method, a gait closed-loop stimulation output method, a motion prediction model training device, a motion prediction device, a gait closed-loop stimulation output device, an electronic device, a storage medium and a computer program product. Background Art

[0002] With the development of science and technology, the application of motion prediction is becoming more and more extensive. For example, in the rehabilitation treatment of lower limbs, walking exercises are an important process in motor rehabilitation. By outputting stimulation signals to the target object when it is in motion, the excitability of its muscles and neurons can be increased to accelerate motor rehabilitation. This requires accurate prediction of the motion state of the target object to output stimulation signals in real time.

[0003] However, in related technologies, external devices such as infrared cameras and inertial sensors are usually used to collect gait information and analyze the walking status offline. Doctors can use this information to formulate stimulation treatment strategies, and then connect the stimulation device for stimulation regulation. In this process, patient feedback is needed to improve the timing of stimulation, and continuous training is carried out to help the recovery of muscle or nerve function. However, the above method of detecting gait through external devices has a time lag, and the time resolution is difficult to guarantee for the complete walking process, which leads to low accuracy. In addition, the stimulation output depends on the guidance of the doctor, and has high requirements for the doctor's experience. Summary of the invention

[0004] The present invention has been made in view of the above-mentioned problems.

[0005] According to the first aspect of the present invention, a motion prediction model training method is provided. The motion prediction model training method includes: obtaining physiological data and image data collected simultaneously during the movement of an animal; marking the motion state of the animal in each frame of the image data, wherein the motion state includes a first action state and a second action state; dividing the physiological data collected during the movement into multiple physiological data segments along the time axis; for each physiological data segment in at least part of the physiological data segments, marking the motion state of the animal in the physiological data segment according to the motion state of the animal marked in the first image of the image data, wherein the first image is an image collected at a first acquisition time, and the first acquisition time is the acquisition time after the acquisition time of the physiological data segment and closest to the acquisition time corresponding to the physiological data segment; using the marked physiological data segments as training sample data to train the motion prediction model to obtain a trained motion prediction model.

[0006] Exemplarily, before using the labeled physiological data segments as training sample data to train the motion prediction model, the method also includes: determining the ratio of sample data marked as a first action state and sample data marked as a second action state in the training sample data based on the motion state of the animal marked in the training sample data; based on the ratio, sampling the sample data with a high ratio or resampling the sample data with a low ratio to balance the sample data marked as different states to update the training sample data.

[0007] Exemplarily, marking the movement state of the animal in each frame of the image data includes: using an image classification model to classify the image data according to the movement state of the animal in each frame of the image data to obtain a classification result; and marking the movement state of the animal in each frame of the image according to the classification result.

[0008] Exemplarily, marking the movement state of the animal in the physiological data segment based on the movement state of the animal marked in the first image of the image data includes: determining the movement state period of the first acquisition moment, the movement state period includes a first period and a second period, the movement state of the animal marked in the image acquired during the first period is the first action state, and the movement state of the animal marked in the image acquired during the second period is the second action state; when the first acquisition moment is in the first period, marking the movement state of the animal in the physiological data of the physiological data segment as the first action state; when the first acquisition moment is in the second period, marking the movement state of the animal in the physiological data of the physiological data segment as the second action state.

[0009] Exemplarily, marking the movement state of the animal in each frame of the image data includes: obtaining key point position information of the animal in each frame of the image data; for each frame of the image data, marking the movement state of the animal in the current frame according to the key point position information of the animal in the current frame and the key point position information of the animal in the previous frame.

[0010] Exemplarily, the key point position information includes toe position information; marking the movement state of the animal in the current frame image according to the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image includes: determining the displacement of the toes of the animal from the acquisition time of the previous frame image to the acquisition time of the current frame image according to the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image; determining and marking the movement state of the animal in the current frame image according to the displacement.

[0011] Exemplarily, the displacement includes horizontal displacement and vertical displacement; and determining and marking the motion state of the animal in the current frame image according to the displacement includes: determining and marking the motion state of the animal in the current frame image according to the following formula:

[0012] Label i =[sign(abs(x i -x i-1 ))]&[sign(abs(y i -y i-1 ))](i=2,3,……,N)

[0013] Among them, Label i represents the motion state of the animal in the current frame image; sign() represents the sign function; abs() represents the function of obtaining the absolute value; x i -x i-1 represents the horizontal displacement of the toes of the animal; y i -y i-1 represents the vertical displacement of the toes of the animal; N is the total number of image frames in the image data.

[0014] Exemplarily, before segmenting the physiological data into a plurality of physiological data segments, the motion prediction model training method further includes: denoising and / or filtering the physiological data.

[0015] Exemplarily, the physiological data includes sciatic nerve signal data, electromyographic signal data and / or brain motor cortex nerve signal data.

[0016] Exemplarily, the first action state is a leg-raising state, and the second action state is a standing state.

[0017] According to a second aspect of the present invention, a motion prediction method is provided. The motion prediction method comprises: using the above-mentioned motion prediction model training method to obtain the trained motion prediction model; inputting the currently obtained physiological data segment into the trained motion prediction model to obtain a prediction result, wherein the prediction result indicates the predicted motion state of the animal in the next period at the current moment.

[0018] According to a third aspect of the present invention, a gait closed-loop stimulation output method is provided, which comprises: obtaining the prediction result by using the above-mentioned motion prediction method; and outputting a stimulation signal to the animal when the prediction result indicates the first action state.

[0019] According to a fourth aspect of the present invention, there is also provided an ultrasonic image segmentation device, comprising:

[0020] An acquisition module, used to acquire physiological data and image data collected simultaneously during the movement of the animal;

[0021] A first marking module, for marking the motion state of the animal in each frame of the image data, wherein the state includes a first motion state and a second motion state;

[0022] A segmentation module, used for segmenting the physiological data collected during the exercise into a plurality of physiological data segments along a time axis;

[0023] a second marking module, configured to mark, for each physiological data segment in at least part of the physiological data segments, the motion state of the animal in the physiological data segment according to the motion state of the animal marked in a first image, wherein the first image is an image in the image data acquired at a first acquisition time, and the first acquisition time is an acquisition time of the image data that is closest to the acquisition time corresponding to the physiological data segment;

[0024] The training module is used to train the motion prediction model using the marked physiological data segments as training sample data to obtain a trained motion prediction model.

[0025] According to a fifth aspect of the present invention, there is also provided a motion prediction device, comprising:

[0026] A motion prediction model training device, used to obtain the trained motion prediction model using the above-mentioned motion prediction model training method;

[0027] The prediction module is used to input the currently obtained physiological data segment into the trained motion prediction model to obtain a prediction result, wherein the prediction result represents the prediction of the motion state of the animal in the next time period at the current moment.

[0028] According to a sixth aspect of the present invention, there is also provided a gait closed-loop stimulation output device, comprising:

[0029] A motion prediction device, used to obtain the prediction result by using the above-mentioned motion prediction method;

[0030] A stimulation module is used to output a stimulation signal to the animal when the prediction result indicates the first action state.

[0031] According to the seventh aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method when the processor is running.

[0032] According to an eighth aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method when running.

[0033] According to the ninth aspect of the present invention, there is also provided a computer program product, comprising computer program instructions, which are used to execute the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method when running.

[0034] In the above technical solution, the physiological data corresponding to the time is marked using the image data of the animal, and the motion prediction model is trained according to the marked physiological data. The trained motion prediction model can predict the animal's motion state in real time according to the animal's physiological data, and improves the accuracy and robustness of the prediction.

[0035] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1A schematic diagram showing an ultrasound image in an ultrasound image sequence according to an embodiment of the present invention;

[0038] Figure 2 A schematic flow chart of updating training sample data according to an embodiment of the present invention is shown;

[0039] Figure 3 A schematic flow chart showing the motion state of an animal in each frame of image data of a marking image according to an embodiment of the present invention is shown;

[0040] Figure 4 A schematic flow chart of marking the motion state of an animal in the physiological data segment according to the motion state of an animal marked in a first image of the image data according to an embodiment of the present invention is shown;

[0041] Figure 5 A schematic flow chart showing the motion state of an animal in each frame of image data of a marking image according to an embodiment of the present invention is shown;

[0042] Figure 6 A schematic flow chart of marking the motion state of an animal in a current frame image according to an embodiment of the present invention is shown;

[0043] Figure 7 A schematic flow chart of a motion prediction method according to an embodiment of the present invention is shown;

[0044] Figure 8 A schematic diagram showing a motion prediction model output prediction result according to an embodiment of the present invention;

[0045] Fig. 9 A schematic flow chart of a gait closed-loop stimulation output method according to an embodiment of the present invention is shown;

[0046] Fig.10 A schematic block diagram of a motion prediction model training device according to an embodiment of the present invention is shown;

[0047] Fig.11 A schematic block diagram of a motion prediction device according to an embodiment of the present invention is shown;

[0048] Fig.12 A schematic block diagram of a gait closed-loop stimulation output device according to an embodiment of the present invention is shown;

[0049] Fig.13 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more obvious, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

[0051] During the movement process, nerve signals in different tissue parts reflect the process of the brain sending commands to the muscles. For example, nerve signals in the motor cortex of the brain reflect the generation of movement intentions, sciatic nerve signals are important structures in the descending conduction nerves, and electromyographic signals also reflect the movement state of the muscles. However, the conventional external device detection method cannot fully display the complete transmission process of the nerve signals of movement control, and cannot accurately predict the movement state of the animal and give the corresponding nerve signal stimulation in time, which will slow down the recovery process of the animal's motor nerves.

[0052] In order to at least partially solve the above problems, a motion prediction model training method is proposed. The method obtains physiological data and image data collected simultaneously during the movement of an animal, and then marks the motion state of the animal in each frame of the image data, wherein the motion state includes a first motion state and a second motion state. After the physiological data collected during the movement is divided into multiple physiological data segments along the time axis, for each physiological data segment in at least part of the physiological data segments, the motion state of the animal in the physiological data segment is marked according to the motion state of the animal marked in the first image of the image data, wherein the first image is an image collected at a first acquisition time, and the first acquisition time is the acquisition time closest to the acquisition time corresponding to the physiological data segment after the acquisition time of the physiological data segment, and finally the marked physiological data segment is used as training sample data to train the motion prediction model to obtain a trained motion prediction model. The motion prediction model trained by the method can predict the motion state of the animal in real time according to the physiological data of the animal, and improves the accuracy and robustness of the prediction.

[0053] Figure 1 FIG. 2 shows a schematic flow chart of a motion prediction model training method according to an embodiment of the present invention. Figure 1 As shown, the motion prediction model training method may include steps S110 to S150.

[0054] In step S110, physiological data and image data collected simultaneously during the movement of the animal are obtained.

[0055] The animal may be any animal that can move normally, or may be a designated animal. For example, when the motion prediction model trained is expected to be used to predict the motion state of a patient, the animal may be a human. When the motion prediction model trained is expected to be used to predict the motion state of an experimental rat, the animal may be an experimental rat.

[0056] Physiological data can be determined based on the neural signals transmitted in the neural tissue to be detected. It is understandable that during the movement of an animal, the brain neural signals will be transmitted to the target tissue part used to support the movement, so as to stimulate the target tissue part to support the movement process of the animal. For example, for humans and experimental rats, the neural tissue to be detected can be brain neural tissue, leg muscle neural tissue, foot muscle neural tissue, etc.

[0057] Exemplarily, the physiological data includes sciatic nerve signal data, electromyographic signal data and / or brain motor cortex nerve signal data.

[0058] When a creature is exercising, nerve signals are usually sent from the motor cortex nerves of the brain and transmitted to the muscles used for exercise through the sciatic nerve tissue. Therefore, by collecting sciatic nerve signal data, electromyographic signal data and / or motor cortex nerve signal data, physiological data of nerve signal changes during exercise can be recorded. Physiological data can be represented by a waveform diagram of the corresponding nerve signal changes.

[0059] The image data may be a video or an image sequence composed of images with corresponding acquisition moments. The images in the image data may be RGB images, grayscale images, or images in other formats. These images may also be images of any suitable size and resolution. These images may be original images directly acquired by a photographic device or images after a preprocessing operation is performed on the original images. The preprocessing operation may include digitization, geometric transformation, normalization, filtering, and other operations on the original images to obtain image data for subsequent steps of the motion prediction model training method provided by the present invention.

[0060] In step S120, the motion state of the animal in each frame of the image data is marked, wherein the motion state includes a first motion state and a second motion state.

[0061] When animals are moving, their posture will change. The animal's movement state can be determined by analyzing the animal's posture in each frame of the image.

[0062] For example, for organisms that move with their legs, such as humans, experimental rats, etc., the movement state of the animal can be determined based on the movement posture of the legs and feet. For example, the movement state of the animal can be determined based on the degree of joint bending of the legs, feet, etc. For another example, the movement state of the animal can be determined based on the position changes of the legs, feet, etc. in the image.

[0063] Optionally, the neural network model may be directly used to analyze the image information in each frame of the image to determine the movement state of the animal in each frame of the image.

[0064] The motion state of the animal may include jumping, leg lifting, standing, squatting, etc. The first motion state may be a state representing dynamic motion among these states, such as jumping, leg lifting, etc. The second motion state may be a state representing static motion among these states, such as standing, squatting, etc.

[0065] Exemplarily, the first action state is a leg-lifting state, and the second action state is a standing state.

[0066] During normal movement of animals that use their legs to move, there are usually only two states: raising the legs and taking steps, and standing still. Therefore, when the first movement state is the leg raising state and the second movement state is the standing state, it is possible to avoid the movement state with little relevance to the movement process affecting the marking of the animal's movement state in each frame of the image.

[0067] In step S130, the physiological data collected during the exercise process is divided into a plurality of physiological data segments along the time axis.

[0068] The physiological data can be divided into a plurality of physiological data segments according to the collection time of the physiological data, and each physiological data segment includes at least one physiological data of the collection time. The collection time of the physiological data in each physiological data segment determines the duration of the physiological data segment. For example, each physiological data segment has physiological data of the same duration, which is conducive to reducing the computational complexity.

[0069] The frequency of collecting physiological data is usually higher than that of collecting image data. For example, the duration of each physiological data segment can also be determined according to the time interval between adjacent collection moments of physiological data, so that each physiological data segment can correspond to the image data collected at different collection moments. For example, when the sampling rate of the camera to capture image data is 100 Hz, 100 frames of images are captured in 1 second as image data. The sampling rate of physiological data is usually very high, for example, 1000 Hz, that is, 1000 samples are sampled in 1 second to obtain 1000 frames of physiological data. At this time, 10 frames of physiological data collected at continuous moments can be used as a physiological data segment, and the physiological data collected during the exercise process can be divided into multiple physiological data segments along the time axis.

[0070] Exemplarily, before the physiological data is segmented into a plurality of physiological data segments, the physiological data may be denoised and / or filtered.

[0071] Usually, the physiological data of each target tissue part can be obtained by using electrodes set at the target tissue part of the animal, and the target tissue part usually has multiple electrodes set to simultaneously collect the physiological data of the target tissue part using multiple channels. The physiological data can be denoised by using methods such as inter-channel common averaging method and multi-channel fusion method.

[0072] For example, for the physiological data of each target tissue site, the external noise of the physiological data can be removed by using the inter-channel co-average method and the following formula 1:

[0073]

[0074] in, is the signal of the ith channel, num ch is the number of channels, is the signal value of the ith channel calculated after common average reference.

[0075] Physiological data obtained from different target tissues can be filtered using filtering methods such as Chebyshev filtering method and Butterworth second-order filtering method. For example, electromyographic signals can be band-pass filtered at 30-250 Hz, sciatic nerve signals can be high-pass filtered at 0.5 Hz, and cerebral cortex nerve signals can be band-pass filtered at 300-6000 Hz.

[0076] For example, for the filtered physiological data segments collected by each channel, the threshold value can also be calculated using the RMS method and the following formula 2:

[0077]

[0078] Among them, x i represents the amplitude of the physiological data collected at the ith collection moment of the channel, n represents the length of the physiological data of the channel, and threshold represents the threshold. The length of the physiological data of the channel is the total number of moments corresponding to the collection moment.

[0079] After the threshold is calculated in this way, the amplitude of the physiological data segment that is less than the threshold can be modified to 1, and the amplitude of the physiological data segment that is smaller than the threshold can be modified to 0, so as to obtain the physiological data segment with updated amplitude after denoising and / or filtering.

[0080] In step S140, for each physiological data segment in at least part of the physiological data segments, the movement state of the animal in the physiological data segment is marked according to the movement state of the animal marked in the first image of the image data, wherein the first image is an image acquired at a first acquisition moment, and the first acquisition moment is an acquisition moment that is after the acquisition time of the physiological data segment and closest to the acquisition time corresponding to the physiological data segment.

[0081] At least part of the physiological data segments may be physiological data segments collected by using multiple channels at the same collection time, or may be physiological data segments collected by a single channel.

[0082] The acquisition time corresponding to each physiological data in the physiological data segment can be determined first, and the acquisition time of the physiological data in the physiological data segment can be determined. Then, starting from the last acquisition time, the acquisition time after the last acquisition time and closest to the last acquisition time can be determined from the acquisition time corresponding to each frame of the image data as the first acquisition time of the first image.

[0083] For each physiological data segment in at least part of the physiological data segments, the motion state of the animal marked in the first image corresponding to the physiological data segment reflects the motion state that the animal is expected to be in at the acquisition moment corresponding to the physiological data segment. By marking the motion state of the animal in the physiological data segment according to the motion state of the animal marked in the first image of the image data, a relationship between the motion state that the animal is expected to be in after the physiological data segment and the physiological data segment can be established. In this way, when the marked physiological data segments are used as training sample data, the ability of the motion prediction model to analyze the relationship can be improved, and the accuracy of the predicted motion state of the animal can be improved.

[0084] In step S150, the motion prediction model is trained using the marked physiological data segments as training sample data to obtain a trained motion prediction model.

[0085] The motion prediction model can be any neural network model for predicting motion state, such as ConvLSTM model, CNN-LSTM model, etc. For each physiological data segment in the partially labeled physiological data segment, the labeled physiological data segment can be input into the motion prediction model to re-predict the motion state of the animal in the physiological data segment, and then the loss is calculated based on the predicted motion state of the animal in the physiological data segment and the motion state of the animal in the labeled physiological data segment. The parameters of the motion prediction model are optimized according to the calculated loss until the calculated loss remains stable.

[0086] For example, the above steps can be used to obtain labeled physiological data segments other than training sample data as a validation set. The validation set is used to validate the model and adjust the hyperparameters of the motion prediction model. Finally, the model parameters with higher accuracy are selected as the model parameters of the trained motion prediction model.

[0087] In the above technical solution, the physiological data and image data collected simultaneously during the movement of the animal are obtained, and then the movement state of the animal in each frame of the image data is marked as the first state or the second state. After the physiological data collected during the movement is divided into multiple physiological data segments along the time axis, for each physiological data segment in at least part of the physiological data segments, the movement state of the animal in the physiological data segment is marked according to the movement state of the animal marked in the first image of the image data, wherein the first image is an image collected at the first acquisition time, and the first acquisition time is the acquisition time closest to the acquisition time corresponding to the physiological data segment after the acquisition time of the physiological data segment, and finally the marked physiological data segment is used as training sample data to train the motion prediction model to obtain the trained motion prediction model. The motion prediction model trained by this method can predict the movement state of the animal in real time according to the physiological data of the animal, and improves the accuracy and robustness of the prediction.

[0088] Figure 2 FIG. 4 shows a schematic flow chart of updating training sample data according to an embodiment of the present invention. Figure 2 As shown, the motion prediction model training method may further include steps S141 to S142.

[0089] In step S141, according to the movement states of the marked animals in the training sample data, the ratio of the sample data marked as the first movement state to the sample data marked as the second movement state in the training sample data is determined.

[0090] The ratio of the sample data marked as the first action state to the sample data marked as the second action state may be determined according to the amount of physiological data in the sample data marked as the first action state and the amount of physiological data in the sample data marked as the second action state.

[0091] For example, the sample data marked as the first action state can be used as the first sample data, and the sample data marked as the second action state can be used as the second sample data. Then, according to the total number of collection moments corresponding to the physiological data in the first sample data, the first number of physiological data in the first sample data is determined. According to the total number of collection moments corresponding to the physiological data in the second sample data, the second number of physiological data in the second sample data is determined. Then, the ratio of the first number to the second number is used as the ratio of the sample data marked as the first action state to the sample data marked as the second action state.

[0092] In step S142, according to the ratio, the sample data with a high proportion is sampled or the sample data with a low proportion is resampled to update the training sample data by balancing the sample data marked as different states.

[0093] Through sampling, some sample data can be screened out from the sample data with a high proportion to replace the original sample data with a high proportion. The number of the screened sample data is less than the original sample data. For example, during sampling, some sample data can be randomly selected to replace the original sample data with a high proportion, or some sample data can be selected at equal intervals to replace the original sample data with a high proportion according to the time interval and the order of the collection moments corresponding to the physiological data in the sample data with a high proportion, so as to reduce the proportion of such sample data. For example, when the sample data with a high proportion is sample data marked as the second action state and the sample data contains physiological data collected at 10 consecutive collection moments and accounts for 2 / 3, physiological data collected at 5 collection moments can be randomly selected from the sample data as new sample data marked as the second action state. Physiological data collected at the 2nd, 4th, 6th, 8th, 10th collection moments or the 1st, 3rd, 5th, 7th, 9th collection moments can also be selected at equal intervals as new sample data marked as the second action state.

[0094] By resampling, the physiological data segments of the sample data with a low proportion can be copied to obtain more sample data of this type. For example, when the physiological data in the physiological data segments of the sample data with a low proportion accounts for 1 / 3, the physiological data in the physiological data segments of the sample data with a low proportion can be repeatedly collected or copied, and the repeatedly collected or copied sample data can be added to the sample data of this type to increase the proportion of the sample data of this type.

[0095] The above sampling or resampling can make the number of sample data in the sample data marked as different states close, so as to update the training sample data by balancing the sample data marked as different states. Among them, the best balancing effect is that the number of physiological data in the sample data of different states is the same.

[0096] For example, when the first action state is the leg-raising state, the second action state is the standing state, the above-mentioned first sample data contains physiological data collected at 5 collection moments, and the above-mentioned second sample data contains physiological data collected at 50 collection moments, the ratio of sample data marked as the first action state to sample data marked as the second action state is 1:10. At this time, the sample data marked as the first action state is relatively small, and the sample data marked as the second action state accounts for a higher proportion. The second sample data can be sampled to select 1 / 10 of the physiological data as the new second sample data to replace the original second sample data, and the first sample data remains unchanged. Alternatively, the first sample data can be resampled, and the first sample data that has been repeatedly collected or copied 9 times is added to the original first sample data, and the second sample data remains unchanged. In this way, the proportion of the first sample data and the second sample data can be the same. Afterwards, the first sample data and the second sample data with the same proportion can be used as updated training sample data.

[0097] For example, a label smoothing method may be used to assign weights to the labeled motion states in the first sample data and the second sample data, and the inverse of the data volume ratio may also be used as the weight.

[0098] In the above technical solution, according to the movement state of the marked animal in the training sample data, the ratio of the sample data marked as the first movement state and the sample data marked as the second movement state in the training sample data is determined, and according to the ratio, the sample data with a high ratio is sampled or the sample data with a low ratio is resampled to update the training sample data by balancing the sample data marked as different states. The updated training sample data can make the trained motion prediction model have a good prediction effect for both the first motion state and the second motion state.

[0099] Figure 3 FIG. 4 is a schematic flow chart showing the motion state of an animal in each frame of image data according to an embodiment of the present invention. Figure 3 As shown, the above step S120 may include steps S310 to S320.

[0100] In step S310, the image classification model is used to classify the image data according to the movement state of the animal in each frame of the image data to obtain a classification result.

[0101] The image classification model can be any trained neural network model for image classification, such as a traditional machine learning model, a deep learning model, etc. The image data of the classification result can be predetermined as a training set. The image data is then input into the image classification model to output the classification result. Afterwards, the loss of the output classification result and the determined image data can be calculated, and the parameters of the image classification model can be optimized according to the loss to obtain the trained image classification model. The image classification model can determine the motion state of the animal in each frame of the image data as the classification result by analyzing the action posture of the animal in each frame of the image data. The classification result is a first motion state or a second motion state.

[0102] Optionally, the image data may be classified by the movement state of the animal in each frame of the image data using a semantic segmentation model.

[0103] In step S320, the movement state of the animal in each frame of the image is marked according to the classification result.

[0104] For example, only one frame of the image data can be input into the image classification model each time, and then the image classification model can be used to output the first action state or the second action state as the classification result of the motion state of the animal in the frame of the image. The motion state of the animal in the image can then be marked as the motion state corresponding to the classification result. The above operation is repeated for each frame of the image data, that is, the motion state of the animal in each frame of the image can be marked.

[0105] For example, only multiple frames of the image data can be input into the image classification model each time, and then the image classification model can be used to output the first action state or the second action state as the classification result of the motion state of the animal in each frame of the image according to the acquisition time corresponding to each frame of the image. Repeat the above operation for other images of the image data, that is, the motion state of the animal in each frame of the image can be marked.

[0106] In the above technical solution, the image data is classified by the motion state of the animal in each frame of the image data using the image classification model to obtain a classification result, and the motion state of the animal in each frame of the image is marked according to the classification result. Using the image processing model to mark the motion state of the animal in each frame of the image can reduce the complexity of calculation and improve the marking efficiency.

[0107] Figure 4 FIG. 1 is a schematic flow chart of marking the motion state of an animal in a physiological data segment according to the motion state of an animal marked in a first image of the image data according to an embodiment of the present invention. Figure 4 As shown, the above step S140 may include steps S410 to S430.

[0108] In step S410, the motion state period of the first acquisition moment is determined, the motion state period includes a first period and a second period, the motion state of the animal marked in the image collected in the first period is a first action state, and the motion state of the animal marked in the image collected in the second period is a second action state.

[0109] The processing operation of step S120 can be used to predetermine the motion state of the animal in each image, and then divide the acquisition time of the image marked with the first motion state into a first period, and divide the acquisition time of the image marked with the second motion state into a second period. After determining the first acquisition time, the first acquisition time can be matched in the first period and the second period to determine the motion state period of the first acquisition time, and then the motion state period of the first acquisition time can be determined.

[0110] In step S420, when the first collection moment is in the first time period, the movement state of the animal in the physiological data of the physiological data segment is marked as a first action state.

[0111] When the first acquisition moment is in the first time period, it means that the motion state of the animal marked in the first image is the first motion state, and the motion state of the animal in the physiological data of the physiological data segment can be marked as the first motion state.

[0112] In step S430, when the first collection time is in the second time period, the movement state of the animal in the physiological data of the physiological data segment is marked as the second action state.

[0113] When the first acquisition moment is in the second time period, it means that the motion state of the animal marked in the first image is the second motion state, and the motion state of the animal in the physiological data of the physiological data segment can be marked as the second motion state.

[0114] In the above technical solution, the motion state period of the first acquisition moment is determined, and the motion state period includes a first period and a second period. The motion state of the animal marked in the image collected in the first period is the first action state, and the motion state of the animal marked in the image collected in the second period is the second action state. According to the motion state of the animal corresponding to the motion state period of the first acquisition moment, the motion state of the animal in the physiological data of the physiological data segment is marked. In this way, there is no need to read the mark in the first image, and the motion state of the animal in the physiological data of the physiological data segment can be marked only according to the acquisition moment of the first image.

[0115] Figure 5 FIG. 4 is a schematic flow chart showing the motion state of an animal in each frame of image data according to an embodiment of the present invention. Figure 5As shown, the above step S120 may include steps S510 to S520.

[0116] In step S510, key point position information of the animal in each frame of the image data is obtained.

[0117] The key point position information of the animal includes the position information of the target tissue parts that can be used to identify the movement state of the animal during the movement process and support the animal to move. For example, the key point position information of the animal may include the position information of the target tissue parts such as the toes, legs, ankles, knees, buttocks and back of the animal in the image, and the key point position information of the animal may be the position information of only one of these target tissue parts in the image. The key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image correspond to the same target tissue part of the animal.

[0118] Exemplarily, the key point position information includes toe position information.

[0119] Understandably, when an animal is in the stepping stage during movement, it usually lifts the heel first and then the toe to move forward. However, actions such as lifting the heel and bending the leg do not necessarily mean that the animal is moving forward, but may also be changing the standing posture. Therefore, using the toe position information as the key point position information is conducive to more accurately distinguishing the animal's movement state.

[0120] For example, deep learning tools can be used to extract key point information in gait videos.

[0121] In step S520, for each frame of the image data, the motion state of the animal in the current frame is marked according to the key point position information of the animal in the current frame and the key point position information of the animal in the previous frame.

[0122] The key point position information of the animal in the current frame image may be compared with the key point position information of the animal in the previous frame image to determine the difference between the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image.

[0123] For example, the collection time of the current frame image can be set as the first moment, and the collection time of the previous frame image can be set as the second moment. When the difference is not 0, it means that the position of the target tissue part of the animal at the first moment is different from the position at the second moment, so the target tissue part of the animal is still moving at the first moment, and the movement state of the animal in the current frame image can be marked as the first action state. When the difference is not 0, it means that the position of the target tissue part of the animal at the first moment is the same as the position at the second moment, so the target tissue part of the animal is no longer moving at the first moment, and the movement state of the animal in the current frame image can be marked as the second action state.

[0124] In the above technical solution, the key point position information of the animal in each frame of the image data is obtained, and for each frame of the image data, the motion state of the animal in the current frame is marked according to the key point position information of the animal in the current frame and the key point position information of the animal in the previous frame. Based on the reliable key point position information, the motion state of the animal in the current frame can be marked more accurately.

[0125] Figure 6 FIG. 4 is a schematic flow chart of marking the motion state of an animal in a current frame image according to an embodiment of the present invention. Figure 6 As shown, the above step S520 may include steps S610 to S620.

[0126] In step S610, the displacement of the animal's toes from the acquisition time of the previous frame image to the acquisition time of the current frame image is determined based on the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image.

[0127] According to the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image, the first position coordinates of the key point of the animal in the current frame image and the second position coordinates of the key point of the animal in the previous frame image can be determined. According to the difference between the first position coordinates and the second position coordinates, the displacement of the toes of the animal from the acquisition time of the previous frame image to the acquisition time of the current frame image can be determined.

[0128] Optionally, the distance between the first position coordinate and the second position coordinate may be directly used as the displacement of the toe of the animal from the acquisition time of the previous frame image to the acquisition time of the current frame image.

[0129] Optionally, the horizontal displacement and the vertical displacement may be determined according to the abscissa distance between the first position coordinate and the second position coordinate, and the ordinate distance between the first position coordinate and the second position coordinate, respectively.

[0130] In step S620, the motion state of the animal in the current frame image is determined and marked according to the displacement.

[0131] Optionally, when the determined displacement is 0, the motion state of the animal in the current frame image may be determined and marked as the second state. When the determined displacement is not 0, the motion state of the animal in the current frame image may be determined and marked as the first state.

[0132] Optionally, considering the influence of the shooting environment when shooting the image data, a displacement threshold may be preset. When the determined displacement is less than the displacement threshold, the motion state of the animal in the current frame image may be determined and marked as the second state. When the determined displacement is greater than or equal to the displacement threshold, the motion state of the animal in the current frame image may be determined and marked as the first state.

[0133] Exemplarily, when the displacement includes horizontal displacement and vertical displacement, the motion state of the animal in the current frame image may be determined and marked according to the following formula 3:

[0134] Label i =[sign(abs(x i -x i-1 ))]&[sign(abs(y i -y i-1 ))](i=2,3,……,N) Formula 3

[0135] Among them, Label i Indicates the motion state of the animal in the current frame image; sign() indicates the sign function; abs() indicates the function of finding the absolute value; x i -x i-1 Represents the horizontal displacement of the animal's toes; y i -y i-1 represents the vertical displacement of the animal's toes; N is the total number of frames of the image data. The current frame image is the i-th frame image among the N frames of the image data.

[0136] For example, the first motion state may be a leg-lifting state, and the second motion state may be a standing state. i -x i-1 and i -y i-1 When both are not 0, the animal's movement state is usually in the leg-lifting state. According to formula 1, we can get Label i The value of is 1, and the animal's motion state in the current frame image can be marked as 1 or leg-lifting state. i -x i-1 and\ory i -y i-1 When it is 0, the animal's movement state is usually standing. According to formula 1, we can get the Label i The value of is 0, and the movement state of the animal in the current frame image can be marked as 0 or standing state.

[0137] In the above technical solution, the displacement of the animal's toes from the acquisition time of the previous frame image to the acquisition time of the current frame image is determined based on the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image, and then the motion state of the animal in the current frame image is determined and marked based on the displacement. The motion state of the animal in the current frame image can be determined more accurately based on the displacement, thereby improving the accuracy of marking.

[0138] Figure 7 FIG. 2 shows a schematic flow chart of a motion prediction method according to an embodiment of the present invention. Figure 7 As shown, the motion prediction method may include steps S710 to S720.

[0139] In step S710, the motion prediction model training method described above is used to obtain a trained motion prediction model.

[0140] Step S710 is similar to the steps of the above-mentioned motion prediction model training method and will not be described in detail here.

[0141] In step S710, the currently acquired physiological data segment is input into the trained motion prediction model to obtain a prediction result, which indicates the predicted motion state of the animal in the next time period at the current moment.

[0142] The currently obtained physiological data segment may be a physiological data segment obtained up to the current moment. The next time period at the current moment may only include the next moment after the currently obtained physiological data segment, or may include other moments including the next moment after the currently obtained physiological data segment. It can be understood that the relationship between the expected motion state of the animal after the physiological data segment and the physiological data segment has been established in the training data used to train the motion prediction model, so the trained motion prediction model can determine the relationship between the expected motion state of the animal after the currently obtained physiological data segment and the physiological data segment based on the input physiological data segment. Among them, because the end time of the currently obtained physiological data segment is the current moment, the next time period at the current moment is the time period after the currently obtained physiological data segment. Thus, the trained motion prediction model can predict the motion state of the animal in the next time period at the current moment. Among them, the motion state of the animal in the next time period at the current moment includes the first motion state or the second motion state.

[0143] Exemplarily, a deep learning model calling plug-in may be created in the Open Ephys software, and the plug-in may be used to call the trained motion prediction model to perform the above steps S710 to S720.

[0144] Figure 8A schematic diagram showing a motion prediction model outputting prediction results according to an embodiment of the present invention is shown.

[0145] like Figure 8 As shown, the motion prediction model can adopt the CNN-LSTM model. The physiological data segments of the currently acquired multiple channels can be convoluted and filtered using four different convolution kernels of different sizes: (1, 1), (4, 1), (16, 1) and (64, 1), where ks refers to the convolution kernel size. Then, a convolution kernel of size (32, 1) is used to perform depth convolution on the matrix in the channel direction, and a convolution kernel of size (1, 10) is used for pooling to obtain a 1*4 vector.

[0146] The vector is then input into the long short-term memory network to predict the motion state at the next moment. The long short-term memory network consists of an input gate, a forget gate, a cell state, and an output gate. For example, the long short-term memory network can determine the motion state h at the current moment according to the following formula 4-9: t :

[0147] i t =σ(W i [x t ,h t-1 ]+b i ) Formula 4

[0148] f t =σ(W f [x t ,h t-1 ]+b f ) Formula 5

[0149] c t ′=tanh(W c [x t ,h t-1 ]+b c ) Formula 6

[0150] c t =f t *c t-1 +i t *c t ′ Formula 7

[0151] o t =σ(W o [x t ,h t-1 ]+b o ) Formula 8

[0152] h t =o t *tanh(c t ) Formula 9

[0153] Among them, i t Represents the output result of the input gate, x t Indicates the currently acquired physiological data segment, h t-1 Indicates the stepping state at the previous moment, W i Represents the weight matrix of the input gate, plus the input b i represents the bias coefficient, and σ represents the sigmoid function. t represents the output of the forget gate, W f represents the forgetting matrix, b f represents the forgetting bias. t ′ represents the candidate value of the cell state to be updated, tanh represents the hyperbolic tangent function, W c and b c Represent the weights and biases related to cell state updates, respectively. t is the updated cell state, c t-1 is the cell state at the previous time step. t is the output of the output gate, W o and b o are the weight and bias of the output gate respectively.

[0154] After the long short-term memory network outputs the current motion state, the current motion state is input into the fully connected layer and reduced to a 2-dimensional vector to obtain the probability values ​​of two types of states (the first motion state and the second motion state). Then the softmax layer can determine the predicted state according to the following formula:

[0155]

[0156] Among them, p i is the i-th sample, pj is the j-th sample, p m is the value with the highest probability among the probability values ​​of the two types of states, p j -p m To avoid overflow of the result, n is the number of categories and is equal to 2. The classification probability is normalized by the logsoftmax function, and the sum of all sample probabilities is 1. The predicted state can be obtained by calculating the maximum probability state through the logsoftmax function. The predicted state can include the first movement state and the second movement state, namely the leg-raising state and the standing state.

[0157] Then, the distance between the predicted state and the actual mark is calculated according to the following formula, and the marks are weighted to determine the mark corresponding to the motion state at the next moment:

[0158] NLLLoss(p y , y) = -w y *log(p y )

[0159] Among them, p y is the predicted probability of the classification with label y, w y is the weight of the label y. For example, the label y may be a mark "1" indicating the first motion state (leg-lifting state) or a mark "0" indicating the second motion state (standing state).

[0160] Then, the motion state at the next moment can be determined based on the determined label.

[0161] In the above technical solution, the above motion prediction model training method is used to obtain a trained motion prediction model, and the currently obtained physiological data segment is input into the trained motion prediction model to obtain a prediction result, which indicates the predicted motion state of the animal in the next period at the current moment. In this way, the action that the animal is expected to perform at the next moment can be accurately determined, which is conducive to more accurately stimulating the animal with the stimulation signal.

[0162] Fig. 9 FIG. 4 is a schematic flow chart of a closed-loop gait stimulation output method according to an embodiment of the present invention. Fig. 9 As shown, the gait closed-loop stimulation output method may include steps S910 to S920.

[0163] In step S910, the above motion prediction method is used to obtain a prediction result.

[0164] Step S910 is similar to the steps of the above-mentioned motion prediction method and will not be described in detail here.

[0165] In step S920, when the prediction result indicates the first action state, a stimulation signal is output to the animal.

[0166] When the prediction result indicates the first action state, it indicates that the desired movement state of the animal is the leg-lifting state, that is, the animal wants to lift its legs. At this time, a stimulation signal can be output to the target tissue part of the animal to stimulate the target tissue part for leg-lifting to perform corresponding movements. For example, leg-lifting can be achieved by stimulating nerve tissues in target tissue parts such as toes, legs, ankles, knees, buttocks and back. The stimulation signal can be obtained based on the physiological data analysis when the movement state of the animal during the movement is the leg-lifting state.

[0167] For example, the animal's sciatic nerve signal, electromyographic signal or motor cortex electroencephalogram signal can be collected and input into the Open Ephys software to output the prediction result in real time. When the prediction result indicates the first action state, the stimulation device is used to synchronously output the stimulation. The time of the stimulation output is usually very short, such as 0.1ms.

[0168] In the above technical solution, the above motion prediction method is used to obtain a prediction result, and when the prediction result indicates the first action state, a stimulation signal is output to the animal. In this way, the stimulation signal can be used to accurately and timely stimulate the animal to perform the action of the first action state.

[0169] Exemplarily, taking rats as an example, A92 rats are implanted with sciatic nerve electrodes and motor cortex brain area electrodes, B36 rats are implanted with electromyographic electrodes, and B39 rats are implanted with sciatic nerve electrodes. According to the steps of the above-mentioned motion prediction model training method, the motion prediction model is trained. The motion states of A92 rats, B36 rats and B39 rats are predicted respectively using the trained motion prediction models. Among them, the prediction accuracy of A92 sciatic nerve signals is 95.4%, the prediction accuracy of motor cortex is 90.1%, the prediction accuracy of B36 electromyographic signals is 89.4%, and the prediction accuracy of B39 sciatic nerve signals is 90.3%. It can be seen from this that the motion prediction model trained by the above-mentioned motion prediction model training method can more accurately predict the motion state of animals.

[0170] Fig.10 FIG. 1 is a schematic block diagram of a motion prediction model training device 1000 according to an embodiment of the present invention. Fig.10 As shown, the motion prediction model training device 1000 includes an acquisition module 1010 , a first marking module 1020 , a segmentation module 1030 , a second marking module 1040 , and a training module 1050 .

[0171] The acquisition module 1010 is used to acquire physiological data and image data collected simultaneously during the movement of the animal;

[0172] The first marking module 1020 is used to mark the motion state of the animal in each frame of the image data, wherein the state includes a first motion state and a second motion state;

[0173] The segmentation module 1030 is used to segment the physiological data collected during the exercise into multiple physiological data segments along the time axis;

[0174] The second marking module 1040 is used to mark the motion state of the animal in the physiological data segment for each physiological data segment in at least part of the physiological data segments according to the motion state of the animal marked in the first image, wherein the first image is an image in the image data collected at a first collection time, and the first collection time is a collection time of the image data closest to the collection time corresponding to the physiological data segment;

[0175] The training module 1050 is used to train the motion prediction model using the marked physiological data segments as training sample data to obtain a trained motion prediction model.

[0176] The motion prediction model training device 1000 may further include a ratio calculation submodule and an equalization submodule.

[0177] The ratio calculation submodule is used to determine the ratio of sample data marked as the first action state and sample data marked as the second action state in the training sample data according to the movement state of the marked animal in the training sample data before training the motion prediction model using the marked physiological data segments as training sample data. The balancing submodule is used to sample the sample data with a high ratio or resample the sample data with a low ratio according to the ratio, so as to balance the sample data marked as different states to update the training sample data.

[0178] The first marking module 1020 may include a classification submodule and a motion state marking submodule. The classification submodule is used to classify the image data by the motion state of the animal in each frame of the image data using an image classification model to obtain a classification result. The motion state marking submodule is used to mark the motion state of the animal in each frame of the image according to the classification result.

[0179] The second marking module 1040 may include a first determination submodule, a first marking submodule, and a second marking submodule. The first determination submodule is used to determine the motion state period of the first acquisition moment, the motion state period includes a first period and a second period, the motion state of the animal marked in the image collected in the first period is a first action state, and the motion state of the animal marked in the image collected in the second period is a second action state. The first marking submodule is used to mark the motion state of the animal in the physiological data of the physiological data segment as the first action state when the first acquisition moment is in the first period. The second marking submodule is used to mark the motion state of the animal in the physiological data of the physiological data segment as the second action state when the first acquisition moment is in the second period.

[0180] The first marking module 1020 may include a position information acquisition submodule and a third marking submodule. The position information acquisition submodule is used to acquire key point position information of the animal in each frame of the image data. The third marking submodule is used to mark the motion state of the animal in the current frame of the image for each frame of the image data according to the key point position information of the animal in the current frame of the image and the key point position information of the animal in the previous frame of the image.

[0181] Exemplarily, the key point position information includes toe position information, and the third marking submodule may include a displacement determination submodule and a fourth marking submodule. The displacement determination submodule is used to determine the displacement of the animal's toes from the acquisition time of the previous frame image to the acquisition time of the current frame image based on the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image. The fourth marking submodule is used to determine and mark the movement state of the animal in the current frame image based on the displacement.

[0182] Exemplarily, the displacement includes horizontal displacement and vertical displacement; the fourth marking submodule includes a motion state calculation module. The motion state calculation module is used to determine and mark the motion state of the animal in the current frame image according to the following formula:

[0183] Label i =[sign(abs(x i -x i-1 ))]&[sign(abs(y i -y i-1 ))](i=2,3,……,N)

[0184] Among them, Label i Indicates the motion state of the animal in the current frame image; sign() indicates the sign function; abs() indicates the function of finding the absolute value; x i -x i-1 Represents the horizontal displacement of the animal's toes; y i -y i-1 represents the vertical displacement of the animal's toes; N is the total number of image frames in the image data.

[0185] Exemplarily, the motion prediction model training device 1000 may further include a data optimization module. The data optimization module is used to perform denoising and / or filtering on the physiological data before segmenting the physiological data into a plurality of physiological data segments.

[0186] Exemplarily, the physiological data includes sciatic nerve signal data, electromyographic signal data and / or brain motor cortex nerve signal data.

[0187] Exemplarily, the first action state is a leg-lifting state, and the second action state is a standing state.

[0188] Fig.11 FIG. 1 is a schematic block diagram of a motion prediction device 1100 according to an embodiment of the present invention. Fig.11 As shown, the motion prediction device 1100 includes a motion prediction model training device 1000 and a prediction module 1110 .

[0189] The motion prediction model training device 1000 is used to obtain a trained motion prediction model using the above-mentioned motion prediction model training method;

[0190] The prediction module 1110 is used to input the currently obtained physiological data segment into the trained motion prediction model to obtain a prediction result, which indicates the predicted motion state of the animal in the next time period at the current moment.

[0191] Fig.12 FIG. 1 shows a schematic block diagram of a gait closed-loop stimulation output device 1200 according to an embodiment of the present invention. Fig.12 As shown, the gait closed-loop stimulation output device 1200 includes a motion prediction device 1100 and a stimulation module 1210 .

[0192] The motion prediction device 1100 is used in the above-mentioned motion prediction method to obtain a prediction result;

[0193] The stimulation module 1210 is used to output a stimulation signal to the animal when the prediction result indicates the first action state.

[0194] According to another aspect of the present invention, an electronic device is provided. Fig.13 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Fig.13 As shown, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the motion prediction model training method, motion prediction method or gait closed-loop stimulation output method described above when the processor is running.

[0195] In addition, according to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method of the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned motion prediction model training device, motion prediction device or gait closed-loop stimulation output device according to the embodiment of the present invention. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0196] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions, which are used to execute the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method when running.

[0197] A person of ordinary skill in the art can understand the specific implementation and beneficial effects of the above-mentioned motion prediction model training method, motion prediction method or gait closed-loop stimulation output method by reading the above-mentioned specific description of the motion prediction model training method, motion prediction method or gait closed-loop stimulation output method. For the sake of brevity, they will not be repeated here.

[0198] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0199] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0201] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0202] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0203] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0204] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0205] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all functions of some modules in a motion prediction model training device, a motion prediction device, or a gait closed-loop stimulation output device according to an embodiment of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0206] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0207] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A motion prediction model training method, characterized in that: include: Acquire physiological data and image data collected simultaneously during the movement of animals; marking the motion state of the animal in each frame of the image data, wherein the motion state includes a first motion state and a second motion state; Segmenting the physiological data collected during the exercise into a plurality of physiological data segments along the time axis; For each physiological data segment in at least part of the physiological data segments, marking the motion state of the animal in the physiological data segment according to the motion state of the animal marked in a first image of the image data, wherein the first image is an image acquired at a first acquisition time, and the first acquisition time is an acquisition time that is after the acquisition time of the physiological data segment and closest to the acquisition time corresponding to the physiological data segment; The motion prediction model is trained using the marked physiological data segments as training sample data to obtain a trained motion prediction model.

2. The motion prediction model training method according to claim 1, characterized in that: Before using the marked physiological data segments as training sample data to train the motion prediction model, the method further includes: According to the movement states of the animal marked in the training sample data, determining the ratio of sample data marked as a first movement state to sample data marked as a second movement state in the training sample data; According to the ratio, sample data with a high proportion are sampled or sample data with a low proportion are resampled to update the training sample data by balancing the sample data marked as different states.

3. The motion prediction model training method according to claim 1, characterized in that: The marking of the movement state of the animal in each frame of the image data comprises: Using an image classification model, classifying the image data according to the motion state of the animal in each frame of the image data to obtain a classification result; According to the classification result, the movement state of the animal in each frame of image is marked.

4. The motion prediction model training method according to claim 1, characterized in that: The step of marking the movement state of the animal in the physiological data segment according to the movement state of the animal marked in the first image of the image data includes: Determine the motion state period at the first acquisition moment, the motion state period includes a first period and a second period, the motion state of the animal marked in the image acquired in the first period is the first action state, and the motion state of the animal marked in the image acquired in the second period is the second action state; When the first acquisition moment is in the first time period, marking the movement state of the animal in the physiological data of the physiological data segment as the first action state; When the first collection moment is in the second time period, the movement state of the animal in the physiological data of the physiological data segment is marked as the second action state.

5. The motion prediction model training method according to claim 1, characterized in that: The marking of the movement state of the animal in each frame of the image data comprises: Acquire key point position information of the animal in each frame of the image data; For each frame of the image data, the movement state of the animal in the current frame is marked according to the key point position information of the animal in the current frame and the key point position information of the animal in the previous frame.

6. The motion prediction model training method according to claim 5, characterized in that: The key point position information includes toe position information; The step of marking the motion state of the animal in the current frame image according to the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image includes: Determine the displacement of the toes of the animal from the acquisition time of the previous frame image to the acquisition time of the current frame image according to the key point position information of the animal in the current frame image and the key point position information of the animal in the previous frame image; According to the displacement, the movement state of the animal in the current frame image is determined and marked.

7. The motion prediction model training method according to claim 6, characterized in that: The displacement includes horizontal displacement and vertical displacement; Determining and marking the motion state of the animal in the current frame image according to the displacement includes: The motion state of the animal in the current frame image is determined and marked according to the following formula: Label i =[sign(abs(x i -x i-1 ))]&[sign(abs(y i -y i-1 ))] (i=2,3,……,N) Among them, Label i represents the motion state of the animal in the current frame image; sign() represents the sign function; abs() represents the function of obtaining the absolute value; x i -x i-1 represents the horizontal displacement of the toes of the animal; y i -y i-1 represents the vertical displacement of the toes of the animal; N is the total number of image frames in the image data.

8. The motion prediction model training method according to claim 1, characterized in that: Before segmenting the physiological data into a plurality of physiological data segments, the motion prediction model training method further includes: The physiological data is denoised and / or filtered.

9. The motion prediction model training method according to claim 1, characterized in that: The physiological data includes sciatic nerve signal data, electromyographic signal data and / or brain motor cortex nerve signal data.

10. The motion prediction model training method according to claim 1, characterized in that: The first action state is a leg-lifting state, and the second action state is a standing state.

11. A motion prediction method, characterized in that: include: Using the motion prediction model training method according to claims 1 to 10, obtaining the trained motion prediction model; The currently acquired physiological data segment is input into the trained motion prediction model to obtain a prediction result, wherein the prediction result indicates a prediction of the motion state of the animal in the next time period at the current moment.

12. A gait closed-loop stimulation output method, characterized in that: include: Using the motion prediction method according to claim 11, obtaining the prediction result; When the prediction result indicates the first action state, a stimulation signal is output to the animal.

13. A motion prediction model training device, characterized in that: include: An acquisition module, used to acquire physiological data and image data collected simultaneously during the movement of the animal; A first marking module, for marking the motion state of the animal in each frame of the image data, wherein the state includes a first motion state and a second motion state; A segmentation module, used for segmenting the physiological data collected during the exercise into a plurality of physiological data segments along a time axis; a second marking module, configured to mark, for each physiological data segment in at least part of the physiological data segments, the motion state of the animal in the physiological data segment according to the motion state of the animal marked in a first image, wherein the first image is an image in the image data acquired at a first acquisition time, and the first acquisition time is an acquisition time of the image data that is closest to the acquisition time corresponding to the physiological data segment; The training module is used to train the motion prediction model using the marked physiological data segments as training sample data to obtain a trained motion prediction model.

14. A motion prediction device, characterized in that: include: A motion prediction model training device, used to obtain the trained motion prediction model using the motion prediction model training method according to claims 1 to 10; The prediction module is used to input the currently obtained physiological data segment into the trained motion prediction model to obtain a prediction result, wherein the prediction result represents the prediction of the motion state of the animal in the next time period at the current moment.

15. A gait closed-loop stimulation output device, characterized in that: include: A motion prediction device, configured to obtain the prediction result by using the motion prediction method according to claim 11; A stimulation module is used to output a stimulation signal to the animal when the prediction result indicates the first action state.

16. An electronic device, comprising: A processor and a memory, characterized in that The memory stores computer program instructions, which are used by the processor to execute the motion prediction model training method according to any one of claims 1 to 10, the motion prediction method according to claim 11, or the gait closed-loop stimulation output method according to claim 12 when the processor is running.

17. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the motion prediction model training method according to any one of claims 1 to 10, the motion prediction method according to claim 11, or the gait closed-loop stimulation output method according to claim 12 when running.

18. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the motion prediction model training method according to any one of claims 1 to 10, the motion prediction method according to claim 11, or the gait closed-loop stimulation output method according to claim 12 when running.

Images