Mouse gait evaluation method based on deep learning and related equipment

Abstract

The invention discloses a mouse gait assessment method and related equipment based on deep learning, and the method comprises the steps: obtaining original video data, and carrying out the corresponding preprocessing, and obtaining to-be-processed data; detecting to-be-processed data according to the trained target detection model to obtain standard video stream data of three view angles, namely a look-up view angle, a left view angle and a right view angle; inputting the obtained standard video stream data into the trained deep learning model for feature extraction, and determining key point data of mouse movement in the video; performing data adjustment processing according to the key point data and preset parameters to obtain two-dimensional trajectory data under different viewing angles; and performing calculation according to the obtained two-dimensional trajectory data to obtain corresponding three-dimensional motion data, and performing index calculation according to the obtained three-dimensional motion data to determine an evaluation result of mouse motion. The embodiment of the invention can improve the evaluation accuracy. The method can be widely applied to the technical field of data processing.

Description

A deep learning-based method and related equipment for mouse gait assessment Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a deep learning-based method and related equipment for evaluating mouse gait. Background Technology

[0002] Mouse models of spinal cord injury are widely used to study nerve regeneration and functional recovery. By assessing the behavior of mice in an open environment, the severity of spinal cord injury can be evaluated, and the assessment results can provide a basis for relevant research and rehabilitation strategies. However, existing mouse behavioral assessments usually rely on the subjective judgment of observers, resulting in low accuracy. Summary of the Invention

[0003] The main objective of this application is to propose a deep learning-based method and related equipment for evaluating mouse gait, which can improve the accuracy of the evaluation.

[0004] To achieve the above objectives, one aspect of this application proposes a deep learning-based method for evaluating mouse gait. The method includes: acquiring raw video data; preprocessing the raw video data to determine data to be processed; inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data; wherein the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; inputting the standard video stream data into a trained deep learning model for processing to obtain keypoint data, and processing the keypoint data and preset parameters to obtain two-dimensional trajectory data; calculating based on the two-dimensional trajectory data to determine three-dimensional motion data; and calculating based on the three-dimensional motion data to obtain an evaluation result.

[0005] In some embodiments, the step of inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data specifically includes: performing frame-by-frame inference on the data to be processed according to the trained target detection model to determine detection data; wherein, the detection data includes instance bounding boxes and instance masks; performing consistency screening based on the detection data and a preset foreground mask to determine candidate instance data, and dividing the candidate instance data according to a preset three-band region to obtain three-band region data; performing identity tracking and short-term missed detection completion processing on the three-band region data to obtain target three-band region data, and performing smoothing calculation on the target three-band region data to obtain bounding box data; and cropping based on the bounding box data and a preset margin to obtain the standard video stream data.

[0006] In some embodiments, the step of inputting the standard video stream data into a trained deep learning model for processing to obtain keypoint data specifically includes: extracting features from the standard video stream data according to the trained deep learning model, determining branch feature maps, and performing feature decoding processing on the branch feature maps to obtain contour maps; performing forward computation on the contour maps to obtain multi-view output data; wherein, the multi-view output data includes keypoint heatmaps, sub-pixel offset data, and visibility confidence data; and calculating based on the multi-view output data to determine the keypoint data.

[0007] In some embodiments, the step of calculating and determining the key point data based on the multi-view output data specifically includes: parsing the multi-view output data to determine key point heatmap data, sub-pixel offset data, and visibility confidence scores for each viewpoint; analyzing the key point heatmap data to determine peak key points, and calculating and determining the sub-pixel coordinates of key points based on the peak key points and the sub-pixel offset data; matching the sub-pixel coordinates of key points with the visibility confidence scores to determine the key point confidence scores, and filtering the peak key points according to a preset confidence threshold to determine the key point data.

[0008] In some embodiments, the step of calculating two-dimensional trajectory data based on the key point data and preset parameters specifically includes: denoising the key point data to obtain standard key point data, and updating the trajectory based on the standard key point data to obtain first two-dimensional trajectory data; wherein, the first two-dimensional trajectory data includes two-dimensional trajectory data from three perspectives, namely, an upward view, a left view, and a right view; parsing the first two-dimensional trajectory data to determine the upward view key point data and the left and right view key point data; performing coordinate mapping and dimension unification on the upward view key point data according to the preset parameters to obtain standard upward view key point data, and performing mirror geometry constraints and dimension unification on the left and right view key point data to obtain standard left and right view key point data; updating the trajectory based on the standard upward view key point data and the standard left and right view key point data to obtain the two-dimensional trajectory data.

[0009] In some embodiments, according to the method of claim 1, the step of calculating and determining three-dimensional motion data based on the two-dimensional trajectory data specifically includes: performing linear triangulation processing on the two-dimensional trajectory data to obtain initial three-dimensional coordinate data; optimizing the initial three-dimensional coordinate data according to a preset optimization algorithm and a preset error threshold to obtain target three-dimensional coordinate data; and processing the target three-dimensional coordinate data and preset constraints to determine the three-dimensional motion data.

[0010] In some embodiments, the step of calculating the evaluation result based on the three-dimensional motion data specifically includes: parsing the three-dimensional motion data to determine the target three-dimensional coordinate data; calculating the walking index data based on the target three-dimensional coordinate data and a preset gait cycle; wherein the walking index data includes stride length, stride frequency, and stride height; calculating the hind limb joint angle data based on the target three-dimensional coordinate data and a first preset formula; calculating the hind limb phase difference data based on the target three-dimensional coordinate data, a second preset formula, and a preset gait cycle; and determining the evaluation result based on the walking index data, the hind limb joint angle data, and the hind limb phase difference data.

[0011] To achieve the above objectives, another aspect of this application proposes a mouse gait evaluation system based on deep learning. The system includes: an acquisition module for acquiring raw video data, preprocessing the raw video data, and determining data to be processed; a detection module for inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data; wherein the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; an extraction module for inputting the standard video stream data into a trained deep learning model for processing to obtain key point data, and processing the key point data and preset parameters to obtain two-dimensional trajectory data; and an evaluation module for calculating based on the two-dimensional trajectory data to determine three-dimensional motion data, and calculating based on the three-dimensional motion data to obtain an evaluation result.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a mouse gait assessment method, system, electronic device, storage medium, and program product based on deep learning. This solution acquires raw video data and performs corresponding preprocessing to obtain data to be processed; it then detects the data to be processed using a trained target detection model to obtain standard video stream data including three perspectives: upward, left, and right; the obtained standard video stream data is input into a trained deep learning model for feature extraction to determine key point data of mouse movement in the video; data adjustment processing is performed based on the key point data and preset parameters to obtain two-dimensional trajectory data from different perspectives; calculations are performed based on the obtained two-dimensional trajectory data to obtain corresponding three-dimensional motion data, and index calculations are performed based on the obtained three-dimensional motion data to determine the assessment result of mouse movement; by performing target detection processing on the raw video data, video data from different perspectives is generated; a deep learning model is used to extract features from the video data from different perspectives to obtain key point data; and two-dimensional motion trajectories and three-dimensional motion data are determined based on the key point data, achieving quantifiable assessment and improving assessment accuracy. Attached Figure Description

[0016] Figure 1 is a flowchart of a mouse gait evaluation method based on deep learning provided in an embodiment of this application; Figure 2 is a schematic diagram of a testing device in a mouse gait evaluation method based on deep learning provided in an embodiment of this application; Figure 3 is another schematic diagram of a testing device in a mouse gait evaluation method based on deep learning provided in an embodiment of this application; Figure 4 is a flowchart of step 102 in Figure 1; Figure 5 is a flowchart of step 103 in Figure 1; Figure 6 is a flowchart of step 503 in Figure 5; Figure 7 is another flowchart of step 103 in Figure 1; Figure 8 is a flowchart of step 104 in Figure 1; Figure 9 is another flowchart of step 104 in Figure 1; Figure 10 is a flowchart of a specific embodiment provided in an embodiment of this application; Figure 11 is a flowchart of an embodiment of this application. Figure 12 is a schematic diagram of the upward trajectory in the embodiment of this application; Figure 13 is a distribution diagram of the side trajectory in the embodiment of this application; Figure 14 is a diagram of the change in stride distance between the forelimbs and hindlimbs of the mouse in the embodiment of this application; Figure 15 is a diagram of the change in angle between two points of the hindlimb and the base of the tail in the embodiment of this application; Figure 16 is a curve of the change in angle of the mouse ankle joint in the embodiment of this application; Figure 17 is a schematic diagram of the structure of a mouse gait evaluation system based on deep learning provided in the embodiment of this application; Figure 18 is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of this application; wherein, 1 is an acrylic track, 2 is a constant illuminance LED light strip, 3 is a low-iron ultra-white glass plane mirror, 4 is an acrylic darkroom, 5 is a lens fixing buckle, 6 is an organic glass experimental platform, 7 is a high-speed camera, and 8 is a shockproof bracket. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0022] Figure 1 is an optional flowchart of a deep learning-based mouse gait assessment method provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S101 to S104.

[0023] Step S101: Acquire raw video data, preprocess the raw video data, and determine the data to be processed; Step S102: Input the data to be processed into a trained object detection model for detection to obtain standard video stream data; wherein, the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; Step S103: Input the standard video stream data into a trained deep learning model for processing to obtain key point data, and process it according to the key point data and preset parameters to obtain two-dimensional trajectory data; Step S104: Calculate based on the two-dimensional trajectory data to determine three-dimensional motion data; and calculate based on the three-dimensional motion data to obtain the evaluation result.

[0024] Steps S101 to S104, as illustrated in this embodiment, involve acquiring raw video data of the mice to be evaluated using the testing device shown in Figures 2 and 3. This testing device uses a transparent track to constrain the mouse's movement, and a strong light source is placed around the transparent track to induce the mouse to walk autonomously. 45-degree plane mirrors are placed on both sides of the track to refract the movement images from both sides of the mouse. A high-speed camera is placed below the track to acquire video data, resulting in raw video data with three simultaneous perspectives. The acquired raw video data is preprocessed to remove reflections caused by the strong light source reflected from the transparent track and background interference caused by the strong light source. To improve the accuracy of subsequent evaluations, the preprocessed video data is then input into a trained target detection model for further processing. Frame-by-frame inference is performed on the three perspectives within the video data to detect mice. The original data is then divided into three video streams with different perspectives, serving as standard video streams. These standard video streams are then input into a trained deep learning model for feature extraction, yielding keypoint data that reflects the mouse's movement posture. Based on this keypoint data, calculations are performed to obtain the mouse's two-dimensional trajectory and three-dimensional motion data from different perspectives. Finally, the three-dimensional motion data is analyzed to obtain the corresponding evaluation results.

[0025] Please refer to Figure 4. In some embodiments, step S102 may include, but is not limited to, steps S401 to S404: Step S401: Perform frame-by-frame inference on the data to be processed according to the trained target detection model to determine the detection data; wherein, the detection data includes instance bounding boxes and instance masks; Step S402: Perform consistency screening based on the detection data and the preset foreground mask to determine candidate instance data, and divide the candidate instance data according to the preset three-band region to obtain three-band region data; Step S403: Perform identity tracking and short-term missed detection completion processing on the three-band region data to obtain target three-band region data, and perform smoothing calculation on the target three-band region data to obtain bounding box data; Step S404: Perform cropping based on the bounding box data and the preset margin to obtain standard video stream data.

[0026] In step S401 of some embodiments, the preprocessed video data is input into the trained object detection model for frame-by-frame inference to identify mouse instances from each viewpoint in each frame of video, and corresponding bounding boxes and instance masks are generated based on the identified mouse instances. In this embodiment, the YOLOv11 model is used as the object detection model to process the preprocessed video stream data. The object detection model obtains a three-view joint annotation set by annotating images of mice from different viewpoints. Then, the YOLOv11 is trained with the annotation set to obtain the initial weights. "Pseudo-labels" are automatically generated on the entire database of videos. Frames with low confidence or abnormal shapes are manually annotated, and the model is retrained again until the detection is stable. The trained model is retained as the object detection model, and the object detection model outputs at most three mouse instances.

[0027] In step S402 of some embodiments, the instance masks of each view output by the target detection model and the set foreground mask are screened for IoU consistency, and mouse instances with sufficient overlap are retained as candidate instances; then, the screened candidate instances are divided according to the vertical coordinate of the instance rectangle output by the target detection model, and the candidate instances of each view are divided into the set three-zone region to obtain the corresponding three-zone region data, so as to manage the position of the mouse by zone.

[0028] In step S403 of some embodiments, IoU processing is performed based on the center distance of the instance bounding boxes of mouse instances in each adjacent video frame to track the identity of mouse instances between different video frames, so as to ensure the continuity of mouse movement in the video stream; at the same time, when no mouse instance is detected between consecutive frames, the corresponding mouse instance is obtained as a supplement by nearest neighbor matching with the current foreground connected component in the same band in the previous video frame, so as to ensure the continuity of mouse movement; then, after identity tracking and instance supplementation, trajectory calculation exponential moving average bounding box is performed on each band of data in the three bands, which is used to subsequently divide video data from different perspectives.

[0029] In step S404 of some embodiments, after obtaining the bounding box data, regions of interest (ROI) for three perspectives are generated based on the video data, and each perspective in the same video frame is cropped based on the ROI to obtain independent video stream data for each perspective. Since it is obtained by cropping from the same video frame, the timing of the video stream data for each perspective is strictly aligned, while retaining the original video frame number and the corresponding timestamp.

[0030] Please refer to Figure 5. In some embodiments, step S103 may include, but is not limited to, steps S501 to S503: Step S501: Extract features from standard video stream data according to the trained deep learning model, determine branch feature maps, and perform feature decoding on the branch feature maps to obtain contour maps; Step S502: Perform forward calculation on the contour maps to obtain multi-view output data; wherein, the multi-view output data includes key point heatmaps, sub-pixel offset data, and visibility confidence data; Step S503: Calculate and determine key point data based on the multi-view output data.

[0031] In step S501 of some embodiments, after cropping the video data from each viewpoint, high-precision denoising is performed on the video data from each viewpoint to improve the accuracy of subsequent evaluation. In this embodiment, fastNlMeansDenoisingMulti and fastNlMeansDenoisingColored are used for spatiotemporal joint denoising to reduce the graininess caused by high-speed shooting. Flat-field correction and mild color equalization are performed to improve the contrast of structures such as mouse paws and hair. Barrel and lens distortion correction is completed by using calibrated internally involved distortion coefficients through cv.undistort. Temporal registration super-resolution and blind deblurring (motion kernel estimation + Non-blind deconvolution is used to enhance the sharpness of the foot and joint edges; then, the denoised data from each viewpoint is input into a trained deep learning model for feature extraction; in this embodiment, the trained DeepLabCut model is used to extract key points from the three-view video data on the ROI with three-view synchronization; in this embodiment, the model uses professional personnel to obtain training samples from different movement postures, different time stages, and multiple posture angles of the mouse, and performs corresponding key point annotations to train the constructed model and obtain a deep learning model; the model extracts the geometric features of the binary contour of the mouse in the three-view video data through multi-layer convolution and residual connections, and outputs branch feature maps of each viewpoint; then, feature decoding is performed on the branch feature maps of each viewpoint to obtain the corresponding contour map data.

[0032] In step S502 of some embodiments, the deep learning model then performs forward computation on the obtained contour map data, parses the branch feature map of each viewpoint, and obtains key point heatmap data, sub-pixel offset data and visibility confidence for each viewpoint, which are used for subsequent screening of key points that can characterize mouse movement.

[0033] In step S503 of some embodiments, calculations and analyses are performed based on the obtained key point heatmap data, subpixel offset data and visibility confidence for each viewpoint, and typical key points that can characterize the mouse's movement posture are selected from multiple key point data output by the deep learning model, such as key points at the mouse's limb ends and key points at limb joints.

[0034] Please refer to Figure 6. In some embodiments, step S503 may include, but is not limited to, steps S601 to S603: Step S601: parse the multi-view output data to determine the key point heatmap data, sub-pixel offset data, and visibility confidence of each view; Step S602: analyze the key point heatmap data to determine the peak key points, and calculate the sub-pixel coordinates of the key points based on the peak key points and the sub-pixel offset data; Step S603: match the sub-pixel coordinates of the key points with the visibility confidence to determine the key point confidence, and filter the peak key points according to a preset confidence threshold to determine the key point data.

[0035] In step S601 of some embodiments, the feature data of each perspective extracted by the deep learning model are analyzed to obtain key point heatmap data, sub-pixel offset data and visibility confidence of each perspective, so as to screen typical key point data that can characterize mouse movement.

[0036] In step S602 of some embodiments, the key point heatmap data from each viewpoint are analyzed to extract the peak data; then, the extracted peak data is matched and analyzed with the sub-pixel offset data to determine the sub-pixel coordinates of the key points.

[0037] In step S603 of some embodiments, the confidence level of each key point is determined by matching the obtained sub-pixel coordinates of the key points with the visibility confidence level; then, the confidence level of each key point is filtered according to the set confidence level threshold, and key points with a confidence level greater than or equal to the confidence level threshold are retained as key point data for subsequent evaluation.

[0038] Please refer to Figure 7. In some embodiments, step S103 may include, but is not limited to, steps S701 to S704: Step S701: Denoise the key point data to obtain standard key point data, and update the trajectory based on the standard key point data to obtain first two-dimensional trajectory data; wherein, the first two-dimensional trajectory data includes two-dimensional trajectory data from three perspectives, namely, an upward view, a left view, and a right view; Step S702: Analyze the first two-dimensional trajectory data to determine the key point data from the upward view and the key point data from the left and right views; Step S703: Perform coordinate mapping and dimension unification on the key point data from the upward view according to preset parameters to obtain standard upward key point data, and perform mirror geometry constraints and dimension unification on the key point data from the left and right views to obtain standard left and right view key point data; Step S704: Update the trajectory based on the standard upward key point data and the standard left and right view key point data to obtain two-dimensional trajectory data.

[0039] In step S701 of some embodiments, the obtained key point data is denoised to reduce the impact of mouse movement, testing device or camera shake, and standard key point data is obtained. In this embodiment, by performing translation-invariant wavelet multi-scale decomposition on the time series of key point data from each viewpoint, noise is estimated at a fine scale and an adaptive soft threshold is applied to the detail coefficients, followed by inverse transformation reconstruction, so that key events are not affected while suppressing high-frequency jitter, and standard key point data is obtained. Then, the trajectory is updated according to the standard key point data to generate a continuous and stable two-dimensional trajectory. In this embodiment, the trajectory is updated with the limb bone length approximately constant as a soft constraint to obtain the two-dimensional trajectory of each viewpoint.

[0040] In step S702 of some embodiments, in order to facilitate subsequent quantifiable evaluation, the dimensions of the two-dimensional trajectories from different perspectives are unified; since the testing device sets up a mirror to reflect the motion images of the mouse on the left and right sides, the obtained two-dimensional trajectories from each perspective are analyzed to obtain the two-dimensional trajectories on the left and right sides and the two-dimensional trajectories from the upward view, and the corresponding key point data are extracted from the two-dimensional trajectory data.

[0041] In step S703 of some embodiments, the pixel coordinates corresponding to the key point data of the upward view are mapped according to the intrinsic parameters, distortion coefficients, mirror equivalent extrinsic parameters and homography matrix of the bottom view plane calibrated in the testing device, and uniformly mapped to the runway world plane. The coordinates are then dimensionally unified according to the known scale to obtain standard upward view key point data. The key point data of the left and right views are constrained to the same world plane according to mirror geometry for contact event verification and anomaly removal, resulting in key point data with consistent dimensions on the left and right sides.

[0042] In step S704 of some embodiments, the two-dimensional trajectory is updated based on the standard upward view key point data after unification of dimensions and the key point data with consistent dimensions on the left and right sides, so as to obtain the two-dimensional trajectory data for subsequent three-dimensional reconstruction.

[0043] Please refer to Figure 8. In some embodiments, step S104 may include, but is not limited to, steps S801 to S804: Step S801, performing linear triangulation processing on the two-dimensional trajectory data to obtain initial three-dimensional coordinate data; Step S802, optimizing the initial three-dimensional coordinate data according to a preset optimization algorithm and a preset error threshold to obtain target three-dimensional coordinate data; Step S803, processing the target three-dimensional coordinate data and preset constraints to determine three-dimensional motion data.

[0044] In step S801 of some embodiments, the obtained two-dimensional trajectory data is linearly triangulated based on the "real camera + two-mirror virtual camera" model of the test device. The key point data in the two-dimensional trajectory from three perspectives are arranged and linearly triangulated to obtain the initial value of the three-dimensional coordinates.

[0045] In step S802 of some embodiments, the obtained initial three-dimensional coordinate values ​​are optimized using global Bundle Adjustment (BA) and external parameter disturbances such as the test device are optimized to minimize the total reprojection error between the calculated three-dimensional coordinates and the actual measured three-dimensional coordinates. The Huber / Charbonnier robust loss and RANSAC elimination of error key points generated during the optimization process are then processed to obtain the target three-dimensional coordinate data.

[0046] In step S803 of some embodiments, three-dimensional reconstruction is performed based on the obtained target three-dimensional coordinate data to obtain three-dimensional motion data of the mouse in the testing device; in this embodiment, bone length constraints and smoothing priors are applied in the 3D domain to obtain continuous three-dimensional motion data of the mouse in the testing device.

[0047] Please refer to Figure 9. In some embodiments, step S104 may include, but is not limited to, steps S901 to S904: Step S901: parse the three-dimensional running data to determine the target three-dimensional coordinate data; Step S902: calculate and determine the walking index data based on the target three-dimensional coordinate data and a preset gait cycle; wherein, the walking index data includes stride length, stride frequency, and stride height; Step S903: calculate and determine the hind limb joint angle data based on the target three-dimensional coordinate data and a first preset formula; Step S904: calculate and determine the hind limb phase difference data based on the target three-dimensional coordinate data, a second preset formula, and a preset gait cycle, and determine the evaluation result based on the walking index data, hind limb joint angle data, and hind limb phase difference data.

[0048] In step S901 of some embodiments, after obtaining the three-dimensional motion data, calculations are performed based on relevant indicators of mouse gait assessment; by parsing the three-dimensional motion data, the three-dimensional coordinate data is extracted to calculate various indicators.

[0049] In step S902 of some embodiments, motion data during mouse movement is calculated based on three-dimensional coordinate data, including stride length, stride frequency, and stride height. Stride length represents the directional displacement of the same hind foot along the forward direction within a gait cycle, reflecting propulsion ability and stride length adjustment, and is sensitive to changes in hind limb weight-bearing and propulsion function in SCI. The stride length data of the mouse movement within a gait cycle is calculated according to the following formula: ,in, Step size, for The three-dimensional coordinates at time [time]. for The three-dimensional coordinates at time t, where T is the transpose matrix. The unit vector represents the direction of the mouse's movement; gait frequency refers to the reciprocal of the gait cycle defined by two consecutive foot strikes, representing the gait rhythm and central rhythm output. In this embodiment, gait frequency data is calculated according to the following formula: ,in, For step frequency, For gait period, , The step height refers to the moment of two consecutive foot strikes; step height is the increment of the highest foot height relative to the initial foot strike height within a gait cycle, reflecting limb joint strength and dorsiflexion control; in this embodiment, step height is calculated according to the following formula: ,in, For BBK, for The height of the foot at that moment, for The height of the foot at that moment.

[0050] In step S903 of some embodiments, spinal cord injury in mice affects the coordination and mobility of hind limb force exertion. Therefore, the flexion-extension range of motion and coordinated control of the hind limbs can be assessed by calculating the hind limb joint angles. In this embodiment, the hind limb joint angles are calculated using the following formula:

[0051] in, Let f be the angle of the hind limb joint, and let f, k, and a be the coordinates of the knee, ankle, and foot of the hind limb, respectively.

[0052] In step S904 of some embodiments, spinal cord injury in mice can affect not only the movement of the hind limb on the same side, but also the movement incoordination of both hind limbs. In this embodiment, the coordination of hind limb movement is assessed by calculating the phase difference between the left and right hind limbs. This phase difference is calculated using the following formula:

[0053] in, For phase difference, The moment the right hind foot landed. The moment of landing of the left hind foot. The period is the duration; the mouse gait is evaluated by integrating the obtained walking index data, the hind limb joint angle data, and the hind limb phase difference data.

[0054] The following is a detailed description and explanation of the embodiments of the present invention, using specific application examples: In some embodiments, a device for testing mice with spinal cord injuries is constructed and tested, as shown in Figure 10. A 50*10*6 transparent acrylic track is horizontally installed on a glass experimental table. The track contains a sliding drawer-type cage with quick-release latches, allowing the experimenter to release and slide the cage out with one hand at any time for safe and quick mouse retrieval. Magnetic locking doors are installed at both ends of the track to prevent escape. Constant illuminance LED light strips (color temperature 5600 K, luminous flux ≥1000 lm) are installed around the track. The light source is driven by a constant current controller, providing uniform and flicker-free backlighting. Two 45° inclined reflectors are located on both sides of the track and are bolted to the track side plates via mirror mounts, reflecting the left and right side views of the mice to the area below the track. The DJI Action 5 camera (4.6K resolution, 100 fps, 1 / 3840 s shutter speed, fixed focal length low distortion lens) located at the center of the bottom of the track is mounted on a shockproof bracket and secured to the bottom groove of the track with vibration damping pads. A square lens hood and polarizer are used to suppress specular highlights and stray light. The camera and mirrors can jointly form a three-view composite image. Since they originate from the same camera and have a consistent time base, there is no need to calibrate the timing. 30-60 frames of Charuco / checkerboard images are collected within the camera's field of view. The intrinsic parameters of Action 5, distortion parameters, and equivalent extrinsic parameters of the two mirrors are jointly solved to establish a "real camera + two mirror virtual camera" model. After nonlinear optimization, the average reprojection error is ≤0.3-0.5 px. The intrinsic and extrinsic parameters, mirror normal, temperature and humidity, and equipment serial number are determined. At the same time, to enhance batch consistency, red fluorescent markers with a diameter of 2 mm are pasted around the track, along the edge of the mirrors, and along the edge of the base plate. The markers can still be stably recorded by the camera in white light environment and serve as a fixed calibration benchmark within the scene.The entire recording then proceeded to 4.6K master video at 60–100 fps, simultaneously recording intrinsic and extrinsic parameters, specular normals, exposure, and shooting metadata. A particular video frame included a three-view image, as shown in Figure 11. The captured video data was processed using a three-Gaussian mixture model to suppress reflections and background interference, then input into a YOLOv11 model for further processing, outputting three synchronized video streams. High-precision denoising and distortion correction were then applied to the three video streams to improve detail clarity. The obtained Sanlu data was then input into a pre-trained DLC model, which was trained using manually labeled training data to learn the geometric features of mouse videos. Output the corresponding key point data; then, perform calculations based on the key point data to obtain two-dimensional trajectory data from each perspective, such as the upward trajectory data shown in Figure 12 and the side trajectory distribution map shown in Figure 13; perform dimension unification on the obtained two-dimensional trajectory data, then perform linear triangulation and BA optimization to obtain three-dimensional motion data; calculate the evaluation index based on the three-dimensional motion data, and draw corresponding charts for visualization, such as the mouse forelimb and hindlimb stride change diagram shown in Figure 14, the angle change diagram between the two points of the hindlimb and the tail base shown in Figure 15, and the mouse ankle joint angle change curve shown in Figure 16.

[0055] The embodiments of this application include at least the following beneficial effects: This application provides a mouse gait assessment method, system, electronic device, storage medium, and program product based on deep learning. This solution acquires raw video data and performs corresponding preprocessing to obtain data to be processed; it then detects the data to be processed using a trained target detection model to obtain standard video stream data including three perspectives: upward, left, and right; the obtained standard video stream data is input into a trained deep learning model for feature extraction to determine key point data of mouse movement in the video; data adjustment processing is performed based on the key point data and preset parameters to obtain two-dimensional trajectory data from different perspectives; calculations are performed based on the obtained two-dimensional trajectory data to obtain corresponding three-dimensional motion data, and index calculations are performed based on the obtained three-dimensional motion data to determine the assessment result of mouse movement; by performing target detection processing on the raw video data, video data from different perspectives is generated; a deep learning model is used to extract features from the video data from different perspectives to obtain key point data; and two-dimensional motion trajectories and three-dimensional motion data are determined based on the key point data, achieving quantifiable assessment and improving assessment accuracy.

[0056] Please refer to Figure 17. This application embodiment also provides a mouse gait evaluation system based on deep learning, which can implement the above method. The system includes: an acquisition module for acquiring raw video data, preprocessing the raw video data, and determining data to be processed; a detection module for inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data; wherein, the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; an extraction module for inputting the standard video stream data into a trained deep learning model for processing to obtain key point data, and processing it according to the key point data and preset parameters to obtain two-dimensional trajectory data; and an evaluation module for calculating based on the two-dimensional trajectory data to determine three-dimensional motion data; and calculating based on the three-dimensional motion data to obtain an evaluation result.

[0057] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0059] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0060] Please refer to Figure 18, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 1801, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 1802, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 1802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1802 and is called and executed by the processor 1801. The input / output interface 1803 is used to implement information input and output. The communication interface 1804 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 1805 transmits information between various components of the device (such as processor 1801, memory 1802, input / output interface 1803 and communication interface 1804). The processor 1801, memory 1802, input / output interface 1803 and communication interface 1804 are connected to each other within the device through the bus 1805.

[0061] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0062] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0063] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0064] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0065] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0067] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0068] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0070] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0071] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0073] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A deep learning-based method for evaluating mouse gait, characterized in that, The method includes: acquiring raw video data; preprocessing the raw video data to determine data to be processed; inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data; wherein, the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; inputting the standard video stream data into a trained deep learning model for processing to obtain key point data, and processing the key point data and preset parameters to obtain two-dimensional trajectory data; calculating based on the two-dimensional trajectory data to determine three-dimensional motion data; and calculating based on the three-dimensional motion data to obtain an evaluation result.

2. The method according to claim 1, characterized in that, The step of inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data specifically includes: performing frame-by-frame inference on the data to be processed based on the trained target detection model to determine detection data; wherein, the detection data includes instance bounding boxes and instance masks; performing consistency screening based on the detection data and a preset foreground mask to determine candidate instance data, and dividing the candidate instance data according to a preset three-band region to obtain three-band region data; performing identity tracking and short-term missed detection completion processing on the three-band region data to obtain target three-band region data, and performing smoothing calculation on the target three-band region data to obtain bounding box data; and cropping based on the bounding box data and a preset margin to obtain the standard video stream data.

3. The method according to claim 1, characterized in that, The step of inputting the standard video stream data into a trained deep learning model for processing to obtain keypoint data specifically includes: extracting features from the standard video stream data using the trained deep learning model to determine branch feature maps, and performing feature decoding on the branch feature maps to obtain contour maps; performing forward computation on the contour maps to obtain multi-view output data; wherein, the multi-view output data includes keypoint heatmaps, sub-pixel offset data, and visibility confidence data; and calculating and determining the keypoint data based on the multi-view output data.

4. The method according to claim 3, characterized in that, The step of calculating and determining the key point data based on the multi-view output data specifically includes: parsing the multi-view output data to determine the key point heatmap data, sub-pixel offset data, and visibility confidence scores for each viewpoint; analyzing the key point heatmap data to determine peak key points, and calculating and determining the sub-pixel coordinates of key points based on the peak key points and the sub-pixel offset data; matching the sub-pixel coordinates of key points with the visibility confidence scores to determine the key point confidence scores, and filtering the peak key points according to a preset confidence threshold to determine the key point data.

5. The method according to claim 1, characterized in that, The step of calculating two-dimensional trajectory data based on the key point data and preset parameters specifically includes: denoising the key point data to obtain standard key point data, and updating the trajectory based on the standard key point data to obtain first two-dimensional trajectory data; wherein, the first two-dimensional trajectory data includes two-dimensional trajectory data from three perspectives, namely, an upward view, a left view, and a right view; parsing the first two-dimensional trajectory data to determine the upward view key point data and the left and right view key point data; performing coordinate mapping and dimension unification on the upward view key point data according to the preset parameters to obtain standard upward view key point data, and performing mirror geometry constraints and dimension unification on the left and right view key point data to obtain standard left and right view key point data; updating the trajectory based on the standard upward view key point data and the standard left and right view key point data to obtain the two-dimensional trajectory data.

6. The method according to claim 1, characterized in that, The step of calculating and determining the three-dimensional motion data based on the two-dimensional trajectory data specifically includes: performing linear triangulation processing on the two-dimensional trajectory data to obtain initial three-dimensional coordinate data; optimizing the initial three-dimensional coordinate data according to a preset optimization algorithm and a preset error threshold to obtain target three-dimensional coordinate data; and processing the target three-dimensional coordinate data and preset constraints to determine the three-dimensional motion data.

7. The method according to claim 1, characterized in that, The step of calculating the evaluation result based on the three-dimensional motion data specifically includes: parsing the three-dimensional motion data to determine the target three-dimensional coordinate data; calculating and determining walking index data based on the target three-dimensional coordinate data and a preset gait cycle; wherein the walking index data includes stride length, stride frequency, and stride height; calculating and determining hind limb joint angle data based on the target three-dimensional coordinate data and a first preset formula; calculating and determining hind limb phase difference data based on the target three-dimensional coordinate data, a second preset formula, and a preset gait cycle; and determining the evaluation result based on the walking index data, the hind limb joint angle data, and the hind limb phase difference data.

8. A mouse gait assessment system based on deep learning, characterized in that, The system includes: an acquisition module for acquiring raw video data, preprocessing the raw video data, and determining data to be processed; a detection module for inputting the data to be processed into a trained target detection model for detection to obtain standard video stream data; wherein the standard video stream data includes left-view video stream, right-view video stream, and upward-view video stream; an extraction module for inputting the standard video stream data into a trained deep learning model for processing to obtain key point data, and processing the key point data and preset parameters to obtain two-dimensional trajectory data; and an evaluation module for calculating based on the two-dimensional trajectory data to determine three-dimensional motion data, and calculating based on the three-dimensional motion data to obtain an evaluation result.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; when said at least one program is executed by said at least one processor, such that said at least one processor implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images