Kinematics index determination method and device based on autonomous navigation and related equipment

Abstract

The invention provides a kinematics index determination method and device based on autonomous navigation and related equipment, and the method comprises the steps: moving to a target position in a preset area based on a target path, the target path being a path obtained by re-planning a first path according to the obtained environment information of the preset area, the first path is a path generated according to the current position of the robot and the target position; an image data set of a target object is obtained, the image data set comprises a plurality of image frames arranged according to a first time sequence, and each image frame comprises the target object; generating three-dimensional attitude model data according to the image data set, wherein the three-dimensional attitude model data is determined based on a basic model of the target object and an evolution trajectory of the basic model along with time change; and determining a corresponding kinematics index of the target part of the target object in the three-dimensional attitude model data. And the accuracy of the determined kinematics index is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation medicine, and in particular, to a method, device and related equipment for determining kinematic indexes based on autonomous navigation. Background Art

[0002] With the development of medical technology, intelligent auxiliary robots are gradually popularized in hospitals. While the robots improve the nursing efficiency and reduce the workload of medical staff, there are also some problems. For example, in the process of evaluating the functional status of patients (such as posture, joint range of motion, etc.), mostly the manual observation and evaluation method is still adopted. Even in some solutions, the robot-assisted evaluation is used, but usually the action capture is realized based on the method of wearing marker points. It causes certain invasiveness and discomfort to patients and is difficult to be applied to long-term and high-frequency continuous monitoring and evaluation; moreover, the kinematic indexes determined based on the method of wearing marker points are less accurate.

[0003] It can be seen that there is a problem of poor accuracy in determining kinematic indexes in the prior art. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device and related equipment for determining kinematic indexes based on autonomous navigation to solve the problem of poor accuracy in determining kinematic indexes in the prior art.

[0005] To solve the above technical problems, the present invention is implemented as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for determining kinematic indexes based on autonomous navigation, which is applied to a robot system. The method includes:

[0007] Moving to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position;

[0008] Obtaining an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object;

[0009] Generating three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on the basic model of the target object and the evolution trajectory of the basic model changing with time;

[0010] Determining the kinematic indexes corresponding to the target part of the target object in the three-dimensional pose model data.

[0011] Optionally, generating the three-dimensional pose model data according to the image data set includes:

[0012] Iterating the Gaussian kernel parameters of the general three-dimensional model according to the image data set to obtain the basic model, where the general three-dimensional model is pre-trained according to the general action data, and the basic model includes a plurality of Gaussian kernels, and each Gaussian kernel corresponds to a body part of the target object;

[0013] Determining the target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to the variable information of each Gaussian kernel in the basic model, where the variable information includes the increment or decrement of the Gaussian kernel position, rotation, and scale;

[0014] Generating the three-dimensional pose model data according to the basic model and the target evolution trajectory.

[0015] Optionally, before moving to the target position in the preset area based on the target path, the method further includes:

[0016] Inputting the obtained environmental information of the preset area into a pre-trained navigation model to obtain navigation decision data, where the navigation decision data includes a speed instruction and a steering instruction when moving along the first path;

[0017] Re-planning the first path according to the navigation decision data to generate the target path.

[0018] Optionally, the training process of the navigation model includes:

[0019] Using the robot to perform stochastic gradient descent training based on the sample environmental information of the preset area to obtain the first model parameters after updating the initial navigation model, where the initial navigation model is a neural network model for path planning and dynamic obstacle avoidance;

[0020] Aggregating the first model parameters corresponding to at least two of the robots to obtain second model parameters;

[0021] Adjusting the initial navigation model based on the second model parameters to obtain the navigation model.

[0022] Optionally, determining the target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to the variable information of each Gaussian kernel in the basic model is expressed as the following formula:

[0023] (Δc, Δr, Δs) = T φ (sin(ω x x), sin(ω y y), sin(ω z z), sin(ωt t));

[0024] Wherein, Δc is the position variable of the first Gaussian kernel, Δr is the rotation variable of the first Gaussian kernel, Δs is the scale variable of the first Gaussian kernel, and T φ is a parameterized neural network, and sin(ω x x), sin(ω y y), sin(ω z z), and sin(ω t t) are high-frequency sine encoding functions for spatial and temporal dimensions, and the first Gaussian kernel is any one of multiple Gaussian kernels.

[0025] Optionally, after determining the kinematic index corresponding to the target part of the target object in the three-dimensional pose model data, the method further includes:

[0026] Calculating the score of the kinematic index according to a functional evaluation scale to obtain a standardized evaluation report of the target object. The functional evaluation scale includes the Barthel Index, the Modified Ashworth Scale, and the Fugl-Meyer Scale. The Barthel Index is used to evaluate the ability of daily living activities, the Modified Ashworth Scale is used to assess the degree of muscle spasm, and the Fugl-Meyer Scale is used to evaluate the limb motor function after stroke.

[0027] In a second aspect, an embodiment of the present invention provides a kinematic index determination device based on autonomous navigation, which is applied to a robot system. The device includes:

[0028] A motion module, configured to move to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position;

[0029] An acquisition module, configured to acquire an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object;

[0030] A generation module, configured to generate three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on the basic model of the target object and the evolution trajectory of the basic model changing with time;

[0031] A determination module, configured to determine the kinematic index corresponding to the target part of the target object in the three-dimensional pose model data.

[0032] In a third aspect, an embodiment of the present invention provides a robot system, including a transceiver and a processor,

[0033] The processor is configured to move to a target position in a preset area based on a target path, where the target path is obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is generated according to the current position of the robot and the target position;

[0034] The transceiver is configured to acquire an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object;

[0035] The processor is further configured to generate three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on a basic model of the target object and an evolution trajectory of the basic model changing over time;

[0036] The processor is further configured to determine kinematic indexes corresponding to a target part of the target object in the three-dimensional pose model data.

[0037] In a fourth aspect, an embodiment of the present invention provides a robot system, including: a processor, a memory, and a program stored on the memory and executable on the processor, where when the program is executed by the processor, the steps of the method for determining kinematic indexes based on autonomous navigation as described in the first aspect are implemented.

[0038] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for determining kinematic indexes based on autonomous navigation as described in the first aspect are implemented.

[0039] In the embodiments of the present invention, first, a first path is quickly generated according to the current position of the robot and a specified target position. During the movement to the target position, the first path can be re-planned according to the real-time acquired environmental information to obtain a target path, and newly emerging obstacles on the movement path can be detected and avoided in a timely manner, improving the accuracy and robustness of autonomous navigation; after reaching the target position, an image data set of the target object is acquired, and the target object can be automatically recognized through a semantic segmentation or target detection model, improving the accuracy of the initial information of the contour and pose of the target object in the image data set; then, the multi-frame data in the image data set is jointly optimized to obtain a basic model, and a time factor is introduced on the basis of the basic model, and the target object in the image data set is represented by three-dimensional pose model data of "space + time", which can accurately capture the trajectory of the action of the target object evolving over time; then, the target part of the target object is automatically located and tracked, so as to obtain the kinematic indexes corresponding to the target part in the three-dimensional pose model data. The accuracy of the determined kinematic indexes is improved. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 is one of the flowcharts of a method for determining kinematic indicators based on autonomous navigation provided by an embodiment of the present invention;

[0042] Figure 2 is one of the schematic structural diagrams of a robot system provided by an embodiment of the present invention;

[0043] Figure 3 is the second flowchart of a method for determining kinematic indicators based on autonomous navigation provided by an embodiment of the present invention;

[0044] Figure 4 is the schematic structural diagram of a device for determining kinematic indicators based on autonomous navigation provided by an embodiment of the present invention;

[0045] Figure 5 is the second schematic structural diagram of a robot system provided by an embodiment of the present invention. Detailed Embodiments

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0047] See Figure 1 , Figure 1 is one of the flowcharts of a method for determining kinematic indicators based on autonomous navigation provided by an embodiment of the present invention, applied to a robot system, as Figure 1 shown, the method includes the following steps:

[0048] Step 101, Move to the target position in the preset area based on the target path, where the target path is the path obtained by re-planning the first path according to the acquired environmental information of the preset area, and the first path is the path generated according to the current position of the robot and the target position;

[0049] A robot system, i.e., a robot system applicable to the ward environment with autonomous navigation and markerless objective function evaluation for patients, such as Figure 2 As shown, it includes a mobile robot body, an autonomous navigation module, an automatic identification and positioning module, a markerless motion capture module, and a function evaluation module. The robot body includes: a mobile chassis (e.g., differential wheel or omnidirectional wheel type), an RGB camera and a 360-degree lidar (Light Laser Detection and Ranging, LiDAR) installed on the top of the robot, a built-in inertial measurement unit (Inertial Measurement Unit, IMU), and an embedded computing module with edge computing capabilities (e.g., an embedded graphics processing unit (Graphics Processing Unit, GPU) module). Each module is deployed within the computing module, and through real-time data interaction, a complete automated closed-loop for the robot's autonomous navigation and the evaluation of the patient's functional status is achieved.

[0050] In this step, the preset area can include areas such as hospital corridors and wards; the target location can be a specific room or location, i.e., the location where the target object is located. The robot (i.e., the robot body) moves to the target location based on the target path to achieve autonomous navigation, facilitating the determination of subsequent kinematic indicators.

[0051] Specifically, after the robot is started, it collects environmental information in real time through cameras, lidars, and IMU sensors, including visual images, spatial layouts, and robot postures. In the initial deployment stage of the robot system, the hospital ward structure (such as main corridors, ward layouts, etc.) can be pre-stored in the robot locally in the form of a digital map or structured information, so that the robot can quickly locate and perform initial path planning to obtain the first path. The first path is a path generated based on the current location of the robot and the target location.

[0052] Considering the frequent dynamic changes in the preset area (such as medical staff walking, bed movement, temporary placement of medical equipment, etc.), the robot not only relies on the pre-stored map information but also needs to perform simultaneous localization and mapping (SLAM) or online environment update on the local environment according to real-time sensor data during operation. Therefore, the first path can be re-planned based on the obtained environmental information of the preset area to obtain the target path. In this way, moving to the target location in the preset area based on the target path achieves more flexible and robust autonomous navigation.

[0053] Step 102, obtain an image dataset of the target object, where the image dataset includes multiple image frames arranged in a first time sequence, and each image frame includes the target object;

[0054] In this step, after the robot completes autonomous navigation and reaches the designated target position, it automatically conducts patient identification and markerless motion capture: First, the robot uses the equipped RGB camera to collect images of the patient and the surrounding environment, and through a deep learning semantic segmentation model, it separates the patient area from the background in real time to accurately obtain the patient's position. After the identification is completed, the robot can perform fine-tuning positioning and pitch angle adjustment to ensure that the patient is within the best field of view of the camera. In this way, without the need for the patient to wear any sensors or markers on the body, the robot can continuously capture the patient's movements from multiple angles, thereby obtaining an image dataset of the target object, including multiple image frames arranged in the first time sequence, and each image frame includes the target object, which is convenient for subsequent three-dimensional markerless motion capture and pose reconstruction based on the image dataset.

[0055] Step 103: Generate three-dimensional pose model data according to the image dataset, and the three-dimensional pose model data is determined based on the basic model of the target object and the evolution trajectory of the basic model over time;

[0056] In this step, three-dimensional human pose reconstruction technology can be used to reconstruct the three-dimensional model of the target object according to the image dataset obtained in step 102, and the basic model of the target object is obtained by optimizing the model parameters, which improves the accuracy of the three-dimensional pose reconstruction of the target object.

[0057] Furthermore, when the target object performs continuous actions, the coherence in the time series needs to be considered. Therefore, a dynamic deformation field can be combined to additionally introduce a time factor into the basic model, and represent the target object in the image dataset with "space + time" three-dimensional pose model data. In this way, the trajectory of the target object's action evolving over time, that is, the evolution trajectory of the basic model over time, can be accurately captured. By jointly optimizing multiple frames of data, the time continuity and spatial consistency of the three-dimensional reconstruction result are maintained, ensuring accurate, stable, and drift-free motion capture of the target object.

[0058] Step 104: Determine the kinematic indexes corresponding to the target part of the target object in the three-dimensional pose model data.

[0059] In this step, the basic model can present each part of the target object (such as the head, torso, limb joints, etc.). By automatically positioning and tracking the target part of the target object on the three-dimensional basic model, the kinematic indexes corresponding to the target part in the three-dimensional pose model data can be obtained. The kinematic indexes include range of motion (ROM), action speed, symmetry, etc.

[0060] In the embodiments of the present invention, first, a first path is quickly generated according to the current position of the robot and the specified target position. During the movement to the target position, the first path can be replanned according to the real-time acquired environmental information to obtain a target path, and newly emerging obstacles on the movement path can be detected and avoided in a timely manner, improving the accuracy and robustness of autonomous navigation. After reaching the target position, an image data set of the target object is acquired, and the target object can be automatically recognized through a semantic segmentation or object detection model, improving the accuracy of the initial information of the contour and pose of the target object in the image data set. Then, the basic model is obtained by jointly optimizing multiple frames of data in the image data set, and a time factor is introduced on the basis of the basic model. The target object in the image data set is represented by three-dimensional pose model data of "space + time", and the trajectory of the target object's action evolving with time can be accurately captured. Then, the target part of the target object is automatically located and tracked, so as to obtain the kinematic indexes corresponding to the target part in the three-dimensional pose model data. The accuracy of the determined kinematic indexes is improved.

[0061] In addition, by means of markerless three-dimensional motion capture technology, non-invasive and high-precision determination of kinematic indexes is realized, overcoming the inconvenience of traditional manual evaluation and wearing sensors.

[0062] It should be noted that the images and kinematic indexes or other evaluation data of the target object are only stored and processed locally by the robot, and the original video data is not transmitted externally. Moreover, when quantifying kinematic indexes, data anonymization and encryption technologies can be adopted to ensure the privacy security and compliance of the data. Identity authentication and permission management are set on the robot side to ensure that only authorized medical staff can access and view the evaluation results of the target object.

[0063] Optionally, generating the three-dimensional pose model data according to the image data set includes:

[0064] Iterating the Gaussian kernel parameters of the general three-dimensional model according to the image data set to obtain the basic model. The general three-dimensional model is pre-trained according to general action data. The basic model includes multiple Gaussian kernels, and each Gaussian kernel corresponds to a body part of the target object;

[0065] According to the variable information of each Gaussian kernel in the basic model, determining the target evolution trajectory of each Gaussian kernel changing according to the first time sequence. The variable information includes the increment or decrement of the Gaussian kernel position, rotation and scale;

[0066] Generating the three-dimensional pose model data according to the basic model and the target evolution trajectory.

[0067] In this embodiment, in the initial stage, general action data can be used for large-scale offline training to obtain a general three-dimensional model with broad adaptability. After going online, for some special target objects (such as limb deformities, movement disorders, etc.), the Gaussian kernel parameters of the general three-dimensional model can be iterated according to the characteristic actions extracted from the image dataset, so that the Gaussian kernel better fits the actual posture of the target object. Specifically, see the following description:

[0068] Each body part of the target object is represented by a number of Gaussian kernels in the general three-dimensional model, and each Gaussian kernel corresponds to a body part of the target object. Then, using the differentiable rendering algorithm, the three-dimensional Gaussian kernels are projected onto the two-dimensional image plane of the image frames in the image dataset, and the difference is optimized with the actual image to iteratively update the Gaussian kernel parameters, obtaining a basic model that better fits the actual posture of the target object, thereby realizing high-precision three-dimensional reconstruction of the target object. For a single frame or a single pose, a static three-dimensional model can complete the representation. By optimizing the model parameters, the basic model of the target object is obtained, improving the accuracy of three-dimensional reconstruction of the target object's posture.

[0069] Among them, the formula for any Gaussian kernel is as follows:

[0070]

[0071] In the formula, α k represents the weight coefficient of the k-th Gaussian kernel function, K represents the number of Gaussian kernels, p represents any point in three-dimensional space, and Pc k represents the two-dimensional projection coordinates of the center point of the k-th Gaussian kernel function, represents the Mahalanobis distance defined using the deformation covariance matrix Σ k defined.

[0072] Furthermore, when the target object performs continuous actions, the coherence in the time series needs to be considered. Therefore, a dynamic deformation field can be combined with the three-dimensional Gaussian kernel representation to improve the accuracy of generating three-dimensional pose model data. Specifically, a time factor is additionally introduced into the basic model, and the variable information of each Gaussian kernel is estimated through a neural network. In this way, according to the variable information of each Gaussian kernel in the basic model, the target evolution trajectory of each Gaussian kernel changing according to the first time series can be determined, improving the consistency of the reconstruction results between consecutive multiple frames. Furthermore, three-dimensional pose model data is generated based on the basic model and the target evolution trajectory. The target object in the image dataset is represented by three-dimensional pose model data of "space + time". Through the joint optimization of differentiable rendering and dynamic deformation field, high-precision and markerless capture of the target object's actions is achieved, improving the accuracy of the kinematic indexes determined subsequently.

[0073] Optionally, based on the variable information of each Gaussian kernel in the base model, determine the target evolution trajectory of each Gaussian kernel changing according to the first time series, which is expressed by the following formula:

[0074] (Δc, Δr, Δs) = T φ (sin(ω x x), sin(ω y y), sin(ω z z), sin(ω t t));

[0075] Among them, Δc is the position variable of the first Gaussian kernel, Δr is the rotation variable of the first Gaussian kernel, Δs is the scale variable of the first Gaussian kernel, T φ is a parameterized neural network, sin(ω x x), sin(ω y y), sin(ω z z), and sin(ω t t) are high-frequency sine encoding functions for the spatial and temporal dimensions, and the first Gaussian kernel is any one of multiple Gaussian kernels. In this way, through the above process, the temporal coherence and accuracy in the dynamic pose reconstruction of the target object are effectively improved. The target object in the image dataset is represented as "space + time" three-dimensional pose model data.

[0076] In addition, if multiple robots are deployed in the same preset area, they can independently iterate the Gaussian kernel parameters of the general three-dimensional model and determine the evolution trajectory of each Gaussian kernel changing according to the time series; then share and aggregate the iterated parameters through the federated learning method, so as to further improve the temporal coherence and accuracy in the dynamic pose reconstruction of the target object.

[0077] Optionally, before moving to the target position in the preset area based on the target path, the method further includes:

[0078] Input the obtained environmental information of the preset area into a pre-trained navigation model to obtain navigation decision data, where the navigation decision data includes a speed command and a steering command when moving along the first path;

[0079] Re-plan the first path according to the navigation decision data to generate the target path.

[0080] In this embodiment, a target path can be generated by a pre-trained navigation model, which can be a multi-modal fusion deep neural network model. The navigation model can include an input layer, a feature extraction layer, a feature fusion layer, and an output layer. The environmental information of the preset area can include RGB camera images, 360-degree lidar point clouds, and inertial measurement unit data, etc. These acquired environmental information are input into the input layer of the navigation model to perceive the environmental layout and obstacle distribution within the preset area; the feature extraction layer can adopt a Convolutional Neural Networks (CNN) combined with an Attention mechanism to extract high-dimensional spatial features for visual and lidar data, enhancing the robustness in a changing environment; the feature fusion layer can accumulate and aggregate multi-modal features from vision, radar, and IMU during the fusion stage to obtain a comprehensive feature representation that takes into account both spatial structure and motion information; finally, the output layer obtains navigation decision data based on the fused features, and the navigation decision data includes motion control instructions (such as speed instructions and steering instructions) when moving along the first path. The first path generated in the initial stage is re-planned according to the navigation decision data to obtain the target path. In this way, moving to the target position in the preset area based on the target path realizes more flexible and robust autonomous navigation.

[0081] Optionally, the training process of the navigation model includes:

[0082] Using the robot to perform stochastic gradient descent training based on the sample environmental information of the preset area to obtain the first model parameters after updating the initial navigation model, where the initial navigation model is a neural network model for path planning and dynamic obstacle avoidance;

[0083] Aggregating the first model parameters corresponding to at least two of the robots to obtain the second model parameters;

[0084] Adjusting the initial navigation model based on the second model parameters to obtain the navigation model.

[0085] In this embodiment, considering the requirements of data privacy security and model generalization performance, a federated learning training framework combining local model update (DPASGD) and model parameter aggregation (FedAvg) can be adopted:

[0086] First, in the local model training stage: The robot uses the sample environment information in the preset area for stochastic gradient descent training to obtain the first model parameters after updating the initial navigation model. The sample environment information can include the locally collected environmental perception data (images, radar point clouds, IMU information) and the actual motion control records of the robot, etc. Conduct several rounds of stochastic gradient descent iterations locally, and continuously update the parameters of the navigation model according to the environmental data in the current preset area to obtain the updated first model parameters It can be expressed as:

[0087]

[0088] In the formula, is the parameter of the navigation model of the i-th robot at the t-th iteration, i is greater than or equal to 2, η is the learning rate, n i is the data volume of the i-th robot, d i,j is the j-th data sample of the i-th robot, is the gradient of the loss function, and ε is a constant used to prevent the denominator from being 0.

[0089] Then, after obtaining the first model parameters after updating the initial navigation model Furthermore, model parameter aggregation is performed: Model parameter aggregation is performed on the first model parameters corresponding to at least two robots to obtain the second model parameters. Specifically, the first model parameters obtained after the local training of all robot nodes are periodically exchanged and weighted averaged on a P2P or lightweight federated server to obtain the global model parameters, that is, the second model parameters

[0090]

[0091] In the formula, λ i is the weight coefficient calculated by the i-th robot based on the data volume or model update effect, and δ is a smoothing constant to ensure the stability of the set. Among them, the weighting coefficient can be dynamically adjusted according to the data volume or training effect of each node.

[0092] Finally, the second model parameters are distributed back to each robot for the next round of training or direct inference. So that each robot adjusts the initial navigation model based on the second model parameters to obtain the navigation model, improving the adaptability and accuracy of the autonomous navigation network in various scenarios.

[0093] In this way, during the initial deployment, the robot can pre-store the layout maps of the main wards and corridors in the hospital for rapid positioning in stable areas and global path planning, and quickly generate the first path. However, since the ward environment is often dynamic and changeable, the robot still needs to perform online SLAM or local map updates based on real-time sensor (camera, LiDAR, etc.) data to ensure timely detection and avoidance of newly emerging obstacles. When the robot detects temporary obstacles (such as moving beds, walking medical staff, etc.), it automatically triggers the adaptive path planning algorithm to perform local path replanning, that is, replan the first path according to the navigation decision data determined by the real-time environment information to generate the target path, ensuring the safety and continuity of the navigation process. In addition, if multiple robots are deployed in the same preset area, they can independently collect navigation data and train models, and then share and aggregate model parameters through federated learning to improve the overall navigation accuracy and robustness.

[0094] Optionally, after determining the kinematic index corresponding to the target part of the target object in the three-dimensional pose model data, the method further includes:

[0095] Calculating the score of the kinematic index according to the functional evaluation scale to obtain the standardized evaluation report of the target object. The functional evaluation scale includes the Barthel index, the Modified Ashworth Scale, and the Fugl-Meyer Scale. The Barthel index is used to evaluate the ability of daily living activities, the Modified Ashworth Scale is used to assess the degree of muscle spasm, and the Fugl-Meyer Scale is used to evaluate the limb motor function after stroke.

[0096] In this embodiment, the kinematic index can be obtained through the function evaluation module in the robot system, and during the process of calculating the score of the kinematic index, combined with the clinically recognized standard function evaluation scale (such as the Barthel index, the Modified Ashworth grading, and the Fugl-Meyer Scale), a standardized evaluation report of the target object is automatically generated. After being anonymized and securely encrypted, the evaluation report is automatically uploaded to the hospital information system and pushed to the terminals of relevant medical staff in real time for medical staff to view and make clinical decisions. During the functional evaluation of the patient (i.e., the target object), if the system detects a significant abnormality or a trend of degradation in the patient's functional status (such as a significant decrease in the range of joint movement or abnormal movements), the robot system automatically sends a warning message to the nurse station terminal or the doctor's mobile terminal in real time through the wireless network. Based on the evaluation report and the real-time warning information, medical staff can take appropriate clinical intervention measures in a timely manner, such as adjusting the patient's rehabilitation training plan or treatment plan, to ensure timely and effective medical care intervention.

[0097] In a specific embodiment, such as Figure 3As described above, a method for determining kinematic indicators based on autonomous navigation is provided. This method enables the robot system to autonomously complete the positioning and identification of patients in the ward, and uses non-invasive and markerless visual capture technology to objectively evaluate the functional status of patients. The specific implementation process is as follows:

[0098] Task reservation and startup: According to the actual clinical needs, medical staff submit an assessment task request to the robot system through the hospital mobile terminal APP. The task information includes the unique identification number (patient ID) of the patient, the bed location, and the type of assessment task. The robot system automatically receives and parses the task information, combines it with the pre-stored hospital ward map, automatically plans the navigation path, and activates the autonomous navigation module to autonomously move to the designated patient's bed area.

[0099] Automatic identification and functional assessment process: After the robot arrives at the target bed area, the patient automatic identification module and the markerless motion capture module are automatically activated. First, the robot uses the built-in RGB camera to continuously collect patient image data, and through the built-in deep learning semantic segmentation model, automatically extracts the patient area mask and removes background interference to accurately identify and determine the patient's spatial position and pose information; based on the recognition result, the robot adjusts its own position and shooting angle in real time to ensure the best viewing angle and distance for subsequent accurate motion capture. Then, without the patient wearing any sensors or markers on the body, the robot continuously shoots the patient's movements from multiple angles through the camera; using three-dimensional human pose reconstruction technology (such as the reconstruction method based on three-dimensional Gaussian kernel) and dynamic deformation field network technology, the patient's motion data is reconstructed in real-time three-dimensionally and continuously to ensure the stability and accuracy of motion capture. Then, the function evaluation module extracts the three-dimensional reconstructed dynamic motion data in real time, automatically calculates and obtains the functional indicators required clinically, including the patient's range of motion (ROM), movement speed, movement frequency, and symmetry, etc.; combined with clinically recognized standard function evaluation scales (such as Barthel index, modified Ashworth scale, Fugl-Meyer scale), a standardized patient function evaluation report is automatically generated; after the evaluation report is anonymized and securely encrypted, it is automatically uploaded to the hospital information system and pushed to the relevant medical staff terminal in real time for medical staff to view and make clinical decisions.

[0100] Clinical intervention and feedback mechanism: During the patient function evaluation process, if the system detects that the patient's functional status shows obvious abnormalities or trend degradation (such as a significant decrease in the range of joint motion or abnormal movements), the robot system automatically sends a warning message to the nurse station terminal or the doctor's mobile terminal in real time through the wireless network. Based on the evaluation report and the real-time warning information, medical staff promptly take appropriate clinical intervention measures, such as adjusting the patient's rehabilitation training plan or treatment plan, to ensure timely and effective medical care intervention.

[0101] Automatic return after task completion: After the evaluation task is completed, the robot system automatically calls the autonomous navigation module to plan and execute the return path in real time, and autonomously navigates back to the designated standby point or charging station in the hospital to complete the automated closed-loop process of this clinical function evaluation task.

[0102] See Figure 4 , Figure 4 is a schematic structural diagram of a kinematic index determination device based on autonomous navigation provided by an embodiment of the present invention, which is applied to a robot system, such as Figure 4 shown, the kinematic index determination device 400 based on autonomous navigation includes:

[0103] A motion module 401 for moving to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position;

[0104] An acquisition module 402 for acquiring an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object;

[0105] A generation module 403 for generating three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on a basic model of the target object and an evolution trajectory of the basic model changing over time;

[0106] A determination module 404 for determining kinematic indexes corresponding to a target part of the target object in the three-dimensional pose model data.

[0107] Optionally, the generation module 403 is specifically configured to:

[0108] Iterate the Gaussian kernel parameters of the general three-dimensional model according to the image data set to obtain the basic model, where the general three-dimensional model is pre-trained according to general action data, and the basic model includes a plurality of Gaussian kernels, and each Gaussian kernel corresponds to a body part of the target object;

[0109] Determine a target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to the variable information of each Gaussian kernel in the basic model, where the variable information includes increments or decrements of Gaussian kernel position, rotation, and scale;

[0110] Generate the three-dimensional pose model data according to the basic model and the target evolution trajectory.

[0111] Optionally, the device further includes:

[0112] An input module, configured to input the obtained environmental information of the preset area into a pre-trained navigation model to obtain navigation decision data, where the navigation decision data includes a speed command and a steering command when moving along the first path;

[0113] A planning module, configured to re-plan the first path according to the navigation decision data to generate the target path.

[0114] Optionally, the training process of the navigation model includes:

[0115] Using a robot to perform stochastic gradient descent training based on the sample environmental information of the preset area to obtain first model parameters after updating the initial navigation model, where the initial navigation model is a neural network model for path planning and dynamic obstacle avoidance;

[0116] Aggregating the first model parameters corresponding to at least two of the robots to obtain second model parameters;

[0117] Adjusting the initial navigation model based on the second model parameters to obtain the navigation model.

[0118] Optionally, according to the variable information of each Gaussian kernel in the base model, determining a target evolution trajectory for each Gaussian kernel to change according to the first time sequence, expressed by the following formula:

[0119] (Δc, Δr, Δs) = T φ (sin(ω x x), sin(ω y y), sin(ω z z), sin(ω t t));

[0120] Wherein, Δc is the position variable of the first Gaussian kernel, Δr is the rotation variable of the first Gaussian kernel, Δs is the scale variable of the first Gaussian kernel, T φ is a parameterized neural network, sin(ω x x), sin(ω y y), sin(ω z z) and sin(ω t t) are high-frequency sine encoding functions for the spatial and temporal dimensions, and the first Gaussian kernel is any one of multiple Gaussian kernels.

[0121] Optionally, the device further includes:

[0122] An evaluation module, configured to calculate scores of the kinematic indicators according to a functional evaluation scale, so as to obtain a standardized evaluation report of the target object. The functional evaluation scale includes the Barthel Index, the Modified Ashworth Scale, and the Fugl-Meyer Scale. The Barthel Index is used to evaluate the ability of performing daily living activities, the Modified Ashworth Scale is used to assess the degree of muscle spasm, and the Fugl-Meyer Scale is used to evaluate the limb motor function after stroke.

[0123] The kinematic indicator determination device 400 based on autonomous navigation can implement each process of the above-mentioned embodiments of the method for determining kinematic indicators based on autonomous navigation. The technical features correspond one by one and can achieve the same technical effects. To avoid repetition, they will not be elaborated here.

[0124] An embodiment of the present invention further provides a robot system, including: a processor, a memory, and a program stored on the memory and executable on the processor. When the program is executed by the processor, it implements each process of the above-mentioned embodiment of the method for determining kinematic indicators based on autonomous navigation and can achieve the same technical effects. To avoid repetition, they will not be elaborated here.

[0125] Specifically, referring to Figure 5 , an embodiment of the present invention further provides a robot system, including a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505, and a memory 506.

[0126] Wherein, the processor 505 is configured to move to a target position in a preset area based on a target path, and the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position;

[0127] The transceiver 502 is configured to acquire an image data set of a target object, and the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object;

[0128] The processor 505 is further configured to generate three-dimensional pose model data according to the image data set, and the three-dimensional pose model data is determined based on a basic model of the target object and an evolution trajectory of the basic model changing over time;

[0129] The processor 505 is further configured to determine kinematic indicators corresponding to a target part of the target object in the three-dimensional pose model data.

[0130] Optionally, the generating three-dimensional pose model data according to the image data set includes:

[0131] Iterate the Gaussian kernel parameters of the general 3D model according to the image dataset to obtain the base model. The general 3D model is pre-trained according to general action data. The base model includes multiple Gaussian kernels, and each Gaussian kernel corresponds to a body part of the target object;

[0132] According to the variable information of each Gaussian kernel in the base model, determine the target evolution trajectory of each Gaussian kernel changing according to the first time sequence. The variable information includes the increment or decrement of the Gaussian kernel position, rotation, and scale;

[0133] Generate the 3D pose model data according to the base model and the target evolution trajectory.

[0134] Optionally, the processor 505 is further configured to:

[0135] Input the obtained environmental information of the preset area into a pre-trained navigation model to obtain navigation decision data, where the navigation decision data includes a speed instruction and a steering instruction when moving along the first path;

[0136] Re-plan the first path according to the navigation decision data to generate the target path.

[0137] Optionally, the training process of the navigation model includes:

[0138] Use the robot to perform stochastic gradient descent training based on the sample environmental information of the preset area to obtain the first model parameters after updating the initial navigation model. The initial navigation model is a neural network model for path planning and dynamic obstacle avoidance;

[0139] Aggregate the first model parameters corresponding to at least two of the robots to obtain second model parameters;

[0140] Adjust the initial navigation model based on the second model parameters to obtain the navigation model.

[0141] Optionally, the determining the target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to the variable information of each Gaussian kernel in the base model is expressed by the following formula:

[0142] (Δc, Δr, Δs) = T φ (sin(ω x x), sin(ω y y), sin(ω z z), sin(ω t t));

[0143] where, Δc is the position variable of the first Gaussian kernel, Δr is the rotation variable of the first Gaussian kernel, Δs is the scale variable of the first Gaussian kernel, and T φ is a parameterized neural network, and sin(ω x x), sin(ω y y), sin(ω z z), and sin(ω t t) are high-frequency sine encoding functions for spatial and temporal dimensions, and the first Gaussian kernel is any one of multiple Gaussian kernels.

[0144] Optionally, the processor 505 is further configured to:

[0145] Calculate the score of the kinematic index according to the functional evaluation scale to obtain a standardized evaluation report of the target object. The functional evaluation scale includes the Barthel index, the Modified Ashworth Scale, and the Fugl-Meyer Scale. The Barthel index is used to evaluate the ability of daily living activities, the Modified Ashworth Scale is used to assess the degree of muscle spasm, and the Fugl-Meyer Scale is used to evaluate the limb motor function after stroke.

[0146] In Figure 5 , the bus architecture (represented by bus 501), bus 501 may include any number of interconnected buses and bridges. Bus 501 links together various circuits including one or more processors represented by processor 505 and a memory represented by memory 506. Bus 501 can also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and thus will not be further described herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. Transceiver 502 can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 505 is transmitted over a wireless medium via antenna 503. Further, antenna 503 also receives data and transmits the data to processor 505.

[0147] Processor 505 is responsible for managing bus 501 and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 506 can be used to store data used by processor 505 when performing operations.

[0148] Optionally, the processor 505 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or a Complex Programmable Logic Device (CPLD).

[0149] The embodiments of the present invention also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements each process of the above embodiments of the method for determining kinematic indexes based on autonomous navigation and can achieve the same technical effects. To avoid repetition, it will not be elaborated here. Among them, the computer-readable storage medium may be, for example, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0150] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0152] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the purpose of the present invention and the scope protected by the claims, and all of them belong to the protection scope of the present invention.

Claims

1. A method for determining kinematic indicators based on autonomous navigation, characterized in that Applied to a robot system, the method includes: Moving to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position; Obtaining an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object; Generating three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on a basic model of the target object and an evolution trajectory of the basic model changing over time; Determining kinematic indexes corresponding to a target part of the target object in the three-dimensional pose model data.

2. The method according to claim 1, characterized in that, The generating three-dimensional pose model data according to the image data set includes: Iterating Gaussian kernel parameters of a general three-dimensional model according to the image data set to obtain the basic model, where the general three-dimensional model is pre-trained according to general action data, and the basic model includes a plurality of Gaussian kernels, and each Gaussian kernel corresponds to a body part of the target object; Determining a target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to variable information of each Gaussian kernel in the basic model, where the variable information includes increments or decrements of Gaussian kernel position, rotation, and scale; Generating the three-dimensional pose model data according to the basic model and the target evolution trajectory.

3. The method according to claim 1, wherein Before moving to the target position in the preset area based on the target path, the method further includes: Inputting the acquired environmental information of the preset area into a pre-trained navigation model to obtain navigation decision data, where the navigation decision data includes a speed instruction and a steering instruction when moving along the first path; Re-planning the first path according to the navigation decision data to generate the target path.

4. The method according to claim 3, wherein The training process of the navigation model includes: Using the robot to perform stochastic gradient descent training based on the sample environmental information of the preset area to obtain first model parameters after updating the initial navigation model, where the initial navigation model is a neural network model for path planning and dynamic obstacle avoidance; Aggregating the first model parameters corresponding to at least two robots to obtain second model parameters; Adjusting the initial navigation model based on the second model parameters to obtain the navigation model.

5. The method according to claim 2, wherein The determining a target evolution trajectory of each Gaussian kernel changing according to the first time sequence according to variable information of each Gaussian kernel in the basic model is expressed by the following formula: (Δc, Δr, Δs) = T φ (sin(ω x x), sin(ω y y), sin(ω z z), sin(ω t t)); where Δc is the position variable of the first Gaussian kernel, Δr is the rotation variable of the first Gaussian kernel, Δs is the scale variable of the first Gaussian kernel, and T φ is a parameterized neural network, sin(ω x x), sin(ω y y), sin(ω z z), and sin(ω t t) are high-frequency sine encoding functions for the spatial and temporal dimensions, and the first Gaussian kernel is any one of a plurality of Gaussian kernels.

6. The method according to claim 1, characterized in that, After determining the kinematic indexes corresponding to the target part of the target object in the three-dimensional pose model data, the method further includes: Calculate the scores of the kinematic indicators according to the functional evaluation scale to obtain a standardized evaluation report of the target object. The functional evaluation scale includes the Barthel Index, the Modified Ashworth Scale, and the Fugl-Meyer Scale. The Barthel Index is used to evaluate the ability of activities of daily living, the Modified Ashworth Scale is used to assess the degree of muscle spasm, and the Fugl-Meyer Scale is used to evaluate the limb motor function after stroke.

7. A kinematic index determination device based on autonomous navigation, characterized in that, Applied to a robot system, the device includes: A motion module, configured to move to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position; An acquisition module, configured to acquire an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object; A generation module, configured to generate three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on the basic model of the target object and the evolution trajectory of the basic model changing over time; A determination module, configured to determine the kinematic indicators corresponding to the target part of the target object in the three-dimensional pose model data.

8. A robot system, characterized in that, Includes a transceiver and a processor, The processor is configured to move to a target position in a preset area based on a target path, where the target path is a path obtained by re-planning a first path according to the acquired environmental information of the preset area, and the first path is a path generated according to the current position of the robot and the target position; The transceiver is configured to acquire an image data set of a target object, where the image data set includes a plurality of image frames arranged in a first time sequence, and each image frame includes the target object; The processor is further configured to generate three-dimensional pose model data according to the image data set, where the three-dimensional pose model data is determined based on the basic model of the target object and the evolution trajectory of the basic model changing over time; The processor is further configured to determine the kinematic indicators corresponding to the target part of the target object in the three-dimensional pose model data.

9. A robot system, characterized in that, Includes: A processor, a memory, and a program stored on the memory and executable on the processor. When the program is executed by the processor, the steps of the method for determining kinematic indicators based on autonomous navigation according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the method for determining kinematic indicators based on autonomous navigation according to any one of claims 1 to 6 are implemented.

Images