A sensor-based companion robot intelligent prediction method, device and medium

Abstract

This application discloses a sensor-based intelligent prediction method, device, and medium for accompanying robots, belonging to the technical field of accompanying robots. The method includes: collecting user behavior, visual, and environmental data; filtering and standardizing the data using a Kalman filter algorithm to generate standard data; processing the standard data using an action recognition algorithm to determine user behavior; constructing a digital model based on standard environmental data, and combining this model with a preset target and reward function to select a navigation path; predicting the user's next position and direction of movement based on historical environmental data and a parallel interactive multi-model Kalman filter algorithm; and calculating the accompanying robot's motion commands based on the navigation strategy and user prediction information. This application effectively integrates data from multiple sensors, improves the accuracy of user behavior recognition, and enables the development of flexible and efficient navigation strategies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of companion robots, and particularly relates to a sensor-based intelligent prediction method for companion robots, a device and a medium. BACKGROUND

[0002] Under the background of rapid development of modern science and technology, companion robots, as an important member of the intelligent service field, are gradually integrated into people's daily life. Such robots are designed to follow and serve specific users in various environments, such as providing navigation, assisting walking, carrying objects, etc. However, it is not easy to achieve efficient and safe companion services, which requires robots to accurately understand the user's behavior intention, perceive the changes in the surrounding environment in real time, and make reasonable navigation decisions accordingly.

[0003] Traditional companion robot systems often rely on single sensor data to identify user behavior and perceive the environment, which leads to limitations in understanding complex scenarios. For example, relying solely on visual sensors may not accurately determine the user's motion state, especially in cases of light changes or occlusions; while using only inertial sensors may have difficulty distinguishing subtle differences in user actions. In addition, traditional navigation strategies are often based on fixed rules or simple algorithms, lacking adaptability and flexibility to dynamic changes in the environment, and are prone to collisions or getting lost in crowded or obstacle-rich environments.

[0004] Therefore, how to effectively fuse multiple sensor data, improve the accuracy of user behavior recognition, and develop flexible and efficient navigation strategies has become a technical problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a sensor-based intelligent prediction method for companion robots, a device and a medium, to solve the technical problems of how to effectively fuse multiple sensor data, improve the accuracy of user behavior recognition, and develop flexible and efficient navigation strategies.

[0006] In a first aspect, the embodiments of the present application provide a sensor-based intelligent prediction method for a companion robot, applied to a companion robot and a wearable device, the companion robot being provided with a camera unit, a laser radar and a depth sensor, and the wearable device being provided with an inertial measurement unit, the method comprising: collecting behavior data of a user based on the wearable device, collecting visual data of the user based on the camera unit, and determining environmental data around the companion robot based on the laser radar and the depth sensor; wherein the behavior data comprises acceleration and angular velocity, and the environmental data comprises at least pedestrian position, pedestrian speed, pedestrian distance and obstacles; filtering the behavior data, the image data and the environmental data based on a preset Kalman filtering algorithm, and performing standardization processing to generate standard data; wherein the standard data comprises standard behavior data, standard visual data and standard environmental data; processing the standard data based on a preset action recognition algorithm to determine a user behavior of the user; wherein the user behavior comprises at least one of the following data: standing, walking, turning and running; constructing a digital model based on the standard environmental data; processing the user behavior and the digital model based on a preset target and a reward function to determine a navigation path; wherein the target comprises safely following the user, avoiding obstacles and crowd dense areas, and the reward function comprises high scores for successfully avoiding collisions and rewards for efficiently reaching a destination; processing the behavior data and the navigation path based on the standard environmental data and historical environmental data of the standard environmental data to determine a navigation strategy; processing the behavior data of the user based on a preset parallel interactive multi-model Kalman filtering algorithm and a motion pattern to predict a position and a motion direction of the user at a next unit time; wherein the unit time is a prediction interval of the companion robot; and calculating a motion instruction of the companion robot based on the navigation strategy and the position and the motion direction of the user at the next unit time; wherein the motion instruction comprises robot speed, robot acceleration and robot turning angle.

[0007] In an implementation manner of the present application, the behavior data, the image data and the environmental data are filtered and standardized to generate standard data, specifically comprising: processing the behavior data based on a preset adaptive Kalman filtering algorithm to adjust filtering parameters of the adaptive Kalman filtering algorithm according to dynamic change characteristics of the behavior data to generate preliminary standard behavior data; extracting a feature vector of the image data based on a preset feature extraction algorithm to generate preliminary standard visual data; processing the environmental data based on the adaptive Kalman filtering to generate preliminary standard environmental data; and normalizing the preliminary standard behavior data, the preliminary standard visual data and the preliminary standard environmental data to the same data range to generate the standard data.

[0008] In one implementation of this application, standard data is processed based on a preset action recognition algorithm to determine the user's behavior. Specifically, this includes: constructing a preliminary action recognition model; wherein the preliminary action recognition model is a neural network model obtained by training based on user behavior data; training the preliminary action recognition model based on historical data of the behavior data processed by a preset transfer learning algorithm to obtain the action recognition model; wherein the action recognition model is equipped with an action recognition algorithm; and inputting standard data into the action recognition model to determine the user's behavior.

[0009] In one implementation of this application, constructing a digital model based on standard environmental data specifically includes: mapping the standard environmental data to a preset three-dimensional spatial coordinate system to construct an initial environmental model; wherein the three-dimensional spatial coordinate system is divided into multiple voxels; processing the initial environmental model based on the standard environmental data to fill environmental attributes into the multiple voxels to construct a digital model; wherein the environmental attributes include walkability, population density, and obstacles.

[0010] In one implementation of this application, user behavior and a digital model are processed based on a preset goal and a reward function to determine a navigation path. Specifically, this includes: starting with the accompanying robot as the starting point and determining a target point based on the user and a preset distance; determining the goal and constructing a reward function based on the goal; wherein the goal includes safely following the user, avoiding obstacles, and avoiding densely populated areas, and the reward function is set with different weights; processing the starting point, target point, and digital model based on a preset path planning algorithm to determine multiple preliminary navigation paths; and evaluating the multiple preliminary navigation paths based on the weights of the reward function to determine the final navigation path.

[0011] In one implementation of this application, behavioral data and navigation paths are processed based on standard environmental data and historical environmental data of the standard environmental data to determine a navigation strategy. Specifically, this includes: retrieving a preset historical environmental database based on the standard environmental data to determine if historical environmental data of the accompanying robot exists; if not, using the navigation path as the navigation strategy; if yes, obtaining the current time of the search for the accompanying robot and retrieving the historical database based on the current time to determine if historical environmental data of the same moment exists; if not, using the navigation path as the navigation strategy; if yes, evaluating the navigation path and behavioral data based on the historical environmental data to generate an evaluation result; wherein the evaluation result includes potential obstacles; and modifying the navigation path based on the evaluation result to determine the navigation strategy.

[0012] In one implementation of this application, user behavior data is processed based on a preset parallel interactive multi-model Kalman filter algorithm and motion patterns to predict the user's position and direction of movement in the next unit of time. Specifically, this includes: constructing multiple motion models; where each motion model corresponds to a user behavior, including but not limited to uniform straight-line walking, accelerating running, decelerating to a stop, and sudden turning; processing multiple motion models based on the Kalman filter algorithm, updating the predicted state of the multiple motion models in real time using the user's historical behavior data; where the predicted state includes position, velocity, and acceleration; processing user behavior data based on a preset motion pattern recognition algorithm to evaluate the matching degree of the multiple motion models; processing behavior data based on the motion model with the highest matching degree to determine a preliminary prediction; and calculating the user's position and direction of movement in the next unit of time based on the preliminary prediction and the unit of time.

[0013] In one implementation of this application, the motion command of the accompanying robot is calculated based on the navigation strategy and the user's position and direction of motion in the next unit of time. Specifically, this includes: calculating the target position and direction that the accompanying robot needs to reach based on the path points determined by the navigation strategy and the user's predicted position in the next unit of time; determining the constraints of the accompanying robot; wherein the constraints include maximum speed, maximum acceleration, steering angle range, and motion smoothness requirements; processing the current state of the accompanying robot based on the target position and direction to generate preliminary motion commands; and processing the preliminary motion commands based on the constraints to determine the motion commands.

[0014] Secondly, embodiments of this application also provide a sensor-based intelligent prediction device for accompanying robots, applied to accompanying robots and wearable devices. The accompanying robot is equipped with a camera unit, a lidar, and a depth sensor, and the wearable device is equipped with an inertial measurement unit. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: collect user behavior data from the wearable device; collect user visual data from the camera unit; and determine environmental data around the accompanying robot using the lidar and depth sensor. The behavior data includes acceleration and angular velocity, and the environmental data includes at least pedestrian position, pedestrian speed, and pedestrian distance. The behavior data, image data, and environmental data are filtered and standardized using a preset Kalman filter algorithm to generate standard data. The standard data includes standard behavior data, standard visual data, and standard environmental data. The system processes standard data based on a pre-defined motion recognition algorithm to determine user behavior. User behavior includes at least one of the following: standing, walking, turning, or running. A digital model is constructed based on standard environmental data. User behavior and the digital model are processed based on pre-defined objectives and reward functions to determine the navigation path. Objectives include safely following the user, avoiding obstacles and densely populated areas, and reward functions include high scores for successfully avoiding collisions and rewards for efficiently reaching the destination. Behavioral data and navigation paths are processed based on standard environmental data and historical environmental data to determine the navigation strategy. User behavior data is processed based on a pre-defined parallel interactive multi-model Kalman filter algorithm and motion patterns to predict the user's position and direction of movement in the next unit of time. The unit of time is the prediction interval for the accompanying robot. Based on the navigation strategy and the user's position and direction of movement in the next unit of time, motion commands for the accompanying robot are calculated. Motion commands include machine speed, machine acceleration, and machine turning angle.

[0015] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for intelligent prediction of accompanying robots based on sensors, applied to accompanying robots and wearable devices. The accompanying robot is equipped with a camera unit, a lidar, and a depth sensor, while the wearable device is equipped with an inertial measurement unit storing computer-executable instructions. The computer-executable instructions are configured to: collect user behavior data based on the wearable device; collect user visual data based on the camera unit; and determine environmental data around the accompanying robot based on the lidar and depth sensor. The behavior data includes acceleration and angular velocity, and the environmental data includes at least pedestrian position, pedestrian speed, and pedestrian distance. The behavior data, image data, and environmental data are filtered and standardized using a preset Kalman filter algorithm to generate standard data. The standard data includes standard behavior data, standard visual data, and standard environmental data. The standard data is processed using a preset action recognition algorithm to determine... User behavior; wherein user behavior includes at least one of the following data: standing, walking, turning, running; constructing a digital model based on standard environmental data; processing user behavior and the digital model based on preset goals and reward functions to determine the navigation path; wherein the goals include safely following the user, avoiding obstacles and densely populated areas, and the reward functions include high scores for successfully avoiding collisions and rewards for efficiently reaching the destination; processing behavior data and navigation paths based on standard environmental data and historical environmental data of standard environmental data to determine the navigation strategy; processing user behavior data based on preset parallel interactive multi-model Kalman filter algorithm and motion patterns to predict the user's position and direction of movement in the next unit of time; wherein the unit of time is the prediction interval of the accompanying robot; calculating the motion commands of the accompanying robot based on the navigation strategy and the user's position and direction of movement in the next unit of time; wherein the motion commands include: machine speed, machine acceleration, and machine turning angle.

[0016] The sensor-based intelligent prediction method, device, and medium for accompanying robots provided in this application have at least the following technical effects:

[0017] By integrating data from multiple sensors, accurate collection of user behavior and environmental data is achieved to a certain extent, thereby improving data reliability and diversity. Adaptive Kalman filtering and feature extraction algorithms are used to filter and standardize the data to generate standard data; transfer learning algorithms are used to optimize the action recognition model, accurately identifying user behavior and enhancing the robot's understanding of user behavior. The construction of digital models and the application of path planning algorithms enable the robot to intelligently plan navigation paths, effectively avoiding obstacles and densely populated areas to a certain extent. Parallel interactive multi-model Kalman filtering algorithms are used to predict the user's position and direction of movement in the next unit of time; by combining navigation strategies and user prediction information, motion commands that meet constraints are calculated, ensuring the robot's safe and efficient following to a certain extent. In summary, this application can effectively integrate data from multiple sensors, improve the accuracy of user behavior recognition, and formulate flexible and efficient navigation strategies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A flowchart of a sensor-based intelligent prediction method for accompanying robots provided in this application embodiment;

[0020] Figure 2 This is a schematic diagram of the internal structure of a sensor-based intelligent prediction device for a companion robot, provided as an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a sensor-based intelligent prediction method, device, and medium for accompanying robots, which aims to solve the following technical problems: how to effectively integrate data from multiple sensors to improve the accuracy of user behavior recognition and to formulate flexible and efficient navigation strategies.

[0023] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1This document provides a flowchart of a sensor-based intelligent prediction process for a companion robot, as illustrated in an embodiment of this application. Figure 1 As shown in the figure, the sensor-based intelligent prediction method for accompanying robots provided in this application specifically includes the following steps:

[0025] The embodiments of this application are applied to accompanying robots and wearable devices. The accompanying robot is equipped with a camera unit, a lidar and a depth sensor, and the wearable device is equipped with an inertial measurement unit.

[0026] In this embodiment of the application, the accompanying robot is used to follow the user to achieve the effects of accompanying the elderly and helping to place goods, etc., and the wearable device is used to collect the user's action information, that is, to identify the user's actions.

[0027] The accompanying robot is equipped with a camera unit, lidar, and depth sensor.

[0028] The camera unit is mounted on the head of the accompanying robot to capture real-time image information of the user's facial features, body posture, and surrounding environment.

[0029] The lidar is mounted on the side of the accompanying robot to scan the surrounding environment and generate a 3D point cloud map.

[0030] Depth sensors work in conjunction with lidar to supplement lidar's limited detection capabilities at short distances.

[0031] An inertial measurement unit is built into a wearable device (such as a smartwatch or smart bracelet; in this embodiment, the wearable device is an ankle bracelet or leg wrap) and includes an accelerometer and a gyroscope.

[0032] Accelerometers are used to measure a user’s linear acceleration in three-dimensional space, reflecting the user’s walking speed and running rhythm.

[0033] Gyroscopes are used to measure the rotational motion of a user's body or limbs, i.e., angular velocity, and to identify the user's turning intentions and changes in body posture.

[0034] Step 1: Collect user behavior data based on wearable devices, collect user visual data using camera units, and determine environmental data around the accompanying robot using LiDAR and depth sensors; among which, behavior data includes acceleration and angular velocity, and environmental data includes at least pedestrian position, pedestrian speed, and pedestrian distance.

[0035] In an embodiment of this application relating to step 1: the companion robot is used to follow the elderly person to help them load items and to keep them company.

[0036] The elderly wear an ankle bracelet (wearable device), which connects to the accompanying robot via Bluetooth. The wearable device collects the elderly’s behavioral data through its built-in inertial measurement unit. The accompanying robot follows the elderly 2 meters behind and uses a camera (video unit) to identify the elderly’s body posture (visual data). The accompanying robot uses lidar and depth sensors to identify the position, speed, distance of people around the accompanying robot, and obstacle information.

[0037] Step 2: Filter and standardize the behavioral data, image data, and environmental data to generate standard data; wherein, the standard data includes standard behavioral data, standard visual data, and standard environmental data.

[0038] Step 21: Process the behavioral data based on the preset adaptive Kalman filter algorithm, and adjust the filtering parameters of the adaptive Kalman filter algorithm according to the dynamic change characteristics of the behavioral data to generate preliminary standard behavioral data.

[0039] The adaptive Kalman filter algorithm is a Kalman filter algorithm that can automatically adjust the filtering parameters according to the dynamic changes in the data.

[0040] First, initialize the parameters of the adaptive Kalman filter algorithm.

[0041] Furthermore, the acceleration and angular velocity output by the inertial measurement unit in the wearable device are collected in real time.

[0042] Furthermore, the collected acceleration and angular velocity are input as measurements into the adaptive Kalman filter algorithm for state estimation and error update.

[0043] Furthermore, the filtering parameters of the adaptive Kalman filter algorithm are adjusted in real time based on the dynamic characteristics of the behavioral data.

[0044] Furthermore, preliminary standard behavioral data is output.

[0045] Step 22: Extract feature vectors from image data based on a preset feature extraction algorithm to generate preliminary standard visual data.

[0046] First, image data output from the camera unit on the accompanying robot is acquired in real time.

[0047] Furthermore, the acquired image data is preprocessed to reduce noise and enhance contrast.

[0048] Furthermore, the scale-invariant feature transform algorithm is applied to extract features from the preprocessed image data.

[0049] Furthermore, preliminary standard visual data is output.

[0050] Step 23: Process the environmental data based on adaptive Kalman filtering to generate preliminary standard environmental data.

[0051] First, environmental data output from the lidar and depth sensor on the accompanying robot is collected in real time. Then, the collected environmental data is used as measurement values ​​and input into an adaptive Kalman filter algorithm to output preliminary standard environmental data. This is the same as step 21 and will not be repeated here.

[0052] Step 24: Normalize the preliminary standard behavioral data, preliminary standard visual data, and preliminary standard environmental data to the same data range to generate standard data.

[0053] First, a data range analysis was conducted on the preliminary standard behavioral data, preliminary standard visual data, and preliminary standard environmental data to determine the data range.

[0054] Furthermore, select a data range as the target range, which can accommodate the variation range of all preliminary standard data.

[0055] Furthermore, linear normalization is applied to normalize the preliminary standard behavioral data, preliminary standard visual data, and preliminary standard environmental data to the target range, and then standard data is output.

[0056] Step 3: Process standard data based on a preset action recognition algorithm to determine the user's behavior; wherein, the user behavior includes at least one of the following: standing, walking, turning around, running.

[0057] Step 31: Construct a preliminary action recognition model; wherein, the preliminary action recognition model is a neural network model obtained by training based on user behavior data.

[0058] First, user behavior data is collected as training samples, including various behaviors such as standing, walking, turning, and running. These training samples are used to train a preliminary, universally applicable motion recognition model. For example, images of people standing and data on the human body while standing are obtained from the internet.

[0059] Furthermore, the collected user behavior data is preprocessed, such as denoising and normalization, to improve the accuracy and efficiency of model training.

[0060] Furthermore, the model is designed based on the characteristics of user behavior data.

[0061] Furthermore, the neural network is trained using preprocessed user behavior data. By adjusting network parameters and optimizing the algorithm, the model can accurately identify different user behaviors.

[0062] Furthermore, after training, a preliminary action recognition model is obtained, which can initially identify the user's standing, walking, turning, running and other behaviors.

[0063] Understandably, this preliminary action recognition model is a universally applicable model.

[0064] Step 32: Train a preliminary action recognition model based on historical data of behavior data processed by a preset transfer learning algorithm to obtain the action recognition model; wherein, the action recognition model is equipped with an action recognition algorithm.

[0065] First, collect historical data on user behavior.

[0066] Furthermore, by applying transfer learning algorithms, the knowledge in the initial action recognition model is transferred to historical data, and the model is further trained by combining the current training data and historical data.

[0067] Furthermore, during the training process, the parameters and optimization strategies of the transfer learning algorithm are adjusted to fully utilize information from historical data and improve the model's generalization ability and recognition accuracy.

[0068] Furthermore, after training, an action recognition model is obtained.

[0069] Step 33: Input the standard data into the action recognition model to determine the user's behavior.

[0070] First, standard user data is collected in real time.

[0071] Furthermore, the collected standard data is input into the action recognition model as input data for the action recognition model.

[0072] Furthermore, the action recognition model processes the input standard data and determines the user's current behavior through forward propagation of the neural network and calculation by the action recognition algorithm.

[0073] Furthermore, it outputs the user's behavior.

[0074] Step 4: Construct a digital model based on standard environmental data.

[0075] Step 41: Map the standard environmental data to a preset three-dimensional spatial coordinate system to construct an initial environmental model; wherein, the three-dimensional spatial coordinate system is divided into multiple voxels.

[0076] First, standard environmental data is determined, which has been generated in step 2.

[0077] Furthermore, a suitable three-dimensional spatial coordinate system is set, and the origin, coordinate axis directions, and unit length of the coordinate system are determined. This process is done manually.

[0078] Furthermore, each object or region in the standard environmental data is mapped to its corresponding position in a three-dimensional spatial coordinate system.

[0079] Furthermore, based on the preset voxel size, the three-dimensional space is divided into multiple voxels, each voxel representing a small region in the space.

[0080] Furthermore, based on the mapped data and voxel segmentation results, an initial environment model is constructed. This model only contains the spatial location and basic geometry of objects, but has not yet been assigned environmental attributes.

[0081] Step 42: Based on standard environmental data processing, an initial environmental model is created by filling multiple voxels with environmental attributes to construct a digital model; the environmental attributes include walkability, population density, and obstacles.

[0082] First, based on actual needs, define the types of environmental attributes that need to be filled, such as walkable, densely populated, and obstacles, and set corresponding value ranges or standards for each attribute.

[0083] Furthermore, standard environmental data is analyzed to extract environmental attribute information related to each voxel. For example, terrain data is analyzed to determine whether a voxel is walkable; population flow data is used to determine the population density of a voxel; and obstacle data is used to identify whether there are obstacles within a voxel.

[0084] Furthermore, the environmental attribute information obtained from the analysis is filled into the corresponding voxels in the initial environmental model, and each voxel is assigned a specific attribute label.

[0085] Furthermore, after attribute filling, the final digital model is obtained.

[0086] Step 5: Process user behavior and digital models based on preset goals and reward functions to determine the navigation path to choose; where the goals include safely following the user, avoiding obstacles and densely populated areas, and the reward functions include getting a high score for successfully avoiding collisions and getting a reward for efficiently reaching the destination.

[0087] Step 51: Starting from the accompanying robot, determine the target point based on the user and preset distance.

[0088] First, before navigation begins, the current position of the accompanying robot is used as the starting point for navigation.

[0089] Furthermore, the user's location information can be obtained in real time through sensors (such as camera units and infrared sensors).

[0090] Furthermore, a safe distance threshold is set, and the distance between the user's location and the accompanying robot's current location is calculated.

[0091] If the distance exceeds the safe distance threshold, a point a certain distance in front of the user's current location (such as 1 meter or 2 meters, which can be adjusted according to actual needs) will be used as the target point; if the distance is within the safe distance range, the current following state will be maintained, and no specific target point will be set or a fine-tuned target point will be set to maintain an appropriate distance.

[0092] Step 52: Determine the objectives and construct a reward function based on them; the objectives include safely following the user, avoiding obstacles, and avoiding densely populated areas, and the reward function is set with different weights.

[0093] First, clarify the goals to be achieved during navigation, namely, safely following the user and avoiding obstacles and densely populated areas.

[0094] Furthermore, construct the reward function:

[0095] Set up reward items related to user distance, such as a positive reward for maintaining an appropriate distance and a negative reward for being too close or too far away; a negative reward for a collision and a positive reward for successfully avoiding one; a negative reward for entering a crowded area and a positive reward for avoiding one.

[0096] Furthermore, different weights can be assigned to each item in the reward function based on actual needs. For example, safely following the user has a higher weight because it is a basic requirement for navigation; while the weight of avoiding densely populated areas may be adjusted depending on the scenario, such as in a shopping mall, where avoiding densely populated areas has a lower weight.

[0097] Step 53: Process the starting point, target point, and digital model based on the preset path planning algorithm to determine multiple preliminary navigation paths.

[0098] Choose a suitable path planning algorithm according to actual needs. The path planning algorithm used in this embodiment is the A* algorithm.

[0099] First, the starting point and target point determined in step 51 are used as inputs to the path planning algorithm.

[0100] Furthermore, the digital model constructed in step 4 is used as the environmental information input for the path planning algorithm.

[0101] Furthermore, a path planning algorithm (such as the A* algorithm) is used to calculate multiple preliminary navigation paths from the starting point to the target point.

[0102] Step 54: Evaluate multiple preliminary navigation paths based on the weights of the reward function to determine the navigation path.

[0103] First, for each initial navigation path, its reward value is calculated based on the reward function. The higher the reward value, the more the path meets the target requirements.

[0104] Furthermore, the reward values ​​of each initial navigation path are compared to identify the path with the highest reward value.

[0105] Furthermore, the path with the highest reward value is used as the final navigation path for the accompanying robot to execute.

[0106] In this embodiment, the accompanying robot starts at the mall entrance and determines its starting position using a positioning system. The location information of the customer (user) is acquired in real time via a camera. Based on the customer's position and the accompanying robot's current position, the distance between them is calculated, and a safe distance threshold of 1.5 meters is set. If the distance exceeds the threshold, a point 1.5 meters in front of the customer is taken as the target point; if the distance is within the safe range, the current following state is maintained.

[0107] Define navigation objectives: safely follow customers, avoid obstacles (such as shelves and display cases), and avoid densely populated areas (such as promotional areas and rest areas).

[0108] Construct the reward function:

[0109] Safely follow customers: +10 points / second for maintaining an appropriate distance, -5 points / second for being too close or too far away.

[0110] Avoiding obstacles: Collision results in a -20 score, while successful avoidance results in a +5 score.

[0111] Avoid crowded areas: Entering a crowded area earns -10 minutes / second, while avoiding it earns +2 minutes / second.

[0112] Set the weights: Safely follow customers with a weight of 1.0, avoid obstacles with a weight of 0.8, and avoid densely populated areas with a weight of 0.6 (considering that there may be many densely populated areas in the mall, the weights should be appropriately reduced to avoid excessive avoidance. The weights can be set according to the actual situation).

[0113] The starting point, the target point, and the digital model (including the mall layout, obstacle locations, and crowd density) are used as inputs to the A* algorithm.

[0114] Run the A* algorithm to calculate multiple preliminary navigation paths from the starting point to the target point.

[0115] For each initial navigation path, its reward value is calculated based on the reward function.

[0116] Compare the reward values ​​of each path and find the path with the highest reward value.

[0117] The path with the highest reward value is used as the final navigation path for the accompanying robot to execute.

[0118] Step 6: Process behavioral data and navigation paths based on standard environmental data and historical environmental data of standard environmental data to determine navigation strategies.

[0119] Step 61: Search the preset historical environment database based on standard environment data to determine whether the accompanying robot has historical environment data based on standard environment data.

[0120] First, obtain the standard environmental data for the current environment according to step 2.

[0121] Furthermore, the acquired standard environmental data is used as a query condition to search a preset historical environmental database to find whether there is historical environmental data similar to the current environment.

[0122] Furthermore, if similar historical environmental data is found, proceed to step 63; otherwise, proceed to step 62.

[0123] Step 62: If not, use navigation path as the navigation strategy.

[0124] First, in the absence of similar historical environmental data, the accompanying robot will rely on the current navigation path as its navigation strategy.

[0125] Furthermore, the robot moves along the navigation path while simultaneously updating environmental data in real time.

[0126] Step 63: If yes, obtain the current time of the search companion robot and search the historical database based on the current time to determine whether there is historical environmental data at the same time.

[0127] First, the current time is obtained through the robot's internal clock or an external time synchronization system.

[0128] Furthermore, by using the current time as a query condition, the historical environment database is searched to find whether there is historical environment data that is the same as or similar to the current time.

[0129] Furthermore, if historical environmental data at the same time is found, proceed to step 65; otherwise, proceed to step 64.

[0130] Step 64: If not, use navigation path as the navigation strategy.

[0131] The same as step 62, so I will not repeat it here.

[0132] Step 65: If so, evaluate the navigation path and behavioral data based on historical environmental data to generate an evaluation result, which includes potential obstacles.

[0133] First, extract historical environmental data that is the same as or close to the current time from the historical environmental database.

[0134] Furthermore, historical environmental data, current navigation paths, and behavioral data are compared to consider the impact of possible changes in the historical environment on the navigation path. The specific consideration process is illustrated in the embodiments of this application.

[0135] Furthermore, based on the results of the comprehensive analysis, an assessment report is generated, which clearly identifies potential obstacles and risk points.

[0136] Step 66: Modify the navigation path based on the evaluation results to determine the navigation strategy.

[0137] First, carefully review the assessment report to understand the specific location and extent of impact of potential obstacles and risks.

[0138] Furthermore, based on the evaluation results, necessary adjustments are made to the original navigation path, such as changing the movement route, adjusting the movement speed, and adding obstacle avoidance actions.

[0139] Furthermore, the adjusted navigation path will be used as a new navigation strategy to guide the accompanying robot in navigation.

[0140] In the embodiments of this application:

[0141] As the robot is activated within the industrial park, it acquires standard environmental data of the current environment, including obstacle locations, road widths, and crowd density.

[0142] The acquired standard environmental data is used as the query criteria to search the preset historical environmental database to find if similar historical environmental data exists.

[0143] (If no similar historical environmental data is found)

[0144] If no similar historical environment data is found in the historical environment database, the accompanying robot will rely on the current navigation path for navigation.

[0145] The navigation path is calculated by the path planning algorithm (A* algorithm), which plans an optimal path from the starting point to the target point.

[0146] If similar historical environmental data is found in the historical environmental database, the accompanying robot will obtain the current time and further search the historical database based on the current time. It will then look for historical environmental data with the same or similar time to the current time to account for the impact of time on the navigation path.

[0147] (If no historical environmental data for the same moment is found) If no historical environmental data for the same moment is found in the historical database, the accompanying robot will still rely on the current navigation path for navigation.

[0148] If historical environmental data from the same moment is found in the historical database, the robot extracts this data and performs a comprehensive analysis with the current navigation path and behavior data.

[0149] The navigation path is evaluated, taking into account potential changes in the historical environment (such as temporary construction or road closures), and an evaluation result is generated. The evaluation result identifies potential obstacles and risk points, such as construction areas and areas with high population density.

[0150] Based on the evaluation results, the original navigation path is adjusted and optimized. For example, it avoids construction areas and bypasses areas with large crowds. The adjusted navigation path is used as the new navigation strategy to guide the accompanying robot.

[0151] Step 7: Process the user's behavior data based on the preset parallel interactive multi-model Kalman filter algorithm and motion pattern to predict the user's position and direction of movement in the next unit time; where the unit time is the prediction interval of the accompanying robot.

[0152] Step 71: Construct multiple motion models; each motion model corresponds to a user behavior, and the motion models include, but are not limited to, walking in a straight line at a constant speed, running at an accelerated speed, stopping at a deceleration speed, and turning around suddenly.

[0153] First, by analyzing users' historical behavior data and actual application scenarios, we can determine the main types of behaviors that users may take, such as walking in a straight line at a constant speed, running at a faster speed, slowing down and stopping, and turning around suddenly.

[0154] Furthermore, a corresponding motion model is constructed for each type of user behavior. For example, the uniform straight-line walking model can describe the state of a user moving along a straight line at a constant speed; the accelerating running model can describe the process of a user gradually accelerating while running; the decelerating stopping model can describe the state of a user gradually decelerating until stopping; and the sudden turning model can describe the behavior of a user suddenly changing the direction of movement.

[0155] Furthermore, parameters are set for each motion model, including initial position, velocity, and acceleration.

[0156] Step 72: Process multiple motion models based on the Kalman filter algorithm, and update the predicted state of multiple motion models in real time with the user's historical behavior data; wherein, the predicted state includes position, velocity and acceleration.

[0157] First, based on the user's historical behavior data, an initial state is set for each motion model.

[0158] Furthermore, as user behavior continues, new observation data (such as user position and velocity) is constantly acquired. Using the Kalman filter algorithm, combined with the observation data and the dynamic equations of the motion model, the predicted state of each motion model is updated in real time.

[0159] Step 73: Process the user's behavior data based on the preset motion pattern recognition algorithm to evaluate the matching degree of multiple motion models.

[0160] First, extract key features from user behavior data, such as the rate of change of position, the rate of change of velocity, and the rate of change of acceleration.

[0161] Furthermore, the extracted features are input into a preset motion pattern recognition algorithm, which then uses these features to determine what kind of behavior the user is currently engaging in.

[0162] Furthermore, for each motion model, its degree of matching with the user's current behavior is calculated. The degree of matching can be determined by comparing the difference between the predicted state of the motion model and the user's actual behavior data.

[0163] Step 74: Process the behavioral data based on the motion model with the highest matching degree to determine the initial prediction.

[0164] First, based on the matching degree evaluation results, the motion model with the highest matching degree is selected as the best model.

[0165] Furthermore, the optimal model is used to process the user's behavioral data to obtain preliminary predictions for the user's next unit of time, including position and direction of movement.

[0166] Step 75: Calculate the user's position and direction of movement for the next unit of time based on the preliminary prediction and the unit time.

[0167] First, determine the appropriate unit time length based on the actual application requirements and the performance of the accompanying robot.

[0168] Furthermore, based on the preliminary prediction results and the unit time, the user's specific position and direction of movement in the next unit time are calculated.

[0169] Furthermore, the calculated future state is output as a prediction result for the accompanying robot to make subsequent decisions and take actions.

[0170] Step 8: Based on the navigation strategy and the user's position and direction of movement in the next unit of time, calculate the motion commands of the accompanying robot; wherein, the motion commands include: machine speed, machine acceleration and machine turning angle.

[0171] Step 81: Based on the waypoints determined by the navigation strategy and the user's predicted position in the next unit of time, calculate the target position and direction that the accompanying robot needs to reach.

[0172] First, a series of waypoints are extracted from the preset navigation strategy. These waypoints represent the robot's expected trajectory.

[0173] Furthermore, user behavior prediction algorithms are used to calculate the predicted position of the user in the next unit of time.

[0174] Furthermore, based on the user's predicted location and navigation path points, the target location that the robot needs to reach is calculated. This is typically achieved by selecting the path point closest to the user's predicted location or by interpolating based on the distribution of path points and the user's movement trends.

[0175] Furthermore, based on the target location and the robot's current position, the target direction that the robot needs to face is calculated. This can be obtained by calculating the direction vector between the two points or by using more complex path planning algorithms.

[0176] Step 82: Determine the constraints of the accompanying robot; the constraints include maximum speed, maximum acceleration, steering angle range, and motion smoothness requirements.

[0177] First, based on the robot's hardware specifications and motion capabilities, determine its maximum speed, maximum acceleration, and turning angle range.

[0178] Furthermore, to ensure user comfort and safety, requirements are set for the smoothness of the robot's movements, such as limiting the rate of change of acceleration and avoiding sharp turns.

[0179] Furthermore, the constraints on the robot are determined by comprehensively considering both the robot's performance and user experience requirements.

[0180] Step 83: Process the current state of the accompanying robot based on the target position and orientation to generate preliminary motion commands.

[0181] First, the robot's current position, velocity, acceleration, and direction of motion are obtained through its sensor system.

[0182] Furthermore, based on the target position and orientation, as well as the robot's current state, preliminary motion parameters such as the robot's speed, acceleration, and steering angle that need to be adjusted are calculated. This can be achieved using a proportional-integral-derivative (PID) control algorithm or other motion control algorithms.

[0183] Furthermore, the calculated preliminary motion parameters are converted into motion command formats that the robot can understand, such as motor control signals and servo motor rotation angles.

[0184] Step 84: Process the initial motion commands based on the constraints to determine the motion commands.

[0185] First, the initial motion commands are compared with the robot's constraints to check whether they exceed the maximum speed, maximum acceleration, steering angle range, or violate motion smoothness requirements.

[0186] Furthermore, if the initial motion command violates the constraints, it is adjusted. For example, if the initial velocity exceeds the maximum speed limit, it is reduced to the maximum speed; if the initial acceleration changes too rapidly, it is smoothed out.

[0187] Furthermore, the adjusted motion instructions are sent to the robot's control system as the actual motion control instructions executed by the robot to perform the corresponding motion.

[0188] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a sensor-based intelligent prediction device for accompanying robots, the structure of which is as follows: Figure 2 As shown.

[0189] Figure 2 This is a schematic diagram of the internal structure of a sensor-based intelligent prediction device for a companion robot, provided as an embodiment of this application. Figure 2 As shown, the device includes:

[0190] At least one processor 201;

[0191] And a memory 202 that is communicatively connected to at least one processor;

[0192] The memory 202 stores instructions executable by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to:

[0193] Wearable devices collect user behavior data, camera units collect user visual data, and LiDAR and depth sensors determine environmental data around the accompanying robot. Behavioral data includes acceleration and angular velocity, while environmental data includes at least pedestrian position, speed, and distance. A pre-defined Kalman filter algorithm is used to filter and standardize the behavioral, visual, and environmental data to generate standard data. This standard data includes standard behavioral, visual, and environmental data. A pre-defined action recognition algorithm processes the standard data to determine user behavior. User behavior includes at least one of the following: standing, walking, turning, or running. A digital model is constructed based on the standard environmental data. Based on a pre-defined target and... The reward function processes user behavior and digital models to determine the navigation path. Objectives include safely following the user, avoiding obstacles and densely populated areas. Rewards include high scores for successfully avoiding collisions and rewards for efficiently reaching the destination. Behavioral data and navigation paths are processed based on standard environmental data and historical environmental data to determine the navigation strategy. User behavior data is processed based on a pre-defined parallel interactive multi-model Kalman filter algorithm and motion patterns to predict the user's position and direction of movement in the next unit of time. The unit of time is the prediction interval for the accompanying robot. Based on the navigation strategy and the user's position and direction of movement in the next unit of time, motion commands for the accompanying robot are calculated. These commands include robot speed, robot acceleration, and robot turning angle.

[0194] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for intelligent prediction of accompanying robots based on sensors, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0195] Wearable devices collect user behavior data, camera units collect user visual data, and LiDAR and depth sensors determine environmental data around the accompanying robot. Behavioral data includes acceleration and angular velocity, while environmental data includes at least pedestrian position, speed, and distance. A pre-defined Kalman filter algorithm is used to filter and standardize the behavioral, visual, and environmental data to generate standard data. This standard data includes standard behavioral, visual, and environmental data. A pre-defined action recognition algorithm processes the standard data to determine user behavior. User behavior includes at least one of the following: standing, walking, turning, or running. A digital model is constructed based on the standard environmental data. Based on a pre-defined target and... The reward function processes user behavior and digital models to determine the navigation path. Objectives include safely following the user, avoiding obstacles and densely populated areas. Rewards include high scores for successfully avoiding collisions and rewards for efficiently reaching the destination. Behavioral data and navigation paths are processed based on standard environmental data and historical environmental data to determine the navigation strategy. User behavior data is processed based on a pre-defined parallel interactive multi-model Kalman filter algorithm and motion patterns to predict the user's position and direction of movement in the next unit of time. The unit of time is the prediction interval for the accompanying robot. Based on the navigation strategy and the user's position and direction of movement in the next unit of time, motion commands for the accompanying robot are calculated. These commands include robot speed, robot acceleration, and robot turning angle.

[0196] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0197] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0198] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0200] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0201] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0202] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0203] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0204] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0205] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0206] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A sensor-based intelligent prediction method for accompanying robots, applied to accompanying robots and wearable devices, wherein the accompanying robot is equipped with a camera unit, a lidar, and a depth sensor, and the wearable device is equipped with an inertial measurement unit, characterized in that, The method includes: The wearable device collects user behavior data, the camera unit collects user visual data, and the lidar and depth sensor determine environmental data around the accompanying robot; wherein, the behavior data includes acceleration and angular velocity, and the environmental data includes at least pedestrian position, pedestrian speed, pedestrian distance, and obstacles; The behavioral data, image data, and environmental data are filtered and standardized to generate standard data; wherein, the standard data includes standard behavioral data, standard visual data, and standard environmental data. Construct a preliminary action recognition model; wherein, the preliminary action recognition model is a neural network model obtained by training based on user behavior data; The preliminary action recognition model is trained based on a preset transfer learning algorithm and historical data of the behavior data to obtain the action recognition model; wherein, the action recognition model is equipped with an action recognition algorithm. The standard data is input into the action recognition model to determine the user's behavior; wherein the user behavior includes at least one of the following: standing, walking, turning, running; A digital model is constructed based on the aforementioned standard environmental data; The user behavior and the digital model are processed based on preset goals and reward functions to determine the navigation path; wherein, the goals include safely following the user, avoiding obstacles and densely populated areas, and the reward functions include getting a high score for successfully avoiding collisions and getting a reward for efficiently reaching the destination; The behavioral data and navigation path are processed based on the standard environmental data and the historical environmental data of the standard environmental data to determine the navigation strategy; Construct multiple motion models; each motion model corresponds to a user behavior, and the motion models include, but are not limited to, constant speed straight walking, accelerated running, deceleration to stop, and sudden turning. The Kalman filter algorithm is used to process multiple motion models, and the predicted state of the multiple motion models is updated in real time using the user's historical behavior data; wherein, the predicted state includes position, velocity and acceleration; The user's behavior data is processed based on a preset motion pattern recognition algorithm to evaluate the matching degree of multiple motion models; The behavioral data is processed based on the motion model with the highest matching degree to determine a preliminary prediction; The user's position and direction of movement for the next unit of time are calculated based on the preliminary prediction and the unit of time; wherein, the unit of time is the prediction interval of the accompanying robot; Based on the navigation strategy and the user's position and direction of movement in the next unit of time, the motion commands of the accompanying robot are calculated; wherein, the motion commands include: machine speed, machine acceleration, and machine turning angle.

2. The sensor-based intelligent prediction method for accompanying robots according to claim 1, characterized in that, The behavioral data, image data, and environmental data are filtered and standardized to generate standard data, specifically including: The behavioral data is processed based on a preset adaptive Kalman filter algorithm, and the filtering parameters of the adaptive Kalman filter algorithm are adjusted according to the dynamic change characteristics of the behavioral data to generate preliminary standard behavioral data. The feature vectors of the image data are extracted based on a preset feature extraction algorithm to generate preliminary standard visual data; The environmental data is processed based on the adaptive Kalman filter to generate preliminary standard environmental data; The preliminary standard behavioral data, preliminary standard visual data, and preliminary standard environmental data are normalized to the same data range to generate the standard data.

3. The sensor-based intelligent prediction method for accompanying robots according to claim 1, characterized in that, Constructing a digital model based on the aforementioned standard environmental data specifically includes: The standard environmental data is mapped to a preset three-dimensional spatial coordinate system to construct an initial environmental model; wherein the three-dimensional spatial coordinate system is divided into multiple voxels; The initial environmental model is processed based on the standard environmental data to populate environmental attributes on multiple voxels to construct a digital model; wherein the environmental attributes include walkability, population density, and obstacles.

4. The sensor-based intelligent prediction method for accompanying robots according to claim 1, characterized in that, Based on a preset objective and a reward function, the user behavior and the digital model are processed to determine the navigation path, specifically including: Starting from the accompanying robot, the target point is determined based on the user and a preset distance. Define the objectives and construct a reward function based on the objectives; wherein the objectives include safely following the user, avoiding obstacles, and avoiding densely populated areas, and the reward function is assigned different weights; The starting point, target point, and digital model are processed based on a preset path planning algorithm to determine multiple preliminary navigation paths; Multiple preliminary navigation paths are evaluated based on the weights of the reward function to determine a navigation path.

5. The sensor-based intelligent prediction method for accompanying robots according to claim 1, characterized in that, Based on the standard environmental data and its historical environmental data, the behavioral data and navigation path are processed to determine the navigation strategy, specifically including: Based on the standard environmental data, a preset historical environmental database is retrieved to determine whether the accompanying robot has historical environmental data of the standard environmental data. If not, the navigation strategy is based on the navigation path; If so, obtain the current time of the search companion robot, and search the historical database based on the current time to determine whether there is historical environmental data at the same time. If not, the navigation strategy is based on the navigation path; If so, the navigation path and behavioral data are evaluated based on the historical environmental data to generate an evaluation result; wherein the evaluation result includes potential obstacles; The navigation path is modified based on the evaluation results to determine the navigation strategy.

6. The sensor-based intelligent prediction method for accompanying robots according to claim 1, characterized in that, Based on the navigation strategy and the user's position and direction of movement in the next unit of time, the motion commands of the accompanying robot are calculated, specifically including: Based on the path points determined by the navigation strategy and the user's predicted position in the next unit of time, the target position and direction that the accompanying robot needs to reach are calculated. Determine the constraints of the accompanying robot; wherein the constraints include maximum speed, maximum acceleration, steering angle range, and motion smoothness requirements; The current state of the accompanying robot is processed based on the target position and orientation to generate preliminary motion commands; The initial motion command is processed based on the constraints to determine the motion command.

7. A sensor-based intelligent prediction device for accompanying robots, applied to accompanying robots and wearable devices, wherein the accompanying robot is equipped with a camera unit, a lidar, and a depth sensor, and the wearable device is equipped with an inertial measurement unit, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The wearable device collects user behavior data, the camera unit collects user visual data, and the lidar and depth sensor determine environmental data around the accompanying robot; wherein, the behavior data includes acceleration and angular velocity, and the environmental data includes at least pedestrian position, pedestrian speed, pedestrian distance, and obstacles; The behavioral data, image data, and environmental data are filtered and standardized based on a preset Kalman filter algorithm to generate standard data; wherein, the standard data includes standard behavioral data, standard visual data, and standard environmental data. Construct a preliminary action recognition model; wherein, the preliminary action recognition model is a neural network model obtained by training based on user behavior data; The preliminary action recognition model is trained based on a preset transfer learning algorithm and historical data of the behavior data to obtain the action recognition model; wherein, the action recognition model is equipped with an action recognition algorithm. The standard data is input into the action recognition model to determine the user's behavior; wherein the user behavior includes at least one of the following: standing, walking, turning, running; A digital model is constructed based on the aforementioned standard environmental data; The user behavior and the digital model are processed based on preset goals and reward functions to determine the navigation path; wherein, the goals include safely following the user, avoiding obstacles and densely populated areas, and the reward functions include getting a high score for successfully avoiding collisions and getting a reward for efficiently reaching the destination; The behavioral data and navigation path are processed based on the standard environmental data and the historical environmental data of the standard environmental data to determine the navigation strategy; Construct multiple motion models; each motion model corresponds to a user behavior, and the motion models include, but are not limited to, constant speed straight walking, accelerated running, deceleration to stop, and sudden turning. The Kalman filter algorithm is used to process multiple motion models, and the predicted state of the multiple motion models is updated in real time using the user's historical behavior data; wherein, the predicted state includes position, velocity and acceleration; The user's behavior data is processed based on a preset motion pattern recognition algorithm to evaluate the matching degree of multiple motion models; The behavioral data is processed based on the motion model with the highest matching degree to determine a preliminary prediction; The user's position and direction of movement for the next unit of time are calculated based on the preliminary prediction and the unit of time; wherein, the unit of time is the prediction interval of the accompanying robot; Based on the navigation strategy and the user's position and direction of movement in the next unit of time, the motion commands of the accompanying robot are calculated; wherein, the motion commands include: machine speed, machine acceleration, and machine turning angle.

8. A non-volatile computer storage medium for intelligent prediction of accompanying robots based on sensors, storing computer-executable instructions, applied to accompanying robots and wearable devices, wherein the accompanying robot is equipped with a camera unit, a lidar, and a depth sensor, and the wearable device is equipped with an inertial measurement unit, characterized in that, The computer-executable instructions are set as follows: The wearable device collects user behavior data, the camera unit collects user visual data, and the lidar and depth sensor determine environmental data around the accompanying robot; wherein, the behavior data includes acceleration and angular velocity, and the environmental data includes at least pedestrian position, pedestrian speed, pedestrian distance, and obstacles; The behavioral data, image data, and environmental data are filtered and standardized based on a preset Kalman filter algorithm to generate standard data; wherein, the standard data includes standard behavioral data, standard visual data, and standard environmental data. Construct a preliminary action recognition model; wherein, the preliminary action recognition model is a neural network model obtained by training based on user behavior data; The preliminary action recognition model is trained based on a preset transfer learning algorithm and historical data of the behavior data to obtain the action recognition model; wherein, the action recognition model is equipped with an action recognition algorithm. The standard data is input into the action recognition model to determine the user's behavior; wherein the user behavior includes at least one of the following: standing, walking, turning, running; A digital model is constructed based on the aforementioned standard environmental data; The user behavior and the digital model are processed based on preset goals and reward functions to determine the navigation path; wherein, the goals include safely following the user, avoiding obstacles and densely populated areas, and the reward functions include getting a high score for successfully avoiding collisions and getting a reward for efficiently reaching the destination; The behavioral data and navigation path are processed based on the standard environmental data and the historical environmental data of the standard environmental data to determine the navigation strategy; Construct multiple motion models; each motion model corresponds to a user behavior, and the motion models include, but are not limited to, constant speed straight walking, accelerated running, deceleration to stop, and sudden turning. The Kalman filter algorithm is used to process multiple motion models, and the predicted state of the multiple motion models is updated in real time using the user's historical behavior data; wherein, the predicted state includes position, velocity and acceleration; The user's behavior data is processed based on a preset motion pattern recognition algorithm to evaluate the matching degree of multiple motion models; The behavioral data is processed based on the motion model with the highest matching degree to determine a preliminary prediction; The user's position and direction of movement for the next unit of time are calculated based on the preliminary prediction and the unit of time; wherein, the unit of time is the prediction interval of the accompanying robot; Based on the navigation strategy and the user's position and direction of movement in the next unit of time, the motion commands of the accompanying robot are calculated; wherein, the motion commands include: machine speed, machine acceleration, and machine turning angle.

Images