Anti-falling method and device and robot

Abstract

The invention discloses an anti-falling method and device and a robot, and the method comprises the steps: obtaining multi-modal data for a scene where a target object is located in a process of following the target object; evaluating risk parameters of the target object based on the multi-modal data; the risk parameter is used for reflecting the falling risk of the target object; generating a behavior strategy of the robot based on the risk parameters; the behavior strategy is used for supporting the target object when the target object is about to fall down.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a fall prevention method, device and robot. Background Art

[0002] With the aging of the population and the increasing attention paid by society to the quality of life of the elderly, how to effectively ensure the safety and health of the elderly has become an urgent problem to be solved. In scenarios such as home-based elderly care and community care, the elderly frequently fall due to decreased physical function, the impact of disease or accidents, which not only seriously threatens the life and health of the elderly, but also brings a heavy burden to the family and society. Traditional anti-fall solutions use a single sensor technology, such as accelerometers, gyroscopes or infrared sensors. Although they can achieve instant detection of falls to a certain extent, the false alarm rate is high due to the limited data dimensions they rely on. In addition, traditional anti-fall solutions only have detection functions and lack the ability to actively intervene, and cannot provide timely assistance when the elderly are about to fall. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide a fall prevention method, device, robot, storage medium and program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] On the one hand, an embodiment of the present application provides a fall prevention method, which includes: in the process of following a target object, obtaining multimodal data for a scene where the target object is located; evaluating risk parameters of the target object based on the multimodal data; the risk parameters are used to reflect the fall risk of the target object; generating a behavior strategy of the robot based on the risk parameters; and the behavior strategy is used to support the target object when the target object is about to fall.

[0006] On the other hand, an embodiment of the present application provides an anti-fall device, which is applied to a robot, and the device includes: an acquisition module, which is used to acquire multimodal data for the scene where the target object is located during the process of following the target object; an evaluation module, which is used to evaluate the risk parameters of the target object based on the multimodal data; a generation module, which is used to generate a behavior strategy of the robot based on the risk parameters; the behavior strategy is used to support the target object when the target object is about to fall.

[0007] On the other hand, an embodiment of the present application provides a robot, comprising a multimodal sensor, a memory and a processor, wherein the multimodal sensor is used to collect real-time multimodal data; the memory stores a computer program that can be run on the processor; and the processor implements some or all of the steps in the above method when executing the program.

[0008] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements part or all of the steps in the above method when executed by a processor.

[0009] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method.

[0010] In the embodiment of the present application, by acquiring multimodal data in the process of following the target object, the information of the target object and the scene in which it is located can be understood in real time and comprehensively, including visual images, sounds, radar signals, infrared information, etc., which provides a rich data basis for subsequent risk assessment; at the same time, by evaluating the risk parameters of the target object based on these multimodal data, the risk of falling of the target object can be accurately reflected, providing a key basis for the robot's behavioral decision-making; further, the robot's behavioral strategy is generated based on the risk parameters, so that when the target object is about to fall, supporting measures can be taken quickly, thereby effectively preventing the occurrence of falling events.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0013] Figure 1 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 1 ;

[0014] Figure 2 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 2 ;

[0015] Figure 3 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 3 ;

[0016] Figure 4 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 4 ;

[0017] Figure 5 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 5 ;

[0018] Figure 6 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 6 ;

[0019] Figure 7 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 7 ;

[0020] Figure 8 A schematic diagram of the structure of an anti-fall device provided in an embodiment of the present application;

[0021] Fig. 9 A schematic diagram of a hardware entity of a robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0025] The embodiment of the present application provides a method for preventing falling, which can be executed by a processor of a robot. The robot can be a device with autonomous movement capability, sensor integration, and data processing capability. These devices can perform complex tasks, including but not limited to navigation, path planning, object recognition and manipulation, environmental perception, etc.

[0026] In some embodiments, the robot may be a device that not only has basic data processing capabilities, but also integrates sensors (such as cameras, laser radars, ultrasonic sensors, gyroscopes, accelerometers, etc.), actuators (such as motors, wheels, robotic arms, etc.), and algorithms and software for controlling and optimizing these components. Exemplarily, the robot may be a service robot (such as a household sweeping robot, a restaurant food delivery robot, a hospital nursing robot, etc.), an industrial robot (a robot that performs processing, assembly, and handling tasks with high repetitiveness and strict precision requirements on a production line), an exploration robot (for exploration and reconnaissance tasks in extreme environments or dangerous areas), and a mobile robot (self-driving cars, drones, etc.).

[0027] Figure 1 A schematic diagram of an implementation process of a fall prevention method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes the following steps S101 to S103:

[0028] Step S101: in the process of following a target object, obtaining multimodal data for a scene where the target object is located.

[0029] In some embodiments, the multimodal data refers to various types of information acquired by the robot through multimodal sensors (including different types of sensors) at the same time or scene, such as radar signals, infrared information, visual images, sounds, etc. In the embodiments of the present application, the multimodal data is integrated to more fully understand and respond to the environment. It is understandable that since the target object is in the environment, the multimodal data reflects not only the environmental information, but also the information of the target object.

[0030] In some embodiments, the target object is an entity that the robot needs to pay attention to, track, or interact with. For example, the target object can be an individual who needs to be monitored to prevent falling, such as an elderly person, a patient, or a worker in a specific working environment. The scene is the physical environment where the target object is located, and for example, the scene can include a home environment, a medical institution, a workplace, etc.

[0031] In some embodiments, in order to comprehensively and accurately obtain multimodal data of the scene where the target object is located, the robot can capture various information of the target object and the scene where it is located in real time by integrating multiple sensors and data acquisition devices, such as cameras, microphones, radars, infrared sensors, etc. In other embodiments, the above-mentioned multiple sensors and data acquisition devices can exist independently of the robot, and by establishing a communication channel between the robot and the sensors and data acquisition devices, various information of the target object and the scene where it is located can be obtained.

[0032] In some embodiments, while the robot is following a target object, all integrated sensors (and data acquisition devices) can be activated and data can be collected in real time, wherein the camera captures visual information of the target object, the microphone collects sound information, and the radar and infrared sensors provide position, speed, and depth information of the target object, thereby obtaining the multimodal data.

[0033] For example, in a home environment, the robot is following an elderly person as he walks. The camera on the robot captures the elderly person's posture data, the microphone collects the elderly person's voice and environmental noise, and the radar and infrared sensors monitor the elderly person's position and movement speed. All of these data are collected as multimodal data in real time and transmitted to the robot's processing unit.

[0034] Step S102: evaluating a risk parameter of the target object based on the multimodal data; the risk parameter is used to reflect the fall risk of the target object.

[0035] Among them, the risk parameter may include a quantitative value for reflecting the possibility of the target object (such as an elderly person) falling in the current state. The risk parameter is the result obtained by in-depth analysis and processing of multimodal data, which can provide a key basis for the robot's behavioral decision-making. It is understandable that the level of the risk parameter directly reflects the severity of the target object's risk of falling. Of course, the risk of falling refers to the possibility that the target object faces falling due to various reasons (such as decreased physical function, environmental factors, etc.). The present application can evaluate the risk of falling due to decreased physical function through the relevant information of the target object in the multimodal data, and can also evaluate the risk of falling due to environmental factors through the relevant information of the environment in the multimodal data.

[0036] In some embodiments, the target object's body contour and key point information can be extracted from the visual image in the multimodal data; a posture recognition algorithm is applied to analyze whether the target object's body posture is stable, such as whether there are signs of shaking or loss of balance; and based on the posture analysis results, the target object's risk of falling due to its own factors is assessed.

[0037] In some embodiments, the object recognition results in the visual image can be analyzed to identify risk factors in the environment that may cause falls, such as obstacles and slippery surfaces. By combining radar signals, infrared information, sound and other data, the distance and relative position of the target object and surrounding objects can be monitored to assess the risk of falling of the target object due to environmental factors.

[0038] In some embodiments, the risk parameter of the target object may be generated based on any one of the above-mentioned fall risks caused by personal factors and the fall risks caused by environmental factors, or a comprehensive risk parameter may be generated by combining the two fall risks.

[0039] Step S103: generating a behavior strategy of the robot based on the risk parameter; the behavior strategy is used to support the target object when the target object is about to fall.

[0040] In some embodiments, the behavior strategy may include a sequence of actions formulated by the robot based on risk parameters, including subtasks such as path planning, contact force control, and balance compensation. Generally speaking, the behavior strategy needs to meet real-time and safety constraints, such as quickly adjusting its posture and providing physical support when the target object loses balance.

[0041] In some embodiments, the process of the robot assisting the target object may include: the robot applies a controllable external force to the target object through an actuator such as a robotic arm, a trunk or an exoskeleton to restore the target object's balance.

[0042] In the embodiment of the present application, by acquiring multimodal data in the process of following the target object, the information of the target object and the scene in which it is located can be understood in real time and comprehensively, including visual images, sounds, radar signals, infrared information, etc., which provides a rich data basis for subsequent risk assessment; at the same time, by evaluating the risk parameters of the target object based on these multimodal data, the risk of falling of the target object can be accurately reflected, providing a key basis for the robot's behavioral decision-making; further, the robot's behavioral strategy is generated based on the risk parameters, so that when the target object is about to fall, supporting measures can be taken quickly, thereby effectively preventing the occurrence of falling events.

[0043] Figure 2 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 2 .based on Figure 1 , Figure 1 Step S102 in can be updated to step S201 to step S202, combining Figure 2 The steps shown are explained.

[0044] Step S201: Detecting center of gravity change data of the target object based on the multimodal data.

[0045] In some embodiments, the center of gravity change data refers to a record of changes in the center of gravity position of the target object in space over time; the center of gravity change data is used to evaluate the stability, balance ability and potential movement risks of the target object.

[0046] In some embodiments, the multimodal data may include at least one of the following: laser data, depth images, and visual images. The laser data is collected by a laser radar device and contains three-dimensional coordinate information of the surface of objects in the environment. The laser data is presented in the form of a point cloud, which can accurately reflect the shape and position of objects (including target objects) in the environment; the value of each pixel in the depth image represents the depth of the corresponding point in the scene (i.e., the distance from the depth camera); the visual image is used to record the appearance characteristics of objects in the scene, such as color, texture, and shape.

[0047] In some implementation scenarios, laser radar can be used to collect laser data of the target object and its surrounding environment to generate a point cloud map; at the same time, a depth camera and a conventional camera are used to obtain a depth image and a visual image respectively; then, the laser data is filtered and denoised to improve the accuracy of the point cloud, and the depth image is corrected and smoothed to reduce noise and errors, and the visual image is preprocessed, such as grayscale, edge detection, etc., to facilitate subsequent target recognition and tracking. In the visual image, a computer vision algorithm (such as target detection and tracking algorithm) is used to identify the target object and track its position changes in the image. At the same time, the position of the target object is further confirmed by combining the laser data and the depth image; according to the identified target object, the point cloud data belonging to the object is extracted from the laser data, and the current center of gravity position of the target object is calculated using a geometric algorithm (such as a centroid algorithm), or the depth data belonging to the object is extracted from the depth image, and the current center of gravity position of the target object is calculated by constructing a three-dimensional model of the target object. Here, at least one of the above two center of gravity positions can be used to determine the final center of gravity position. The center of gravity position of the target object is continuously calculated over time to form a center of gravity change data set, that is, the center of gravity change data of the target object is obtained.

[0048] Step S202: predicting risk parameters of the target object based on the center of gravity change data of the target object.

[0049] In some embodiments, the risk parameter of the target object may be predicted from at least one of the following aspects: a stability parameter, a balance ability parameter, and a potential risk prediction parameter.

[0050] Among them, by analyzing the fluctuation amplitude and frequency of the center of gravity change data, the stability parameters of the target object can be evaluated.

[0051] Among them, the balance ability parameter is used to characterize the target object's ability to maintain body stability when subjected to external interference. The target object's balance ability can be evaluated by analyzing the changing trend of the center of gravity change data in the time series, that is, the balance ability parameter can be obtained; for example, when the center of gravity position of the target object shifts significantly while walking or standing, it may indicate that its balance ability is impaired.

[0052] Among them, the historical behavior data of the target object (such as walking speed, gait characteristics, etc.) and the current center of gravity change data can be combined to build a prediction model using machine learning or deep learning algorithms to predict the future movement trend of the target object and obtain the potential risk prediction parameters.

[0053] In some embodiments, the above risk parameters may be predicted using a prediction model based on time series analysis, which may be, but is not limited to, support vector machine (SVM), random forest (RF), neural network (such as LSTM), etc.

[0054] In some embodiments, the prediction model can be trained using historical center of gravity change data and known risk parameters (such as fall records, stability scores, etc.) as training sets.

[0055] In some embodiments, after obtaining the trained prediction model, the risk parameters of the target object can be predicted based on the new center of gravity change data input into the trained prediction model. The prediction model may include a stability parameter prediction network, a balance ability parameter prediction network, and a potential risk prediction parameter prediction network, as well as a fusion module for weighted fusion of the outputs of each prediction network.

[0056] In the embodiment of the present application, through the detection method based on multimodal data (which may include laser data, depth images, visual images, etc.), the center of gravity change data of the target object in a dynamic environment can be fully and accurately captured, so that the stability and motion characteristics of the target object in different states can be obtained. At the same time, using these center of gravity change data as input, the risk parameters are predicted through the model, and the quantitative assessment of the potential risk of the target object can be achieved. Since the prediction not only takes into account the change of the center of gravity position, but also may integrate the rate and direction of the center of gravity change, it is possible to more comprehensively reflect the motion state and risk level of the target object.

[0057] Figure 3 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 3 .based on Figure 1 , Figure 1 Step S102 in can be updated to step S301 to step S303, combining Figure 3 The steps shown are explained.

[0058] Step S301: Identify obstacle information in the scene where the target object is located based on the multimodal data.

[0059] In some embodiments, the multimodal data includes depth images, laser point cloud data, and visual image data. In the scene where the robot follows the target object, the target object is an object close to the robot, and the obstacle object in the environment where the target object is located is an object far from the robot. In order to make full use of the advantages of different sensors. In the embodiment of the present application, the target object (such as the elderly) can be accurately identified by the depth camera to obtain the posture information of the target object, such as walking posture, gestures, etc. At this time, the laser point cloud data obtained by the laser radar is used as an auxiliary reference to provide additional three-dimensional spatial information for the recognition of the depth camera, thereby enhancing the accuracy and robustness of the recognition. For obstacle objects in the distance, the embodiment of the present application can mainly rely on the laser radar for detection. Due to the characteristics of the laser radar with its long-distance and high-precision, it can accurately scan and construct a three-dimensional map of the surrounding environment, including obstacles in the distance, terrain changes, etc., and then obtain comprehensive obstacle information. At this time, the depth image collected by the depth camera is used as an auxiliary reference to improve the comprehensiveness of environmental understanding. It can be understood that the visual image data in the multimodal data can be used as auxiliary information when obtaining obstacle information and posture information, so as to improve the accuracy of obstacle information and posture information.

[0060] In some embodiments, the obstacle information is a collection of attributes such as the type of objects that are potential obstacles to the movement of the target object (such as an elderly person) in the scene where the target object (such as an elderly person) is located, and their relative positions, sizes, shapes, etc. The obstacle information will affect the risk parameters of the target object. On the other hand, the obstacle information can also affect the robot's planning of a safe following path.

[0061] In some embodiments, in the process of identifying obstacle information, laser point cloud data is used as the main data, and depth images and visual images are used as supplementary data to solve the accuracy and efficiency problems of long-distance obstacle detection. Among them, laser radar can quickly scan a large range of environments and generate dense point clouds with its long detection distance (up to tens of meters) and high spatial resolution; depth images can assist in correcting noise or missing point clouds by supplementing fine depth information of local areas (such as the edge of steps); visual image data is used to provide semantic information (such as distinguishing between "walls" and "curtains"), reducing the probability of misjudging harmless objects as obstacles.

[0062] In some embodiments, the environment can be scanned by a lidar to generate point cloud data covering a certain range in front of the robot (such as a range of 10-30 meters); then, for low-density areas in the laser point cloud (such as narrow corners), the depth image is used to supplement local details (such as small obstacle outlines); the visual image data is recognized through a semantic segmentation model (such as Mask R-CNN) to obtain the obstacle category (such as "chair", "threshold") and associate it with the point cloud position.

[0063] Step S302: Identify the posture information of the target object based on the multimodal data.

[0064] In some embodiments, the above-mentioned posture information is the real-time status of each part of the target object in three-dimensional space. Taking the target object as a person as an example, the posture information may include the coordinates of skeleton key points (such as head, shoulder, hip), joint angles and movement speed.

[0065] In some embodiments, in the process of identifying posture information, the accuracy and robustness of near-field posture estimation are solved by using depth images as the main method and laser point cloud data and visual images as the auxiliary methods. Among them, the depth image provides high-resolution close-range human body contour information, which can accurately extract skeleton key points; the laser point cloud data is used to assist in correcting the occluded area in the depth image and provide absolute scale calibration (to avoid the scale error of the depth sensor); the visual image data is used to provide color and texture information to assist in the acquisition of human body contour information.

[0066] In some embodiments, a depth image of a target object can be captured by a depth camera. Generally, the target object is in the near field range of the depth camera (e.g., within 1-3 meters). For the collected visual image data, the human body contour information of the target object is determined by utilizing the extracted color and texture information of the target object in combination with the captured depth image. The above-mentioned skeleton key points are obtained based on the human body contour information. High-density point cloud sampling is performed on the area surrounding the target object (1-3 meters), and after alignment with the depth image, the three-dimensional coordinates of the skeleton key points are corrected, and then the coordinates of the skeleton key points can be obtained. The joint angles can be determined based on the coordinates of the skeleton key points, and the movement speed can be determined based on the relative position differences between the coordinates of the skeleton key points at multiple moments.

[0067] Step S303: Generate risk parameters of the target object using the obstacle information and the posture information.

[0068] The risk parameter of the target object is a comprehensive indicator of the possibility of the target object falling due to its own posture imbalance or interference from environmental obstacles. As described in the above embodiment, the risk parameter can be expressed as a quantitative value. In some embodiments, the risk parameter can be expressed as a probability value (such as 0%-100%) or a risk level (such as low, medium, high).

[0069] In the embodiment of the present application, in order to improve the assessment accuracy of risk parameters, a dynamic modeling method that jointly analyzes the state of the target object itself and environmental threats can be used to predict the risk of falling in complex scenarios. For example, the present application mainly considers the following exemplary scenarios: Even if the posture of the target object is temporarily stable, if there is a slippery ground or steps (obstacle information) in front, a risk parameter with a higher risk still needs to be generated.

[0070] In some embodiments, the balance state of the target object can be evaluated based on posture information (including the coordinates of skeleton key points, joint angles and movement speed); the potential interference of obstacles in the environment on the target object's movements can be evaluated based on obstacle information (including the type of object, relative position with respect to the target object, size, shape, etc.); and the interaction between the posture change trend of the target object and the spatial distribution of obstacles can be analyzed by using data-driven models in related technologies (such as neural networks), so as to predict the probability of the target object falling in the next few seconds and generate the risk parameter.

[0071] In the embodiment of the present application, by identifying obstacle information in the scene where the target object is located based on multimodal data, the complementary advantages of depth images, laser point cloud data and visual image data can be fully utilized to accurately obtain key attributes such as the type, position, size and shape of objects in the environment that pose potential obstacles to the movement of the target object, thereby improving the ability to understand the surrounding environment and the accuracy of obstacle detection; at the same time, by identifying the posture information of the target object through multimodal data, the posture changes of the target object in three-dimensional space can be captured in real time, including the coordinates of the key points of the skeleton, joint angles and movement speeds, etc., providing an important basis for evaluating the balance state of the target object; based on these detailed obstacle information and posture information, the risk parameters of the target object are generated by a data-driven model, which can comprehensively consider the target object's own state and environmental threats, predict the risk of falling in complex scenarios, and provide strong support for timely preventive measures.

[0072] In some embodiments, the above-mentioned behavior strategy includes at least one of the following: a position movement strategy of the robot and a posture change strategy of the robot.

[0073] Among them, the position movement strategy is used to control the robot to move to the target position, so that the robot can implement the supporting action for the target object at the target position; the posture change strategy is used to control the robot to switch to the target posture, so that the robot can implement the supporting action for the target object in the target posture.

[0074] In some embodiments, the position movement strategy is used to adjust the position of the robot itself in the physical space to reach the best support position (such as the side, rear, or near the support point of the target object) to ensure that physical intervention can be applied quickly and stably when the target object is unbalanced. The position movement strategy can be generated by the robot through path planning and motion control algorithms.

[0075] In some embodiments, the target orientation may include the target direction of the robot relative to the target object. The target direction is determined based on the prediction of the center of gravity trajectory. Based on the posture information of the target object, the center of gravity landing point in the next 1-2 seconds can be predicted to determine the target direction in which the robot needs to move. Exemplarily, if the center of gravity is predicted to shift to the left front, the robot needs to move to a 45° position in the left front. In an example scenario, when the center of gravity of the elderly shifts to the right front and his right foot is about to step on the slippery ground, the robot determines through trajectory prediction that he may fall to the right front, and then plans a path to bypass the sofa on the left, moves to 1 meter in front of the right, and prepares to provide support from the right side.

[0076] In some embodiments, the posture change strategy is used to adjust the robot's own mechanical structure (such as the arm joint angle, chassis height, etc.), switch to a physical posture suitable for performing supporting actions, ensure that the direction of force matches the imbalance direction of the target object, and maintain its own stability.

[0077] In some embodiments, the optimal force angle and contact point position can be calculated based on the imbalance direction (such as leaning back, tilting) and force analysis of the target object. For example, if you fall backward, the robotic arm needs to lift you from the back, and the force direction is vertically upward; if you tilt, you need to support it laterally from the armpit. It should be noted that based on the robot dynamics model, the projection of its own center of gravity in the adjusted posture can be located within the support polygon (such as the wheelbase range of a four-wheeled robot) to prevent tipping over when force is applied. In an example scenario, when an old man falls to the left due to weakness in his left leg, the robot quickly lowers the chassis height (reduces the center of gravity height), extends the left robotic arm to his armpit and maintains a 15° tilt angle (opposite to the direction of the fall), and the right robotic arm opens slightly to balance the reaction force.

[0078] In the embodiment of the present application, the position movement strategy can ensure that the robot moves safely and accurately to the target position in a complex environment to prepare for the assisting action. At the same time, the posture change strategy can ensure that the robot provides stable and comfortable support when performing the assisting task. Through the above strategy, the robot can significantly reduce the risk of falling injury to the target object in complex dynamic scenes.

[0079] Figure 4 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 4 .based on Figure 1 , when the behavior strategy includes a position movement strategy, the target orientation includes a distance between the robot and the target object and a direction of the robot; Figure 1 Step S103 in can be updated to step S401 to step S403, combining Figure 4 The steps shown are explained.

[0080] Step S401: determining a distance between the robot and the target object based on the risk parameter; the distance is negatively correlated with the risk parameter.

[0081] Among them, the distance between the robot and the target object is the three-dimensional space interval between the robot and the target object, usually calculated as Euclidean distance (meters). In the support scenario, this distance needs to be dynamically adjusted to balance safety monitoring and emergency intervention needs. In the embodiment of the present application, the higher the risk parameter, the smaller the distance between the robot and the target object; the lower the risk parameter, the larger the distance between the robot and the target object.

[0082] In some embodiments, in order to solve the problem of intervention delay or excessive interference caused by the fixed following distance of the robot, the distance can be adjusted in real time through the risk parameter, shortening the distance to increase the intervention speed when the risk is high, and maintaining a safe distance to reduce the sense of oppression when the risk is low. For example, the risk parameter can be converted into a distance setting value through a linear or nonlinear mapping function (such as an exponential decay function).

[0083] For example, in the case of low risk (0% to 30%), the default distance of 1.5 meters is maintained, and in the case of medium risk (30% to 70%), the distance is dynamically adjusted to 1.5 to 0.8 meters; in the case of high risk (70% to 100%), the minimum safety distance is locked to 0.5 meters. In an example scenario, when the elderly walk on a slippery ground, the risk parameter rises to 80%. The robot gradually approaches from the initial distance of 1.2 meters to 0.6 meters, ready to provide support at any time.

[0084] Step S402: determining the orientation between the robot and the target object based on the risk parameter; the angular difference between the orientation and the direction in which the robot points to the target object is negatively correlated with the risk parameter.

[0085] The orientation between the robot and the target object is the spatial orientation of the robot chassis or the robot arm actuator relative to the target object; the angle difference is the angle between the current orientation of the robot and the direction pointing to the target object. In the embodiment of the present application, the higher the risk parameter, the smaller the angle difference (the robot orientation needs to be more accurately aligned with the target object); the lower the risk parameter, the larger the angle difference.

[0086] In some embodiments, in order to solve the problem of failure to provide timely assistance due to robot orientation deviation, the angle difference can be adjusted through risk parameters. When the risk is high, the robot can be forced to accurately align with the imbalance direction of the target object (such as leaning back corresponds to the front of the robot facing the back of the target).

[0087] Exemplarily, in the case of low risk (0% to 30%), the allowed angle difference is ≤30°; in the case of medium risk (30% to 70%), the allowed angle difference is ≤15°; in the case of high risk (70% to 100%), the allowed angle difference is ≤5°.

[0088] Step S403: generating the position movement strategy based on the distance between the robot and the target object and the orientation of the robot.

[0089] Among them, the position movement strategy includes parameters such as the robot's target position, moving path, speed curve, etc., which are used to achieve dynamic adjustment of distance and direction.

[0090] In some embodiments, an optimal position movement strategy is calculated by an algorithm based on the distance between the robot and the target object and the orientation of the robot. This strategy is intended to ensure that the robot can maintain a safe and effective distance when approaching the target object and approach it in a suitable orientation in order to perform subsequent operations or tasks. By comprehensively considering the distance and orientation, the robot's movement path can be optimized, unnecessary movements can be reduced, and the efficiency of task execution can be improved.

[0091] In an embodiment of the present application, by dynamically adjusting the distance between the robot and the target object, the incidence of collisions and other potential risk events can be effectively reduced, thereby improving the safety of the robot in complex environments; at the same time, by optimizing the orientation of the robot, this step enables the robot to interact with the target object more efficiently and accurately, reducing operational errors or risks caused by improper orientation; the position movement strategy generated based on these adjustments (including parameters such as target position, moving path, speed curve, etc.) can achieve accurate monitoring and efficient intervention of the robot in assistance scenarios.

[0092] Figure 5 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 5 .based on Figure 1 , the position movement strategy satisfies the following range; the method further includes steps S501 to S502, combining Figure 5 The steps shown are explained.

[0093] Step S501, obtaining a data collection range of each sensor device corresponding to the multimodal data; the data collection range is determined based on a collection method of the sensor device for a target object and a setting position of the sensor device in the robot.

[0094] The data collection range is the spatial area where the sensor device can effectively obtain the target object or environmental data, which is determined by the physical characteristics of the sensor (such as field of view, detection distance) and the installation location. For example, when the horizontal scanning angle of the laser radar is 270° and the detection distance is 20 meters, its data collection range can be described as a fan-shaped three-dimensional space with a radius of 20 meters centered on the sensor.

[0095] In some embodiments, the effective data collection range of each sensing device can be calculated through geometric modeling based on the physical parameters of the sensing device (field of view angle, resolution, detection distance) and the installation location (such as the top or front side of the robot).

[0096] Step S502: Based on the data collection range corresponding to each of the sensor devices and the data collection weight of each of the sensor devices, a following range based on the target object is generated.

[0097] Wherein, at any position in the following range of the robot, there is at least one sensor device whose collection range includes the target object.

[0098] The robot following range is the spatial area where the robot can maintain effective monitoring and interaction when following the target object. The robot following range is determined based on the data collection range and / or data collection weight of each sensor device. The data collection weight of the sensor device is the importance or credibility of the data collected by the sensor device in the decision-making process, which is used to characterize the quality and reliability of the data collected by the sensor device. The position movement strategy is used to control the robot to move within the data collection range.

[0099] In order to solve the problem of how to determine a following range that can cover the target object and ensure effective monitoring when the robot follows the target object. The above step S502 constructs a following range based on the target object by comprehensively considering the data collection range and data collection weight of each sensor device. The following range can enable the robot to effectively sense and monitor the target object through at least one sensor device no matter where it is in the following process, thereby improving the accuracy and reliability of following.

[0100] In some embodiments, a sub-following range corresponding to each sensor device and centered on the target object can be created according to the data collection range of each sensor device, and the sub-following range represents the area where the sensor device can effectively sense and monitor the target object. The sub-following ranges of all sensor devices are fused to obtain the following range.

[0101] In some embodiments, after obtaining the above-mentioned following range, the position movement strategy can be generated in combination with the distance between the robot and the target object and the orientation of the robot, wherein at least one to-be-selected position that satisfies the distance between the target object and the orientation of the robot can be determined within the above-mentioned following range, and then, based on the data collection weights of the sensing devices where the target positions are located, a target position is determined in at least one to-be-selected position, and then based on the target position and a preset path planning strategy, the position movement strategy is generated.

[0102] In the embodiments of the present application, by obtaining the data collection range of each sensor device corresponding to the multimodal data, the perception boundary of each sensor device can be accurately defined according to the sensor device's collection method of the target object and its setting position in the robot; at the same time, based on these data collection ranges and the data collection weight of each sensor device, a following range based on the target object can be comprehensively evaluated and generated to ensure that any position of the robot within the following range can be effectively covered by at least one sensor device to the target object, thereby improving the accuracy and reliability of following. Based on the embodiments provided by the present application, the robot can achieve more intelligent and flexible following of the target object, optimizing the information perception and processing capabilities during the following process.

[0103] In some embodiments, the detection index value of the robot at any position in the following range meets a preset threshold; the detection index value is determined by the following method: obtaining a detection weight corresponding to each of the sensing devices; performing weighted summation of sub-index values ​​corresponding to each of the sensing devices based on the detection weight corresponding to each of the sensing devices to obtain the detection index value;

[0104] Among them, the detection index value is determined based on the sub-indicator value corresponding to each of the sensing devices. When the robot is within the data collection range corresponding to the sensing device, the sub-indicator value is positive; when the robot is outside the data collection range corresponding to the sensing device, the sub-indicator value is negative.

[0105] In some embodiments, the detection index value is a comprehensive quantitative index used to measure the detection effectiveness of the robot's position relative to the data collection range of each sensor device when following the target object. The detection index value is calculated by comprehensively considering the detection weights of each sensor device and its corresponding sub-index values, and is used to evaluate whether the detection performance of the robot at any position in the following range meets the preset requirements.

[0106] In some embodiments, the detection weight is the relative importance or credibility index assigned to each sensor device when constructing the detection index value. This weight is usually determined based on the detection performance parameters of the sensor device (such as detection distance, accuracy, stability, etc.) and the specific requirements of the application scenario, and is used to reflect the contribution of different sensor devices to the overall detection performance in the weighted summation process.

[0107] Among them, the sub-indicator value is a basic quantitative value that reflects the detection state of the robot within or outside the data collection range of a specific sensor device. When the robot is within the data collection range corresponding to the sensor device, the sub-indicator value is positive, indicating that the sensor device can effectively detect the target object; and when the robot is outside the data collection range corresponding to the sensor device, the sub-indicator value is negative, indicating that the sensor device cannot effectively detect the target object. In some embodiments, the closer the distance between the robot and the target object, the larger the sub-indicator value, on the contrary, the farther the distance between the robot and the target object, the smaller the sub-indicator value.

[0108] In order to ensure that the detection performance of the robot can meet the preset threshold requirements at any position in the following range, the embodiment of the present application comprehensively considers the detection weights of various sensing devices and their corresponding sub-index values, calculates a comprehensive detection index value, and compares the value with the preset threshold value, so that the detection performance of the robot can meet certain standards no matter where it is in the following process, thereby improving the accuracy and reliability of following.

[0109] In an embodiment of the present application, by obtaining the detection weight corresponding to each sensor device, the characteristics of each sensor device and its effectiveness in different scenarios can be fully utilized to provide an accurate weight allocation basis for the subsequent weighted summation; at the same time, based on these detection weights, the sub-indicator values ​​corresponding to each sensor device are weighted and summed to obtain a detection index value that comprehensively reflects the detection performance of the robot at any position in the following range, wherein the positive and negative settings of the sub-indicator values ​​can intuitively reflect whether the robot is within the effective data collection range of the sensor device, thereby enhancing the sensitivity and practicality of the detection index value; in addition, by setting a preset threshold and comparing it with the detection index value, the robot's following strategy can be flexibly adjusted to ensure that the robot always maintains effective detection of the target object during the following process.

[0110] Figure 6 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 6 .based on Figure 1 The method further includes steps S601 to S603, combining Figure 6 The steps shown are explained.

[0111] Step S601: Execute the behavior strategy.

[0112] In some embodiments, the above-mentioned behavior strategy includes at least one of the following: the position movement strategy of the robot and the posture change strategy of the robot. Among them, the position movement strategy is used to control the robot to move to the target position, so that the robot can implement the supporting action for the target object at the target position; the posture change strategy is used to control the robot to switch to the target posture, so that the robot can implement the supporting action for the target object in the target posture. The posture change strategy is used to adjust the robot's own mechanical structure (such as the joint angle of the robot arm, the height of the chassis, etc.), switch to a physical posture suitable for performing the supporting action, ensure that the force direction matches the imbalance direction of the target object, and maintain its own stability.

[0113] It can be understood that the above-mentioned behavior strategy is used to enable the robot to quickly and accurately provide support to the target object at critical moments.

[0114] For example, the robot continuously monitors the movements and postures of the elderly through cameras and sensors, assesses their risk of falling, obtains risk parameters, and starts the behavior strategy execution program based on the risk parameters, including: according to the position movement strategy, the robot plans and executes movement instructions, quickly moves from the current position to the vicinity of the elderly, and ensures that support can be implemented in the best position. At the same time, according to the posture change strategy, the robot adjusts the joint angle and chassis height of the robotic arm and switches to the most suitable posture for supporting the elderly. In the end, the robot successfully prepared to support the elderly before they fell.

[0115] Step S602: In response to the risk parameter satisfying a preset risk condition, generating a supporting action based on the current posture of the robot and the current posture of the target object.

[0116] Among them, the preset risk condition can be that the risk parameter exceeds the preset risk threshold, for example, the probability of falling is ≥ 70%. Posture is a comprehensive description of the position and posture of an object in space. In the embodiment of the present application, the current posture of the robot and the current posture of the target object are important references for generating supporting actions. The supporting action is a series of actions or posture adjustments taken by the robot to assist the target object in preventing it from falling. In some embodiments, the supporting action may include moving to the vicinity of the target object, adjusting the robot posture to match the unbalanced state of the target object, applying appropriate force to stabilize the target object, etc.

[0117] In some embodiments, the robot monitors in real time whether the risk parameters meet the preset risk conditions while following the elderly. Once the conditions are met, the robot immediately starts the posture acquisition program and obtains the current posture information of itself and the target object in real time through integrated sensors (such as cameras, lidar, inertial measurement units, etc.). Subsequently, the robot uses the posture analysis algorithm, combined with the imbalance state of the target object, its own mechanical structure and movement ability, to generate appropriate assistance actions.

[0118] Step S603: Execute the supporting action.

[0119] The supporting action includes at least one of the following: shortening the distance to the target object; changing the robot action to match the current posture of the target object.

[0120] In the embodiment of the present application, in order to prevent the target object from falling, the robot can reduce the spatial distance between the robot and the target object by moving its own position when performing the supporting action, so as to provide support and stability more directly and effectively; and / or adjust its own action mode according to the current posture and action of the target object to better adapt to and cooperate with the needs of the target object. This includes adjusting the robot's motion trajectory, speed, strength and other parameters to ensure the coordination and effectiveness of the supporting action.

[0121] In an embodiment of the present application, by executing the behavioral strategy, the robot's action plan can be flexibly adjusted according to the state of the target object and the environmental conditions, providing suitable prerequisites for subsequent supporting actions, thereby improving the effectiveness of robot assistance to a certain extent; at the same time, in response to the risk parameters satisfying the preset risk conditions, supporting actions are generated based on the current posture of the robot and the current posture of the target object, which can assess risks in real time and formulate appropriate supporting strategies, helping the robot to provide assistance more accurately and reduce adverse consequences caused by misjudgment or delayed response; when performing supporting actions, by shortening the distance to the target object and changing the robot's actions to match the current posture of the target object, physical support can be provided to the target object more effectively, maintaining its stability, while enhancing the coordination and comfort of the actions.

[0122] Figure 7 This is a schematic diagram of the implementation process of an anti-fall method provided in an embodiment of the present application. Figure 7 .based on Figure 1 The method further includes steps S701 to S703, combining Figure 7 The steps shown are explained.

[0123] Step S701: In response to the risk parameter satisfying a preset risk condition, the collected multimodal data is stored.

[0124] In some embodiments, the multimodal data collected can be stored while generating the assisting action based on the current position of the robot and the current position of the target object. In the process of real-time monitoring of risk parameters, once it is detected that the risk parameters reach or exceed the preset risk conditions, the data storage program is immediately started to save the currently collected multimodal data (such as video recordings, audio recordings, sensor data, etc.) to a designated database locally or on a server.

[0125] Step S702: when the multimodal data indicates that a fall event has occurred to the target object, generate event record data corresponding to the fall event based on the stored multimodal data.

[0126] In some embodiments, when it is necessary to determine whether a fall event has occurred in the target object, the stored multimodal data is first loaded, and then analysis is performed based on the multimodal data. If a fall event is confirmed to have occurred, detailed event record data is generated based on the timestamp of the data, the status information of the target object, and environmental conditions, and saved in an event log database.

[0127] Step S703: when the multimodal data indicates that the target object has not experienced a fall event, delete the stored multimodal data.

[0128] In some embodiments, multimodal data that does not represent a fall event can be automatically cleaned up after a preset period of time. For example, the stored multimodal data can be checked regularly and data records that are not associated with a fall event can be deleted to free up storage space.

[0129] In an embodiment of the present application, by responding to risk parameters that meet preset risk conditions and storing the collected multimodal data in a timely manner, it can be ensured that when the target object may be at risk of falling, key information can be retained for subsequent analysis; at the same time, when the multimodal data clearly indicates that the target object has a fall event, detailed event record data is generated based on these data, which can accurately record the specific circumstances of the fall event and provide strong support for subsequent emergency response and accident analysis; and when the multimodal data does not show that the target object has a fall event, timely deleting these data can effectively free up storage space and avoid unnecessary resource occupation.

[0130] In some embodiments, when the multimodal data indicates that a fall event has occurred to the target object, the method further includes step S704 and step S705.

[0131] Step S704: Evaluate the injury level of the target object based on the event record data.

[0132] In some embodiments, the event record data includes multimodal data corresponding to a fall event. The multimodal data corresponding to the fall event not only includes relevant data of the target object collected after the fall, but also includes multimodal data of the target object before and during the fall. In this way, the fall event of the target object can be analyzed throughout the process based on the multimodal data collected during the entire fall process to obtain a more accurate injury level.

[0133] Among them, the injury level is divided according to the degree of injury suffered by the target object in the fall incident. The injury level can be judged based on medical standards and experience to guide subsequent rescue measures.

[0134] In some embodiments, the injury level may include: mild injury, the target object can stand up on its own or with only slight help after falling, without obvious trauma or pain; moderate injury, the target object cannot stand up on its own after falling, and needs assistance from others, or is accompanied by mild trauma and pain; severe injury, the target object falls with serious trauma, pain, or suspected fractures, internal organ damage, etc., and requires immediate first aid and is sent to the hospital for professional treatment as soon as possible; critical injury, the target object falls and has serious vital signs abnormalities such as loss of consciousness and respiratory arrest, which falls into the category of emergency rescue and requires immediate initiation of the first aid process to save lives.

[0135] Step S705: executing a rescue strategy of a corresponding level based on the injury level.

[0136] The rescue strategy includes at least one of the following: generating rescue information and sending it to a rescue object of a corresponding rescue level; sending the event record data to a rescue object of a corresponding rescue level; and executing a lifting strategy.

[0137] The rescue objects corresponding to different levels are also different. For example, the rescue objects may include medical staff, family members, etc. When the injury level is low, the corresponding rescue object may be family members; when the injury level is high, the corresponding rescue object may be medical staff.

[0138] In some embodiments, when the injury level is a mild injury, the rescue target is a family member or a caregiver, and the corresponding rescue strategy is to send an alarm message containing the time, location and brief description of the fall to the family member's mobile phone; at the same time, the robot executes the strategy of helping up. If it is determined to be a moderate injury, the rescue target is the community doctor and family member, and the strategy includes sending event record data such as the fall process video and impact force curve to the community doctor terminal. The robot assists in helping up and monitors heart rate, breathing and other indicators in real time. For severe injuries, the rescue targets are emergency centers and hospitals, which automatically trigger emergency calls and send patient location, basic information and multimodal data (such as thermal imaging temperature maps, joint twisting angle analysis). If it is a critical injury, the rescue targets include the emergency center and the nearest AED equipment administrator. The robot can guide nearby personnel to use the AED equipment through indoor navigation, or perform standardized compressions through a robotic arm, and vital signs data are transmitted back to the emergency platform in real time for remote guidance by doctors.

[0139] In an embodiment of the present application, the injury level of the target object can be objectively and accurately assessed based on event record data (such as fall videos, sensor data, etc.), thereby providing a strong reference basis for subsequent rescue; at the same time, according to the assessed injury level, the corresponding level of rescue strategy can be quickly matched and executed. For example, for minor injuries, it may only be necessary to send simple rescue information to family members, while for severe or critical injuries, it is necessary to immediately initiate an emergency rescue process to ensure that rescue measures can be implemented in a targeted manner.

[0140] Based on the foregoing embodiments, an embodiment of the present application provides an anti-fall device, which includes the various units included and the various modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0141] Figure 8 A schematic diagram of the structure of an anti-fall device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the anti-fall device 800 includes: an acquisition module 810, an evaluation module 820, and a generation module 830, wherein:

[0142] An acquisition module 810 is used to acquire multimodal data of a scene where the target object is located during the process of following the target object;

[0143] An evaluation module 820, configured to evaluate risk parameters of the target object based on the multimodal data;

[0144] The generation module 830 is used to generate a behavior strategy of the robot based on the risk parameter; the behavior strategy is used to support the target object when the target object is about to fall.

[0145] In some embodiments, the evaluation module 820 is further used to detect center of gravity change data of the target object based on the multimodal data; and predict the risk parameter of the target object based on the center of gravity change data of the target object.

[0146] In some embodiments, the multimodal data includes depth data, and the evaluation module 820 is further used to identify obstacle information in the scene where the target object is located based on the multimodal data; identify posture information of the target object based on the multimodal data; and generate risk parameters of the target object using the obstacle information and the posture information.

[0147] In some embodiments, the behavior strategy includes at least one of the following: a position movement strategy of the robot and a posture change strategy of the robot; wherein the position movement strategy is used to control the robot to move to a target position so that the robot can implement a supporting action for the target object at the target position; the posture change strategy is used to control the robot to switch to a target posture so that the robot can implement a supporting action for the target object in the target posture.

[0148] In some embodiments, when the behavior strategy includes a position movement strategy, the target orientation includes the distance between the robot and the target object and the orientation of the robot; the generation module 830 is also used to determine the distance between the robot and the target object based on the risk parameter; the distance is negatively correlated with the risk parameter; the orientation between the robot and the target object is determined based on the risk parameter; the angle difference between the orientation and the direction in which the robot points to the target object is negatively correlated with the risk parameter; the position movement strategy is generated based on the distance between the robot and the target object and the orientation of the robot.

[0149] In some embodiments, the position movement strategy satisfies the following range; the generation module 830 is also used to obtain the data collection range of each sensor device corresponding to the multimodal data; the data collection range is determined based on the collection method of the sensor device for the target object and the setting position of the sensor device in the robot; based on the data collection range corresponding to each of the sensor devices and the data collection weight of each of the sensor devices, a following range based on the target object is generated; wherein, at any position of the robot in the following range, there is at least one sensor device whose collection range includes the target object.

[0150] In some embodiments, the detection index value of the robot at any position in the following range satisfies a preset threshold; the generation module 830 is also used to obtain the detection weight corresponding to each of the sensing devices; based on the detection weight corresponding to each of the sensing devices, the sub-indicator values ​​corresponding to each of the sensing devices are weighted and summed to obtain the detection index value; wherein, the detection index value is determined based on the sub-indicator value corresponding to each of the sensing devices, and when the robot is within the data collection range corresponding to the sensing device, the sub-indicator value is positive; when the robot is outside the data collection range corresponding to the sensing device, the sub-indicator value is negative.

[0151] In some embodiments, the anti-fall device 800 includes an execution module, which is used to execute the behavior strategy; in response to the risk parameter satisfying a preset risk condition, generating a supporting action based on the current posture of the robot and the current posture of the target object; executing the supporting action; wherein the supporting action includes at least one of the following: shortening the distance to the target object; changing the robot action to match the current posture of the target object.

[0152] In some embodiments, the execution module is used to store the collected multimodal data in response to the risk parameter satisfying a preset risk condition; when the multimodal data indicates that a fall event has occurred to the target object, generate event record data corresponding to the fall event based on the stored multimodal data; when the multimodal data indicates that a fall event has not occurred to the target object, delete the stored multimodal data.

[0153] In some embodiments, when the multimodal data represents a fall event of the target object, the execution module is used to evaluate the injury level of the target object based on the event record data; execute a rescue strategy of a corresponding level based on the injury level; the rescue strategy includes at least one of the following: generating rescue information and sending it to a rescue object of a corresponding rescue level; sending the event record data to a rescue object of a corresponding rescue level; and executing a lifting strategy.

[0154] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0155] It should be noted that in the embodiments of the present application, if the above-mentioned anti-fall method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0156] An embodiment of the present application provides a robot, comprising a multimodal sensor, a memory and a processor, wherein the multimodal sensor is used to collect real-time multimodal data; the memory stores a computer program that can be run on the processor; and the processor implements some or all of the steps in the above method when executing the program.

[0157] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0158] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0159] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0160] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0161] Fig. 9 A schematic diagram of a hardware entity of a robot provided in an embodiment of the present application, such as Fig. 9 As shown, the hardware entity of the robot 900 includes: a processor 901, a memory 902 and a multimodal sensor 903, wherein the memory 902 stores a computer program that can be run on the processor 901, and the processor 901 implements the steps in the method of any of the above embodiments when executing the program; the multimodal sensor 903 is used to collect real-time multimodal data.

[0162] The memory 902 stores computer programs that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and various modules in the robot 900 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0163] When the processor 901 executes the program, the steps of any of the above-mentioned anti-falling methods are implemented. The processor 901 generally controls the overall operation of the robot 900.

[0164] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the anti-fall method in any of the above embodiments.

[0165] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0166] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0167] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0168] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0169] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0171] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0172] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0173] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device corresponding to a robot (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0174] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for preventing falling, characterized in that: Applied to a robot, the method comprises: In the process of following the target object, obtaining multimodal data for the scene where the target object is located; Evaluating a risk parameter of the target object based on the multimodal data; the risk parameter is used to reflect the fall risk of the target object; A behavior strategy of the robot is generated based on the risk parameter; the behavior strategy is used to support the target object when the target object is about to fall.

2. The method according to claim 1, characterized in that The multimodal data includes depth data, and the risk parameter of the target object is evaluated based on the multimodal data, including at least one of the following: detecting center of gravity change data of the target object based on the multimodal data, and predicting a risk parameter of the target object based on the center of gravity change data of the target object; Obstacle information in the scene where the target object is located is identified based on the multimodal data, posture information of the target object is identified based on the multimodal data, and risk parameters of the target object are generated using the obstacle information and the posture information.

3. The method according to claim 1, characterized in that The behavior strategy includes at least one of the following: a position movement strategy of the robot and a posture change strategy of the robot; wherein the position movement strategy is used to control the robot to move to a target position so that the robot can perform a supporting action for the target object at the target position; and the posture change strategy is used to control the robot to switch to a target posture so that the robot can perform a supporting action for the target object at the target posture. In the case where the behavior strategy includes a position movement strategy, the target orientation includes a distance between the robot and the target object and a direction of the robot; and generating the behavior strategy of the robot based on the risk parameter includes: Determining a distance between the robot and the target object based on the risk parameter; the distance is negatively correlated with the risk parameter; determining the orientation between the robot and the target object based on the risk parameter; the angular difference between the orientation and the direction in which the robot points to the target object is negatively correlated with the risk parameter; The position movement strategy is generated based on the distance between the robot and the target object and the orientation of the robot.

4. The method according to any one of claims 1 to 3, characterized in that: The position movement strategy satisfies the following range; the method further comprises: Acquire a data collection range of each sensor device corresponding to the multimodal data; the data collection range is determined based on a collection method of the sensor device for a target object and a setting position of the sensor device in the robot; Based on the data collection range corresponding to each of the sensor devices and the data collection weight of each of the sensor devices, generating a following range based on the target object; Wherein, at any position of the robot in the following range, there is at least one sensor device whose collection range includes the target object.

5. The method according to claim 4, characterized in that The detection index value of the robot at any position in the following range meets the preset threshold; the detection index value is determined by the following method: Obtaining a detection weight corresponding to each of the sensing devices; Performing weighted summation of the sub-index values ​​corresponding to each of the sensing devices based on the detection weight corresponding to each of the sensing devices to obtain the detection index value; Among them, the detection index value is determined based on the sub-indicator value corresponding to each of the sensing devices. When the robot is within the data collection range corresponding to the sensing device, the sub-indicator value is positive; when the robot is outside the data collection range corresponding to the sensing device, the sub-indicator value is negative.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: executing said behavioral strategy; In response to the risk parameter satisfying a preset risk condition, generating a supporting action based on a current posture of the robot and a current posture of the target object; Performing the supporting action; The supporting action includes at least one of the following: shortening the distance to the target object; changing the robot action to match the current posture of the target object.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to the risk parameter satisfying a preset risk condition, storing the collected multimodal data; In a case where the multimodal data indicates that a fall event has occurred to the target object, generating event record data corresponding to the fall event based on the stored multimodal data; In a case where the multimodal data indicates that the target object has not experienced a fall event, the stored multimodal data is deleted.

8. The method according to claim 7, characterized in that In the case where the multimodal data indicates that the target object has fallen, the method further includes: assessing the injury level of the target object based on the event record data; Executing a rescue strategy of a corresponding level based on the injury level; The rescue strategy includes at least one of the following: generating rescue information and sending it to a rescue object of a corresponding rescue level; sending the event record data to a rescue object of a corresponding rescue level; and executing a lifting strategy.

9. A fall prevention device, characterized in that: Applied to a robot, the device comprises: An acquisition module, used for acquiring multimodal data of a scene where the target object is located during the process of following the target object; An evaluation module, configured to evaluate risk parameters of the target object based on the multimodal data; A generation module is used to generate a behavior strategy of the robot based on the risk parameter; the behavior strategy is used to support the target object when the target object is about to fall.

10. A robot, characterized in that: It includes a multimodal sensor, a memory and a processor, wherein: The multimodal sensor is used to collect real-time multimodal data; The memory stores a computer program executable on the processor; When the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

Images