A fall detection method, device, controller and storage medium

Abstract

Embodiments of the present application provide a fall detection method, device, controller, and storage medium. The method includes: after acquiring an image sequence, sending the image sequence and its posture information to a fall detection terminal, so that when the fall detection terminal determines that the image sequence contains an image representing a fall, it sends target posture information corresponding to the image representing the fall to the controller; after receiving the target posture information, controlling a robot to capture a fall image based on the target posture information, and sending the fall image to a ranging terminal, so that the ranging terminal determines the position information between the robot and the fallen object in the fall image based on the fall image; based on the position information, determining the target coordinate information in a digital map coordinate system, and controlling the robot's movement based on the target coordinate information. This embodiment not only enables flexible fall detection at different locations, but also allows timely arrival at the fallen object after a fall is determined, facilitating subsequent fall processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent robot technology, and in particular to a fall detection method, device, controller, and storage medium. Background Art

[0002] With the aging population, the demand for home-based elderly care services is growing, and the physical and mental health of the elderly is receiving increasing attention. If a fall is not detected in time, it can be dangerous or even life-threatening. Therefore, timely fall detection is necessary for the elderly.

[0003] In the related art, multiple cameras are placed indoors and fall detection is performed based on images collected by the cameras.

[0004] However, the above method requires too many cameras and is easily affected by external factors (such as changes in lighting and motion occlusion). Summary of the Invention

[0005] Embodiments of the present application provide a fall detection method, device, controller, and storage medium for accurately performing fall detection.

[0006] In a first aspect, an embodiment of the present application provides a fall detection method, which is applied to a controller in a robot. The method includes:

[0007] After acquiring the image sequence, sending the image sequence and the posture information of the image sequence to the fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller;

[0008] After receiving the target posture information, controlling the robot to collect a fall image based on the target posture information, and sending the fall image to the ranging end, so that the ranging end determines the position information between the robot and the fallen object in the fall image based on the fall image; wherein the fall image is a three-channel color and depth combination (RGB-D) image;

[0009] Based on the position information, target coordinate information in a digital map coordinate system is determined, and the robot is controlled to move based on the target coordinate information.

[0010] In the above scheme, the robot flexibly patrols the detection area and collects images in different postures at the detection position to obtain an image sequence from multiple perspectives; then, when there is an image representing a fall in the image sequence, the fall image is captured based on the posture when the image is collected, so that the ranging end can determine the position information between the robot and the fallen object based on the fall image; after receiving the above position information, the position conversion is performed to obtain the target coordinate information in the digital map coordinate system, and the movement of the robot is controlled based on the target coordinate information. This embodiment can not only flexibly detect falls at different positions, but also reach the fallen object in time after the fall is determined, which is convenient for subsequent fall processing.

[0011] In some optional implementations, acquiring an image sequence includes:

[0012] When the robot is in the detection position, the robot body is controlled to rotate multiple times in the horizontal direction based on the preset rotation angles to obtain images corresponding to the preset rotation angles;

[0013] The images corresponding to all the rotation angles are determined as the image sequence corresponding to the detection position.

[0014] In some optional implementations, obtaining images corresponding to each preset rotation angle includes:

[0015] For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

[0016] In some optional embodiments, the position information includes an offset angle and a straight-line distance of the robot relative to the fallen object; and determining target coordinate information in a digital map coordinate system based on the position information includes:

[0017] Determining whether the offset angle is less than a preset angle threshold;

[0018] If yes, determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance;

[0019] Otherwise, the robot body is controlled to rotate by the offset angle, a new fall image is reacquired, and the target coordinate information is determined based on new position information corresponding to the new fall image.

[0020] In some optional implementations, determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance includes:

[0021] Determine the difference between the direct distance and the preset distance as the target distance;

[0022] The target coordinate information is determined based on the current coordinate information, the target distance, and the target angle; wherein the target angle is the angle between the X-axis and the target connecting line in the digital map coordinate system, and the target connecting line is the connecting line between the robot and the fallen object.

[0023] In some optional implementations, after controlling the robot to move based on the target coordinate information, the method further includes:

[0024] After determining that the robot has moved to the position represented by the target coordinate information, a first notification message is notified through a first preset notification method; wherein the first notification message is a message representing a fall inquiry.

[0025] In some optional implementations, after notifying the first notification message in the first preset notification manner, the method further includes:

[0026] If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

[0027] In a second aspect, an embodiment of the present application further provides a fall detection device, which is applied to a controller in a robot, comprising:

[0028] a fall determination module, configured to, after acquiring an image sequence, send the image sequence and the posture information of the image sequence to a fall detection terminal, so that when the fall detection terminal determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller;

[0029] a distance determination module, configured to, after receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to a distance measurement terminal, so that the distance measurement terminal determines, based on the fall image, position information between the robot and the fallen object in the fall image; wherein the fall image is an RGB-D image;

[0030] The movement control module is used to determine target coordinate information in a digital map coordinate system based on the position information, and control the movement of the robot based on the target coordinate information.

[0031] In some optional implementations, the fall determination module is specifically configured to:

[0032] When the robot is in the detection position, controlling the body of the robot to rotate multiple times in the horizontal direction based on preset rotation angles to obtain images corresponding to each preset rotation angle;

[0033] The images corresponding to all the rotation angles are determined as the image sequence corresponding to the detection position.

[0034] In some optional implementations, the fall determination module is specifically configured to:

[0035] For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

[0036] In some optional embodiments, the position information includes an offset angle and a straight-line distance of the robot relative to the fallen object; and the movement control module is specifically configured to:

[0037] Determining whether the offset angle is less than a preset angle threshold;

[0038] If yes, determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance;

[0039] Otherwise, the robot body is controlled to rotate by the offset angle, a new fall image is reacquired, and the target coordinate information is determined based on new position information corresponding to the new fall image.

[0040] In some optional implementations, the movement control module is specifically configured to:

[0041] Determine the difference between the direct distance and the preset distance as the target distance;

[0042] The target coordinate information is determined based on the current coordinate information, the target distance, and the target angle; wherein the target angle is the angle between the X-axis and the target connecting line in the digital map coordinate system, and the target connecting line is the connecting line between the robot and the fallen object.

[0043] In some optional implementations, a notification module is further included, configured to:

[0044] After the movement control module controls the movement of the robot based on the target coordinate information, if it is determined that the robot moves to the position represented by the target coordinate information, a first notification message is notified through a first preset notification method; wherein, the first notification message is a message representing a fall inquiry.

[0045] In some optional implementations, after notifying the first notification message in the first preset notification manner, the notification module is further configured to:

[0046] If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

[0047] In a third aspect, an embodiment of the present application provides a controller comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes any of the fall detection methods described in the first aspect.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer. When the program runs on the computer, the computer executes the fall detection method as described in any one of the first aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of the flow of the first fall detection method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of a second fall detection method provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the horizontal rotation process provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the vertical swing process provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of a flow chart of a third fall detection method provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of the offset angle provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of the position in the digital map coordinate system provided in an embodiment of the present application;

[0058] Figure 9 A flowchart of a fourth fall detection method provided in an embodiment of the present application;

[0059] Figure 10 A schematic diagram of the structure of a fall detection device provided in an embodiment of the present application;

[0060] Figure 11 A schematic diagram of the structure of the controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0062] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0063] In the description of this application, it should be noted that, unless otherwise specified or limited, the term "connection" should be understood in a broad sense. For example, it can mean direct connection, indirect connection through an intermediate medium, or internal communication between two devices. Those skilled in the art will understand the specific meaning of the above terms in this application based on specific circumstances.

[0064] If the elderly are not discovered in time after falling, they are likely to be in danger or even life-threatening. Therefore, it is necessary to detect falls for the elderly in a timely manner.

[0065] In the related art, multiple cameras are placed indoors and fall detection is performed based on images collected by the cameras.

[0066] However, the above method requires too many cameras and is easily affected by external factors (such as changes in lighting and motion occlusion).

[0067] In view of this, embodiments of the present application provide a fall detection method, device, controller, and storage medium for accurately performing fall detection.

[0068] See Figure 1 As shown, an application scenario provided by an embodiment of the present application includes a robot, a fall detection terminal, and a ranging terminal; the robot includes a controller, an image acquisition device, and other components;

[0069] The image acquisition device is used to acquire an image sequence and send the image sequence to the controller;

[0070] The controller is configured to, after acquiring the image sequence, send the image sequence and the posture information of the image sequence to the fall detection terminal;

[0071] After receiving the image sequence, the fall detection end determines whether there is an image representing a fall (a fallen object) in the image sequence. If there is an image representing a fall, the target posture information corresponding to the image representing the fall is sent to the controller;

[0072] After receiving the target posture information sent by the fall detection end, the controller adjusts the posture of the robot based on the target posture information and controls the image acquisition device to capture the fall image; and sends the fall image to the ranging end;

[0073] The ranging end determines position information between the robot and the fallen object in the fall image based on the fall image, and sends the position information to the controller;

[0074] The controller determines target coordinate information in the digital map coordinate system based on the position information, and controls the movement of the robot based on the target coordinate information.

[0075] This embodiment does not specifically limit the specific implementation method of the above-mentioned fall detection terminal, such as one or more servers with fall detection capabilities (such as including a fall detection algorithm).

[0076] This embodiment does not specifically limit the fall detection method of the fall detection terminal. For any image in the image sequence, fall detection can be performed in the following ways, but not limited to:

[0077] 1. Perform skeleton point detection and human body detection on the image to obtain skeleton point detection results and human body detection results. Specifically, extract 18 skeleton point information from the image through Openpose (an open source human posture recognition project), input the skeleton point information into the skeleton point detection model, and obtain the first classification result and first confidence level (skeleton point detection result) output by the skeleton point detection model; and input the image into the Yolov5 (a single-stage target detection algorithm) model, perform human body selection and action recognition through the Yolov5 model, and obtain the second classification result and second confidence level (human body detection result).

[0078] 2. Determine whether the image contains a fallen object based on the skeleton point detection results and the human body detection results. Specifically, determine a target confidence level based on the first classification result and the first confidence level. If the first classification result indicates a fall, the target confidence level is equal to the product of the first confidence level and a preset coefficient; otherwise, the target confidence level is 0. If the second classification result indicates a fall, and the second confidence level is greater than the target confidence level, determine that a fallen object exists (i.e., the image indicates a fall).

[0079] The above fall detection process is only an example description. In practice, fall detection may also be performed by other methods (such as a single detection method), which will not be described in detail here.

[0080] This embodiment does not specifically limit the specific implementation of the above-mentioned ranging end, such as one or more servers with distance detection capabilities (such as including a distance detection algorithm).

[0081] During implementation, a fall may be misjudged, or the fallen object may stand up on its own. Therefore, in some optional implementations, if the ranging end does not detect the fallen object from the fall image, the fall image needs to be sent to relevant personnel for confirmation.

[0082] This embodiment does not specifically limit the distance detection method of the distance measuring end. The specific detection method is related to the type of fall image. For example, the image acquisition device is a binocular camera (RGB-D camera) that fuses color and depth. After acquiring the depth image and color image, the depth image and the color image are aligned to obtain an RGB-D image (i.e., the fall image). Correspondingly, distance fall detection can be performed by, but not limited to, the following methods:

[0083] 1. Input the color image into the detectron2 framework (a deep learning framework) and use the instance segmentation model trained on the COCO dataset (a dataset for object detection, segmentation, and characters) to perform instance segmentation, separate the segmentation mask map of the fallen object, and calculate the center of gravity of the fallen object.

[0084] 2. Input the center of gravity coordinates (u, v) of the fallen object and the camera intrinsic parameters into the depth image to obtain the world coordinates (x, y, z) of the fallen object.

[0085] 3. Calculate the robot's offset angle and straight-line distance relative to the fallen object based on the world coordinates (x, y, z) of the fallen object.

[0086] The above distance detection process is only an example description. In practice, distance detection can also be performed in other ways, which will not be described in detail here.

[0087] In the above scheme, the robot flexibly patrols the detection area and collects images in different postures at the detection position to obtain an image sequence from multiple perspectives; then, when there is an image representing a fall in the image sequence, the fall image is captured based on the posture when the image is collected, so that the ranging end can determine the position information between the robot and the fallen object based on the fall image; after receiving the above position information, the position conversion is performed to obtain the target coordinate information in the digital map coordinate system, and the movement of the robot is controlled based on the target coordinate information. This embodiment can not only flexibly detect falls at different positions, but also reach the fallen object in time after the fall is determined, which is convenient for subsequent fall processing.

[0088] The following will be combined with the accompanying drawings and specific embodiments to explain in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0089] The embodiment of the present application provides a first fall detection method, which is applied to the above controller, such as Figure 2 As shown, the following steps are included:

[0090] Step S201: After acquiring an image sequence, the image sequence and the posture information of the image sequence are sent to a fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, the target posture information corresponding to the image representing the fall is sent to the controller.

[0091] During implementation, the robot can patrol the detection area flexibly, and collect images in different postures at the detection position (a specific position, or the position at preset time intervals during random patrols) to obtain image sequences under multiple perspectives (that is, comprehensively collect images under different perspectives), and send the image sequence and posture information to the fall detection end. In this way, the fall detection end can determine whether there is a fallen object under different perspectives at the detection position; if the fall detection end determines that there is an image representing a fall (the image contains a fallen object), the target posture information corresponding to the image is sent to the controller, and the controller controls the robot to collect the fall image for ranging based on the target posture information.

[0092] Step S202: After receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to the ranging end, so that the ranging end determines the position information between the robot and the falling object in the fall image based on the fall image.

[0093] For example, if the fall detection end determines that there is an image representing a fall, it means that the robot has collected an image containing a fallen object and needs to return to the original position to shoot a measurable image again to obtain a fall image.

[0094] As mentioned above, the distance detection method of the ranging end is related to the type of fall image. When the image acquisition device is an RGB-D camera, the fall image is an RGB-D image obtained by aligning the depth image and the color image.

[0095] The specific implementation of the distance detection method at the ranging end can refer to the above embodiment and will not be repeated here.

[0096] Step S203: determining target coordinate information in a digital map coordinate system based on the position information, and controlling the movement of the robot based on the target coordinate information.

[0097] Since misjudgment may occur during the entire detection process, or the fallen object may have stood up, or the fallen object may need assistance, the robot needs to further interact with the fallen object. Based on this, after determining the target coordinate information, this embodiment controls the robot to move to the fallen object to facilitate subsequent fall processing.

[0098] In the above scheme, the robot flexibly patrols the detection area and collects images in different postures at the detection position to obtain an image sequence from multiple perspectives; then, when there is an image representing a fall in the image sequence, the fall image is captured based on the posture when the image is collected, so that the ranging end can determine the position information between the robot and the fallen object based on the fall image; after receiving the above position information, the position conversion is performed to obtain the target coordinate information in the digital map coordinate system, and the movement of the robot is controlled based on the target coordinate information. This embodiment can not only flexibly detect falls at different positions, but also reach the fallen object in time after the fall is determined, which is convenient for subsequent fall processing.

[0099] The embodiment of the present application provides a second fall detection method, such as Figure 3 As shown, the following steps are included:

[0100] Step S301: When the robot is at a detection position, the robot body is controlled to rotate multiple times in a horizontal direction based on preset rotation angles, and an image corresponding to each preset rotation angle is acquired.

[0101] During implementation, the robot is in the detection area, so the robot is surrounded by all areas to be photographed. In order to photograph the entire detection area, multiple preset rotation angles are set. In this way, the robot body is controlled to rotate multiple times in the horizontal direction, and a picture is taken every time it rotates to a preset rotation angle, thus ensuring that the detection area is fully photographed in the horizontal direction.

[0102] The robot's horizontal rotation process can be found in Figure 4 shown.

[0103] This embodiment does not specifically limit the preset rotation angle. During implementation, the preset rotation angle can be set based on the horizontal field of view of the image acquisition device. For example, when the horizontal field of view is large, a smaller preset rotation angle can be set, while when the horizontal field of view is small, a larger preset rotation angle can be set. In some embodiments, the preset rotation angles include 0°, 90°, 180°, and 270°.

[0104] In some optional implementations, obtaining images corresponding to each preset rotation angle may be achieved by, but not limited to, the following methods:

[0105] For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

[0106] During implementation, the inspection area may be at a high altitude, or the vertical field of view of the image acquisition device may be small. In this case, even if the robot body is controlled to rotate horizontally multiple times, it is difficult to ensure that the inspection area is fully captured.

[0107] Based on this, this embodiment sets multiple preset swing angles. In this way, after controlling the robot body to rotate to a preset rotation angle, it is also necessary to control the image acquisition device to swing multiple times in the vertical direction, and a shot must be taken every time it swings to a preset swing angle. This ensures that the detection area is fully photographed in both the horizontal and vertical directions.

[0108] In some embodiments, the image acquisition device is set on the head of the robot, and the process of the robot swinging in the vertical direction can be referred to Figure 5 shown.

[0109] This embodiment does not specifically limit the preset swing angle. During implementation, it can be set according to the vertical field of view of the image acquisition device. For example, when the vertical field of view is large, fewer preset swing angles are set, and when the vertical field of view is small, more preset swing angles are set.

[0110] Step S302: Determine the images corresponding to all rotation angles as the image sequence corresponding to the detection position.

[0111] In this embodiment, the detection area is fully photographed to obtain a sequence of images captured at different viewing angles in all directions at the detection position.

[0112] Step S303: sending the image sequence and the posture information of the image sequence to the fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller.

[0113] Step S304: After receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to the ranging end, so that the ranging end determines the position information between the robot and the fallen object in the fall image based on the fall image.

[0114] Step S305: determining target coordinate information in a digital map coordinate system based on the position information, and controlling the movement of the robot based on the target coordinate information.

[0115] The specific implementation of steps S303 to S305 can refer to the above embodiment and will not be described again here.

[0116] In the above scheme, since the robot is in the detection area, all areas to be photographed are surrounded by the robot. In order to photograph the entire detection area, multiple preset rotation angles are set. In this way, the robot body is controlled to rotate multiple times in the horizontal direction, and a picture is taken every time it rotates to a preset rotation angle. This ensures that the detection area is fully photographed in the horizontal direction, and a sequence of images collected at different perspectives at the detection position is obtained.

[0117] In some optional implementations, the position information includes an offset angle and a straight-line distance of the robot relative to the fallen object;

[0118] Correspondingly, the embodiment of the present application provides a third fall detection method, which is applied to the above controller, such as Figure 6 As shown, the following steps are included:

[0119] Step S601: After acquiring an image sequence, the image sequence and the posture information of the image sequence are sent to a fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, the target posture information corresponding to the image representing the fall is sent to the controller.

[0120] Step S602: Control the robot to collect a fall image based on the target posture information, and send the fall image to the ranging end, so that the ranging end determines the position information between the robot and the fallen object in the fall image based on the fall image.

[0121] The specific implementation of steps S601 to S602 can refer to the above embodiment and will not be repeated here.

[0122] Step S603: Determine whether the offset angle is smaller than a preset angle threshold.

[0123] See Figure 7 As shown, the above-mentioned offset angle is the angle α formed by the optical axis of the image acquisition device in the robot and the target connection line (the connection line between the robot and the fallen object).

[0124] During implementation, the robot may capture the falling image while facing the falling object directly, that is, the optical axis of the image acquisition device coincides with the target connection line (the connection line between the robot and the falling object) (the offset angle is 0); or the robot may capture the falling image while facing the falling object at a relatively small angle, that is, the angle between the optical axis of the image acquisition device and the target connection line is relatively small. In these cases, the ranging end can more accurately determine the straight-line distance of the robot relative to the falling object based on the falling image. Therefore, the subsequent step S604 is executed to directly determine the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance.

[0125] Of course, the robot may also take a falling image facing the falling object at a large inclination, that is, the angle between the optical axis of the image acquisition device and the target connecting line is large. In this case, the straight-line distance error determined by the ranging end based on the falling image is large. Therefore, to execute the subsequent step S605, it is necessary to control the robot's body to rotate the above-mentioned offset angle, re-face the falling object to take the falling image, and return to step S602 again.

[0126] Step S604: Determine the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance.

[0127] As mentioned above, in this scenario, the ranging end can more accurately determine the straight-line distance of the robot relative to the fallen object based on the fall image. Therefore, the target coordinate information can be directly determined based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance.

[0128] In some optional implementations, step S604 may be implemented by, but not limited to, the following methods:

[0129] Determine the difference between the direct distance and the preset distance as the target distance;

[0130] The target coordinate information is determined based on the current coordinate information, the target distance, and the target angle; wherein the target angle is the angle between the X-axis and the target connecting line in the digital map coordinate system, and the target connecting line is the connecting line between the robot and the fallen object.

[0131] In practice, the robot can reach a specific location by avoiding obstacles based on the location information in the digital map coordinate system. Therefore, it is necessary to determine the target coordinate information of the location that the robot is expected to reach in the digital map coordinate system.

[0132] For example, in order to prevent the robot from being too close to the falling object, a preset distance (e.g., 0.5 meters) is set, and the difference between the direct distance and the preset distance is determined as the target distance;

[0133] See Figure 8 As shown, the target horizontal coordinate x′=x0+D*cosβ; the target vertical coordinate y′=y0+D*sinβ; wherein x0 is the horizontal coordinate in the current coordinate information (the horizontal coordinate of the robot in the digital map coordinate system), y0 is the vertical coordinate in the current coordinate information (the vertical coordinate of the robot in the digital map coordinate system), D is the above-mentioned target distance, and β is the above-mentioned target angle.

[0134] In implementation, the arc information corresponding to the target angle in the digital map coordinate system can be determined by the following methods:

[0135] Radian information θ=β*π / 180.

[0136] Step S605: Control the body of the robot to rotate by the offset angle, and execute again the step of collecting the fall image based on the target posture information in step S602, and sending the fall image to the ranging end.

[0137] As described above, in this case, the straight-line distance determined by the ranging end based on the fall image has a large error. It is difficult to accurately determine the target coordinate information directly based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance. Therefore, it is necessary to control the robot's body to rotate the above-mentioned offset angle, re-take the fall image facing the fall object, and return to step S602 again.

[0138] Step S606: Control the movement of the robot based on the target coordinate information.

[0139] The specific implementation of step S606 can refer to the above embodiment and will not be repeated here.

[0140] In the above scheme, the robot may take a falling image facing the falling object, that is, the optical axis of the image acquisition device coincides with the target connecting line; or, the robot may take a falling image facing the falling object at a smaller inclination, that is, the angle between the optical axis of the image acquisition device and the target connecting line is smaller. In these cases, the ranging end can more accurately determine the straight-line distance of the robot relative to the falling object based on the falling image. Therefore, the target coordinate information is determined directly based on the current coordinate information and straight-line distance of the robot in the digital map coordinate system; the robot may also take a falling image facing the falling object at a larger inclination, that is, the angle between the optical axis of the image acquisition device and the target connecting line is larger. In this case, the straight-line distance error determined by the ranging end based on the falling image is larger. Therefore, it is necessary to control the robot body to rotate the above-mentioned offset angle, re-take the falling image facing the falling object, and determine the target coordinate information based on the new position information corresponding to the new falling image.

[0141] The present application embodiment provides a fourth fall detection method, such as Figure 9 As shown, the following steps are included:

[0142] Step S901: After acquiring an image sequence, the image sequence and the posture information of the image sequence are sent to the fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, the target posture information corresponding to the image representing the fall is sent to the controller.

[0143] Step S902: After receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to the ranging end, so that the ranging end determines the position information between the robot and the falling object in the fall image based on the fall image.

[0144] Step S903: determining target coordinate information in a digital map coordinate system based on the position information, and controlling the movement of the robot based on the target coordinate information.

[0145] Step S904: After determining that the robot has moved to the position represented by the target coordinate information, a first notification message is sent through a first preset notification method.

[0146] The first notification message is a message indicating a fall inquiry.

[0147] As mentioned above, during the entire detection process, there may be misjudgments or the fallen subject may have stood up, or the fallen subject may need assistance, which requires the robot to further interact with the fallen subject;

[0148] Based on this, after determining the target coordinate information, this embodiment controls the robot to move to the fallen object, and notifies the first notification message through the first preset notification method to achieve interaction with the fallen object.

[0149] This embodiment does not specifically limit the first notification message and the first preset notification method. For example, the first notification message is a voice message representing a fall inquiry, such as "Are you falling?", "Do you need assistance?", etc., and the voice message is played through the robot's speaker.

[0150] In implementation, after the first notification message is sent, it is also necessary to determine the situation of the fallen object based on the interaction result and perform corresponding processing;

[0151] Based on this, in some optional implementations, after the above step S904, the following steps are further performed:

[0152] If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

[0153] Exemplarily, if a confirmation instruction for the first notification message is received within the preset time period of triggering the first notification message, it indicates that the fallen object has confirmed that a fall event has occurred and fall assistance is required; if no response is received or no valid response is received (no key words expressing confirmation or denial are recognized) within the preset time period of triggering the first notification message, it indicates that the fallen object is unable to interact effectively and fall assistance is required; in these cases, the second notification message needs to be notified through the second preset notification method so that the management personnel can provide assistance;

[0154] If a denial instruction for the first notification message is received within the preset time period of triggering the first notification message, it means that the fallen object has confirmed that no fall event has occurred and no fall assistance is required.

[0155] This embodiment does not specifically limit the second notification message and the second preset notification method. For example, a video call with the administrator is established, and subtitles such as "Someone has fallen" are prompted; or an alarm is initiated to the rescue party.

[0156] The above notification method is only an example and this application does not make any specific limitations on it.

[0157] Based on the same inventive concept, the present application embodiment provides a fall detection device, see Figure 10 As shown, the fall detection device 1000 includes:

[0158] A fall determination module 1001 is configured to, after acquiring an image sequence, send the image sequence and the pose information of the image sequence to a fall detection terminal, so that when the fall detection terminal determines that the image sequence contains an image representing a fall, it sends the target pose information corresponding to the image representing the fall to the controller;

[0159] a distance determination module 1002 configured to, after receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to a distance measurement terminal, so that the distance measurement terminal determines, based on the fall image, position information between the robot and the fallen object in the fall image; wherein the fall image is an RGB-D image;

[0160] The movement control module 1003 is configured to determine target coordinate information in a digital map coordinate system based on the position information, and control the movement of the robot based on the target coordinate information.

[0161] In some optional implementations, the fall determination module 1001 is specifically configured to:

[0162] When the robot is in the detection position, controlling the body of the robot to rotate multiple times in the horizontal direction based on preset rotation angles to obtain images corresponding to each preset rotation angle;

[0163] The images corresponding to all the rotation angles are determined as the image sequence corresponding to the detection position.

[0164] In some optional implementations, the fall determination module 1001 is specifically configured to:

[0165] For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

[0166] In some optional implementations, the position information includes an offset angle and a straight-line distance of the robot relative to the fallen object; the movement control module 1003 is specifically configured to:

[0167] Determining whether the offset angle is less than a preset angle threshold;

[0168] If yes, determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance;

[0169] Otherwise, the robot body is controlled to rotate by the offset angle, a new fall image is reacquired, and the target coordinate information is determined based on new position information corresponding to the new fall image.

[0170] In some optional implementations, the movement control module 1003 is specifically configured to:

[0171] Determine the difference between the direct distance and the preset distance as the target distance;

[0172] The target coordinate information is determined based on the current coordinate information, the target distance, and the target angle; wherein the target angle is the angle between the X-axis and the target connecting line in the digital map coordinate system, and the target connecting line is the connecting line between the robot and the fallen object.

[0173] In some optional implementations, the system further includes a notification module 1004 for:

[0174] After the movement control module 1003 controls the movement of the robot based on the target coordinate information, if it is determined that the robot moves to the position represented by the target coordinate information, a first notification message is notified through a first preset notification method; wherein, the first notification message is a message representing a fall inquiry.

[0175] In some optional implementations, after notifying the first notification message in the first preset notification manner, the notification module 1004 is further configured to:

[0176] If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

[0177] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0178] Based on the same technical concept, the embodiment of the present application also provides a controller 1100, such as Figure 11 As shown, it includes at least one processor 1101 and a memory 1102 connected to the at least one processor. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Figure 11 For example, the processor 1101 and the memory 1102 are connected via a bus 1103. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0179] Among them, the processor 1101 is the control center of the controller, which can use various interfaces and lines to connect various parts of the controller, and realize data processing by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 may include one or more processing units. The processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes the issuance of instructions. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 may be implemented on the same chip. In some embodiments, they may also be implemented separately on independent chips.

[0180] The processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiment of the fall detection method can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0181] Memory 1102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Memory 1102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0182] In an embodiment of the present application, the memory 1102 stores a computer program. When the program is executed by the processor 1101, the processor 1101 executes:

[0183] After acquiring the image sequence, sending the image sequence and the posture information of the image sequence to the fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller;

[0184] After receiving the target posture information, controlling the robot to collect a fall image based on the target posture information, and sending the fall image to the ranging end, so that the ranging end determines the position information between the robot and the fallen object in the fall image based on the fall image; wherein the fall image is an RGB-D image;

[0185] Based on the position information, target coordinate information in a digital map coordinate system is determined, and the robot is controlled to move based on the target coordinate information.

[0186] In some optional implementations, the processor 1101 specifically performs:

[0187] When the robot is in the detection position, controlling the body of the robot to rotate multiple times in the horizontal direction based on preset rotation angles to obtain images corresponding to each preset rotation angle;

[0188] The images corresponding to all the rotation angles are determined as the image sequence corresponding to the detection position.

[0189] In some optional implementations, the processor 1101 specifically performs:

[0190] For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

[0191] In some optional implementations, the position information includes an offset angle and a straight-line distance of the robot relative to the fallen object; the processor 1101 specifically executes:

[0192] Determining whether the offset angle is less than a preset angle threshold;

[0193] If yes, determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system and the straight-line distance;

[0194] Otherwise, the robot body is controlled to rotate by the offset angle, a new fall image is reacquired, and the target coordinate information is determined based on new position information corresponding to the new fall image.

[0195] In some optional implementations, the processor 1101 specifically performs:

[0196] Determine the difference between the direct distance and the preset distance as the target distance;

[0197] The target coordinate information is determined based on the current coordinate information, the target distance, and the target angle; wherein the target angle is the angle between the X-axis and the target connecting line in the digital map coordinate system, and the target connecting line is the connecting line between the robot and the fallen object.

[0198] In some optional implementations, after controlling the robot to move based on the target coordinate information, the processor 1101 further executes:

[0199] After determining that the robot has moved to the position represented by the target coordinate information, a first notification message is notified through a first preset notification method; wherein the first notification message is a message representing a fall inquiry.

[0200] In some optional implementations, after notifying the first notification message in the first preset notification manner, the processor 1101 further executes:

[0201] If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

[0202] Since the controller is the controller in the method in the embodiment of the present application, and the principle of solving the problem by the controller is similar to that of the method, the implementation of the controller can refer to the implementation of the method, and the repeated parts will not be repeated.

[0203] Based on the same technical concept, an embodiment of the present application further provides a computer-readable storage medium storing a computer program executable by a computer. When the program runs on the computer, the computer executes the steps of the above-mentioned fall detection method.

[0204] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0205] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0208] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0209] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A fall detection method, characterized in that: Applied to a controller in a robot, the method comprises: After acquiring the image sequence, sending the image sequence and the posture information of the image sequence to the fall detection end, so that when the fall detection end determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller; After receiving the target posture information, controlling the robot to collect a fall image based on the target posture information, and sending the fall image to the ranging end, so that the ranging end determines the position information between the robot and the fallen object in the fall image based on the fall image; wherein the fall image is a three-channel color and depth combined image; determining whether an offset angle of the robot relative to the fallen object is less than a preset angle threshold; If so, determining the difference between the straight-line distance of the robot relative to the fallen object and the preset distance as the target distance, and determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system, the target distance, and the target angle; wherein the target angle is the angle between the X-axis in the digital map coordinate system and the target connecting line, and the target connecting line is the connecting line between the robot and the fallen object; Otherwise, controlling the body of the robot to rotate by the offset angle, reacquiring a new fall image, and determining the target coordinate information based on new position information corresponding to the new fall image; The robot is controlled to move based on the target coordinate information.

2. The method according to claim 1, wherein Get an image sequence, including: When the robot is in the detection position, controlling the body of the robot to rotate multiple times in the horizontal direction based on preset rotation angles to obtain images corresponding to each preset rotation angle; The images corresponding to all the rotation angles are determined as the image sequence corresponding to the detection position.

3. The method according to claim 2, wherein Get the image corresponding to each preset rotation angle, including: For any preset rotation angle, the image acquisition device of the robot is controlled to swing multiple times in the vertical direction based on the preset swing angle, and images captured by the image acquisition device at each preset swing angle are obtained.

4. The method according to any one of claims 1 to 3, wherein: After controlling the movement of the robot based on the target coordinate information, the method further includes: After determining that the robot has moved to the position represented by the target coordinate information, a first notification message is notified through a first preset notification method; wherein the first notification message is a message representing a fall inquiry.

5. The method according to claim 4, wherein After notifying the first notification message in the first preset notification method, the method further includes: If no denial instruction for the first notification message is received within the preset time period of triggering the first notification message, a second notification message is notified through a second preset notification method; wherein the second notification message is a message representing fall assistance.

6. A fall detection device, characterized in that: Controllers used in robots include: a fall determination module, configured to, after acquiring an image sequence, send the image sequence and the posture information of the image sequence to a fall detection terminal, so that when the fall detection terminal determines that the image sequence contains an image representing a fall, it sends the target posture information corresponding to the image representing the fall to the controller; a distance determination module, configured to, after receiving the target posture information, control the robot to collect a fall image based on the target posture information, and send the fall image to a ranging terminal, so that the ranging terminal determines, based on the fall image, position information between the robot and the fallen object in the fall image; wherein the fall image is a three-channel color and depth combined image; a movement control module, configured to determine whether an offset angle of the robot relative to the fallen object is less than a preset angle threshold; If so, determining the difference between the straight-line distance of the robot relative to the fallen object and the preset distance as the target distance, and determining the target coordinate information based on the current coordinate information of the robot in the digital map coordinate system, the target distance, and the target angle; wherein the target angle is the angle between the X-axis in the digital map coordinate system and the target connecting line, and the target connecting line is the connecting line between the robot and the fallen object; Otherwise, controlling the body of the robot to rotate by the offset angle, reacquiring a new fall image, and determining the target coordinate information based on new position information corresponding to the new fall image; The robot is controlled to move based on the target coordinate information.

7. A controller, characterized in that: The system comprises at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that It stores a computer program executable by a computer, and when the program is run on the computer, the computer is enabled to execute the method according to any one of claims 1 to 5.

Images