A method for assisting a user to generate a safe path to a smart toilet and to judge a fall

Abstract

The application discloses a kind of based on artificial intelligence's auxiliary user to the fall alarm method and device of intelligent closestool, including the following steps: receiving real-time image from TOF camera;If receiving real-time image, establish the node position of user on real-time image;If receiving the joint position of user, generate the posture angle of user based on world reference system and the joint position of user;Determine whether the posture angle of user is located in safe angle range;If the posture angle of user exceeds safe angle range, then generate fall signal;If generating fall signal, then fall signal is sent to output device, can be used to identify user posture and issue alarm, so that user can be rescued in first time.

Description

[0001] Cross-references to related applications

[0002] This application is a divisional application based on application number 202111659390X, filed on December 31, 2021, entitled "A Method and Device for Assisting Users to Fall Off the Toilet Based on Artificial Intelligence". Technical Field

[0003] This invention relates to machine vision technology, specifically to an artificial intelligence-based method and device for assisting users in falling off the toilet. Background Technology

[0004] Path planning is one of the main research directions in machine vision. A sequence of points or curves connecting a starting point and an ending point is called a path, and the strategy for constructing a path is called path planning. It has wide applications in many fields, such as obstacle avoidance for vehicles, setting the gripping trajectory for robotic arms, and navigation for users.

[0005] Currently, visually impaired individuals are prone to falls while walking, especially in indoor spaces with many obstacles and poor lighting. Visual impairment can be caused by aging, eye diseases, or congenital factors. For example, elderly people are highly susceptible to falls in toilets due to the confined space, poor lighting, and haphazardly placed obstacles.

[0006] In order to solve the problem of toilet access for visually impaired people, a navigation system for the visually impaired has been developed using existing technology. The system uses shoes equipped with cameras to guide people's way. The system includes the following steps: (1) Before use, the cameras located at the front of the sole and the side of the front of the sole are turned on by a power switch; (2) The camera at the front of the sole captures images of the user's direction of travel and transmits the images to the image processing module; (3) The image processing module processes the images and performs image recognition, and the processor determines whether there are obstacles in the direction of travel.

[0007] However, the existing technology is not perfect. The existing visually impaired navigation systems are all wearable, which means that users must wear the system to use it. The existing visually impaired navigation systems cannot monitor the user's posture. If the user falls, the system cannot recognize the user's posture, which means that the user cannot get help in time. Invention Content

[0008] To overcome the shortcomings and problems of existing technologies, this invention provides an artificial intelligence-based method and device for assisting users in falling off the toilet.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] An artificial intelligence-based method for assisting a user to fall alarm to a toilet, comprising the following steps:

[0011] receiving real-time images from a TOF camera;

[0012] establishing a node position of the user on the real-time images if the real-time images are received;

[0013] generating a posture angle of the user based on a world reference frame and the joint position of the user if the joint position of the user is received;

[0014] determining whether the posture angle of the user is within a safe angle range;

[0015] generating a fall signal if the posture angle of the user exceeds the safe angle range;

[0016] sending the fall signal to an output device if the fall signal is generated.

[0017] Preferably, the node position of the user includes the head, the neck joint, the shoulder joint, the elbow joint, the hand, the hip joint, the knee joint, the ankle joint, and the pelvic joint, wherein the node position on the upper body of the user is used as a priority reference index. If the posture angle established based on the node position is greater than the maximum inclination posture angle threshold of the human body in a normal condition, the control module responds quickly and determines the abnormality.

[0018] Preferably, the world coordinate system is established based on the edge line of the real-time images.

[0019] Preferably, the step of generating the posture angle of the user based on the world reference frame and the joint position of the user if the world reference frame and the joint position of the user are received, comprises:

[0020] converting the joint position of the user into a posture line if the joint position of the user is received;

[0021] generating the posture angle of the user based on the posture line and the world reference frame if the posture line is received.

[0022] Preferably, the line connecting the midpoint between the neck joint position and the two hip joint positions, or the line connecting the midpoint between the neck joint position and the two knee joint positions, or the line connecting the midpoint between the neck joint position and the two ankle joint positions can be used as the posture line.

[0023] Preferably, it further comprises:

[0024] receiving initial images from a TOF camera;

[0025] converting the initial images into initial safe area images;

[0026] if the real-time image is received, segmenting a part of the initial safety area image mapped on the real-time image as a real-time detection area image;

[0027] generating a real-time safety area model according to the initial safety area image and the real-time detection area image;

[0028] judging whether there is a user image on the real-time detection area image;

[0029] if there is a user image on the real-time user image, segmenting the user image on the real-time detection area image;

[0030] converting the user image into a user position;

[0031] receiving request information from an input device, the request information including a destination position;

[0032] generating a safety path from the user position to the destination position on the real-time safety area model.

[0033] As preferred, further comprising:

[0034] generating a predicted pose of the real-time user image by using a Kalman filtering algorithm;

[0035] if the predicted pose is accepted, generating a motion parameter by using a human motion tracking algorithm;

[0036] judging whether the motion parameter exceeds a safety motion range;

[0037] if the safety parameter exceeds the safety motion range, generating a corrected path;

[0038] if the corrected path is generated, sending the corrected path to an output device.

[0039] In another aspect, the present application also provides an artificial intelligence-based auxiliary user-to-toilet fall alarm device for implementing the above-mentioned artificial intelligence-based auxiliary user-to-toilet fall alarm method, comprising:

[0040] a TOF camera for generating a real-time image;

[0041] a control module for receiving the real-time image, establishing a node position of the user on the real-time image if the real-time image is received, generating a pose angle of the user based on a world reference system and the joint position of the user if the joint position of the user is received, judging whether the pose angle of the user is within a safety angle range, and generating a fall signal if the pose angle of the user exceeds the safety angle range;

[0042] an output device for receiving the fall signal.

[0043] As a preference, the output device comprises at least one of a loudspeaker, a display, a vibrator, and a heater.

[0044] The present application has the following prominent and beneficial technical effects compared with the prior art:

[0045] (1) The present application is mainly used to solve the problem that the visually impaired population falls easily when using the toilet, especially the elderly with visual impairment. The control module establishes node positions, calculates posture angles, and determines whether the posture angle is within a safe angle range, thereby determining whether the elderly have fallen in the toilet. If the elderly fall, a fall signal is immediately sent to the output device so that the elderly can receive rescue in the first time.

[0046] (2) In the present application, the processing flow of the control module for real-time images is relatively simple, which can effectively reduce the processing time and reduce the device requirements of the control module. The output device can convert the fall signal into sound, light, vibration, etc. It is suitable for promotion in various user groups and various places, especially in the visually impaired population and the elderly. Moreover, the visually impaired population and the elderly do not necessarily need to wear a TOF camera, a control module, and an output device.

[0047] (3) In the present application, the TOF camera can capture multiple users and multiple destinations within the shooting range, and can monitor the posture of multiple users. Therefore, the present application supports one-to-many use, which helps to reduce the cost of use. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a step flow structure schematic diagram of a fall alarm method for assisting users to the toilet based on artificial intelligence according to the present application;

[0049] Figure 2 is a three-view for guiding users to the toilet according to the present application;

[0050] Figure 3 is a schematic diagram of the initial images overlapped by the TOF camera in multiple views according to the present application;

[0051] Figure 4 is an initial image captured by the TOF camera in the toilet according to the present application;

[0052] Figure 5 is a schematic diagram of a real-time image of a user standing in the toilet captured by the TOF camera and establishing a two-dimensional posture on the user according to the present application;

[0053] Figure 6 is a schematic diagram of a real-time image of a user falling in the toilet captured by the TOF camera and establishing a two-dimensional posture on the user according to the present application;

[0054] Figure 7 is the basis of the present invention Figure 4 the top view of the three-dimensional point cloud converted from the initial image according to

[0055] Figure 8 is the basis of the present invention Figure 5 the top view of the three-dimensional point cloud converted from the real-time image according to

[0056] Figure 9 is the structure diagram of the present invention Figure 7 establishing an initial safety area model;

[0057] Figure 10 is a structure diagram of a fall alarm device for assisting a user to a toilet based on artificial intelligence according to the present invention.

[0058] Figure 11 is a structure diagram of a TOF camera according to the present invention;

[0059] Figure 12 is a cross-sectional structure diagram of a TOF camera according to the present invention;

[0060] Figure 13 is an exploded structure diagram of a TOF camera according to the present invention;

[0061] Figure 14 is a structure diagram of a smart toilet according to the present invention;

[0062] Figure 15 is a structure diagram of a smart toilet according to the present invention, with the gasket lifted on the toilet seat;

[0063] Figure 16 is a flow framework diagram of a fall alarm method for assisting a user to a toilet based on artificial intelligence according to the present invention;

[0064] In the figure: 1-TOF camera, 2-control module, 3-output device, 4-smart toilet, 5-input device, 11-housing, 12-link, 13-wall plate, 14-control mainboard, 15-VCSEL laser, 16-TOF camera module, 17-cover plate, 18-LED light strip, 19-motor, 111-opening, 112-arc surface, 121-slot, 41-toilet seat, 42-gasket, 43-pedal. DETAILED DESCRIPTION

[0065] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with the drawings and specific embodiments.

[0066] As Figures 1 to 16As shown, the application provides an artificial intelligence-based fall alarm device for assisting users to the toilet, which can implement all steps in the artificial intelligence-based fall alarm method for assisting users to the toilet provided by the embodiments of the application, to solve the problem of no one finding and reporting after the user falls, and to avoid the problem of the user being unable to get help in the first time after falling.

[0067] As shown, Figure 10 As shown, the application provides an artificial intelligence-based fall alarm device for assisting users to the toilet, which includes a TOF camera, a control module, an output device, and an input device. The input device, the TOF camera, and the output device are connected to the control module through wireless communication.

[0068] The TOF camera is used to generate a depth image. TOF is an abbreviation of time of flight camera, which is a kind of depth sensor. The depth image refers to an image containing depth data, which can be used for three-dimensional reconstruction of a scene. The depth data refers to the distance information from the surface of the scene to the viewpoint. The scene refers to the objects within the shooting range of the TOF camera. The viewpoint refers to the position of the TOF camera.

[0069] The depth image includes an initial image and a real-time image. The initial image refers to the image captured by the TOF camera in the initial scene. The initial scene refers to a scene in which the positions of initial obstacles such as walls, toilets, and washstands are basically fixed. Comparing the real-time captured scene with the initial scene, the scene may have more users and new obstacles. The user generally refers to a person, and can also be a moving object. The new obstacle refers to the obstacle in the scene except the initial obstacle. The initial obstacle and the new obstacle are collectively referred to as obstacles. In actual use, the initial image can be preprocessed and pre-set in the control module, and the subsequent real-time image only needs to identify the user and the new obstacle.

[0070] As shown, Figure 2 As shown, the path planning device is used to guide the user to the toilet. The TOF camera is installed on the wall surface in the toilet. The shaded area in the figure represents the shooting range of the TOF camera. The door, the intelligent toilet, the washstand, and part of the floor are located in the shooting range of the TOF camera, and the user can step into the shooting range of the TOF camera. The scene in the toilet includes the door, the washstand, the intelligent toilet, the shower room, the floor, the wall surface, and the user, etc. If the user needs to use the toilet, the artificial intelligence-based fall alarm device for assisting users to the toilet can generate a safe path in the toilet. The dotted line with an arrow is used to represent the safe path in the toilet. The starting position of the safe path is located at the user, and the ending position of the safe path is located at the intelligent toilet. If the TOF camera is shooting the toilet, the TOF camera can generate a depth image, which can be used for three-dimensional reconstruction of the captured scene in the toilet.

[0071] Specifically, the TOF camera comprises a VCSEL laser, a TOF camera module and a control mainboard. The VCSEL laser is a component for emitting laser. The laser emitted by the VCSEL laser is high-performance pulsed light, and the pulse can reach about 100MHz. The TOF camera module is a component for converting optical signals into electrical signals. The VCSEL laser and the TOF camera module are electrically connected to the control mainboard respectively, and the control mainboard is used for processing depth images. Compared with a binocular stereo vision camera or a structured light camera, the TOF camera has better anti-strong light capability, and the precision is less affected by the shooting distance, which can basically be maintained at the cm level.

[0072] The TOF camera further comprises a casing, the VCSEL laser, the TOF camera module and the control mainboard are fixedly arranged in the casing, and a motor is also fixedly arranged in the casing. The VCSEL laser, the TOF camera module and the motor are electrically connected to the control mainboard respectively, and a wall hanging rack is fixedly arranged on a machine shaft of the motor. The wall hanging rack is used for being fixedly arranged on a wall or a smart toilet.

[0073] The casing is hollow, and the casing protects the VCSEL laser, the TOF camera module, the control mainboard and the motor. The motor is used for converting electrical energy into mechanical energy. When the motor works, the motor can drive the casing to rotate relative to the wall hanging rack, so that the direction of the VCSEL laser and the TOF camera module on the wall hanging rack can be adjusted. The VCSEL laser, the TOF camera module and the motor are electrically connected to the control mainboard respectively, and the control mainboard is used for controlling the work of the VCSEL laser, the TOF camera module and the motor.

[0074] In actual use, the wall hanging rack is fixedly arranged on a smart toilet or a wall of a toilet, the motor drives the casing to rotate left and right relative to the wall hanging rack, and the TOF camera module can shoot in different directions, thereby improving the shootable range of the TOF camera module.

[0075] In the present application, the motor can drive the casing to rotate relative to the wall hanging rack, the VCSEL laser can emit laser towards different angles, and the TOF camera module can shoot towards different angles, thereby improving the shooting range of the TOF camera, which is helpful for the TOF camera module to shoot in all directions in the toilet, so that the present application has the advantages of wide shooting range, low cost and simple structure.

[0076] The wall hanging rack comprises a wall plate and a connecting rod, the connecting rod is fixedly arranged on the wall plate, the overall structure of the connecting rod is in the shape of "L", one end of the connecting rod is fixedly arranged on the wall plate, and the other end of the connecting rod is fixedly arranged on the machine shaft of the motor.

[0077] Since the shell needs to be transversely rotated relative to the wall or the intelligent toilet, in actual use, the wall plate is fixedly arranged on the wall or the intelligent toilet, and through the design of the "L"-shaped connecting rod, the motor can drive the VCSEL laser and the TOF camera module to be transversely rotated relative to the wall.

[0078] The shaft of the motor is inserted into one end of the connecting rod, and the one end of the connecting rod is provided with a slot, and the shaft of the motor is inserted into the slot.

[0079] The shaft of the motor and the connecting rod are connected by insertion, so that the user can disassemble and assemble the motor and the connecting rod manually without the aid of tools.

[0080] Specifically, the cross section of the slot is a regular hexagon, and the shaft of the motor is adapted in the slot, thereby improving the connection firmness of the shaft of the motor and the connecting rod.

[0081] The shell is provided with an opening, the VCSEL laser and the TOF camera module are respectively directed towards the opening, and a light-transmitting cover plate is fixedly arranged on the shell and covers the opening.

[0082] In actual use, the laser emitted by the VCSEL laser can pass through the cover plate, and the external light can pass through the cover plate and shine on the TOF camera module.

[0083] Further comprising an LED light strip, the LED light strip is electrically connected to the control mainboard, and the shooting range of the TOF camera module is located in the illumination range of the LED light.

[0084] In actual use, the LED light strip can illuminate the toilet, which can avoid the problem that the user cannot see the road condition, and also helps the TOF camera module to collect clear depth images.

[0085] The LED light strip is fixedly arranged on the shell, and the shell comprises an arc surface, and the LED light strip is wrapped around the arc surface, so that the LED light strip can illuminate towards multiple angles, thereby improving the illumination effect. Specifically, the opening is arranged on the arc surface.

[0086] The control module is configured to receive an initial image, a real-time image, and request information. The initial image and the real-time image come from a TOF camera. The request information comes from an input device, and the request information includes a destination position. If the real-time image is received, a node position of a user is established on the real-time image. If a joint position of the user is received, a posture angle of the user is generated based on a world reference system and the joint position of the user. It is determined whether the posture angle of the user is within a safe angle range. If the posture angle of the user exceeds the safe angle range, a fall signal is generated.

[0087] Wherein, the falling of the user is a process of body posture change, and also can be a process of a series of motion changes of the skeleton and the joint. The current posture of the user can be determined by judging the spatial features of the joint of the user, and then whether the user falls can be determined. The node position of the user refers to the coordinates used to simulate the joint position of the user, as a kind of spatial features. The plane reference system refers to the absolute coordinate system in the scene. The posture angle refers to the actual posture of the user in the scene. In some ways, the convolutional neural network is used to establish the joint position of the user on the real-time image.

[0088] Specifically, the two-dimensional posture of the user in the real-time image is estimated by using the convolutional neural network. The two-dimensional posture of the user refers to a line drawing simulating the skeleton of the user in the real-time image, which is composed of a plurality of line segments, and the intersection points between the plurality of line segments and the end points of the line segments can simulate the joint nodes of the user. When the two-dimensional posture of the user is obtained, the joint position of the user can be determined by the positions of the intersection points and the end points of the two-dimensional posture.

[0089] In some ways, the node position of the user includes the head, the neck joint, the shoulder joint, the elbow joint, the hand, the hip joint, the knee joint, the ankle joint and the pelvic joint. Wherein, the node position located on the upper body of the user is used as a priority reference judgment index. If the posture angle established based on the node position is greater than the maximum inclination posture angle threshold of the human body in the normal case, the control module quickly responds and judges the abnormality.

[0090] In some ways, the world coordinate system is established based on the edge line of the real-time image.

[0091] Wherein, the edge line of the real-time image refers to at least one of the upper, lower, left and right boundary lines of the entire real-time image. In actual use, since the TOF camera is installed on the scene in advance, its position is determined, and therefore the edge line of the real-time image is used as the coordinate axis of the coordinate system. As shown in Figure 3 The lower edge line of the real-time image is used as the X coordinate axis, and the left edge line of the real-time image is used as the Y coordinate axis. Figure 5 In the figure, the posture of the user is standing, Figure 6 In the figure, the posture of the user is falling, and the plurality of lines on the user are the two-dimensional posture of the user.

[0092] In some ways, when the world reference system and the joint position of the user are received, the step of generating the posture angle of the user based on the world reference system of the real-time image and the joint position of the user includes:

[0093] If the joint position of the user is received, the joint position of the user is converted into a posture line;

[0094] If the posture line is received, a posture angle of the user is generated based on the posture line and the world reference system.

[0095] The posture line is used to simulate the posture of the user, and facilitates subsequent calculation of the posture angle. The posture line is determined according to the joint positions of the user.

[0096] Specifically, the posture line is determined according to the following method: first, a line is constructed between the two hip joint positions or between the two knee joint positions or between the two ankle joint positions, and finally, a line is constructed between the midpoint of the line between the neck joint position and the two hip joint positions or between the midpoint of the line between the neck joint position and the two knee joint positions or between the midpoint of the line between the neck joint position and the two ankle joint positions. The line between the midpoint of the line between the neck joint position and the two hip joint positions or between the midpoint of the line between the neck joint position and the two knee joint positions or between the midpoint of the line between the neck joint position and the two ankle joint positions can be used as the posture line.

[0097] In the above, the posture angle refers to the included angle between the posture line and the coordinate axis of the world reference system. In the present embodiment, the lower edge line of the real-time image is used as the X coordinate axis of the world reference system, and the posture angle is the included angle between the posture line and the lower edge line of the real-time image.

[0098] When the line between the midpoint of the line between the neck joint position and the two hip joint positions is used as the posture line. If the posture angle exceeds the safe angle range, it is determined that the user has fallen. If the posture angle is within the safe angle range, it is determined that the user has not fallen.

[0099] When the line between the midpoint of the line between the neck joint position and the two knee joint positions is used as the posture line. If the posture angle exceeds the safe angle range, it is determined that the user has fallen. If the posture angle is within the safe angle range, it is determined that the user has not fallen.

[0100] When the line between the midpoint of the line between the neck joint position and the two ankle joint positions is used as the posture line. If the posture angle exceeds the safe angle range, it is determined that the user has fallen. If the posture angle is within the safe angle range, it is determined that the user has not fallen.

[0101] Because the real-time images captured by the TOF camera may only capture a portion of the user's problem, the control module can only simulate some joint positions. To solve this problem, this invention determines which joint positions exist in the real-time image and then decides which joint positions to use to generate attitude lines. Specifically, it first determines whether both hip joints exist simultaneously. If both hip joints exist simultaneously, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two hip joint positions. If neither hip joint exists, it then determines whether both knee joints exist simultaneously. If both knee joints exist simultaneously, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two knee joint positions. If neither knee joint exists, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two ankle joint positions.

[0102] In some methods, upon receiving a real-time image from a TOF camera, the position of the toilet seat and the user's hip joint are determined; it is then determined whether the hip joint position is higher than the toilet seat position; if the hip joint position is higher than the toilet seat position, the user is determined to have fallen; if the hip joint position is lower than the toilet seat position, the user is determined not to have fallen. In this embodiment, the user's posture angle and hip joint position are used to determine whether the user has fallen, achieving a dual determination effect and improving the accuracy of the system's fall detection.

[0103] In some methods, the control module can also track the user's node states in real-time images, determining whether the user's posture is abnormal based on the joint node states. Specifically, assuming there are K joint nodes, and the joint node states include position, velocity, motion angle, and direction, the motion state at time t is... The joints are mutually constrained. Priority is given to adding attention weights to the states of joints in the upper body, such as the head, shoulder, and neck. Attention weights are added to the states of these joints. The processing speed for a single frame of depth image is calculated as p1 frames / second, moving from top to bottom. The position of the object in each frame is calculated based on the state at different times t. Velocity matching is performed based on the difference between the current and previous frame positions, resulting in a 10-frame moving average. The motion angle direction is the direction vector between the current and previous frame positions, stored as the direction vector during walking. Based on a series of fall-related actions, multiple frames of state values ​​are collected and stored for judgment. When falling forward or backward, the joints of the upper body experience a sudden change in position and velocity. The trajectory path of priority joints is tracked. The depth information of the lower body changes slowly, while that of the upper body changes rapidly. A comprehensive judgment is made based on the five values ​​of the state variables to determine an abnormal posture.

[0104] In some ways, the control module is also used to convert the initial image into an initial safety area image if the initial image is received from the TOF camera; if a real-time image is received from the TOF camera, the part of the initial safety area image mapped on the real-time image is segmented as a real-time detection area image; if the real-time detection area image is received, a safety area model is generated according to the initial safety area image and the real-time detection area image; if the real-time detection area image is received, it is judged whether there is a user image on the real-time detection area image; if there is a user image on the real-time user image, the user image is segmented on the real-time detection area image; if the user image segmented on the real-time detection area image is received, the user image is converted into a user position; if the request information from the input device is received, a safety path from the user position to the destination position is generated on the real-time safety area model.

[0105] It should be noted that the present artificial intelligence-based auxiliary user-to-toilet fall alarm device is mainly used in indoor scenes. For example, if the indoor scene is a toilet, the present artificial intelligence-based auxiliary user-to-toilet fall alarm device can plan a path for the user to use the toilet, take a bath, wash, etc. It solves the problem that the user has difficulty moving and is easy to trip due to visual impairment, poor indoor lighting, and obstacles piled up randomly. Among them, visual impairment refers to people who are blind, partially blind, or have visual impairment. The safety area refers to the ground area after removing the obstacle position and the area above the ground. If the user walks in the safety area, the user will not touch the obstacle, thereby achieving the effect of avoiding obstacles. If the user walks outside the safety area, the user is most likely to knock on the obstacle, thereby causing the user to fall. If the user falls, it is most likely to cause scratches, bumps, fractures, or even syncope, especially for users over 75 years old, which is more likely to cause serious injuries.

[0106] Among them, the control module is an integrated chip. Since the shooting range of the TOF camera is limited, if it is necessary to perform three-dimensional reconstruction on the entire scene shot, there is a problem that the entire scene cannot be well expressed through a single view image. In order to solve the above problem, the depth image received by the control module can be a depth image shot by the TOF camera at a single view or a depth image shot by the TOF camera at multiple views. For example, if the TOF camera collects an initial scene image, the TOF camera can shoot the initial scene at multiple views and generate multiple initial images at multiple views. Since the initial images shot at multiple views have overlapping areas, the multiple initial images can be combined into a single initial image by overlapping the overlapping areas of the multiple initial images, thereby solving the problem that the initial image cannot be well reconstructed in three dimensions. In the real-time scene, only the real-time detection area image in the real-time image needs to be processed, in order to reduce the processing load of the control module, the TOF camera only needs to collect an image at a single view.

[0107] The initial safe area image refers to a depth image used to simulate an initial safe area. The initial safe area refers to a ground area that does not contain initial obstacles and an area above the ground area. If no new obstacle is added in the initial safe area and the user walks on the initial safe area, the user will not bump into the initial obstacle. If the user walks outside the safe area, the user is very likely to bump into an obstacle. The initial obstacle refers to an obstacle that is determined in the scene and fixed. The initial obstacle can include a wall, a toilet, a washstand, a shower room, and the like.

[0108] The real-time detection area image refers to a part on the real-time image and at the same position as the safe area image. In the embodiment, the real-time detection area image is segmented from the real-time image by using a mapping method. If the initial safe area image is mapped on the real-time image, the mapped part on the real-time image is segmented along the contour of the initial safe area image.

[0109] The user image refers to a depth image used to simulate a user. Since the TOF camera has a limitation of a shooting range, the user image can be a depth image simulating the entire user or a depth image simulating a part of the user.

[0110] The user position refers to a position used to indicate a position of a user model in a real-time safe area model, and further indicate an actual position of the user in the scene. The user model refers to a three-dimensional grid model used to simulate the user in real time. The user can be a person or a moving object in particular. The destination position refers to a position used to indicate a position of a destination model in the real-time safe area model. The destination model refers to a three-dimensional grid model used to simulate the destination. The destination can be an obstacle or a specific coordinate position.

[0111] The three-dimensional grid model refers to a topology and a spatial structure used to indicate a contour of a surface of a three-dimensional model, which is defined by a set of polygons. The polygon can be a triangle. For example, if the shooting scene of the TOF camera is a toilet, the three-dimensional grid model established by the control module according to the depth image simulating the toilet is used to simulate the scene in the toilet.

[0112] The adaptive Gaussian mixture algorithm is used to determine whether there is a user image on the detection region image in real time. The adaptive Gaussian mixture algorithm, also known as an AGMM detection model algorithm, adds a priori judgment of the background Gaussian distribution and an optimal mechanism of the update rate to the traditional Gaussian mixture algorithm. The adaptive Gaussian mixture algorithm can simultaneously determine and distinguish multiple users in an image. In a real-time image, since the user has sharp edge lines, the user image is coarsely segmented by edge detection to obtain a coarse edge contour of the user image. The coarse edge contour is finely segmented by depth information to obtain a fine edge contour of the user image, and the discrete points in the fine edge contour are removed. The image in the fine edge contour is the user image, and the user image can be segmented from the real-time image along the fine edge contour.

[0113] The safe path refers to a relatively safe path from a starting position to a terminal position. The safe path is located on the real-time safe region model, the starting position is the user position, and the terminal position is the destination position. If the user position moves along the safe path, the user position will not collide with the initial obstacle model and the new obstacle model and finally safely arrive at the destination position. If the user position does not move along the safe path, the user position is likely to collide with the obstacle. In actual use, the user can walk in the real-time scene according to the trajectory of the safe path, so as to safely arrive at the destination.

[0114] In some modes, the step of converting the initial image into an initial safe region image specifically includes:

[0115] If the initial image is received, the initial image is three-dimensionally reconstructed and an initial three-dimensional grid model is established;

[0116] The initial safe region model is segmented on the initial three-dimensional grid model;

[0117] The initial safe region model is converted into an initial safe region image.

[0118] The initial safe region model refers to a three-dimensional grid model of a user simulated initial safe region.

[0119] In some modes, if the initial image is received, the step of three-dimensionally reconstructing the initial image and establishing an initial three-dimensional grid model specifically includes:

[0120] If the initial image is received, the initial image is preprocessed and converted into an initial three-dimensional point cloud;

[0121] If the initial three-dimensional point cloud is received, the initial three-dimensional point cloud is preprocessed and a normal vector of the initial three-dimensional point cloud is established;

[0122] If the initial three-dimensional point cloud with the normal vector is received, a camera pose is estimated by using an ICP algorithm.

[0123] If the camera pose is received, then the initial three-dimensional point cloud with normal vectors is fused according to the camera pose to generate an initial fused point cloud;

[0124] If the initial fused point cloud is received, then the initial fused point cloud is converted into an initial three-dimensional mesh model by using a Poisson reconstruction algorithm.

[0125] Wherein, the camera pose is estimated by using the ICP algorithm, specifically, receiving the initial images of multiple perspectives taken by the TOF camera, and converting the initial images of multiple perspectives into the initial three-dimensional point clouds with normal vectors, in the initial three-dimensional point clouds with normal vectors of multiple perspectives, the initial three-dimensional point cloud with normal vectors of the previous perspective and the initial three-dimensional point cloud with normal vectors of the next perspective are matched, the matching process is to first extract the key points in the initial three-dimensional point cloud with normal vectors for coarse matching, and then fine matching, and thus the camera pose is solved, and finally the solved camera pose is processed by graph optimization and back-end optimization to obtain the final camera pose.

[0126] Specifically, assuming that the initial images of three perspectives taken by the TOF camera are A1, B1 and C1, 1 / 2 and 1 / 4 downsampling is respectively performed to generate A2, B2, C2 and A3, B3, C3, the corresponding three-dimensional point clouds are denoted by APj, BPj, CPj, j takes 1, 2, 3, and the level number after sampling is denoted by Level, the point clouds in the overlapping areas of A3 and B3 and A3 and C3 are sequentially matched, and the matching process is performed by traversing the point clouds in the overlapping areas i and minimizing the projection difference. According to the minimization formula (R3*BP3i+T3)*BN3-AP3i*AN3i, the rotation matrix R3 and the translation matrix T3 are calculated, the non-overlapping area is projected onto A3 by using the rotation matrix to generate a cube V3. R3 and T3 are taken as the initial values of R2 and T2 and brought into the formula (R2*BP2i+T2)*BN2-AP2i*AN2i to perform minimum optimization, and then R2 and T2 are taken as the initial values of R1 and T1 (R1*BP1i+T1)*BN1-AP1i*AN1i to perform minimum optimization, so as to obtain the final R1 and T1. According to R1, T1 and the coordinates of the overlapping area point clouds, the coordinates of the non-overlapping area point clouds are calculated to complete the matching. After the matching, three-dimensional models V1, V2 and V3 of three scale images are respectively generated, the front perspective VA1, VA2 and VA3 belong to subsets of V1, V2 and V3, and the three-dimensional models of the ground, the wall and the initial obstacle are detected based on V3 and A3, B3 and C3 after the holes are filled, so as to obtain the background area for position marking, generate an initial safety area model, mark the coordinates of the initial safety area model, combine the initial safety area with the three-dimensional model under the front perspective to generate an initial safety area three-dimensional model under the front perspective, and map the initial safety area coordinates back to VA3 under the front perspective, that is, VSafe3. Whether there is a human body or a temporary obstacle (broom, stool, etc.), the initial safety area coordinates are mapped back to VA2 under the front perspective, that is, VSafe2. The initial safety area coordinates are mapped back to VA1 under the front perspective, that is, VSafe1, and the path planning is performed. The fall detection is performed.

[0127] In addition, after the initial image is preprocessed, the three-dimensional point cloud can be calculated through coordinate system conversion. The three-dimensional point cloud is the three-dimensional coordinates of the pixels based on the camera coordinate system.

[0128] In some ways, if the initial image is received, the step of preprocessing the initial image and converting it into an initial three-dimensional point cloud includes:

[0129] If the initial image is received, the initial image is denoised and hole repaired.

[0130] The initial image after denoising and hole repairing is converted into an initial three-dimensional point cloud.

[0131] Since the TOF camera is prone to produce incomplete initial images during shooting due to the reasons such as incomplete shooting of the entire user and reflection of the user's clothing, etc., in order to solve the problem of holes in the initial image, the initial image needs to be repaired. The hole in the initial image refers to the pixel point with a depth information of 0 in the initial image. The hole repair of the initial image refers to searching for the hole in the initial image, determining the position of the hole center and the edge position of the hole, expanding the edge of the hole by 3 pixel points, obtaining the depth information of the three-ring contour and the change of the depth information, generating a diameter line passing through the hole center, and searching for the surrounding effective pixel value in the eight field direction in a spiral curve manner, and establishing a hole function containing the distance value and the weight according to the three-ring contour distance value variable, and completing the depth information on the hole according to the hole function. The coordinates of the converted initial three-dimensional point cloud are based on the camera coordinate system.

[0132] The depth image collected by the TOF camera can also be down-sampled and processed by the image pyramid algorithm, which is helpful for subsequent image recognition, image segmentation and image conversion of the depth image of the TOF camera. The down-sampling of the depth image refers to scaling the original proportion of the depth image by 1 / 2, 1 / 4, etc. to reduce the dimension of the feature and retain the effective information, to a certain extent to avoid overfitting and keep the rotation, translation and scaling invariant. Assuming that the original proportion of the depth image is called Level1 image, the initial image scaled by 1 / 2 is called Level2 image, and the depth image scaled by 1 / 4 is called Level3 image. The Level1 image is the original image with high-definition pixel points, which can be used to determine the user's posture detection, such as judging whether the user falls down. The clarity of the Level2 image is lower than that of the Level1 image, but higher than that of the Level3 image. After denoising processing of the Level2 image, it can be used for tracking of moving objects. Compared with the Level1 image and the Level2 image, the Level3 image has the lowest clarity, which can be used for hole repair and background three-dimensional reconstruction.

[0133] In some ways, if the initial three-dimensional point cloud is received, the steps of preprocessing the initial three-dimensional point cloud and establishing the normal vector of the initial three-dimensional point cloud include:

[0134] If the initial three-dimensional point cloud is received, the initial three-dimensional point cloud is filtered;

[0135] The normal vector of the filtered initial three-dimensional point cloud is calculated.

[0136] The filtering of the initial three-dimensional point cloud can use at least one of a voxel filtering algorithm, a bilateral filtering algorithm, a radius filtering algorithm, and a Gaussian filtering algorithm. For example, if the filtering of the initial three-dimensional point cloud uses the radius filtering algorithm, the center position of the initial three-dimensional point cloud and the position of any three-dimensional point are determined, the distance between the center position and the three-dimensional point is estimated, and the mean of the distance is calculated. It is determined whether the distance between the three-dimensional point and the center position is within the mean minus n times the standard deviation. If the three-dimensional point is within the mean minus n times the standard deviation, the three-dimensional point is retained. If the three-dimensional point is not within the mean minus n times the standard deviation, the three-dimensional point is a discrete point, and the three-dimensional point is removed. Calculating the normal vector of the filtered initial three-dimensional point cloud means determining the angle direction and the positive or negative sign of the initial three-dimensional point cloud. The positive or negative sign of the initial three-dimensional point cloud is obtained in the following manner: the distance information of the model surface is calculated, the surface indicator function is generated according to the distance information of the model surface, the positive or negative sign of the three-dimensional point cloud on the object surface is 0, the positive or negative sign of the three-dimensional point cloud in front of the object surface is negative, and the positive or negative sign of the three-dimensional point cloud behind the object surface is positive.

[0137] In some ways, the step of segmenting the initial safety area model on the initial three-dimensional mesh model specifically includes:

[0138] If the three-dimensional mesh model is received, the initial obstacle model and the background model are segmented on the three-dimensional mesh model.

[0139] If the initial obstacle model and the background model are received, the initial safety area model is segmented on the background model, and the initial obstacle model is not on the initial safety area model.

[0140] The initial obstacle model refers to a three-dimensional mesh model used to simulate an initial obstacle. The background model refers to a three-dimensional mesh model used to simulate the ground. Because the edges of the initial obstacle model are relatively sharp and multi-sided, and the edges of the background model are relatively flat, the initial obstacle model and the background model can be segmented from the three-dimensional mesh model by an interactive image segmentation method. After the initial obstacle model and the background model are determined, the initial safety area model can be determined by removing the obstacle model from the background model.

[0141] In some ways, the step of generating the real-time safety area model according to the initial safety area image and the real-time detection area image specifically includes:

[0142] It is determined whether there is a new obstacle image on the real-time detection area image.

[0143] If there is a new obstacle image on the real-time detection area image, the real-time detection area image is converted into a real-time safety area model.

[0144] If there is no new obstacle image on the real-time detection area image, the initial safety area model is taken as the real-time safety area model.

[0145] The new obstacle image refers to a depth image used to simulate a new obstacle. The real-time safety area model refers to a three-dimensional grid model used to simulate a real-time safety area. The real-time safety area refers to an area in a real-time scene that is safe for a user to pass through. If the user passes through the real-time safety area, the user can safely pass through. If the user passes through outside the real-time safety area, the user is likely to knock into the initial obstacle and the new obstacle.

[0146] Specifically, the initial safety area model is generated by determining ground positions Gi, wall positions Wi, and initial obstacle positions Bi, fitting the edges of the positions to form line segments, extending and intersecting the line segments, so that the multiple line segments enclose a closed polygonal area, which is the safety area Region. The polygon can be referred to as the skeleton of the safety area. The safety path is generated by determining the departure direction of the starting point position, calculating the included angles between the edge lines of the safety area on one side of the starting point position, the included angles between the edge lines of the safety area on the other side of the starting point position, and the end points of the edge lines on both sides of the starting point position, performing first circle fitting according to the two included angles and the four end points to determine the center position and radius length of the first fitted circle, connecting the starting point position and the center position to form a first path S1. The departure direction of the center position is determined again, and the above operation is repeated to form a second path S2. This is repeated until the end point position. The path composed of S1, S2, S3, … connected together is the safety path. Finally, the safety area model and the safety path are mapped to the depth image.

[0147] The real-time safety area model is generated by the following method: if there is no new obstacle image on the real-time detection area image, the initial safety area model is directly taken as the real-time safety area model; if there is a new obstacle image on the real-time detection area image, the ground positions Gi, the wall positions Wi, the initial obstacle positions Bi, and the new obstacle positions Ci are fitted to form line segments based on the initial safety area model generation method.

[0148] In some ways, the step of converting the real-time user image into a user position specifically includes:

[0149] If the real-time user image is received, the real-time user image is three-dimensionally reconstructed and a real-time user model is established;

[0150] The real-time user model is converted into a user position.

[0151] The real-time user model refers to a three-dimensional mesh model used to simulate a user in real time. The real-time user model is converted into a user position by the following method: a minimum circumscribed cube model based on the user model is established, the position of the center of gravity of the minimum circumscribed cube model projected on the ground of the real-time safety area model is determined, and the position of the center of gravity of the minimum circumscribed cube model projected on the ground of the real-time safety area model is the real-time user position. The obstacle position and the destination position can be converted in the above-mentioned manner.

[0152] In some ways, in order to solve the problem that the user deviates from the safety path and may cause bumping and even falling, the present application further comprises the following:

[0153] The Kalman filtering algorithm is used to generate a predicted pose of the real-time user image;

[0154] If the predicted pose is accepted, a motion parameter is generated by using a human motion tracking algorithm;

[0155] It is judged whether the motion parameter exceeds a safety parameter range;

[0156] If the safety parameter exceeds the safety motion range, a corrected path is generated;

[0157] If the corrected path is generated, the corrected path is sent to an output device.

[0158] The Kalman filtering algorithm is used to generate a predicted pose of the real-time user image;

[0159] In the above, the safety parameter range refers to a safety value range in which the predicted pose of the user model moves on the safety area. If the predicted pose of the user model exceeds the safety parameter range, and the user model continues to move in the current state, the user model is likely to collide. In reality, the user is likely to bump and fall.

[0160] The correction path refers to a path for simulating the user's return to the safety path on the real-time safety area model. If the user walks along the correction path, the user can return to the safety path. If the user does not walk along the correction path, it is highly likely that a collision or even a fall will occur.

[0161] In some ways, the motion parameters include at least one of the position, speed, and orientation angle of the user.

[0162] The position in the motion parameters refers to the coordinates of the predicted pose of the user model. In addition, the safety parameter range is defined as follows: the edge line of the safety area is scaled to generate a first area, and the coordinates in the first area are a first set, which can be defined as the safe activity area of the user model in the safety area. A first path is generated at a certain distance on one side of the safety path, and a second path is generated at a certain distance on the other side of the safety path, and the adjacent endpoints of the first path and the second path are connected together to form a second area, and the coordinates in the second area are a second set, which can be defined as the safe activity range of the user model deviating from the safety path. The intersection of the first set and the second set is used as the safety parameter range. The speed in the motion parameters refers to the speed of the predicted pose of the user model. The orientation angle in the motion parameters refers to the orientation angle of the predicted pose of the user model.

[0163] In some ways, the starting position of the correction path is the real-time user position, and the ending position of the correction path is the position closest to the user model on the safety path.

[0164] The above-mentioned correction path is the shortest path between the user model and the safety path, which helps the user to quickly return to the safety path.

[0165] The input device is used to convert sound, kinetic energy, etc. into an electrical signal and send it to the control module. It can be a touch screen, a key, a microphone, etc. In this embodiment, the input device is a touch screen. When the user inputs the request of "toilet" on the input device, the input device can convert the above-mentioned request into request information containing the destination position of the intelligent toilet.

[0166] The output device is used to receive the safety path, which is an electrical signal. If the output device receives the safety path, the output device can convert the safety path into at least one of a sound, a light, a vibration, and a temperature signal for output, which is used to guide the user to reach the destination along the safety path.

[0167] Correspondingly, the output device includes at least one of a loudspeaker, a display, a vibrator, and a heater.

[0168] The output device is also configured to receive a correction path, the correction path being an electrical signal. When the output device receives the correction path, the output device can convert the correction path into at least one of a sound, a light, a vibration, and a temperature and output the at least one of the sound, the light, the vibration, and the temperature to guide the user to follow the correction path and return to the safe path.

[0169] In order to improve the intelligent degree of the intelligent toilet, the intelligent toilet comprises a toilet seat, a toilet pad ring capable of adjusting the thickness of the toilet pad ring on the toilet seat, and a foot pedal capable of adjusting the height of the foot pedal on one side of the toilet seat. The control module is also configured to identify human body size information of the user in a real-time image collected by the TOF camera. The control module is configured to adjust the thickness of the toilet pad ring on the toilet seat and the height of the foot pedal on one side of the toilet seat based on the human body size information of the user. The human body size information comprises the knee height, the thigh length, the calf length, and the angle between the calf and the thigh of the user. The toilet size information comprises the height of the toilet seat. When the human body size information and the toilet size information are received, the thickness of the intelligent toilet pad ring and the height of the foot pedal are adjusted, so that the user can use the toilet more comfortably. Assuming that the height of the foot pedal is H1, the calf length is H2, the thickness of the toilet pad ring is H3, the height of the toilet seat is H4, the thigh length is L1, and the angle between the calf and the thigh is θ, wherein θ is a constant and meets the ergonomic design, the height of the foot pedal can be calculated according to the following formula: H1 = L1*cos(θ) + H3 + H4 - H2.

[0170] Specifically, the toilet pad ring comprises three pad rings, which are respectively hinged above the toilet seat and can be stacked on the toilet seat. When the pad ring rotates on the toilet seat, the pad ring can be opened on the toilet seat or stacked on the toilet seat. By adjusting the number of pad rings stacked on the toilet seat, the thickness of the toilet pad ring can be adjusted. The pad ring can be driven by a first motor, which can be electrically connected to the control module. The control module controls the number of pad rings stacked on the toilet seat through the first motor, thereby improving the automation degree. When one pad ring is stacked on the toilet seat, the thickness of the toilet pad ring is H1; when two pad rings are stacked on the toilet seat, the thickness of the toilet pad ring is 2*H1; and when three pad rings are stacked on the toilet seat, the thickness of the toilet pad ring is 3*H1.

[0171] The control module is also used to identify the identity of the user in the real-time image. The identity of the user includes adults, children and guests. The adult refers to the user who is registered in the system and whose height is higher than 1.5 m, the child refers to the user who is registered in the system and whose height is lower than 1.5 m, and the guest refers to the user who is not registered in the system. The toilet seat can be divided into a first gasket, a second gasket and a third gasket from top to bottom in sequence, the first gasket has a smaller inner hole and two sides with handrails, the first gasket is used for the child, the second gasket is used for the adult, and the third gasket is used for the guest. If the identified user is the child, the first gasket is arranged on the toilet seat, and the height of the foot pedal is determined by the height adjustment formula of the foot pedal. If the identified user is the adult, the second gasket is arranged on the toilet seat, and the height of the foot pedal is determined by the height adjustment formula of the foot pedal. If the identified user is the guest, the third gasket is arranged on the toilet seat, and the height of the foot pedal is determined by the height adjustment formula of the foot pedal.

[0172] The foot pedal comprises at least two pedals, and two or more pedals are stacked together and arranged in front of the toilet seat. By adjusting the number of stacked pedals, the height of the foot pedal can be adjusted. The pedal can rotate relative to the toilet seat, and if the pedal rotates relative to the toilet seat, the adjacent pedals can be separated from each other or stacked together, thereby achieving the effect of adjusting the number of stacked pedals. The pedal can be driven by a second motor, and the second motor can be electrically connected to the control module. The control module controls the number of stacked pedals through the second motor, thereby improving the degree of automation.

[0173] In addition, the control module can also identify the gesture of the user model on the real-time image, and control the intelligent toilet to heat the toilet gasket, automatically flush the toilet, turn on or off the illumination of the toilet seat, open or close the toilet cover on the toilet seat, etc. For example, if the gesture is a fist, the toilet cover is opened on the toilet seat; if the gesture is scissors, the toilet is automatically flushed; and if the gesture is a palm, the toilet gasket is automatically heated.

[0174] Correspondingly, the embodiment of the present application also provides an alarm method for assisting a user to fall to a toilet based on artificial intelligence. The method is executed by using the alarm device for assisting a user to fall to a toilet based on artificial intelligence provided by the embodiment of the present application, and is used to realize the path planning of the user in a safe area.

[0175] As shown in Figure 1 the step flowchart of one embodiment of the alarm method for assisting a user to fall to a toilet based on artificial intelligence, which comprises:

[0176] S1: receiving a real-time image from a TOF camera;

[0177] S2: If the real-time image is received, a node position of the user is established on the real-time image;

[0178] S3: If the joint position of the user is received, a posture angle of the user is generated based on the world reference system and the joint position of the user;

[0179] S4: It is judged whether the posture angle of the user is within a safe angle range;

[0180] S5: If the posture angle of the user exceeds the safe angle range, a fall signal is generated;

[0181] S6: If the fall signal is generated, the fall signal is sent to an output device.

[0182] In some ways, a convolutional neural network is used to establish the joint position of the user on the real-time image.

[0183] In some ways, the node position of the user includes a head, a neck joint, a shoulder joint, an elbow joint, a hand, a hip joint, a knee joint, an ankle joint and a pelvic joint, wherein the node position on the upper body of the user is taken as a priority reference index. If the posture angle established based on the node position is greater than a maximum inclination posture angle threshold value in a normal case of a human body, the control module quickly responds and judges an abnormality.

[0184] In some ways, the world coordinate system is established based on an edge line of the real-time image.

[0185] In some ways, if the world reference system and the joint position of the user are received, the posture angle of the user is generated based on the world reference system and the joint position of the user on the real-time image, comprising:

[0186] If the joint position of the user is received, the joint position of the user is converted into a posture line;

[0187] If the posture line is received, the posture angle of the user is generated based on the posture line and the world reference system.

[0188] In some ways, a line between the midpoint between the neck joint position and the two hip joint positions or a line between the midpoint between the neck joint position and the two knee joint positions or a line between the midpoint between the neck joint position and the two ankle joint positions can be taken as the posture line.

[0189] In some embodiments, it is determined whether two hip joints exist simultaneously, and if so, a line is constructed between the midpoint of a line connecting the cervical joint position and the positions of the two hip joints as the posture line; if two hip joints do not exist, it is determined whether two knee joints exist simultaneously, and if so, a line is constructed between the midpoint of a line connecting the cervical joint position and the positions of the two knee joints as the posture line; if two knee joints do not exist, a line is constructed between the midpoint of a line connecting the cervical joint position and the positions of the two ankle joints as the posture line.

[0190] In some embodiments, the method further comprises:

[0191] receiving an initial image from the TOF camera;

[0192] converting the initial image into an initial safety area image;

[0193] if a real-time image is received, segmenting the part of the initial safety area image mapped on the real-time image as a real-time detection area image;

[0194] generating a real-time safety area model according to the initial safety area image and the real-time detection area image;

[0195] determining whether there is a user image on the real-time detection area image;

[0196] if there is a user image on the real-time user image, segmenting the user image on the real-time detection area image;

[0197] converting the user image into a user position;

[0198] receiving request information from an input device, the request information including a destination position;

[0199] generating a safety path from the user position to the destination position on the real-time safety area model.

[0200] In some embodiments, the step of converting the initial image into an initial safety area image specifically comprises:

[0201] if the initial image is received, performing three-dimensional reconstruction on the initial image and establishing an initial three-dimensional grid model;

[0202] segmenting an initial safety area model on the initial three-dimensional grid model;

[0203] converting the initial safety area model into an initial safety area image.

[0204] In some embodiments, if the initial image is received, the step of performing three-dimensional reconstruction on the initial image and establishing an initial three-dimensional grid model specifically comprises:

[0205] if the initial image is received, pre-processing the initial image and converting the initial image into an initial three-dimensional point cloud;

[0206] if the initial three-dimensional point cloud is received, pre-processing the initial three-dimensional point cloud and establishing a normal vector of the initial three-dimensional point cloud;

[0207] if the initial three-dimensional point cloud with the normal vector is received, estimating a camera pose by using an ICP algorithm;

[0208] if the camera pose is received, performing point cloud fusion on the initial three-dimensional point cloud with the normal vector according to the camera pose and generating an initial fusion point cloud;

[0209] if the initial fusion point cloud is received, converting the initial fusion point cloud into an initial three-dimensional mesh model by using a Poisson reconstruction algorithm.

[0210] In some modes, the step of if the initial image is received, pre-processing the initial image and converting the initial image into an initial three-dimensional point cloud specifically comprises:

[0211] if the initial image is received, performing denoising and hole filling processing on the initial image;

[0212] converting the initial image after denoising and hole filling into an initial three-dimensional point cloud.

[0213] In some modes, the step of if the initial three-dimensional point cloud is received, pre-processing the initial three-dimensional point cloud and establishing a normal vector of the initial three-dimensional point cloud specifically comprises:

[0214] if the initial three-dimensional point cloud is received, filtering the initial three-dimensional point cloud;

[0215] calculating a normal vector of the filtered initial three-dimensional point cloud.

[0216] In some modes, the step of segmenting an initial safety area model on the initial three-dimensional mesh model specifically comprises:

[0217] if the three-dimensional mesh model is received, segmenting an initial obstacle model and a background model on the three-dimensional mesh model;

[0218] if the initial initial obstacle model and the background model are received, segmenting an initial safety area model on the background model, and the initial obstacle model is not on the initial safety area model.

[0219] In some modes, the step of generating a real-time safety area model according to the initial safety area image and the real-time detection area image specifically comprises:

[0220] judging whether there is a new obstacle image on the real-time detection area image;

[0221] If there is a new obstacle image on the real-time detection area image, the real-time detection area image is converted into a real-time safety area model;

[0222] If there is no new obstacle image on the real-time detection area image, the initial safety area model is taken as the real-time safety area model.

[0223] In some ways, the step of converting the real-time user image into a user position specifically includes:

[0224] If the real-time user image is received, the real-time user image is three-dimensionally reconstructed and a real-time user model is established;

[0225] The real-time user model is converted into a user position.

[0226] In some ways, the present method for assisting a user to fall alarm to a toilet based on artificial intelligence is also used to solve the problem that the user deviates from a safety path, and further includes the following steps:

[0227] A predicted pose of the real-time user image is generated by using a Kalman filtering algorithm;

[0228] If the predicted pose is accepted, a motion parameter is generated by using a human motion tracking algorithm;

[0229] It is judged whether the motion parameter exceeds a safety parameter range;

[0230] If the safety parameter exceeds the safety motion range, a corrected path is generated;

[0231] If the corrected path is generated, the corrected path is sent to an output device.

[0232] In some ways, the motion parameter includes at least one of a position, a speed and an orientation angle of the user model.

[0233] In some ways, a starting position of the corrected path is a current position of the user model, and an ending position of the corrected path is a position on the safety path closest to the user model.

[0234] The above embodiments are only the preferred embodiments of the present application, and do not limit the protection scope of the present application. Any equivalent changes made in the structure, shape and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A method for generating a safe path to a smart toilet and detecting falls, characterized in that, include: Receive the initial image from the TOF camera; Convert the initial image into an initial safe region image; If a real-time image is received, the portion of the initial safe region image mapped onto the real-time image is used as the real-time detection region image; A real-time safe zone model is generated based on the initial safe zone image and the real-time detected zone image; Determine whether there is a user image in the real-time detection area image; If there is a user image in the real-time user image, then the user image is segmented from the real-time detection area image; Convert user image into user location; Receive request information from the input device, including the destination location; Generate a safe path from the user's location to the destination location on a real-time safe zone model; receive real-time images from a TOF camera; Establish the user's node location on the real-time image; Generate the user's pose angles based on the world reference frame and the user's joint positions; Determine whether the user's posture angle is within a safe range; If the user's posture angle exceeds the safe angle range, a fall signal is generated; If a fall signal is generated, the fall signal will be sent to the output device.

2. The method for generating a safe path to a smart toilet and determining a fall as described in claim 1, characterized in that, The step of generating a real-time safe region model based on the initial safe region image and the real-time detected region image specifically includes: Determine whether there are newly added obstacle images in the real-time detection area image; If new obstacle images are added to the real-time detection area image, the real-time detection area image will be converted into a real-time safe area model. If no new obstacle image is added to the real-time detection area image, the initial safe area model is used as the real-time safe area model.

3. The method for generating a safe path to a smart toilet and determining a fall, as described in claim 1, is characterized in that... The step of converting the user image into the user's location specifically includes: Perform 3D reconstruction of real-time user images and establish a real-time user model; Transform real-time user models into user locations.

4. The method for generating a safe path to a smart toilet and determining a fall as described in claim 1, characterized in that, The step of converting the initial image into an initial safe region image specifically includes: Perform 3D reconstruction on the initial image and establish an initial 3D mesh model; The initial safe zone model is segmented on the initial 3D mesh model; The initial safe region model is converted into an initial safe region image.

5. The method for generating a safe path to a smart toilet and determining a fall, as described in claim 4, is characterized in that... The steps for performing 3D reconstruction of the initial image and establishing an initial 3D mesh model specifically include: The initial image is preprocessed and converted into an initial 3D point cloud; the initial 3D point cloud is preprocessed and its normal vector is established. Camera pose is estimated using the ICP algorithm; Based on the camera pose, the initial 3D point cloud with normal vectors is fused to generate an initial fused point cloud; The initial fused point cloud is converted into an initial 3D mesh model using the Poisson reconstruction algorithm.

6. The method for generating a safe path to a smart toilet and determining a fall as described in claim 1, characterized in that, A convolutional neural network is used to establish the user's joint position on a real-time image.

7. A method for generating a safe path to a smart toilet and determining a fall, as described in claim 1 or 6, characterized in that... The user's node locations include the head, neck joint, shoulder joint, elbow joint, hand, hip joint, knee joint, ankle joint, and pelvic joint.

8. The method for generating a safe path to a smart toilet and determining a fall as described in claim 1, characterized in that, The world reference frame is established based on the edge lines of real-time images.

9. The method for generating a safe path to a smart toilet and determining a fall as described in claim 1, characterized in that, The steps for generating user pose angles based on a world reference frame of real-time images and user joint positions include: converting user joint positions into pose lines; The user's attitude angles are generated based on the attitude lines and the world reference frame.

10. The method for generating a safe path to a smart toilet and determining a fall, as described in claim 9, is characterized in that... A line drawn between the midpoints of the lines connecting the neck joint position and the two hip joint positions, or between the midpoints of the lines connecting the neck joint position and the two knee joint positions, or between the midpoints of the lines connecting the neck joint position and the two ankle joint positions, can serve as a posture line.

Images