A method and device for assisting a user to a smart toilet based on artificial intelligence

Abstract

The application discloses an artificial intelligence-based method and device for assisting a user to guide a blind person to a smart toilet, which comprises the following steps: receiving an initial image from a TOF camera; converting the initial image into an initial safety area image; receiving a real-time image from the TOF camera; segmenting a part of the initial safety area image mapped on the real-time image as a real-time detection area image; generating a real-time safety area model according to the initial safety area image and the real-time detection area image; judging whether there is a user image on the real-time detection area image; if there is a user image on the real-time detection area image, segmenting the user image on the real-time detection area image; converting the user image into a user position; receiving request information from an input device, wherein the request information comprises a destination position; and generating a safety path from the user position to the destination position on the real-time safety area model.

Description

Technical Field

[0001] This invention relates to machine vision technology, specifically to a method and device for guiding blind users to a smart toilet based on artificial intelligence and machine learning algorithms. Background Technology

[0002] Path planning is one of the main research directions in machine vision. A sequence of points or curves connecting a starting point and an ending point is called a path, and the strategy for constructing a path is called path planning. It has wide applications in many fields, such as obstacle avoidance for vehicles, setting the gripping trajectory for robotic arms, and navigation for users.

[0003] Currently, visually impaired individuals are prone to falls while walking, especially in indoor spaces with many obstacles and poor lighting. Visual impairment can be caused by aging, eye diseases, or congenital factors. For example, elderly people are highly susceptible to falls in toilets due to the confined space, poor lighting, and haphazardly placed obstacles.

[0004] In order to solve the problem of toilet access for visually impaired people, a navigation system for the visually impaired has been developed using existing technology. The system uses shoes equipped with cameras to guide people's way. The system includes the following steps: (1) Before use, the cameras located at the front of the sole and the side of the front of the sole are turned on by a power switch; (2) The camera at the front of the sole captures images of the user's direction of travel and transmits the images to the image processing module; (3) The image processing module processes the images and performs image recognition, and the processor determines whether there are obstacles in the direction of travel.

[0005] However, existing technologies are not perfect and have the following problems:

[0006] (1) In simple scenarios, due to the small number of objects, the existing visually impaired navigation system can identify and avoid obstacles relatively quickly and accurately. However, in the toilet, due to the large number of objects and the dim lighting, the images captured by the camera are dim and the scene is more complex, which makes it difficult for the image processing module to identify obstacles quickly and accurately.

[0007] (2) The existing visually impaired navigation system has a camera installed on the user. The camera needs to capture the surrounding scene in real time and moves with the user. The position of the surrounding scene relative to the camera changes, and the scene captured by the camera is very different each time. The image processing module needs to reprocess the captured image each time. When dealing with small and complex toilets, it is easy to cause the equipment to have a large computing load and low response efficiency. Invention Content

[0008] To overcome the shortcomings and problems of existing technologies, this invention provides a method and device for guiding blind users to a smart toilet based on artificial intelligence.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A method for guiding the blind to a smart toilet based on artificial intelligence includes the following steps:

[0011] Receive the initial image from the TOF camera;

[0012] Convert the initial image into an initial safe region image;

[0013] Receive real-time images from a TOF camera;

[0014] The portion of the initial safe region image mapped onto the real-time image is used as the real-time detection region image;

[0015] A real-time safe zone model is generated based on the initial safe zone image and the real-time detected zone image;

[0016] Determine whether there is a user image in the real-time detection area image;

[0017] If there is a user image in the real-time user image, then the user image is segmented from the real-time detection area image;

[0018] Convert user image into user location;

[0019] Receive request information from the input device, including the destination location;

[0020] Generate a secure path from the user's location to the destination location on a real-time secure zone model.

[0021] Preferably, the step of converting the initial image into an initial safe area image specifically includes:

[0022] If an initial image is received, a 3D reconstruction is performed on the initial image and an initial 3D mesh model is established.

[0023] The initial safe zone model is segmented on the initial 3D mesh model;

[0024] The initial safe region model is converted into an initial safe region image.

[0025] Preferably, if an initial image is received, the steps of performing 3D reconstruction of the initial image and establishing an initial 3D mesh model specifically include:

[0026] If an initial image is received, it is preprocessed and converted into an initial 3D point cloud.

[0027] If an initial 3D point cloud is received, the initial 3D point cloud is preprocessed and a normal vector of the initial 3D point cloud is established.

[0028] If an initial 3D point cloud with normal vectors is received, the camera pose is estimated using the ICP algorithm.

[0029] If the camera pose is received, the initial 3D point cloud with normal vectors is fused according to the camera pose to generate an initial fused point cloud.

[0030] If the initial fused point cloud is received, the Poisson reconstruction algorithm is used to convert the initial fused point cloud into an initial 3D mesh model.

[0031] Preferably, the step of preprocessing the initial image and converting it into an initial 3D point cloud upon receiving the initial image specifically includes:

[0032] If the initial image is received, denoising and hole repair processing are performed on the initial image;

[0033] The initial image after denoising and hole repair is converted into an initial 3D point cloud.

[0034] Preferably, the step of preprocessing the initial 3D point cloud and establishing its normal vectors upon receiving the initial 3D point cloud specifically includes:

[0035] If an initial 3D point cloud is received, then the initial 3D point cloud is filtered.

[0036] Calculate the normal vector of the initial 3D point cloud after filtering.

[0037] Preferably, the step of segmenting the initial safe area model on the initial three-dimensional mesh model specifically includes:

[0038] If a 3D mesh model is received, the initial obstacle model and the background model are segmented on the 3D mesh model;

[0039] If an initial obstacle model and a background model are received, an initial safe zone model is segmented on the background model, and the initial obstacle model is not on the initial safe zone model.

[0040] Preferably, the step of generating a real-time security region model based on the initial security region image and the real-time detection region image specifically includes:

[0041] Determine whether there are newly added obstacle images in the real-time detection area image;

[0042] If new obstacle images are added to the real-time detection area image, the real-time detection area image will be converted into a real-time safe area model.

[0043] If no new obstacle image is added to the real-time detection area image, the initial safe area model is used as the real-time safe area model.

[0044] Preferably, the step of converting real-time user images into user locations specifically includes:

[0045] If a real-time user image is received, a 3D reconstruction is performed on the real-time user image and a real-time user model is established.

[0046] Transform real-time user models into user locations.

[0047] As a preferred embodiment, this AI-based guide method for the visually impaired to the smart toilet also addresses user deviations from the safe path and includes the following steps:

[0048] The Kalman filter algorithm is used to generate the predicted pose of real-time user images;

[0049] If the predicted pose is received, motion parameters are generated using a human motion tracking algorithm;

[0050] Determine whether the motion parameters exceed the safe parameter range;

[0051] If the safety parameters exceed the safe movement range, a correction path will be generated;

[0052] If a corrected path is generated, the corrected path is sent to the output device.

[0053] Preferably, the motion parameters include at least one of the user model's position, velocity, and orientation angle.

[0054] Preferably, the starting position of the corrected path is the current position of the user model, and the ending position of the corrected path is the position on the safe path that is closest to the user model.

[0055] As a preferred option, it also includes:

[0056] If a real-time image is received, the user's node position is established on the real-time image;

[0057] If the user's joint position is received, the user's pose angle is generated based on the world reference frame and the user's joint position.

[0058] Determine whether the user's posture angle is within a safe range;

[0059] If the user's posture angle exceeds the safe angle range, a fall signal is generated;

[0060] If a fall signal is generated, the fall signal will be sent to the output device.

[0061] As a preferred method, a convolutional neural network is used to establish the user's joint position on a real-time image.

[0062] Preferably, the user's node positions include the head, neck joint, shoulder joint, elbow joint, hand, hip joint, knee joint, ankle joint, and pelvic joint, with node positions located on the user's upper body serving as the preferred benchmark. If the posture angle established based on the node positions exceeds the maximum tilt posture angle threshold under normal human conditions, the control module responds quickly and identifies the anomaly.

[0063] Preferably, the world coordinate system is established based on the edge lines of the real-time image.

[0064] Preferably, the step of generating the user's pose angle based on the world reference frame and the user's joint position of the real-time image when the world reference frame and the user's joint position are received includes:

[0065] If the user's joint position is received, the user's joint position is converted into an attitude line;

[0066] If an attitude line is received, the user's attitude angle is generated based on the attitude line and the world reference frame.

[0067] On the other hand, the present invention also proposes a guide device for the blind that assists users to a smart toilet based on artificial intelligence, for implementing the aforementioned guide method for the blind that assists users to a smart toilet based on artificial intelligence, comprising:

[0068] Input device, used to generate request information;

[0069] A TOF camera is used to generate initial and real-time images;

[0070] The control module receives an initial image and a real-time image, converts the initial image into an initial safe region image, segments the portion of the initial safe region image mapped onto the real-time image as a real-time detection region image, generates a real-time safe region model based on the initial safe region image and the real-time detection region image, determines whether there is a user image on the real-time detection region image, and if there is a user image on the real-time user image, segments the user image from the real-time detection region image; converts the user image into a user location; receives request information from an input device, including the destination location; and generates a safe path from the user location to the destination location on the real-time safe region model.

[0071] Output device, used to receive secure paths.

[0072] Preferably, the output device includes at least one of a microphone, a display, a vibrator, and a heater.

[0073] As a preferred option, a smart toilet is also included, which includes a toilet seat, a toilet seat cover, and a foot pedal. The control module can adjust the thickness of the toilet seat cover on the toilet seat, and the control module can also adjust the height of the foot pedal on one side of the toilet seat.

[0074] The outstanding and beneficial technical effects of this invention compared to the prior art are:

[0075] (1) In this invention, the input device, TOF camera, control module and output device are used to solve the functions of user path planning, path correction and navigation, which reduces the probability of the user bumping into obstacles and thus reduces the risk of the user falling, and also helps the user reach the destination on the best path.

[0076] (2) In this invention, the TOF camera acquires initial images and real-time images. The initial images can be preset in the control module. The control module only needs to process the user images and newly added obstacle images based on the initial images, which reduces the number of images that the control module needs to process, reduces the processing pressure of the control module, and thus improves the response efficiency and calculation accuracy of this invention.

[0077] (3) In this invention, the TOF camera can capture multiple users and multiple destinations within its shooting range. It can be used to plan multiple paths at the same time and correct path deviations of multiple users. Therefore, this invention supports one-to-many use and helps to reduce the cost of use. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the steps and flow structure of a guide method for guiding blind users to a smart toilet based on artificial intelligence, as created by the present invention.

[0079] Figure 2 This invention creates a three-view diagram for guiding users to use the toilet;

[0080] Figure 3 This is a schematic diagram showing the overlapping of initial images captured by the TOF camera of this invention from multiple perspectives.

[0081] Figure 4 This is the initial image captured inside a toilet by the TOF camera created in this invention;

[0082] Figure 5 This is a schematic diagram of a real-time image of a user standing in a toilet captured by the TOF camera created in this invention, and a two-dimensional posture established on the user.

[0083] Figure 6 This is a schematic diagram of the TOF camera created in this invention capturing a real-time image of a user falling in the toilet and establishing a two-dimensional posture on the user;

[0084] Figure 7 This is the basis for the invention. Figure 4 The initial image is converted into a top view of a 3D point cloud;

[0085] Figure 8 This is the basis for the invention. Figure 5 A top-down view of a 3D point cloud converted from a real-time image;

[0086] Figure 9 This invention is based on Figure 7 A schematic diagram of the structure for establishing the initial safe zone model;

[0087] Figure 10 This is a schematic diagram of the structure of a guide device for the blind that assists users to a smart toilet based on artificial intelligence, as created by this invention.

[0088] Figure 11 This is a schematic diagram of the structure of the TOF camera created in this invention;

[0089] Figure 12 This is a schematic diagram of the cross-sectional structure of the TOF camera created in this invention;

[0090] Figure 13 This is a schematic diagram of the exploded structure of the TOF camera created in this invention;

[0091] Figure 14 This is a structural schematic diagram of the intelligent toilet created by this invention;

[0092] Figure 15 This is a schematic diagram of the structure of the smart toilet seat created by this invention, where the toilet seat is lifted up.

[0093] Figure 16 This is a flowchart of a method for guiding the blind to a smart toilet based on artificial intelligence, as described in this invention.

[0094] In the diagram: 1-TOF camera, 2-control module, 3-output device, 4-smart toilet, 5-input device, 11-housing, 12-link, 13-panel, 14-control motherboard, 15-VCSEL laser, 16-TOF camera module, 17-lid, 111-opening, 121-slot, 41-toilet seat, 42-washer, 43-pedal. Detailed Implementation

[0095] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0096] like Figures 1 to 16As shown, the present invention provides a guide device for the blind that assists users to a smart toilet based on artificial intelligence. It can realize all the steps of the guide method for the blind that assists users to a smart toilet based on artificial intelligence provided in the embodiments of the present invention, and is used to plan the user's path in a safe area, guide the user to avoid obstacles and reach the destination, and avoid the problem of the user tripping over obstacles.

[0097] It should be noted that this AI-assisted navigation device for the smart toilet is primarily designed for indoor use. For example, in a toilet setting, this device can plan routes for users to use the toilet, shower, and wash up. It addresses issues such as visual impairment, poor lighting, and cluttered spaces causing difficulty in movement and tripping. Visual impairment refers to individuals who are completely blind, partially blind, or have visual impairments. The safe zone refers to the area on the ground excluding obstacles and the area above the ground. When walking within the safe zone, users will not encounter obstacles, achieving obstacle avoidance. Walking outside the safe zone increases the risk of bumping into obstacles and falling. Falls can result in abrasions, bruises, fractures, or even fainting, especially for users over 75 years of age, who are more susceptible to serious injuries.

[0098] like Figure 6 As shown, this invention provides a guide device for the blind that assists users to a smart toilet based on artificial intelligence. It includes an input device, a TOF camera, a control module, an output device, and a smart toilet. The input device, TOF camera, and output device are connected to the control module via wireless communication.

[0099] TOF cameras are used to generate depth images. TOF stands for Time of Flight camera, and it's a type of depth sensor. A depth image is an image containing depth data, which can be used for 3D scene reconstruction. Depth data refers to the distance information from the scene's surface to the viewpoint. The scene refers to objects within the TOF camera's field of view. The viewpoint refers to the location of the TOF camera.

[0100] Depth images include initial images and real-time images. Initial images refer to those captured by the TOF camera within the initial scene. The initial scene refers to a scene where the initial obstacles, such as walls, toilets, and sinks, are essentially stationary. Compared to the initial scene, the real-time captured scene may include the user and newly added obstacles. The user generally refers to a person, but can also be a moving object. Newly added obstacles refer to any obstacles in the scene other than the initial obstacles. Initial and newly added obstacles are collectively referred to as obstacles. In practical use, the initial image can be pre-processed and pre-set in the control module; subsequent real-time images only need to identify the user and newly added obstacles.

[0101] like Figure 2 The diagram shows a three-view drawing of this path planning device used to guide users to the toilet. A Time-of-Flight (TOF) camera is mounted on the wall inside the toilet. The shaded area in the diagram represents the TOF camera's field of view. The door, smart toilet, sink, and part of the floor are within the TOF camera's field of view, allowing the user to enter. The toilet scene includes the door, sink, smart toilet, shower room, floor, walls, and the user. When a user needs to use the toilet, this AI-based guide device generates a safe path within the toilet. The dashed line with an arrow indicates the safe path, starting at the user and ending at the smart toilet. When the TOF camera is capturing images of the toilet, it generates depth images, which can be used for 3D reconstruction of the toilet scene.

[0102] Specifically, the TOF camera includes a VCSEL laser, a TOF camera module, and a control motherboard. The VCSEL laser is used to emit laser light. The laser emitted by the VCSEL laser is high-performance pulsed light, with pulses reaching approximately 100MHz. The TOF camera module is used to convert the light signal into an electrical signal. The VCSEL laser and the TOF camera module are electrically connected to the control motherboard, which is used to process the depth image. Compared to binocular stereo vision cameras or structured light cameras, this TOF camera has better resistance to strong light, and its accuracy is less affected by the shooting distance, generally maintaining a centimeter-level accuracy.

[0103] The TOF camera also includes a housing, with the VCSEL laser, TOF camera module, and control motherboard fixedly installed inside the housing. A motor is also fixedly installed inside the housing. The VCSEL laser, TOF camera module, and motor are electrically connected to the control motherboard. A wall mount is fixedly installed on the motor shaft, and the wall mount is used to fix it to a wall or a smart toilet.

[0104] The housing is hollow, protecting the VCSEL laser, TOF camera module, control board, and motor. The motor converts electrical energy into mechanical energy. When the motor is running, it rotates the housing relative to the wall mount, thus adjusting the orientation of the VCSEL laser and TOF camera module on the wall mount. The VCSEL laser, TOF camera module, and motor are electrically connected to the control board, which controls the operation of these components.

[0105] In practical use, the wall mount is fixedly installed on the wall of the smart toilet or bathroom. The motor drives the housing to rotate left and right relative to the wall mount, allowing the TOF camera module to shoot in different directions, thereby increasing the shooting range of the TOF camera module.

[0106] In this invention, the electric motor can drive the housing to rotate relative to the wall mount, the VCSEL laser can emit lasers at different angles, and the TOF camera module can shoot at different angles, thereby increasing the shooting range of the TOF camera and helping the TOF camera module to shoot from all directions in the toilet. Therefore, this invention has the advantages of wide shooting range, low cost and simple structure.

[0107] The wall mount includes a wall panel and a connecting rod. The connecting rod is fixedly mounted on the wall panel. The overall structure of the connecting rod is "L" shaped. One end of the connecting rod is fixedly mounted on the wall panel, and the other end of the connecting rod is fixedly mounted on the motor shaft.

[0108] Because the casing needs to rotate laterally relative to the wall or smart toilet, in actual use, the wall panel is fixedly installed on the wall or smart toilet. Through the design of the "L"-shaped linkage, the motor can drive the VCSEL laser and TOF camera module to rotate laterally relative to the wall.

[0109] The motor shaft is inserted into one end of the connecting rod, and a slot is provided at one end of the connecting rod, into which the motor shaft is inserted.

[0110] The motor shaft and connecting rod are connected by a plug-in method, allowing users to disassemble and assemble the motor and connecting rod by hand without the need for tools.

[0111] Specifically, the slot has a regular hexagonal cross-section, and the motor shaft fits into the slot, improving the connection strength between the motor shaft and the connecting rod.

[0112] The housing has an opening, with the VCSEL laser and TOF camera module facing the opening respectively. A light-transmitting cover is fixedly installed on the housing and covers the opening.

[0113] In practical use, the laser emitted by the VCSEL laser can pass through the cover plate, and external light can shine through the cover plate onto the TOF camera module.

[0114] It also includes LED light strips, which are electrically connected to the control motherboard. The shooting range of the TOF camera module is within the illumination range of the LED lights.

[0115] In practical use, LED light strips can illuminate the toilet, which on the one hand prevents users from not being able to see the road conditions, and on the other hand helps the TOF camera module to capture clear depth images.

[0116] The LED light strip is fixedly mounted on the housing, which includes an arc-shaped surface. The LED light strip wraps around the arc-shaped surface, allowing it to illuminate from multiple angles and improving the lighting effect. Specifically, the opening is formed on the arc-shaped surface.

[0117] The control module receives an initial image, a real-time image, and a request message. The initial and real-time images are from a Time-of-Flight (TOF) camera. The request message comes from an input device and includes the destination location. If an initial image from the TOF camera is received, it is converted into an initial safe area image. If a real-time image from the TOF camera is received, the portion of the initial safe area image mapped onto the real-time image is segmented to create a real-time detection area image. If a real-time detection area image is received, a safe area model is generated based on the initial and real-time detection area images. If a real-time detection area image is received, it is determined whether a user image is present in the real-time detection area image. If a user image is present in the real-time image, it is segmented from the real-time detection area image. If a segmented user image is received from the real-time detection area image, the user image is converted into a user location. If a request message from the input device is received, a safe path from the user location to the destination location is generated on the real-time safe area model.

[0118] The control module is an integrated chip. Due to the limited shooting range of a Time-of-Flight (TOF) camera, when performing 3D reconstruction of the entire scene, it's difficult to accurately represent the scene from a single viewpoint. To address this, the depth image received by the control module can be either a depth image captured by the TOF camera from a single viewpoint or depth images captured from multiple viewpoints. For example, when acquiring initial scene images, the TOF camera can capture the initial scene from multiple viewpoints, generating multiple initial images. Since there are overlapping areas in these initial images, by overlapping these overlapping areas, multiple initial images can be merged into a single initial image, solving the problem of the initial image not being suitable for 3D reconstruction. In real-time scenes, only the real-time detection area image needs to be processed. To reduce the processing load on the control module, the TOF camera only needs to acquire images from a single viewpoint.

[0119] The initial safe zone image refers to the depth image used to simulate the initial safe zone. The initial safe zone refers to the ground area excluding initial obstacles and the area above it. If no new obstacles are added within the initial safe zone and the user walks within it, the user will not encounter initial obstacles. If the user walks outside the safe zone, the user is highly likely to encounter obstacles. Initial obstacles refer to obstacles that are defined and fixed in the scene. Initial obstacles can include walls, toilets, sinks, shower rooms, etc.

[0120] The real-time detection region image refers to the portion of the real-time image that is located at the same position as the safe region image. In this embodiment, a mapping method is used to segment the real-time detection region image from the real-time image. If the initial safe region image is mapped onto the real-time image, the mapped portion of the real-time image is segmented along the contour of the initial safe region image.

[0121] User images refer to depth images used to simulate a user. Due to the limitations of the shooting range of TOF cameras, user images can be depth images simulating the entire user or depth images simulating a portion of the user.

[0122] User location refers to the position of the user model within the real-time safe zone model, thus representing the user's actual location within the scene. The user model is a 3D mesh model used to simulate the user in real time. The user can specifically be a person or a moving object. Destination location refers to the position of the destination model within the real-time safe zone model. The destination model is a 3D mesh model used to simulate the destination. The destination can be an obstacle or a specific coordinate location.

[0123] A 3D mesh model refers to the topological and spatial structure of a 3D model's surface contours, defined by a set of polygons. Polygons can be triangles. For example, if a TOF camera is capturing a toilet scene, the control module creates a 3D mesh model based on a depth image of the simulated toilet to simulate the scene inside.

[0124] An adaptive Gaussian mixture algorithm (AGMM) is used to determine in real time whether a user image exists in the detection region. AGMM incorporates prior judgment of the background Gaussian distribution and an optimization mechanism for the update rate into the traditional Gaussian mixture algorithm. AGMM can simultaneously identify and distinguish multiple users in an image. In real-time images, users have sharp edges. Edge detection is used to coarsely segment the user image in the real-time image to obtain its coarse edge contour. Then, depth information is used to finely segment the coarse edge contour to obtain the fine edge contour of the user image. Discrete points in the fine edge contour are removed, and the image within the fine edge contour is the user image. Segmentation along the fine edge contour separates the user image from the real-time image.

[0125] A safe path refers to a relatively safe route from the starting point to the destination. The safe path is located on a real-time safe zone model, with the starting point being the user's position and the ending point being the destination. If the user moves along the safe path, they will not encounter either the initial obstacle model or the newly added obstacle model and will eventually reach their destination safely. If the user does not move along the safe path, they are highly likely to encounter obstacles. In practical use, users can refer to the trajectory of the safe path when walking within the real-time scene to safely reach their destination.

[0126] In some methods, the step of converting the initial image into an initial safe region image specifically includes:

[0127] If an initial image is received, a 3D reconstruction is performed on the initial image and an initial 3D mesh model is established.

[0128] The initial safe zone model is segmented on the initial 3D mesh model;

[0129] The initial safe region model is converted into an initial safe region image.

[0130] The initial safe zone model refers to the three-dimensional mesh model of the user's simulated initial safe zone.

[0131] In some methods, if an initial image is received, the steps of performing 3D reconstruction of the initial image and establishing an initial 3D mesh model specifically include:

[0132] If an initial image is received, it is preprocessed and converted into an initial 3D point cloud.

[0133] If an initial 3D point cloud is received, the initial 3D point cloud is preprocessed and a normal vector of the initial 3D point cloud is established.

[0134] If an initial 3D point cloud with normal vectors is received, the camera pose is estimated using the ICP algorithm.

[0135] If the camera pose is received, the initial 3D point cloud with normal vectors is fused according to the camera pose to generate an initial fused point cloud.

[0136] If the initial fused point cloud is received, the Poisson reconstruction algorithm is used to convert the initial fused point cloud into an initial 3D mesh model.

[0137] Specifically, the ICP algorithm is used to estimate the camera pose by receiving initial images from multiple viewpoints captured by a TOF camera and converting these initial images into initial 3D point clouds with normal vectors for each viewpoint. The initial 3D point clouds with normal vectors from the previous viewpoint are matched with those from the next viewpoint. The matching process involves first extracting key points from the initial 3D point clouds with normal vectors for coarse matching, then performing fine matching, and so on iteratively to solve for the camera pose. Finally, graph optimization and backend optimization are performed on the solved camera pose to obtain the final camera pose.

[0138] Specifically, assuming the initial images captured by the TOF camera from three perspectives are A1, B1, and C1, they are downsampled by 1 / 2 and 1 / 4 respectively to generate A2, B2, C2 and A3, B3, C3. The corresponding 3D point clouds are represented by APj, BPj, and CPj, and the unit orientation variables of the point clouds are represented by ANj, BNj, and CNj, where j takes the values ​​1, 2, and 3 to express the level after sampling. Coarse matching is then performed on the overlapping point clouds of A3 and B3, and A3 and C3. This coarse matching process involves traversing the overlapping point cloud i and minimizing the projection difference. The rotation matrix R3 and translation matrix T3 are calculated according to the minimization formula (R3*BP3i+T3)*BN3-AP3i*AN3i. The rotation matrix is ​​then used to project the non-overlapping area onto A3, generating a cube V3. Substituting R3 and T3 as initial values ​​for R2 and T2 into the formula (R2*BP2i+T2)*BN2-AP2i*AN2i and performing minimum optimization, then substituting R2 and T2 into the initial values ​​of R1 and T1 (R1*BP1i+T1)*BN1-AP1i*AN1i and performing minimum optimization again, we obtain the final R1 and T1. Based on R1, T1, and the point cloud coordinates of the overlapping area, we calculate the point cloud coordinates of the non-overlapping area. After matching, we generate 3D models V1, V2, and V3 for three scales of images. The frontal viewpoints VA1, VA2, and VA3 are subsets of V1, V2, and V3. For V3 and the holes-filled A3 and B3,... C3 performs 3D model detection of the ground, walls, and initial obstacles, obtains background area location marking, generates initial safe zone model, marks the coordinates of the initial safe zone model, merges the initial safe zone with the 3D model under the frontal view to generate the initial safe zone 3D model under the frontal view angle, maps the initial safe zone coordinates back to VA3 (VSafe3) under the frontal view, checks for human bodies and temporary obstacles (brooms, stools, etc.), maps the initial safe zone coordinates back to VA2 (VSafe2) under the frontal view; performs path planning, maps the initial safe zone coordinates back to VA1 (VSafe1) under the frontal view; and performs fall detection.

[0139] Furthermore, after preprocessing the initial image, a 3D point cloud can be calculated through coordinate system transformation. The 3D point cloud is the 3D coordinates of pixels based on the camera coordinate system.

[0140] In some embodiments, the step of preprocessing the initial image and converting it into an initial 3D point cloud upon receiving the initial image includes:

[0141] If the initial image is received, denoising and hole repair processing are performed on the initial image;

[0142] The initial image after denoising and hole repair is converted into an initial 3D point cloud.

[0143] Because TOF cameras are prone to issues like incomplete image capture of the user or reflections from clothing during shooting, resulting in holes in the initial image, hole repair is necessary to address these issues. Holes in the initial image refer to pixels with zero depth information. Hole repair involves searching for holes in the initial image, determining the location of the hole's center and edges, expanding the hole's edges outwards by three pixels, acquiring the depth information and its changes from three concentric rings, generating a diameter line passing through the hole's center, and then using this diameter line to search for surrounding valid pixel values ​​in a spiral curve from the eight neighborhood directions. Furthermore, based on the distance values ​​of the three concentric rings, a hole function incorporating distance values ​​and weights is established, and the depth information on the holes is filled in using this function. The coordinates of the resulting initial 3D point cloud are based on the camera coordinate system.

[0144] Downsampling and image pyramid algorithms can also be applied to the depth images acquired by the TOF camera, which helps in subsequent image recognition, image segmentation, and image conversion. Depth image downsampling refers to scaling the original depth image by 1 / 2, 1 / 4, etc., to reduce the dimensionality of features while retaining effective information. This helps avoid overfitting to some extent and maintains stability under rotation, translation, and scaling. Let's call the original depth image Level 1, the image scaled by 1 / 2 Level 2, and the image scaled by 1 / 4 Level 3. The Level 1 image is the original image with high-resolution pixels, which can be used to determine user posture, such as whether a user has fallen. The Level 2 image has lower resolution than Level 1 but higher resolution than Level 3. After denoising, the Level 2 image can be used for tracking moving objects. The Level 3 image has the lowest resolution compared to Level 1 and Level 2 images and can be used for hole repair and 3D background reconstruction.

[0145] In some embodiments, the step of preprocessing the initial 3D point cloud and establishing its normal vectors upon receiving the initial 3D point cloud includes:

[0146] If an initial 3D point cloud is received, then the initial 3D point cloud is filtered.

[0147] Calculate the normal vector of the initial 3D point cloud after filtering.

[0148] The initial 3D point cloud filtering can employ at least one of the following algorithms: voxel filtering, bilateral filtering, radius filtering, and Gaussian filtering. For example, if radius filtering is used, the center point and any arbitrary 3D point are determined. The distance between any arbitrary 3D point and the center point is estimated, and the mean of these distances is calculated. It is then determined whether the distance between a 3D point and the center point is within the mean minus n standard deviations. If the distance is within the mean minus n standard deviations, the 3D point is retained. If it is not, the 3D point is considered a discrete point and is removed. Calculating the normal vector of the filtered initial 3D point cloud refers to determining the angular direction and sign of the angle between the points. The sign of the initial 3D point cloud is obtained as follows: First, calculate the distance information of the model surface, generate a surface indicator function based on the distance information of the model surface, the sign of the 3D point cloud on the object surface is 0, the sign of the 3D point cloud in front of the object surface is negative, and the sign of the 3D point cloud behind the object surface is positive.

[0149] In some methods, the step of segmenting the initial safe zone model on the initial 3D mesh model specifically includes:

[0150] If a 3D mesh model is received, the initial obstacle model and the background model are segmented on the 3D mesh model;

[0151] If an initial obstacle model and a background model are received, an initial safe zone model is segmented on the background model, and the initial obstacle model is not on the initial safe zone model.

[0152] The initial obstacle model refers to the 3D mesh model used to simulate initial obstacles. The background model refers to the 3D mesh model used to simulate the ground. Since the edges of the initial obstacle model are relatively sharp and polygonal, while the edges of the background model are relatively straight, interactive image segmentation methods can be used to segment the initial obstacle model and the background model from the 3D mesh model separately. After determining the initial obstacle model and the background model, the obstacle model can be removed from the background model to determine the initial safe zone model.

[0153] In some embodiments, the step of generating a real-time safe region model based on an initial safe region image and a real-time detected region image specifically includes:

[0154] Determine whether there are newly added obstacle images in the real-time detection area image;

[0155] If new obstacle images are added to the real-time detection area image, the real-time detection area image will be converted into a real-time safe area model.

[0156] If no new obstacle image is added to the real-time detection area image, the initial safe area model is used as the real-time safe area model.

[0157] The newly added obstacle image refers to the depth image used to simulate newly added obstacles. The real-time safe zone model refers to the 3D mesh model used to simulate the real-time safe zone. The real-time safe zone refers to the area within the real-time scene that is safe for the user to pass through. If the user passes through the real-time safe zone, the user can pass through safely. If the user passes outside the real-time safe zone, the user is very likely to encounter problems such as bumping into the initial obstacles and newly added obstacles.

[0158] Specifically, the initial safe zone model is generated as follows: The ground position Gi, wall position Wi, and initial obstacle position Bi are determined. Edge fitting is performed on these positions to form line segments. These line segments are then extended and intersected, forming a closed polygonal region, which is the safe zone region. This polygon can be referred to as the skeleton of the safe zone. The safe path is generated as follows: The starting direction of the starting point is determined. The included angles between the edge lines of the safe zone on one side of the starting point, the included angles between the edge lines of the safe zone on the other side of the starting point, and the endpoints of the edge lines on both sides of the starting point are calculated. Based on the two included angles and four endpoints, a first circle fitting is performed. The center position and radius of the first fitted circle are determined. The starting point and the center position are connected to form the first path S1. The starting direction of the center is then determined again, and the above operation is repeated to form the second path S2. This process is repeated until the endpoint position is reached. The path formed by connecting S1, S2, S3… is the safe path. Finally, the safe zone model and the safe path are mapped onto the depth image.

[0159] The real-time safe zone model is generated using the following method: if there are no new obstacle images in the real-time detection area image, the initial safe zone model is directly used as the real-time safe zone model; if there are new obstacle images in the real-time detection area image, based on the initial safe zone model generation method, the location Ci of the new obstacle is determined, and the ground location Gi, the wall location Wi, the initial obstacle location Bi, and the new obstacle location Ci are edge-fitted to form a line segment.

[0160] In some methods, the step of converting real-time user images into user locations specifically includes:

[0161] If a real-time user image is received, a 3D reconstruction is performed on the real-time user image and a real-time user model is established.

[0162] Transform real-time user models into user locations.

[0163] The real-time user model refers to a 3D mesh model used for real-time user simulation. The real-time user model is converted into the user's location using the following method: A minimum bounding cube model based on the user model is established; the position of the center of gravity of the minimum bounding cube model projected onto the ground of the real-time safe zone model is determined; the position of the center of gravity of the minimum bounding cube model projected onto the ground of the real-time safe zone model is the real-time user's location. Obstacle locations and destination locations can also be converted using the same method.

[0164] In some cases, to address the issue of users deviating from the safe path, potentially causing bumps or even falls, this AI-based guide device for the blind to reach the smart toilet also includes:

[0165] The Kalman filter algorithm is used to generate the predicted pose of real-time user images;

[0166] If the predicted pose is received, motion parameters are generated using a human motion tracking algorithm;

[0167] Determine whether the motion parameters exceed the safe parameter range;

[0168] If the safety parameters exceed the safe movement range, a correction path will be generated;

[0169] If a corrected path is generated, the corrected path is sent to the output device.

[0170] The Kalman filter algorithm tracks the user model in real time, utilizing the spatial structure characteristics of 3D point cloud data and employing an adaptive hierarchical method to accurately determine the trajectory region of the user model. This effectively solves problems such as occlusion tracking, multi-target tracking, and multi-directional tracking, and eliminates the accumulated errors generated by the Kalman filter algorithm in subsequent tracking processes, ultimately ensuring efficient detection and tracking of moving pedestrian targets. Predicted pose refers to the pose of the user model at a future time. The predicted position can be a pose point or a pose trajectory. If the future time is a period of time, the predicted position is the pose trajectory within that period. If the future time is a single point in time, the predicted pose is the pose at that single point in time. In this embodiment, the future time point is one second after the current time.

[0171] In the above, the safety parameter range refers to a safe range of values ​​for the predicted pose of the user model within a safe region. If the predicted pose of the user model exceeds the safety parameter range, the user model is highly likely to collide if it continues to move in its current state. Conversely, in reality, the user is highly likely to bump into things and fall.

[0172] A corrected path refers to a path used to simulate returning a user to a safe path within a real-time safe zone model. If a user follows the corrected path, they can return to the safe path. If a user deviates from the corrected path, they are highly likely to bump into things or even fall.

[0173] In some embodiments, the motion parameters include at least one of the user's position, speed, and orientation angle.

[0174] In motion parameters, position refers to the coordinates of the user model's predicted pose. Additionally, the safety parameter range is defined as follows: A first region is generated by scaling the edge of the safety area according to a certain ratio. The coordinates within this first region form a first set, which can be defined as the safe activity area for the user model within the safety area. A first path is generated at a certain distance on one side of the safety path, and a second path is generated at a certain distance on the other side. The adjacent endpoints of the first and second paths are connected to form a second region. The coordinates within this second region form a second set, which can be defined as the safe activity range for the user model that it can deviate from the safety path. The intersection of the first and second sets is used as the safety parameter range. In motion parameters, velocity refers to the magnitude of the velocity of the user model's predicted pose. In motion parameters, orientation angle refers to the orientation angle of the user model's predicted pose.

[0175] In some embodiments, the starting point of the corrected path is the real-time user location, and the ending point of the corrected path is the location on the safe path that is closest to the user model.

[0176] The aforementioned corrected path is the shortest path between the user model and the safe path, which helps users quickly return to the safe path.

[0177] If a real-time image is received, the user's node position is established on the real-time image;

[0178] If the user's joint position is received, the user's pose angle is generated based on the world reference frame and the user's joint position.

[0179] Determine whether the user's posture angle is within a safe range;

[0180] If the user's posture angle exceeds the safe angle range, a fall signal is generated;

[0181] If a fall signal is generated, the fall signal will be sent to the output device.

[0182] A user's fall is a process of changing body posture, or a series of movements of the bones and joints. By judging the spatial features of the user's joints, the user's current posture can be determined, thus confirming whether the user has fallen. The user's node position refers to the coordinates used to simulate the user's joint position, serving as a spatial feature. The planar reference frame refers to the absolute coordinate system in the scene. The posture angle refers to the angle used to simulate the user's actual posture in the scene. In some methods, convolutional neural networks are used to establish the user's joint position on real-time images.

[0183] Specifically, a convolutional neural network is used to estimate the user's two-dimensional pose in a real-time image. The user's two-dimensional pose refers to a line drawing simulating the user's skeleton in the real-time image, which consists of several line segments. The intersections and endpoints of these lines can simulate the user's joint nodes. Once the user's two-dimensional pose is obtained, the user's joint positions can be determined by the positions of the intersections and endpoints of the two-dimensional pose.

[0184] In some implementations, the user's node positions include the head, neck joint, shoulder joint, elbow joint, hand, hip joint, knee joint, ankle joint, and pelvic joint, with node positions located on the user's upper body serving as a priority benchmark. If the posture angle established based on the node positions exceeds the maximum tilt posture angle threshold under normal human conditions, the control module responds quickly and identifies the anomaly.

[0185] In some embodiments, the world coordinate system is established based on the edge lines of a real-time image.

[0186] In this context, the edge lines of the real-time image refer to at least one of the top, bottom, left, and right boundary lines of the entire real-time image. However, in practical applications, since the TOF camera is pre-mounted on the scene and its position is already determined, the edge lines of the real-time image are used as coordinate axes in the coordinate system. For example... Figure 3 The image shown is a real-time image captured indoors by the TOF camera of this invention. The lower edge of the real-time image is used as the X-axis, and the left edge of the real-time image is used as the Y-axis. Figure 4 The user's posture is standing. Figure 5 The user's posture is that of someone who has fallen down, and the lines on the user in the image represent the user's two-dimensional posture.

[0187] In some embodiments, the step of generating the user's pose angle based on the world reference frame and the user's joint position of the real-time image when the world reference frame and the user's joint position are received includes:

[0188] If the user's joint position is received, the user's joint position is converted into an attitude line;

[0189] If an attitude line is received, the user's attitude angle is generated based on the attitude line and the world reference frame.

[0190] The attitude lines are used to simulate the user's posture, facilitating subsequent calculations of attitude angles. The attitude lines are determined based on the user's joint positions.

[0191] Specifically, the posture line is determined according to the following method: First, a line is constructed between the two hip joint positions, or between the two knee joint positions, or between the two ankle joints. Finally, a line is constructed between the midpoints of the lines connecting the neck joint position and the two hip joint positions, or between the midpoints of the lines connecting the neck joint position and the two knee joint positions, or between the midpoints of the lines connecting the neck joint position and the two ankle joint positions. The lines constructed between the midpoints of the lines connecting the neck joint position and the two hip joint positions, or between the midpoints of the lines connecting the neck joint position and the two knee joint positions, or between the midpoints of the lines connecting the neck joint position and the two ankle joint positions can serve as posture lines.

[0192] In the above, the attitude angle refers to the angle between the attitude line and the coordinate axis of the world reference system. In this embodiment, the lower edge of the real-time image is used as the X-axis of the world reference system, and the attitude angle is the angle between the attitude line and the lower edge of the real-time image.

[0193] When constructing a posture line connecting the midpoints of the lines connecting the neck joint position and the two hip joint positions, if the posture angle exceeds the safe angle range, the user is considered to have fallen. If the posture angle is within the safe angle range, the user is considered not to have fallen.

[0194] A posture line is constructed by connecting the midpoints of the lines between the neck joint position and the two knee joint positions. If the posture angle exceeds the safe angle range, the user is considered to have fallen. If the posture angle is within the safe angle range, the user is considered not to have fallen.

[0195] A posture line is constructed by connecting the midpoints of the lines between the neck joint position and the two ankle joint positions. If the posture angle exceeds the safe angle range, the user is considered to have fallen. If the posture angle is within the safe angle range, the user is considered not to have fallen.

[0196] In some approaches, the control module can track the user's node states in real-time images and determine whether the user's posture is abnormal based on the joint node states. Specifically, assuming there are K joint nodes, and the joint node states include position, velocity, motion angle, and direction, the motion state at time t is... The joints are mutually constrained. Priority is given to adding attention weights to the states of joints in the upper body, such as the head, shoulders, and neck. Attention weights are added to the states of these joints. The processing speed for a single frame of depth image is calculated as p1 frames / second, moving from top to bottom. The position of the object in each frame is calculated based on the state at different times t. Velocity matching is performed based on the difference between the position in each frame and the position in the previous frame, obtaining a sliding average of 10 frames of data. The direction of the motion angle is the direction vector between the position in the current frame and the position in the previous frame, and this is stored as the direction vector during walking. Based on a series of falling actions, multiple frames of state values ​​are collected and stored for judgment. When falling forward or backward, the joints of the upper body experience a sudden change in position and velocity. The trajectory path of priority joints is tracked. The depth information of the lower body changes slowly, while the depth information of the upper body changes rapidly. A comprehensive judgment is made based on the five values ​​of the state variables to determine abnormal posture.

[0197] Because the real-time images captured by the TOF camera may only capture a portion of the user's problem, the control module can only simulate some joint positions. To solve this problem, this invention determines which joint positions exist in the real-time image and then decides which joint positions to use to generate attitude lines. Specifically, it first determines whether both hip joints exist simultaneously. If both hip joints exist simultaneously, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two hip joint positions. If neither hip joint exists, it then determines whether both knee joints exist simultaneously. If both knee joints exist simultaneously, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two knee joint positions. If neither knee joint exists, the attitude line is constructed by connecting the midpoint of the line between the neck joint position and the two ankle joint positions.

[0198] In some methods, upon receiving a real-time image from a TOF camera, the position of the toilet seat and the user's hip joint are determined; it is then determined whether the hip joint position is higher than the toilet seat position; if the hip joint position is higher than the toilet seat position, the user is determined to have fallen; if the hip joint position is lower than the toilet seat position, the user is determined not to have fallen. In this embodiment, the user's posture angle and hip joint position are used to determine whether the user has fallen, achieving a dual determination effect and improving the accuracy of the system's fall detection.

[0199] Input devices can be touchscreens, buttons, microphones, etc., used to convert sound, kinetic energy, etc., into electrical signals and send them to the control module. In this embodiment, the input device is a touchscreen. When the user inputs a request for "toilet" on the input device, the input device can convert the request into request information including the destination location as a smart toilet.

[0200] An output device is used to receive a safety path, which is an electrical signal. If the output device receives a safety path, it can convert the safety path into at least one of the following signals: sound, light, vibration, and temperature, and output it to guide the user to reach their destination along the safety path.

[0201] Accordingly, the output device includes at least one of a loudspeaker, a display, a vibrator, and a heater.

[0202] The output device is also used to receive a correction path, which is an electrical signal. If the output device receives the correction path, it can convert the correction path into at least one of the following signals: sound, light, vibration, and temperature, and output it to guide the user along the correction path and back to the safe path.

[0203] To enhance the intelligence of smart toilets, the system includes a toilet seat, an adjustable toilet seat cover, and an adjustable footrest. This control module also identifies the user's anthropometric information from real-time images captured by a TOF camera. The module uses this information to adjust the toilet seat cover thickness and the footrest height. Anthropometric information includes the user's knee height, thigh length, calf length, and the angle between the calf and thigh. Toilet seat height is also included. Upon receiving both anthropometric and toilet seat dimensions, the module adjusts the toilet seat cover thickness and footrest height for greater user comfort. Assuming the foot pedal height is H1, the lower leg length is H2, the toilet seat thickness is H3, the toilet seat height is H4, the thigh length is L1, and the angle between the lower leg and thigh is θ, where θ is a constant and conforms to ergonomic design, the foot pedal height adjustment can be calculated according to the following formula: H1=L1*cos(θ)+H3+H4-H2.

[0204] Specifically, the toilet seat includes three washers, each hinged above the toilet seat and stackable. When the washers rotate on the toilet seat, they can either lift off the seat or remain stacked. Adjusting the number of washers stacked on the toilet seat allows for adjustment of the toilet seat's thickness. The washers are driven by a first motor, which is electrically connected to a control module. The control module controls the number of washers stacked on the toilet seat via the first motor, improving automation. When one washer is stacked on the toilet seat, the thickness is H1; when two washers are stacked, the thickness is 2*H1; and when three washers are stacked, the thickness is 3*H1.

[0205] The control module is also used to identify users in real-time images. User identities include adults, children, and guests. Adults are users registered in the system who are taller than 1.5m; children are users registered in the system who are shorter than 1.5m; and guests are users not registered in the system. The toilet seat has three gaskets from top to bottom: a first gasket, a second gasket, and a third gasket. The first gasket has a smaller inner hole and armrests on both sides. The first gasket is for children, the second gasket is for adults, and the third gasket is for guests. If the identified user is a child, the first gasket is installed on the toilet seat, and the foot pedal height is determined using a foot pedal height adjustment formula. If the identified user is an adult, the second gasket is installed on the toilet seat, and the foot pedal height is determined using the same formula. If the identified user is a guest, the third gasket is installed on the toilet seat, and the foot pedal height is determined using the same formula.

[0206] The foot pedal includes at least two pedals, stacked together and positioned in front of the toilet seat. The height of the foot pedal can be adjusted by changing the number of stacked pedals. The pedals can rotate relative to the toilet seat; when rotating, adjacent pedals can separate or stack together, thus adjusting the number of stacked pedals. The pedals are driven by a second motor, which is electrically connected to a control module. The control module controls the number of stacked pedals via the second motor, improving automation.

[0207] In addition, the control module can also recognize the gestures of the user model on the real-time image, and control the smart toilet to perform functions such as heating the toilet seat, automatically flushing the toilet, turning the toilet seat light on or off, and opening or closing the toilet seat. For example, if the gesture is a fist, the toilet seat will be open; if the gesture is scissors, the toilet will be flushed automatically; and if the gesture is a palm, the toilet seat will be heated automatically.

[0208] Accordingly, embodiments of the present invention also provide a method for guiding the blind to a smart toilet based on artificial intelligence, which is executed using the guide device for guiding the blind to a smart toilet based on artificial intelligence provided in the embodiments of the present invention. This method is used to plan the user's path within a safe area.

[0209] like Figure 1 The diagram shows a flowchart of an embodiment of the artificial intelligence-assisted user navigation method for smart toilets of the present invention, including:

[0210] S1: Receive the initial image from the TOF camera;

[0211] S2: Convert the initial image into an initial safe region image;

[0212] S3: Receive real-time images from the TOF camera;

[0213] S4: Segment the portion of the initial safe region image mapped onto the real-time image to obtain the real-time detection region image;

[0214] S5: Generate a real-time safe zone model based on the initial safe zone image and the real-time detected zone image;

[0215] S6: Determine whether there is a user image on the real-time detection area image;

[0216] S7: If there is a user image in the real-time user image, then the user image is segmented from the real-time detection area image;

[0217] S8: Convert user image into user location;

[0218] S9: Receive request information from the input device, including the destination location;

[0219] S10: Generate a secure path from the user's location to the destination location on the real-time secure zone model.

[0220] In some methods, the step of converting the initial image into an initial safe region image specifically includes:

[0221] If an initial image is received, a 3D reconstruction is performed on the initial image and an initial 3D mesh model is established.

[0222] The initial safe zone model is segmented on the initial 3D mesh model;

[0223] The initial safe region model is converted into an initial safe region image.

[0224] In some methods, if an initial image is received, the steps of performing 3D reconstruction of the initial image and establishing an initial 3D mesh model specifically include:

[0225] If an initial image is received, it is preprocessed and converted into an initial 3D point cloud.

[0226] If an initial 3D point cloud is received, the initial 3D point cloud is preprocessed and a normal vector of the initial 3D point cloud is established.

[0227] If an initial 3D point cloud with normal vectors is received, the camera pose is estimated using the ICP algorithm.

[0228] If the camera pose is received, the initial 3D point cloud with normal vectors is fused according to the camera pose to generate an initial fused point cloud.

[0229] If the initial fused point cloud is received, the Poisson reconstruction algorithm is used to convert the initial fused point cloud into an initial 3D mesh model.

[0230] In some methods, the step of preprocessing the initial image and converting it into an initial 3D point cloud upon receiving the initial image specifically includes:

[0231] If the initial image is received, denoising and hole repair processing are performed on the initial image;

[0232] The initial image after denoising and hole repair is converted into an initial 3D point cloud.

[0233] In some methods, the step of preprocessing the initial 3D point cloud and establishing its normal vectors upon receiving the initial 3D point cloud specifically includes:

[0234] If an initial 3D point cloud is received, then the initial 3D point cloud is filtered.

[0235] Calculate the normal vector of the initial 3D point cloud after filtering.

[0236] In some methods, the step of segmenting the initial safe zone model on the initial 3D mesh model specifically includes:

[0237] If a 3D mesh model is received, the initial obstacle model and the background model are segmented on the 3D mesh model;

[0238] If an initial obstacle model and a background model are received, an initial safe zone model is segmented on the background model, and the initial obstacle model is not on the initial safe zone model.

[0239] In some embodiments, the step of generating a real-time safe region model based on an initial safe region image and a real-time detected region image specifically includes:

[0240] Determine whether there are newly added obstacle images in the real-time detection area image;

[0241] If new obstacle images are added to the real-time detection area image, the real-time detection area image will be converted into a real-time safe area model.

[0242] If no new obstacle image is added to the real-time detection area image, the initial safe area model is used as the real-time safe area model.

[0243] In some methods, the step of converting real-time user images into user locations specifically includes:

[0244] If a real-time user image is received, a 3D reconstruction is performed on the real-time user image and a real-time user model is established.

[0245] Transform real-time user models into user locations.

[0246] In some methods, this AI-based guide for the visually impaired to the smart toilet also addresses user deviations from safe routes and includes the following steps:

[0247] The Kalman filter algorithm is used to generate the predicted pose of real-time user images;

[0248] If the predicted pose is received, motion parameters are generated using a human motion tracking algorithm;

[0249] Determine whether the motion parameters exceed the safe parameter range;

[0250] If the safety parameters exceed the safe movement range, a correction path will be generated;

[0251] If a corrected path is generated, the corrected path is sent to the output device.

[0252] In some embodiments, the motion parameters include at least one of the user model's position, velocity, and orientation angle.

[0253] In some embodiments, the starting point of the corrected path is the current position of the user model, and the ending point of the corrected path is the position on the safe path that is closest to the user model.

[0254] Some methods also include:

[0255] If a real-time image is received, the user's node position is established on the real-time image;

[0256] If the user's joint position is received, the user's pose angle is generated based on the world reference frame and the user's joint position.

[0257] Determine whether the user's posture angle is within a safe range;

[0258] If the user's posture angle exceeds the safe angle range, a fall signal is generated;

[0259] If a fall signal is generated, the fall signal will be sent to the output device.

[0260] In some approaches, convolutional neural networks are used to establish the user's joint positions on real-time images.

[0261] In some embodiments, the user's node location includes at least two of the following: head, neck, shoulder, elbow, hand, hip, knee, ankle, and pelvis.

[0262] In some embodiments, the world coordinate system is established based on the edge lines of a real-time image.

[0263] In some embodiments, the step of generating the user's pose angle based on the world reference frame and the user's joint position of the real-time image when the world reference frame and the user's joint position are received includes:

[0264] If the user's joint position is received, the user's joint position is converted into an attitude line;

[0265] If an attitude line is received, the user's attitude angle is generated based on the attitude line and the world reference frame.

[0266] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made to the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for guiding the blind to a smart toilet based on artificial intelligence, characterized in that, Includes the following steps: Receive the initial image from the TOF camera; Convert the initial image into an initial safe region image; Receive real-time images from a TOF camera; The portion of the initial safe region image mapped onto the real-time image is used as the real-time detection region image; A real-time safe zone model is generated based on the initial safe zone image and the real-time detected zone image; Determine whether there is a user image in the real-time detection area image; If there is a user image in the real-time user image, then the user image is segmented from the real-time detection area image; Convert user image into user location; Receive request information from the input device, including the destination location; Generate a secure path from the user's location to the destination location on a real-time secure zone model; The step of generating a real-time safe region model based on the initial safe region image and the real-time detected region image specifically includes: Determine whether there are newly added obstacle images in the real-time detection area image; If new obstacle images are added to the real-time detection area image, the real-time detection area image will be converted into a real-time safe area model. If no new obstacle image is added to the real-time detection area image, the initial safe area model is used as the real-time safe area model.

2. The method for guiding the blind to a smart toilet based on artificial intelligence, as described in claim 1, is characterized in that... The step of converting the user image into the user's location specifically includes: If a real-time user image is received, a 3D reconstruction is performed on the real-time user image and a real-time user model is established. Transform real-time user models into user locations.

3. The method for guiding the blind to a smart toilet based on artificial intelligence, as described in claim 1, is characterized in that... The step of converting the initial image into an initial safe region image specifically includes: If an initial image is received, a 3D reconstruction is performed on the initial image and an initial 3D mesh model is established. The initial safe zone model is segmented on the initial 3D mesh model; The initial safe region model is converted into an initial safe region image.

4. The method for guiding the blind to a smart toilet based on artificial intelligence, as described in claim 3, is characterized in that... If an initial image is received, the steps for performing 3D reconstruction of the initial image and establishing an initial 3D mesh model specifically include: If an initial image is received, it is preprocessed and converted into an initial 3D point cloud. If an initial 3D point cloud is received, the initial 3D point cloud is preprocessed and a normal vector of the initial 3D point cloud is established. If an initial 3D point cloud with normal vectors is received, the camera pose is estimated using the ICP algorithm. If the camera pose is received, the initial 3D point cloud with normal vectors is fused according to the camera pose to generate an initial fused point cloud. If the initial fused point cloud is received, the Poisson reconstruction algorithm is used to convert the initial fused point cloud into an initial 3D mesh model.

5. The method for guiding the blind to a smart toilet based on artificial intelligence, as described in claim 1, is characterized in that... Also includes: The Kalman filter algorithm is used to generate the predicted pose of real-time user images; If the predicted pose is received, motion parameters are generated using a human motion tracking algorithm; Determine whether the motion parameters exceed the safe parameter range; If the safety parameters exceed the safe movement range, a correction path will be generated; If a corrected path is generated, the corrected path is sent to the output device.

6. The method for guiding the blind to a smart toilet based on artificial intelligence, as described in claim 5, is characterized in that... The motion parameters include at least one of the user model's position, velocity, and orientation angle.

7. A guide device for the blind that assists users to a smart toilet based on artificial intelligence, characterized in that, include: Input device, used to generate request information; A TOF camera is used to generate initial and real-time images; The control module is used to receive initial and real-time images and convert the initial image into an initial safe area image. The process involves segmenting the portion of the initial safe region image mapped onto the real-time image to create a real-time detection region image; generating a real-time safe region model based on the initial safe region image and the real-time detection region image; determining whether a user image exists in the real-time detection region image; segmenting the user image from the real-time detection region image if a user image exists in the real-time user image; converting the user image into a user location; receiving request information from an input device, including the destination location; and generating a safe path from the user location to the destination location on the real-time safe region model. The step of generating a real-time safe region model based on the initial safe region image and the real-time detection region image specifically includes: determining whether there are any newly added obstacle images in the real-time detection region image; if there are newly added obstacle images in the real-time detection region image, then converting the real-time detection region image into a real-time safe region model; if there are no newly added obstacle images in the real-time detection region image, then using the initial safe region model as the real-time safe region model. Output device, used to receive secure paths.

8. A guide device for the blind based on artificial intelligence to assist users in reaching a smart toilet, as described in claim 7, is characterized in that... The output device includes at least one of a microphone, a display, a vibrator, and a heater.

9. A guide device for the blind based on artificial intelligence to assist users in reaching a smart toilet, as described in claim 7 or 8, characterized in that, It also includes a smart toilet, which consists of a toilet seat, a toilet seat cover, and a foot pedal. The control module can adjust the thickness of the toilet seat cover on the toilet seat, and the control module can also adjust the height of the foot pedal on one side of the toilet seat.

Images