A visual assistance system for bathing tasks and storage medium

Abstract

The application discloses a visual auxiliary system for a bathing task, comprising: an image acquisition module, which is used for acquiring an RGB image and a depth image of a bathing scene and a target human body; and a human body positioning module, which has a deep learning model, wherein the deep learning model is used for optimizing the RGB image to obtain two-dimensional coordinates of key points of the target human body, and then, in combination with the depth image, three-dimensional coordinates of the key points of the target human body are obtained. The application further provides a storage medium based on the visual auxiliary system for the bathing task. The visual auxiliary system improves the accuracy of a human body key point detection algorithm in a special environment with more water mist, improves the registration efficiency, and improves the real-time performance of human body positioning.

Description

Technical Field

[0001] This invention relates to the field of assistive device technology, and in particular to a visual assistive system and storage medium for bathing tasks. Background Technology

[0002] With the increasing aging of the population, the development of care services for the elderly is accelerating, and assisted bathing, as an important demand, takes many forms and models. With the rapid development of artificial intelligence, its application in care is an inevitable trend. By incorporating AI into robotic arms, bathing assistance can be performed more effectively. In the field of computer vision, 3D vision technology is developing rapidly, achieving remarkable results in areas such as intelligent manufacturing, autonomous driving, SLAM, 3D reconstruction, and facial recognition.

[0003] When performing intelligent bathing tasks, robotic arms precisely wash various parts of the body. However, the environment inside the bathing chamber is complex, with fluctuating lighting and abundant water vapor. Existing methods for detecting human key points are rarely capable of locating the human body in such special environments. Due to the unique nature of the bathing scenario, image dehazing is crucial for improving the accuracy of human key point detection. Based on 3D vision, an RGB-D camera is used as the visual sensor. The RGB-D camera captures the same scene using both an RGB camera and a depth camera, obtaining RGB and depth images. Registering the RGB and depth images often requires pixel-by-pixel registration, which is time-consuming and computationally demanding. In addition, bathrooms are also places with a high incidence of falls, making the detection of abnormal behavior a very important aspect.

[0004] Some existing robotic arm-assisted bathing devices lack a visual aid system, making it difficult to accurately locate the person being bathed. Some existing human key point detection systems lack algorithms for detecting key points in a shower cabin filled with water mist and complex lighting conditions. When registering depth images generated by RGB-D cameras with RGB images, most systems perform full-pixel registration, which is time-consuming and computationally demanding. Furthermore, existing robotic arm-assisted bathing devices lack the ability to detect abnormal behavior.

[0005] Therefore, those skilled in the art are dedicated to providing a visual assistance system and storage medium for bathing assistance tasks, enabling three-dimensional spatial positioning of the person being bathed, improving the detection accuracy of the algorithm in scenarios with a lot of water mist, and improving registration efficiency to enhance the real-time performance of positioning. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is how to provide a visual assistance system and storage medium for bathing assistance tasks.

[0007] To achieve the above objectives, the present invention provides a visual assistance system for bathing tasks, comprising:

[0008] The image acquisition module is used to acquire RGB and depth images of the bathing scene and the target human body;

[0009] The human body positioning module has a deep learning model. The deep learning model optimizes and dehazes the RGB image to obtain the two-dimensional coordinates of the key points of the target human body, and then combines the depth image to obtain the three-dimensional coordinates of the key points of the target human body.

[0010] Furthermore, the assisted bathing scenario includes an RGB-D vision sensor and a robotic arm, and the human body positioning module transmits the three-dimensional coordinates of the key points to the robotic arm.

[0011] Furthermore, the RGB-D vision sensor includes an RGB camera and a depth camera, and the image acquisition module acquires image information from the RGB camera and the depth camera to obtain the RGB image and the depth image.

[0012] Furthermore, the image processing procedure of the human body positioning module includes:

[0013] S1. Use the dehazing-human key point detection model to dehaze the RGB image and detect the key points of the target human body to obtain the two-dimensional coordinates of the key points of the target human body in the RGB image.

[0014] S2. Based on the camera calibration of the RGB-D vision sensor, obtain the transformation relationship from the RGB camera spatial coordinates to the depth camera spatial coordinates, and then obtain the three-dimensional coordinates of the key points of the target human body;

[0015] S3. Based on the hand-eye calibration of the RGB-D vision sensor and the robotic arm, the conversion relationship between the spatial coordinates of the RGB-D vision sensor and the spatial coordinates of the robotic arm is obtained, and then the mapping from the spatial coordinates of the RGB camera to the base of the robotic arm is obtained, thus completing the conversion of the key points of the target human body from the camera coordinates to the coordinates of the robotic arm base, and realizing the positioning of the target human body by the robotic arm.

[0016] Furthermore, the construction process of the dehazing-human key point detection model includes:

[0017] The dehazing-human key point detection model is constructed based on a convolutional neural network, including a dehazing module and a human key point detection module. The dehazing module and the human key point detection module are connected in series. The first image is input and dehazed by the dehazing module. Then, the human key point detection module performs human skeleton key point detection on the dehazed image and finally outputs a second image with human skeleton key point markings.

[0018] Take foggy human images in a bathroom scene, select foggy images containing human figures, create a dataset, and label it according to the format required for training human keypoint detection.

[0019] The loss function is selected and the dataset is used to train the dehazing-human keypoint detection model, so that the dehazing-human keypoint detection model can simultaneously perform dehazing and human keypoint detection. By adjusting the parameters, appropriate weights are obtained, the model parameters are saved, and the model is lightweighted and accelerated.

[0020] Preferably, the key points of the human body include: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0021] Furthermore, the process of converting the two-dimensional coordinates of the key points to three-dimensional coordinates in the robotic arm space includes:

[0022] S31. Based on the mapping relationship between the coordinates of the key points on the RGB camera imaging plane and the spatial coordinates of the RGB camera, obtain the spatial coordinates P of the key points on the RGB camera. rgb Specifically, it includes:

[0023] Let point p rgb Let the coordinates of a key point on the RGB camera imaging plane be given. The mapping relationship between the RGB camera spatial coordinates and the RGB camera imaging plane coordinates is as follows:

[0024] p rgb =H rgb P rgb ;

[0025] P rgb =H rgb -1 p rgb ;

[0026] Among them, H rgb This is the intrinsic parameter for converting between RGB camera spatial coordinates and RGB camera imaging plane coordinates;

[0027] S32. Based on the mapping relationship between RGB camera spatial coordinates and depth camera spatial coordinates, and based on the spatial coordinates of the keypoints in the RGB camera, obtain the spatial coordinates P of the keypoints in the depth camera. ir Specifically, it includes:

[0028] The first mapping relationship between RGB camera spatial coordinates and the depth camera spatial coordinates:

[0029] P ir =RP rgb +T;

[0030] Where R is the rotation coordinate and T is the translation matrix;

[0031] Based on the relationship between point P in the global coordinate system and the depth camera spatial coordinate system, and the relationship between point P in the global coordinate system and the RGB camera spatial coordinate system, a second mapping relationship between the RGB camera spatial coordinates and the depth camera spatial coordinates is obtained:

[0032] Mapping relationship between point P in the global coordinate system and the spatial coordinates of the depth camera:

[0033] P ir =R ir P+T ir ;

[0034] Mapping relationship between point P in the global coordinate system and RGB camera space coordinates:

[0035] P rgb =R rgb P+T rgb ;

[0036] Eliminating P from the above equation yields the second mapping relationship between the RGB camera spatial coordinates and the depth camera spatial coordinates:

[0037] P ir =R ir R rgb -1 P rgb +T ir -R ir R rgb -1 T rgb ;

[0038] By combining the first and second mapping relationships between the RGB camera space coordinate system and the depth camera space coordinate system, we can obtain the rotation matrix R and the translation matrix T:

[0039] R = R ir R rgb -1 ;

[0040] T = T ir -R ir R rgb -1 T rgb =T ir -RTrgb ;

[0041] Among them, R ir R rgb -1 T ir T rgb The extrinsic parameter matrix of the RGB-D vision sensor describes the position, orientation, and angle of the camera relative to the observed scene, specifically a translation vector and a rotation matrix, which can be obtained through camera calibration.

[0042] S33. Based on the mapping relationship between the spatial coordinates of the depth camera and the coordinates of the depth image, obtain the point P corresponding to the key point on the depth image. ir Specifically, it includes:

[0043] P ir =H ir P ir ;

[0044] Among them, H ir This is the intrinsic parameter matrix of the depth camera. The depth value of the key point is read according to its coordinate position on the depth image, thus completing the registration from the RGB image of the key point to the depth image.

[0045] S34. Based on the pixel coordinates (x, y) of the keypoints in the RGB image and the depth value Z of the keypoints, and according to perspective projection, obtain the three-dimensional coordinates (X, Y, Z) of the keypoints in the RGB camera coordinate system, specifically including:

[0046]

[0047] Where f is the focal length of the RGB-D vision sensor;

[0048] S35. Based on the hand-eye calibration of the RGB-D vision sensor and the robotic arm, the mapping relationship M between the RGB camera spatial coordinates and the robotic arm spatial coordinates is obtained, and the coordinates of the human body key points in the coordinate system of the robotic arm base are further obtained.

[0049] Furthermore, it also includes a safety warning module, which is used to detect abnormal behavior of the target human body. When the abnormal behavior occurs, the safety warning module issues a warning and suspends the bathing assistance.

[0050] Preferably, the detection process for the abnormal behavior includes:

[0051] Real-time detection of the head coordinates, left hip coordinates, and left ankle coordinates of the target human body. The first feature of abnormal behavior is that the head coordinate velocity v is faster than a certain velocity threshold. The second feature of abnormal behavior is that the vertical distance d1 between the head coordinate and the left ankle coordinate is less than a certain distance threshold. The third feature of abnormal behavior is that the vertical distance d2 between the left hip coordinate and the left ankle coordinate is less than another distance threshold.

[0052] Let V be the first feature threshold of abnormal behavior, D1 be the second feature threshold of abnormal behavior, and D2 be the third feature threshold of abnormal behavior.

[0053] If the detected head coordinate velocity v is greater than the first feature threshold V of the abnormal behavior, then the second and third features are detected.

[0054] If the distance d1 between the head coordinate and the left ankle coordinate of the second feature is less than the second feature threshold D1, and the distance d2 between the left hip coordinate and the left ankle coordinate in the vertical direction of the third feature is less than the third feature threshold D2, then the target human behavior is determined to be abnormal.

[0055] The present invention also provides a computer-storable medium comprising the aforementioned visual assistance system for bathing tasks, wherein modules of the visual assistance system, when executed by a processor, implement the steps of any of the methods described above.

[0056] The present invention has at least the following beneficial technical effects:

[0057] 1. The visual assistance system for bathing assistance tasks provided by the present invention adds a visual assistance system based on RGB-D vision sensor to the robotic arm bathing assistance task. Using an RGB-D camera as the vision sensor, a vision system is added to the robotic arm to realize the three-dimensional spatial positioning of the target human body being bathed, and on this basis, it helps the robotic arm to perform trajectory planning.

[0058] 2. The visual assistance system for bathing tasks provided by this invention proposes a defogging-human key point detection algorithm. Based on human key point detection, the algorithm's defogging capability is increased, and a lightweight defogging algorithm is added, which improves the accuracy of the human key point detection algorithm in special environments with a lot of water mist.

[0059] 3. The visual assistance system for bathing tasks provided by this invention only performs depth image and RGB image registration on the required human body key points, without needing to perform pixel-by-pixel registration on the entire image, which greatly reduces the number of pixels to be registered, improves registration efficiency, reduces registration time, and improves the real-time performance of human body positioning.

[0060] 4. The visual assistance system for bathing tasks provided by this invention is based on human posture detection. It detects the speed and position of specific key points to complete abnormal behavior detection. When abnormal situations such as falls occur, it issues an alarm and stops the bathing task in time to ensure the safety of the person being assisted in bathing.

[0061] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of a robotic arm-assisted bathing scenario according to an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of a visual assistance system for bathing tasks according to an embodiment of the present invention;

[0064] Figure 3 This is a flowchart of the defogging-human key point detection process according to an embodiment of the present invention;

[0065] Figure 4 This is a flowchart of human body positioning according to an embodiment of the present invention;

[0066] Figure 5 This is a flowchart illustrating the conversion of two-dimensional coordinates of key human body points to three-dimensional coordinates in the robotic arm space, according to an embodiment of the present invention. Detailed Implementation

[0067] The preferred embodiments of the present invention are described below to make the technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0068] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0069] This invention provides a vision-assisted system for bathing tasks, specifically for robotic arm-assisted bathing scenarios. For example... Figure 1As shown, the assisted bathing scenario includes an RGB-D vision sensor 100 and a robotic arm 200. The robotic arm 200 provides assisted bathing services to the target human body 300. The robotic arm 200 includes a robotic arm end effector 201 and a robotic arm base 202. The visual range of the RGB-D vision sensor 100 includes the entire motion space of the robotic arm 200 during the assisted bathing task and the target human body 300. The RGB-D vision sensor 100 includes two parts: an RGB camera and a depth camera. The RGB camera acquires RGB images, and the depth camera acquires depth images.

[0070] It should be noted that, in order to protect the privacy of those receiving assisted bathing, the entire visual assistance system operates offline and does not save any data.

[0071] like Figure 2 As shown, the visual assistance system for bathing assistance tasks in this embodiment specifically includes an image acquisition module S40 and a human body positioning module S50. The image acquisition module acquires scene images based on the RGB camera and depth camera of the RGB-D camera 100, obtaining RGB images and depth images of the bathing scene and the target human body; the human body positioning module optimizes and dehazes the images based on a deep learning model and obtains the two-dimensional coordinates of the key points of the target human body, combines the depth information to obtain the three-dimensional coordinates of the key points of the target human body, and transmits them to the robotic arm.

[0072] like Figure 4 As shown, the specific steps for the human body positioning module to locate the human body include:

[0073] S501. After the image acquisition module obtains the RGB image and the depth image, the dehazing-human key point detection model is used to dehaze the RGB image and detect human key points to obtain the two-dimensional coordinates of the target human key points in the RGB image.

[0074] S502. Based on the camera calibration of the RGB-D vision sensor, the transformation relationship from RGB camera spatial coordinates to depth camera spatial coordinates is obtained, and then the three-dimensional coordinates of key points of the target human body are obtained.

[0075] S503. Perform hand-eye calibration between the RGB-D vision sensor and the robotic arm to obtain the conversion relationship between the spatial coordinates of the RGB-D vision sensor and the spatial coordinates of the robotic arm. Then, obtain the mapping from the RGB camera spatial coordinates to the base of the robotic arm. That is, complete the conversion of the key points of the target human body from the camera coordinates to the coordinates of the robotic arm base, and realize the positioning of the target human body by the robotic arm.

[0076] like Figure 3 As shown, the defogging-human key point detection process in this embodiment is as follows:

[0077] Considering the unique characteristic of a bathing scene with abundant water mist, this example combines a dehazing model with a human keypoint detection model. Specifically, the dehazing model is embedded into the human keypoint model to obtain a dehazing-human keypoint detection model. This model optimizes the scene for dehazing and detects and identifies human keypoints. Alternatively, other dehazing methods can be used, such as first performing image enhancement and then using the enhanced image as input for human keypoint detection, performing the detection in two steps.

[0078] A dehazing-human keypoint detection model 900 is constructed based on a convolutional neural network, including a dehazing module 901 and a human keypoint detection module 902. The dehazing module and the human keypoint detection module are connected in series. The input image 800 is dehazed by the dehazing module, and then the human keypoint detection module performs human skeleton keypoint detection on the clear image after dehazing. Finally, a clear image 1000 with human skeleton keypoint markings is output.

[0079] For example, the AOD-Net model is embedded into the HRNet model to obtain AOD-HRNet, which is used to detect human key points on the target human body; the AOD-Net and HRNet are concatenated to construct a dehazing-human key point detection model, namely the example AOD-HRNet. In the AOD-HRNet network structure, the input of the HRNet module is the output of the AOD module, the input of AOD-HRNet is an RGB image, and the output is a key point heatmap.

[0080] To improve network performance, the network structure can be modified. For example, a convolutional layer can be added between AODNet and HRNet as a skip link, which can add the output of AODNet and the original input before feeding it into HRNet; a convolutional layer can be added to the output layer of HRNet as an attention mechanism, which can make the model pay more attention to important regions.

[0081] Before training the model, the dataset needs to be processed to obtain images containing fog and human figures, which are then compiled into a dataset and labeled according to the format required for human keypoint detection training for subsequent training.

[0082] For example, by using Python batch processing to perform fogging on the COCO dataset, a defogging-human keypoint detection model dataset is obtained; other readily conceivable defogging algorithms and human keypoint detection algorithms can also be used to detect human keypoints.

[0083] By selecting an appropriate loss function to train the model, the model can simultaneously perform two tasks: image dehazing and human keypoint detection, thus optimizing the performance of image dehazing and human keypoint detection. By adjusting the parameters to obtain the most suitable weights, the model parameters are saved, and the model is lightweighted and accelerated so that it can be deployed on embedded devices.

[0084] For example, the PyTorch framework is used to train a dehazing-human keypoint detection model. The trained weights are converted to TRT format to accelerate the model. First, the weights are converted to an ONNX model using the onnx.export() function to accelerate the model. Then, TensorRT is used to further accelerate the model, making it lightweight and faster, so that it can achieve real-time detection. Other lightweight methods can also be used to accelerate the model.

[0085] The key points of the human body in this invention are: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0086] like Figure 5 As shown, the process of transforming the two-dimensional coordinates of key points to the three-dimensional coordinates of the robotic arm space includes:

[0087] S5011, The two-dimensional coordinates of the key point refer to the pixel coordinates of the key point on the RGB image. It is necessary to convert the pixel coordinates of the key point on the RGB image into the coordinates of the key point on the RGB camera imaging plane by translation or scaling, so as to obtain the coordinates of the key point on the RGB camera imaging plane.

[0088] S5012. Based on the mapping relationship between the coordinates of the key points on the RGB camera imaging plane and the spatial coordinates of the RGB camera, obtain the spatial coordinates P of the key points on the RGB camera. rgb Specifically, it includes:

[0089] Let point p rgb Let the coordinates of a key point on the RGB camera imaging plane be given. The mapping relationship between the RGB camera spatial coordinates and the RGB camera imaging plane coordinates is as follows:

[0090] p rgb =H rgb P rgb ;

[0091] P rgb =H rgb -1 p rgb ;

[0092] Among them, H rgb This is the intrinsic parameter for converting between RGB camera spatial coordinates and RGB camera imaging plane coordinates;

[0093] S5013. Based on the mapping relationship between RGB camera spatial coordinates and depth camera spatial coordinates, and based on the spatial coordinates of the keypoints in the RGB camera, obtain the spatial coordinates H of the keypoints in the depth camera. rgb Specifically, it includes:

[0094] Mapping relationship between RGB camera spatial coordinates and depth camera spatial coordinates 1:

[0095] P ir =RP rgb +T;

[0096] Where R is the rotation coordinate and T is the translation matrix;

[0097] Based on the relationship between point P in the global coordinate system and the depth camera spatial coordinate system, and the relationship between point P in the global coordinate system and the RGB camera spatial coordinate system, the mapping relationship 2 between the RGB camera spatial coordinates and the depth camera spatial coordinates is obtained:

[0098] Mapping relationship between point P in the global coordinate system and the spatial coordinates of the depth camera:

[0099] P ir =R ir P+T ir ;

[0100] Mapping relationship between point P in the global coordinate system and RGB camera space coordinates:

[0101] P rgb =R rgb P+T rgb ;

[0102] Eliminating P from the above equation yields the mapping relationship 2 between the RGB camera spatial coordinates and the depth camera spatial coordinates:

[0103] P ir =R ir R rgb -1 P rgb +T ir -R ir R rgb -1 T rgb ;

[0104] By combining mapping relationships 1 and 2 between the RGB camera space coordinate system and the depth camera space coordinate system, we can obtain the rotation matrix R and the translation matrix T:

[0105] R = R ir R rgb -1 ;

[0106] T = T ir -R ir R rgb -1 T rgb =T ir -RT rgb ;

[0107] Among them, R ir R rgb -1 T ir T rgb The extrinsic parameter matrix of the RGB-D vision sensor can be obtained through camera calibration.

[0108] According to the above formula, the camera extrinsic parameter R is obtained through camera calibration. ir R rgb -1 T ir T rgb This allows us to obtain the mapping relationship between RGB camera spatial coordinates and depth camera spatial coordinates, and thus obtain the depth camera spatial coordinates P of the keypoint. ir ;

[0109] S5014. Based on the mapping relationship between the spatial coordinates of the depth camera and the coordinates of the depth image, obtain the point P corresponding to the key point on the depth image. ir Specifically, it includes:

[0110] P ir =H ir P ir ;

[0111] Among them, H ir This is the intrinsic parameter matrix of the depth camera. The depth value of the key point is read according to its coordinate position on the depth image, thus completing the registration from the RGB image of the key point to the depth image.

[0112] S5015. Based on the pixel coordinates (x, y) of the keypoints in the RGB image and the depth value Z of the keypoints, and according to perspective projection, obtain the three-dimensional coordinates (X, Y, Z) of the keypoints in the RGB camera coordinate system, specifically including:

[0113]

[0114] Where f is the focal length of the RGB-D vision sensor.

[0115] S5016. Based on the hand-eye calibration of the RGB-D vision sensor and the robotic arm, the mapping relationship M between the RGB camera spatial coordinates and the robotic arm spatial coordinates is obtained. Furthermore, the coordinates of key human body points in the coordinate system of the robotic arm base are obtained, specifically including:

[0116] In this example, the relationship between the robotic arm and the camera is "eye outside the hand". The positions of the camera coordinate system and the robotic arm base coordinate system are fixed. Based on multiple robotic arm pose transformation data and camera calibration parameters, a coordinate transformation formula in the form of "AX = XB" is obtained. The rotation matrix and translation matrix of the mapping relationship between the camera coordinate system and the robotic arm base coordinate system are calculated according to the Tsai two-step method. Other methods can also be used to find the transformation matrix.

[0117] The key points are mapped to the RGB camera space coordinates in the RGB-D vision sensor. Based on the mapping relationship between the RGB camera coordinates and the coordinate system of the robotic arm base, the coordinates of the key points in the coordinate system of the robotic arm base are obtained.

[0118] like Figure 2 As shown, the visual assistance system in this embodiment also includes a safety warning module S60. Based on abnormal behavior detection, the safety warning module promptly issues a warning and suspends the bathing task when the target human body falls, slips, or slides off a chair. The abnormal behavior detection method specifically includes:

[0119] Real-time detection of the head coordinates, left hip coordinates, and left ankle coordinates of the target human body. The first feature of abnormal behavior is that the head coordinate velocity v is faster than a certain velocity threshold. The second feature of abnormal behavior is that the vertical distance d1 between the head coordinate and the left ankle coordinate is less than a certain distance threshold. The third feature of abnormal behavior is that the vertical distance d2 between the left hip coordinate and the left ankle coordinate is less than another distance threshold.

[0120] Let V be the first feature threshold of abnormal behavior, D1 be the second feature threshold of abnormal behavior, and D2 be the third feature threshold of abnormal behavior.

[0121] If the detected head coordinate velocity v is greater than the first feature threshold V of the abnormal behavior, then the second and third features are detected.

[0122] If the distance d1 between the second feature head coordinate and the left ankle coordinate is less than the second feature threshold D1, and the distance d2 between the third feature left hip coordinate and the left ankle coordinate in the vertical direction is less than the third feature threshold D2, then the target human behavior is considered abnormal.

[0123] The present invention also provides a computer-storable medium, which includes a visual assistance system for bathing tasks as described above, wherein when the methods of each module in the visual assistance system for bathing tasks are executed by a processor, any of the aforementioned method steps are implemented.

[0124] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A vision aid system for a bathing task, characterized in that, The application relates to a bathing assistance system, comprising: an image acquisition module for acquiring an RGB image and a depth image of a target human body in a bathing scene; a human body positioning module having a deep learning model, which optimizes the RGB image for defogging to obtain two-dimensional coordinates of key points of the target human body, and combines the depth image to obtain three-dimensional coordinates of the key points of the target human body; the bathing scene comprises an RGB-D vision sensor and a mechanical arm, and the human body positioning module transmits the three-dimensional coordinates of the key points to the mechanical arm; the RGB-D vision sensor comprises an RGB camera and a depth camera, and the image acquisition module acquires image information of the RGB camera and the depth camera to obtain the RGB image and the depth image; the image processing process of the human body positioning module comprises: S1. using a defogging-human key point detection model to perform defogging processing on the RGB image and to detect the key points of the target human body, so as to obtain two-dimensional coordinates of the key points of the target human body in the RGB image; S2. according to camera calibration of the RGB-D vision sensor, a conversion relationship from the RGB camera space coordinates to the depth camera space coordinates is obtained, and then three-dimensional coordinates of the key points of the target human body are obtained; S3. according to hand-eye calibration of the RGB-D vision sensor and the mechanical arm, a conversion relationship from the RGB-D vision sensor space coordinates to the mechanical arm space coordinates is obtained, and then a mapping from the RGB camera space coordinates to the mechanical arm base is obtained, so that the conversion from the camera coordinates of the key points of the target human body to the mechanical arm base coordinates is completed, and the positioning of the mechanical arm on the target human body is realized; the construction process of the defogging-human key point detection model comprises: a convolutional neural network is used to construct the defogging-human key point detection model, which comprises a defogging module and a human key point detection module; the defogging module and the human key point detection module are connected in series, a first image is input into the defogging module for defogging, and then the defogged image is input into the human key point detection module for human skeleton key point detection, and finally a second image with human skeleton key point marks is output; foggy human body images in a bathroom scene are shot, foggy images with human bodies are screened, and a data set is formed; the data set is annotated according to a required format for human key point detection training; a loss function is selected to train the defogging-human key point detection model by using the data set, so that the defogging-human key point detection model can simultaneously complete defogging and human key point detection; appropriate weights are obtained by adjusting parameters, model parameters are saved, and the model is subjected to lightweight processing and acceleration processing; the human key points comprise a head, left shoulders, right shoulders, left elbows, right elbows, left wrists, right wrists, left hips, right hips, left knees, right knees, left ankles and right ankles; a safety warning module is used for detecting abnormal behaviors of the target human body; when the abnormal behaviors occur, the safety warning module issues a warning and pauses the bathing assistance; and the detection process of the abnormal behaviors comprises: Real-time detection is performed on the target human head coordinates, left hip coordinates and left ankle coordinates, the first feature of abnormal behavior is that the head coordinate speed v is faster than a speed threshold, the second feature of abnormal behavior is that the head coordinate in the vertical direction is less than a distance threshold d1 from the left ankle coordinate, and the third feature of abnormal behavior is that the left hip coordinate in the vertical direction is less than another distance threshold d2 from the left ankle coordinate; The first feature threshold V of abnormal behavior, the second feature threshold D1 of abnormal behavior and the third feature threshold D2 of abnormal behavior are set; If the head coordinate speed v is greater than the first feature threshold V of abnormal behavior, the second feature and the third feature are detected; If the second feature head coordinate is less than the second feature threshold D1 from the left ankle coordinate, and the third feature left hip coordinate is less than the third feature threshold D2 from the left ankle coordinate in the vertical direction, it is determined that the target human behavior is abnormal.

2. The visual assistance system for bathing tasks of claim 1, wherein, The process of converting the two-dimensional coordinates of the key points to the three-dimensional coordinates of the mechanical arm space includes: S31, obtaining the spatial coordinates of the key points in the RGB camera according to the mapping relationship between the coordinates of the key points on the imaging plane of the RGB camera and the spatial coordinates of the RGB camera , and specifically comprises: Setting points The mapping relationship between the RGB camera space coordinates and the RGB camera imaging plane coordinates for a certain key point is: ； ； wherein, is the conversion parameter of the RGB camera space coordinates and the RGB camera imaging plane coordinates; S32, according to the mapping relationship between the RGB camera space coordinates and the depth camera space coordinates, and according to the space coordinates of the key points in the RGB camera, the space coordinates of the key points in the depth camera are obtained , specifically comprising: The first mapping relationship between the RGB camera space coordinates and the depth camera space coordinates: ； wherein, is a rotation matrix, is a translation matrix; According to the relationship between the point P in the global coordinates and the depth camera space coordinates, and the relationship between the point P in the global coordinates and the RGB camera space coordinates, the second mapping relationship between the RGB camera space coordinates and the depth camera space coordinates is obtained: The mapping relationship between the point P in the global coordinates and the depth camera space coordinates: ； The mapping relationship between the point P in the global coordinates and the RGB camera space coordinates: ； Eliminate the above formula P, and obtain the second mapping relationship between the RGB camera space coordinates and the depth camera space coordinates: ； The first mapping relationship and the second mapping relationship of the RGB camera space coordinates and the depth camera space coordinates can obtain a rotation matrix , a translation matrix : ； ； wherein, , , , is an extrinsic matrix of the RGB-D vision sensor, which can be obtained by camera calibration; S33, obtaining a corresponding point of the key point on the depth image according to a mapping relationship between the depth camera space coordinates and the depth image coordinates , and specifically comprises: ； wherein, is the intrinsic matrix of the depth camera, the depth value of the key point is read according to the coordinate position of the key point on the depth image, and thus the registration of the key point RGB image to the depth image is completed. S34, according to the pixel coordinates (x, y) of the key points on the RGB image and the depth value Z of the key points, and according to the perspective projection, the three-dimensional coordinates (X, Y, Z) of the key points in the RGB camera space coordinates are obtained, which specifically includes: ； wherein, is the focal length of the RGB-D vision sensor; S35, according to the hand-eye calibration of the RGB-D vision sensor and the mechanical arm, the mapping relationship M between the RGB camera space coordinates and the mechanical arm space coordinates is obtained, and further the coordinate point of the human body key point in the mechanical arm base coordinates is obtained.

Images