Household intelligent old-age care robot system based on multifunctional integration and control method thereof

Abstract

The invention provides a household intelligent old-age care robot system based on multifunctional integration and a control method thereof. The system is provided with four functional modules, and comprises a safety protection module, which is used for constructing an old man body posture model based on media so as to analyze and monitor the safety state of the old man; the daily care module is fused with the yolov8 and the MoveIt so as to recognize garbage in the indoor environment and control the robot to conduct sweeping; the man-machine interaction module is used for carrying out simple voice interaction with the old; and the working stability module is used for carrying out real-time positioning and family map drawing through an ORB-SLAM2 technology and is used for realizing robot path planning, automatic obstacle avoidance and navigation and ensuring the working stability. According to the home intelligent old-age care robot system and the control method thereof, integration of multiple functions of old-age fall detection, daily garbage removal, real-time monitoring and voice emotion interaction is achieved, the working efficiency of the robot is greatly improved, meanwhile, the stability is considered, and services can be better provided for the old.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and robotics technology, and in particular relates to a home intelligent elderly care robot system based on multi-functional integration and a control method thereof. Background Art

[0002] The problem of global population aging is getting worse, and the traditional elderly care model represented by nursing staff is facing many challenges. With the development of society, it has exposed shortcomings such as labor shortage, high cost, and poor service. Therefore, using modern technology to develop an intelligent elderly care robot has become a new solution, which can significantly improve this phenomenon. However, today's elderly care service robot system has a relatively single function, does not integrate multiple functions, and cannot meet the daily needs of the elderly. On the other hand, many of these elderly care robots have poor accuracy and stability, which leads to low work efficiency of the robots and cannot adapt well to the surrounding environment. Therefore, it is very important to integrate deep learning algorithm models into the intelligent elderly care robot system, which can not only integrate multiple functions, but also effectively improve the stability and efficiency of the robot's work. Summary of the invention

[0003] Based on the above problems existing in the prior art, the present invention proposes a home intelligent elderly care robot system based on multi-functional integration and a control method thereof, which are used to solve the problems of single function and unstable operation of the elderly care service robots in the prior art.

[0004] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0005] A home-based intelligent elderly care robot system based on multifunctional integration, characterized in that it includes a central processing unit arranged on the home-based elderly care robot, and a sensor module, an interface module, a network communication module, a voice module, a drive module and a data processing module connected to the central processing unit; the central processing unit is internally provided with four functional modules, including: a safety protection module, a daily care module, a human-computer interaction module and a stability module;

[0006] The safety protection module builds a body posture model of the elderly based on the mediapipe technology to analyze and monitor the elderly's safety status and transmit the monitoring data to the terminal; when the elderly falls, the robot is controlled to sound an alarm and notify the terminal; the terminal includes the children's device;

[0007] The daily care module is used to control the robot to cruise along a specific route, integrate yolov8 and MoveIt to identify garbage in the indoor environment, and control the robot to clean;

[0008] The human-computer interaction module is used to communicate with the elderly through voice and emotion;

[0009] The stability module performs real-time positioning and home mapping based on ORB-SLAM2 technology to achieve robot path planning and automatic obstacle avoidance to ensure working stability.

[0010] More specifically, the sensor module and the data processing module are unidirectionally connected to the central processing unit to input information to the central processing unit;

[0011] The voice module and the driving module are unidirectionally connected to the central processor to receive information output by the central processor;

[0012] The interface module, the network and communication module are bidirectionally connected with the central processing unit to exchange information.

[0013] The present invention also provides a control method for the above system, which specifically includes:

[0014] The method by which the security protection module realizes its functions:

[0015] S11. Record continuous images containing the elderly through the sensor module, input the continuous video frames into the convolutional neural network CNN, and obtain the feature map after convolution. The size calculation formula of the feature map is expressed as:

[0016]

[0017] Among them, W and H are the width and height of the first layer feature map, the number of channels of the feature map is the number of convolution kernels, and W I , H I is the width and height of the input image, W K , H K , P K , S K They are the width, height, padding, and stride of the convolution kernel K respectively;

[0018] S12. Introduce the ReLU activation function to activate the convolutional feature map to obtain the feature map A, which enhances the network expression ability. The formula is expressed as:

[0019] f(a)=max(0,a);

[0020] Among them, a represents the output value of a certain position of the feature map after the convolution layer;

[0021] Perform the maximum pooling operation on the activated feature map to reduce the amount of calculation. The expression is:

[0022]

[0023] Among them, the size of the pooling window is p*p, P i,j,nis the eigenvalue at coordinate (i, j) on the nth channel of the feature map after pooling, A i*p+m,j*p+l,n is the value of the corresponding coordinate of the above activated feature map on the nth channel;

[0024] Repeat the above operations, use the output of the previous layer as the input of the next layer, perform multiple convolution, activation and pooling operations, flatten the final feature map into a one-dimensional vector, and input it into the fully connected layer to integrate the extracted edge features, texture features, local shape features, and semantic features;

[0025] S13, performing heat map regression processing on a series of continuous feature maps obtained in step S12, and extracting key points of the human body from them; using mean square error to define the loss function of the heat map, the formula is expressed as:

[0026]

[0027] Among them, H r , W r H is the height and width of the heat map. t and H f Represent the true heat map value and predicted heat map value of pixel point (i, j) respectively; take the peak point in the heat map as the position of the key point of the human body, reproject the peak coordinate of the heat map back to the original coordinate space of the image, and then get the specific position of the key point of the human body. The formula is expressed as:

[0028]

[0029] Among them, (x, y) is the coordinate of the key point of the human body in the original image, W I , H I is the width and height of the input image, (i peak ,j peak ) represents the coordinates of the peak in the heat map;

[0030] S14, the set of human body key points extracted is recorded as Q = {q 1 ,q 2 ,q 3 ...q n}; The human body key points include head, shoulders, elbows, wrists, knees, ankles; for all the key points of the human body in two consecutive video frames, i To calculate the motion speed and acceleration, the formula is:

[0031]

[0032] Among them, (x 1 ,y 1 ), (x 2 ,y 2 ) are the key points qi The coordinates in two consecutive video frames, Δt is the time interval between the two video frames, v 1 、v 2 is the speed in different video frames, and a is the acceleration;

[0033] S15. Set the speed threshold and acceleration threshold. When the speed and acceleration of 1 / 3 of the key points exceed this threshold, the system will issue an alarm. The threshold judgment expression is as follows:

[0034]

[0035] Among them, v threshold is the speed threshold, a threshold is the acceleration threshold; when D=1, it is considered that the old man falls, and the robot is controlled to issue an alarm and notify the terminal; when D=0, no response is made.

[0036] Furthermore, the method for the daily care module to realize its functions specifically includes the following steps:

[0037] S21. Based on the target detection algorithm yolov8, the garbage is identified and located, and the yolov8 model is trained using a data set containing garbage and trash cans;

[0038] S22, using the sensor module to record the current home ground environment in real time, when garbage appears in the captured image, the trained yolov8 model is used to accurately identify the garbage, and the location of the garbage is regionally located, and then the information is transmitted to the central processor in a timely manner;

[0039] S23, combining the deep learning model algorithm with the robot operating system ROS. After receiving the garbage location information transmitted by the central processor, MoveIt performs collision-free planning of the robot's robotic arm's grasping motion trajectory based on the current relative position of the robot's robotic arm and the garbage and the angles of each joint of the robotic arm.

[0040] S24, ROS receives the path sequence information planned by MoveIt and converts this information into a control signal to transmit to the motor, thereby providing power for the movement of the robot arm to remove the garbage. When a trash can is identified, the driver module controls the robot arm to discard the garbage.

[0041] Furthermore, the method for the human-computer interaction module to realize its function specifically includes the following steps:

[0042] S31, the central processing unit controls the sensor module to collect the voice information of the elderly;

[0043] S32, preprocessing the collected voice information in a data processing module, including noise removal and signal enhancement;

[0044] S33. Convert the elderly’s voice signals into a language that can be executed by the machine itself, match the voice features with the trained model based on deep learning, and complete human-computer interaction through the voice module.

[0045] Furthermore, the method for the stability module to realize its function specifically comprises the following steps:

[0046] S41, extracting ORB features; the ORB features include FAST corner points and BRIEF descriptors; the specific extraction process is:

[0047] S411, determine FAST corner points:

[0048] The camera is used to collect images. For a pixel point p in the image, there are N pixels in a circle with a radius of r = 3 pixels around it, with it as the center. The brightness difference between the pixel point p and the pixels in its area is calculated and compared with the set threshold. The formula is expressed as:

[0049] I p -I i |>threshold;

[0050] Among them, I p Represents the brightness value of pixel p, I i Indicates the brightness value of any point i among the N pixels in the area of ​​pixel p; threshold is the set threshold; if the brightness difference between p and a certain number of pixels exceeds the threshold, p is considered to be a FAST corner point, that is, a feature point;

[0051] S412, obtain the BRIEF descriptor:

[0052] When a pixel point p is considered as a feature point, its direction is used as a reference to define an n*n neighborhood, in which n pairs of pixels are randomly selected, and the following test function is defined:

[0053]

[0054] Among them, I a,m ,I b,m Indicates the brightness value of the mth pair of pixels a and b, where m is 1 to n. T is the set threshold value, which is related to the image brightness distribution and noise level. It can be determined according to the actual situation, and is generally between 10 and 40.

[0055] Arrange the results of the test function from high to low, and the binary number formed is the BRIEF descriptor of the feature point p;

[0056] S42, introducing the Hamming distance, performing feature matching on two consecutive image frames to determine the relative motion between the two adjacent image frames;

[0057] The Hamming distance formula is expressed as:

[0058]

[0059] Where L is the Hamming distance between two consecutive image frames; f 1 (i) f 2 (i) represents the i-th digit of the descriptor of two feature points respectively; Represents XOR operation; the smaller the Hamming distance L is, the more similar the two descriptors are, that is, the more the two feature points match;

[0060] For any pair of matching feature points p 1 and p 2 , satisfying the epipolar geometry constraint, the formula is expressed as:

[0061]

[0062] Among them, E is composed of the rotation matrix R and the translation matrix t, which is a 3*3 matrix, P 2 , P 1 The feature point p 2 、p 1 The two-dimensional coordinate matrix is ​​expanded to a 3*1 homogeneous coordinate matrix.

[0063] Substitute the eight pairs of matched feature points into the epipolar geometry constraint equation, use the eight-point method to solve E, and decompose E to obtain the rotation matrix R and translation matrix t; therefore, through the continuous rotation matrix R and translation matrix t, the position and posture of the camera at different times can be determined, and the movement of the camera can be estimated;

[0064] S43, selecting a suitable key frame: selecting a video frame containing the most new feature points as a key frame to ensure that different environmental states can be better represented. For each selected key frame, a corresponding local map is constructed. The local map is centered on the key frame, based on the feature points in the key frame, and combined with the changes in camera parameters and posture, to determine other feature points within the set distance and angle range of the feature point, and then determine the map points by triangulation method. The local map includes these feature points and map points;

[0065] The local BA optimization is performed on the local map by minimizing the reprojection error. The camera pose and map point positions can be adjusted through local BA optimization so that the map points projected onto the image are as consistent as possible with the actual observed feature points. The objective function of local BA optimization is:

[0066]

[0067] Among them, α is the camera projection function, which can transform the coordinates of points on the three-dimensional plane to the two-dimensional plane, T z is the camera pose of the zth key frame, P q represents the three-dimensional coordinates of the qth map point, p zq represents the two-dimensional coordinates of the map point q observed in the zth key frame;

[0068] S44, gradually obtain the global map by continuously constructing local maps; and perform loop detection to determine whether the camera has returned to the location it has been to; if a closed loop is detected, perform BA optimization on the global map to eliminate the accumulated error;

[0069] More specifically, step S44 includes:

[0070] S441, calculating the average Hamming distance between each descriptor in the current key frame and several descriptors with the highest response strength in the historical key frames, if the obtained average Hamming distance exceeds the set threshold, it is considered that a closed loop has occurred. The setting of the threshold is related to the complexity of the home environment.

[0071] S442. When a closed loop is detected, BA optimization is performed on the global map, and the key frames at both ends of the closed loop and all key frames and map points between them are included in the optimization range, and the camera pose and the position of the map points are re-optimized to minimize the reprojection error.

[0072] Based on the above technical solution, the present invention has the following beneficial effects:

[0073] The present invention closely combines the deep learning algorithm model with the robot functional module, and proposes a multifunctional integrated home-based elderly care robot system and its control method.

[0074] In the safety protection module, a body posture model of the elderly is built based on mediapipe to detect falls of the elderly, breaking through the limitations of traditional safety monitoring, accurately capturing subtle changes in the elderly's body, quickly judging the risk of falling and issuing alarms in time, greatly improving efficiency and accuracy.

[0075] In the daily life care module, the yolov8 model and MoveIt are combined to realize the cleaning of garbage by the robotic arm. The yolov8 deep learning algorithm can accurately and quickly identify garbage, and MoveIt can realize intelligent path planning of the robotic arm. The two can work together to efficiently complete the garbage cleaning task.

[0076] At the same time, based on ORB-SLAM2 technology, the robot can locate itself in the home-based elderly care environment and build a home map in real time to ensure that the robot accurately plans the path and automatically avoids obstacles, thus improving the stability of its work. The overall system can provide all-round assistance to the elderly in terms of safety protection, daily life care, emotional interaction, etc., which greatly improves the robot's work efficiency while ensuring the stability of the robot's work. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0078] Figure 1 This is a functional module diagram of the robot system and its control method proposed in the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0080] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0081] like Figure 1 As shown, the robot system proposed by the present invention includes a central processing unit arranged on a home-based elderly care robot, and a sensor module, an interface module, a network communication module, a voice module, a drive module and a data processing module connected to the central processing unit; four functional modules are arranged inside the central processing unit, including: a safety protection module, a daily care module, a human-computer interaction module and a stability module;

[0082] The safety protection module builds a body posture model of the elderly based on the mediapipe technology to analyze and monitor the elderly's safety status and transmit the monitoring data to the terminal; when the elderly falls, the robot is controlled to sound an alarm and notify the terminal; the terminal includes the children's device;

[0083] The daily care module is used to control the robot to cruise along a specific route, integrate yolov8 and MoveIt to identify garbage in the indoor environment, and control the robot to clean;

[0084] The human-computer interaction module is used to communicate with the elderly through voice and emotion;

[0085] The stability module performs real-time positioning and home mapping based on ORB-SLAM2 technology to achieve robot path planning and automatic obstacle avoidance to ensure working stability.

[0086] In addition, the sensor module and the data processing module are unidirectionally connected to the central processing unit to input information to the central processing unit;

[0087] The voice module and the driving module are unidirectionally connected to the central processor to receive information output by the central processor;

[0088] The interface module, the network and communication module are bidirectionally connected with the central processing unit to exchange information.

[0089] For the above-mentioned intelligent elderly care robot system, this embodiment also proposes a specific control method of the system so that the four functional modules can well realize their own functions, which are given one by one below:

[0090] As a preferred implementation, the method for the security protection module to realize its function specifically includes:

[0091] S11. Record continuous images containing the elderly through the depth camera in the sensor module, input the continuous video frames into the convolutional neural network CNN, and obtain the feature map after convolution. The size calculation formula of the feature map is expressed as:

[0092]

[0093] Among them, W and H are the width and height of the first layer feature map, the number of channels of the feature map is the number of convolution kernels, and W I , H I is the width and height of the input image, W K , H K , P K , S K They are the width, height, padding, and stride of the convolution kernel K respectively;

[0094] S12. Introduce the ReLU activation function to activate the convolutional feature map to obtain the feature map A, which enhances the network expression ability. The formula is expressed as:

[0095] f(a)=max(0,a);

[0096] Among them, a represents the output value of a certain position of the feature map after the convolution layer;

[0097] Perform the maximum pooling operation on the activated feature map to reduce the amount of calculation. The expression is:

[0098]

[0099] Among them, the size of the pooling window is p*p, P i,j,n is the eigenvalue at coordinate (i, j) on the nth channel of the feature map after pooling, A i*p+m,j*p+l,n is the value of the corresponding coordinate of the above activated feature map on the nth channel; the pooling operation does not change the number of channels, but can change the image size;

[0100] Repeat the above operations, use the output of the previous layer as the input of the next layer, perform multiple convolution, activation and pooling operations, flatten the final feature map into a one-dimensional vector, and input it into the fully connected layer to integrate the extracted edge features, texture features, local shape features, and semantic features;

[0101] S13, performing heat map regression processing on a series of continuous feature maps obtained in step S12, and extracting key points of the human body from them; using mean square error to define the loss function of the heat map, the formula is expressed as:

[0102]

[0103] Among them, H r , W r H is the height and width of the heat map. t and H f Represent the true heat map value and predicted heat map value of pixel point (i, j) respectively; take the peak point in the heat map as the position of the key point of the human body, reproject the peak coordinate of the heat map back to the original coordinate space of the image, and then get the specific position of the key point of the human body. The formula is expressed as:

[0104]

[0105] Among them, (x, y) is the coordinate of the key point of the human body in the original image, W I , H I is the width and height of the input image, (i peak ,j peak ) represents the coordinates of the peak in the heat map;

[0106] S14, the set of human body key points extracted is recorded as Q = {q 1 ,q 2 ,q 3 ...q n}; The human body key points include head, shoulders, elbows, wrists, knees, ankles; for all the key points of the human body in two consecutive video frames, i To calculate the motion speed and acceleration, the formula is:

[0107]

[0108] Among them, (x 1 ,y 1 ), (x 2 ,y 2 ) are the key points q i The coordinates in two consecutive video frames, Δt is the time interval between the two video frames, v 1 、v 2 is the speed in different video frames, and a is the acceleration;

[0109] S15. Set the speed threshold and acceleration threshold. When the speed and acceleration of 1 / 3 of the key points exceed the threshold, the central processor control system will issue an alarm. The threshold judgment expression is as follows:

[0110]

[0111] Among them, v threshold is the speed threshold, a threshold is the acceleration threshold; when D=1, it is considered that the old man falls, and the robot is controlled to issue an alarm and notify the terminal; when D=0, no response is made.

[0112] In addition, taking into account the differences in different parts of the human body and the daily habits of the elderly, the threshold of each key point will be different. For example, the daily movements of the human head and neck are relatively stable, so the threshold will be relatively small, while the daily activities of the limbs are large, so a larger threshold will be set. The threshold setting can be set according to the daily habits and physical condition of the elderly.

[0113] In this embodiment, the sensor module first receives and records the external image, and then the data processing module performs preprocessing operations on the image, such as removing noise, adjusting the size, etc., and then identifies the key points of the human body based on the mediaPipe and builds a human posture model, and then analyzes the human posture and body movements. Once the elderly fall, the buzzer alarm will sound an alarm, and the information will be remotely transmitted to the children's mobile phones through the network and communication module to notify the children.

[0114] As a preferred embodiment, the method for the daily care module to realize its function specifically includes:

[0115] S21. Based on the target detection algorithm yolov8, the recognition and positioning of waste paper, fruit peels, bottles and cans is realized. First, the yolov8 model is trained using more than 5,000 data sets containing garbage and trash cans to ensure that the algorithm is ready to recognize garbage and trash cans.

[0116] S22. With the help of the monocular camera in the sensor module, the current ground environment in the home is recorded in real time. When garbage appears in the image captured by the camera, the trained yolov8 model will accurately identify the garbage, locate the location of the garbage, and then transmit this information to the control terminal in time.

[0117] S23. Combine the deep learning model algorithm with the robot operating system ROS. After receiving the garbage location information transmitted by the terminal, MoveIt plans the motion trajectory of the robotic arm's grasping without collision according to the current relative position of the robotic arm and the garbage and the angles of each joint of the robotic arm. Compared with traditional robotic arm planning, MoveIt can plan the optimal path in a complex environment according to the current state of the robotic arm, and can perform collision detection, thereby ensuring the efficiency and safety of the grasping process.

[0118] S24, ROS receives the path sequence information planned by MoveIt and converts this information into a control signal to transmit to the motor, thereby providing power for the movement of the robot arm to remove the garbage. When a trash can is identified, the robot arm is controlled to discard the garbage.

[0119] In this embodiment, when the sensor module captures the image of obstacles or garbage, it recognizes the garbage and locates the area of ​​the garbage based on the yolov8 model. When the central processor receives the recognition information, it uses MoveIt to realize the optimal planning of the robot arm's motion path, and then ROS transmits the control signal to the drive module, so that the drive module drives the robot arm to perform grabbing and discarding actions according to the planned path.

[0120] As a preferred implementation, the method for the human-computer interaction module to realize its function specifically includes:

[0121] S31, the central processing unit controls the sensor module to collect the voice information of the elderly;

[0122] S32, preprocessing the collected voice information in a data processing module, including noise removal and signal enhancement;

[0123] S33. Convert the elderly’s voice signals into a language that can be executed by the machine itself, match the voice features with the trained model based on deep learning, and complete human-computer interaction through the voice module.

[0124] In this embodiment, a simple human-computer interaction function can be implemented based on the trained language model, satisfying the basic needs of the elderly who want to be accompanied by someone, and providing the elderly with certain psychological comfort.

[0125] As a preferred implementation, the method for the stability module to realize its function specifically includes:

[0126] S41, extracting ORB features; the ORB features include FAST corner points and BRIEF descriptors; the specific extraction process is:

[0127] S411, determine FAST corner points:

[0128] The camera is used to collect images. For a pixel point p in the image, there are N pixels in a circle with a radius of r = 3 pixels around it, with it as the center. The brightness difference between the pixel point p and the pixels in its area is calculated and compared with the set threshold. The formula is expressed as:

[0129] |I p -I i |>threshold;

[0130] Among them, I p Represents the brightness value of pixel p, I i Indicates the brightness value of any point i among the N pixels in the area of ​​pixel p; threshold is the set threshold; if the brightness difference between p and a certain number of pixels exceeds the threshold, p is considered to be a FAST corner point, that is, a feature point;

[0131] In this embodiment, the so-called "certain number" is set according to the size of the home space and the placement of items. For example, if the home space is large and there are many furniture items, the certain number can be defined as 55%-60% of the number of pixels in the neighborhood, while for a smaller home environment, it can be defined as about 45%.

[0132] S412, obtain the BRIEF descriptor:

[0133] When a pixel point p is considered as a feature point, its direction is used as a reference to define an n*n neighborhood, in which n pairs of pixels are randomly selected, and the following test function is defined:

[0134]

[0135] Among them, I a,m ,Ib,m Indicates the brightness value of the mth pair of pixels a and b, where m is 1 to n. T is the set threshold value, which is related to the image brightness distribution and noise level. It can be determined according to the actual situation, and is generally between 10 and 40.

[0136] Arrange the results of the test function from high to low, and the binary number formed is the BRIEF descriptor of the feature point p;

[0137] S42, introducing the Hamming distance, performing feature matching on two consecutive image frames to determine the relative motion between the two adjacent image frames;

[0138] The Hamming distance formula is expressed as:

[0139]

[0140] Where L is the Hamming distance between two consecutive image frames; f 1 (i) f 2 (i) represents the i-th digit of the descriptor of two feature points respectively; Represents XOR operation; the smaller the Hamming distance L is, the more similar the two descriptors are, that is, the more the two feature points match;

[0141] For any pair of matching feature points p 1 and p 2 , satisfying the epipolar geometry constraint, the formula is expressed as:

[0142]

[0143] Among them, E is composed of the rotation matrix R and the translation matrix t, which is a 3*3 matrix, P 2 , P 1 The feature point p 2 、p 1 The two-dimensional coordinate matrix is ​​expanded to a 3*1 homogeneous coordinate matrix.

[0144] Substitute the eight pairs of matched feature points into the epipolar geometry constraint equation, use the eight-point method to solve E, and decompose E to obtain the rotation matrix R and translation matrix t; therefore, through the continuous rotation matrix R and translation matrix t, the position and posture of the camera at different times can be determined, and the movement of the camera can be estimated;

[0145] S43, selecting a suitable key frame: selecting a video frame containing the most new feature points as a key frame to ensure that different environmental states can be better represented. For each selected key frame, a corresponding local map is constructed. The local map is centered on the key frame, based on the feature points in the key frame, and combined with the changes in camera parameters and posture, to determine other feature points within the set distance and angle range of the feature point, and then determine the map points by triangulation method. The local map includes these feature points and map points;

[0146] The local BA optimization is performed on the local map by minimizing the reprojection error. The camera pose and map point positions can be adjusted through local BA optimization so that the map points projected onto the image are as consistent as possible with the actual observed feature points. The objective function of local BA optimization is:

[0147]

[0148] Among them, α is the camera projection function, which can transform the coordinates of points on the three-dimensional plane to the two-dimensional plane, T z is the camera pose of the zth key frame, P q represents the three-dimensional coordinates of the qth map point, p zq represents the two-dimensional coordinates of the map point q observed in the zth key frame;

[0149] S44, gradually obtain the global map by continuously constructing local maps; and perform loop detection to determine whether the camera has returned to the location it has been to; if a closed loop is detected, perform BA optimization on the global map to eliminate the accumulated error;

[0150] More specifically, step S44 includes:

[0151] S441, calculating the average Hamming distance between each descriptor in the current key frame and the descriptors with the highest response strength in the historical key frames, if the obtained average Hamming distance exceeds the set threshold, it is considered that a closed loop has occurred. The setting of the threshold is related to the complexity of the home environment;

[0152] S442. When a closed loop is detected, BA optimization is performed on the global map, and the key frames at both ends of the closed loop and all key frames and map points between them are included in the optimization range, and the camera pose and the position of the map points are re-optimized to minimize the reprojection error.

[0153] In this embodiment, the sensor module first obtains continuous image frames of the current environment, the data processing module extracts the ORB features of the image, and the visual ORB-SLAM2 technology is used to build a home map. The control module enables the robot to move back and forth in the home until the robot draws a map of the entire home. After that, the robot can perform positioning navigation and path planning according to the drawn map to ensure that it can reach the designated location accurately and without collision during the work process.

[0154] To sum up, the home intelligent elderly care robot system and its control method proposed in the present invention can realize the multifunctional integration of detecting elderly people's falls, daily garbage removal, real-time monitoring and voice emotional interaction, which greatly improves the working efficiency of the robot while taking into account stability, and can better provide services for the elderly.

[0155] In this specification, the terms "one embodiment", "some embodiments", "examples", "specific examples" or "some examples" refer to at least one embodiment or example described in combination with specific features, structures, materials or characteristics. These described specific features, structures, materials or characteristics can be combined in an appropriate manner in one or more embodiments or examples. In addition, the technician can combine and combine different embodiments or examples and their features described in this specification without contradiction.

[0156] The logic and / or steps shown in the flowchart or described in other ways can be regarded as a sequence of executable instructions for implementing logical functions. These instructions can be implemented in any computer-readable medium for use by instruction execution systems, devices or equipment. These systems, devices or equipment include processor systems or other systems capable of receiving and executing instructions.

[0157] The above embodiments have described the principles and implementations of the present invention in detail, and have used specific examples to illustrate the working principles thereof. These examples are only used to help understand the method of the present invention and its core concept. Meanwhile, according to the concept of the present invention, the actual implementation methods and application scope may vary. Therefore, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A multifunctional integrated home intelligent elderly care robot system, characterized in that: The invention comprises a central processing unit arranged on the home-based elderly care robot, and a sensor module, an interface module, a network communication module, a voice module, a drive module and a data processing module connected to the central processing unit; the central processing unit is internally provided with four functional modules, including: a safety protection module, a daily care module, a human-computer interaction module and a stability module; The safety protection module builds a body posture model of the elderly based on the mediapipe technology to analyze and monitor the elderly's safety status and transmit the monitoring data to the terminal; when the elderly falls, the robot is controlled to sound an alarm and notify the terminal; the terminal includes the children's device; The daily care module is used to control the robot to cruise along a specific route, integrate yolov8 and MoveIt to identify garbage in the indoor environment, and control the robot to clean; The human-computer interaction module is used to communicate with the elderly through voice and emotion; The stability module performs real-time positioning and home mapping based on ORB-SLAM2 technology to achieve robot path planning and automatic obstacle avoidance to ensure working stability.

2. According to claim 1, a multifunctional integrated home intelligent elderly care robot system is characterized in that: The sensor module and the data processing module are unidirectionally connected to the central processing unit to input information to the central processing unit; The voice module and the driving module are unidirectionally connected to the central processor to receive information output by the central processor; The interface module, the network and communication module are bidirectionally connected with the central processing unit to exchange information.

3. The control method of a multifunctional integrated home intelligent elderly care robot system according to claim 1, characterized in that: The control method includes a method for the security protection module to realize its function, which specifically includes the following steps: S11. Record continuous images containing the elderly through the sensor module, input the continuous video frames into the convolutional neural network CNN, and obtain the feature map after convolution. The size calculation formula of the feature map is expressed as: Among them, W 、 H is the width and height of the first layer feature map obtained, the number of channels of the feature map is the number of convolution kernels, and W I , H I is the width and height of the input image, W K , H K , P K , S K They are the width, height, padding, and stride of the convolution kernel K respectively; S12. Introduce the ReLU activation function to activate the convolutional feature map to obtain the feature map A, which enhances the network expression ability. The formula is expressed as: f(a)=max(0,a); Among them, a represents the output value of a certain position of the feature map after the convolution layer; Perform the maximum pooling operation on the activated feature map to reduce the amount of calculation. The expression is: Among them, the size of the pooling window is p*p, P i,j,n is the eigenvalue at coordinate (i, j) on the nth channel of the feature map after pooling, is the value of the corresponding coordinate of the above activated feature map on the nth channel; Repeat the above operations, use the output of the previous layer as the input of the next layer, perform multiple convolution, activation and pooling operations, flatten the final feature map into a one-dimensional vector, and input it into the fully connected layer to integrate the extracted edge features, texture features, local shape features, and semantic features; S13, performing heat map regression processing on a series of continuous feature maps obtained in step S12, and extracting key points of the human body from them; using mean square error to define the loss function of the heat map, the formula is expressed as: Among them, H r , W r H is the height and width of the heat map. t and H f Represent the true heat map value and predicted heat map value of pixel point (i, j) respectively; take the peak point in the heat map as the position of the key point of the human body, reproject the peak coordinate of the heat map back to the original coordinate space of the image, and then get the specific position of the key point of the human body. The formula is expressed as: Among them, (x, y) is the coordinate of the key point of the human body in the original image, W I , H I is the width and height of the input image, (i peak ,j peak ) represents the coordinates of the peak in the heat map; S14, the set of human key points extracted is recorded as Q = {q1, q2, q3...q n }; The human body key points include head, shoulders, elbows, wrists, knees, ankles; for all the key points of the human body in two consecutive video frames, i To calculate the motion speed and acceleration, the formula is: Among them, (x1, y1), (x2, y2) are the key points q i The coordinates in two consecutive video frames, Δt is the time interval between the two video frames, v1 and v2 are the velocities in different video frames, and a is the acceleration; S15. Set the speed threshold and acceleration threshold. When the speed and acceleration of 1 / 3 of the key points exceed the threshold, the central processor control system will issue an alarm. The threshold judgment expression is as follows: Among them, v threshold is the speed threshold, a threshold is the acceleration threshold; when D=1, it is considered that the old man falls, and the robot is controlled to issue an alarm and notify the terminal; when D=0, no response is made.

4. The control method of a multifunctional integrated home intelligent elderly care robot system according to claim 1, characterized in that: The control method includes a method for the daily care module to realize its function, which specifically includes the following steps: S21. Based on the target detection algorithm yolov8, the garbage is identified and located, and the yolov8 model is trained using a data set containing garbage and trash cans; S22, using the sensor module to record the current home ground environment in real time, when garbage appears in the captured image, the trained yolov8 model is used to accurately identify the garbage, and the location of the garbage is regionally located, and then the information is transmitted to the central processor in a timely manner; S23, combining the deep learning model algorithm with the robot operating system ROS. After receiving the garbage location information transmitted by the central processor, MoveIt performs collision-free planning of the robot's robotic arm's grasping motion trajectory based on the current relative position of the robot's robotic arm and the garbage and the angles of each joint of the robotic arm. S24, ROS receives the path sequence information planned by MoveIt, and converts this information into a control signal and transmits it to the motor, thereby providing power for the movement of the robotic arm to remove garbage; when a trash can is identified, the drive module controls the robotic arm to discard the garbage.

5. The control method of a multifunctional integrated home intelligent elderly care robot system according to claim 1, characterized in that: The control method includes a method for the human-computer interaction module to realize its function, which specifically includes the following steps: S31, the central processing unit controls the sensor module to collect the voice information of the elderly; S32, preprocessing the collected voice information in a data processing module, including noise removal and signal enhancement; S33. Convert the elderly’s voice signals into a language that can be executed by the machine itself, match the voice features with the trained model based on deep learning, and complete human-computer interaction through the voice module.

6. The control method of a multifunctional integrated home intelligent elderly care robot system according to claim 1, characterized in that: The control method includes a method for the stability module to realize its function, which specifically includes the following steps: S41, extracting ORB features; the ORB features include FAST corner points and BRIEF descriptors; the specific extraction process is: S411, determine FAST corner points: The camera is used to collect images. For a pixel point p in the image, there are N pixels in a circle with a radius of r = 3 pixels around it, with it as the center. The brightness difference between the pixel point p and the pixels in its area is calculated and compared with the set threshold. The formula is expressed as: |I p -I i |>threshold; Among them, I p Represents the brightness value of pixel p, I i Indicates the brightness value of any point i among the N pixels in the area of ​​pixel p; threshold is the set threshold; if the brightness difference between p and a certain number of pixels exceeds the threshold, p is considered to be a FAST corner point, that is, a feature point; S412, obtain the BRIEF descriptor: When a pixel point p is considered as a feature point, its direction is used as a reference to define an n*n neighborhood, in which n pairs of pixels are randomly selected, and the following test function is defined: Among them, I a,m ,I b,m Indicates the brightness value of the mth pair of pixels a and b, where m is 1 to n; T is the set threshold value, which is related to the image brightness distribution and noise level. It can be determined according to the actual situation, and is generally between 10 and 40. Arrange the results of the test function in order from high to low, and the binary number formed is the BRIEF descriptor of the feature point p; S42, introducing the Hamming distance, performing feature matching on two consecutive image frames to determine the relative motion between the two adjacent image frames; The Hamming distance formula is expressed as: Where L is the Hamming distance between two consecutive image frames; f1(i) and f2(i) represent the i-th digit of the descriptors of the two feature points respectively; Represents XOR operation; the smaller the Hamming distance L is, the more similar the two descriptors are, that is, the more the two feature points match; For any pair of matching feature points p1 and p2, the epipolar geometry constraint is satisfied, and the formula is expressed as: Among them, E is composed of the rotation matrix R and the translation matrix t, which is a 3*3 matrix. P2 and P1 are the two-dimensional coordinate matrices of the feature points p2 and p1 expanded into 3*1 homogeneous coordinate matrices; Substitute the eight pairs of matched feature points into the epipolar geometry constraint equation, use the eight-point method to solve E, and decompose E to obtain the rotation matrix R and translation matrix t; therefore, through the continuous rotation matrix R and translation matrix t, the position and posture of the camera at different times can be determined, and the movement of the camera can be estimated; S43, selecting a suitable key frame: selecting a video frame containing the most new feature points as a key frame to ensure that different environmental states can be better represented. For each selected key frame, a corresponding local map is constructed. The local map is centered on the key frame, based on the feature points in the key frame, and combined with the changes in camera parameters and posture, to determine other feature points within the set distance and angle range of the feature point, and then determine the map points through a triangulation method. The local map includes these feature points and map points; The local BA optimization is performed on the local map by minimizing the reprojection error. The camera pose and map point positions can be adjusted through local BA optimization so that the map points projected onto the image are as consistent as possible with the actual observed feature points. The objective function of local BA optimization is: Among them, α is the camera projection function, which can transform the coordinates of points on the three-dimensional plane to the two-dimensional plane, T z is the camera pose of the zth key frame, P q represents the three-dimensional coordinates of the qth map point, p zq represents the two-dimensional coordinates of the map point q observed in the zth key frame; S44. By continuously constructing local maps, a global map is gradually obtained; and loop detection is performed to determine whether the camera has returned to a location it has been to before; if a closed loop is detected, BA optimization is performed on the global map to eliminate the accumulated error.

7. The control method of a multifunctional integrated home intelligent elderly care robot system according to claim 6, characterized in that: Step S44 specifically includes: S441, calculating the average Hamming distance between each descriptor in the current key frame and several descriptors with the highest response strength in the historical key frames, and if the obtained average Hamming distance exceeds a set threshold, it is considered that a closed loop has occurred; the setting of the threshold is related to the complexity of the home environment; S442. When a closed loop is detected, BA optimization is performed on the global map, and the key frames at both ends of the closed loop and all key frames and map points between them are included in the optimization range, and the camera pose and the position of the map points are re-optimized to minimize the reprojection error.

Images