An intelligent companion robot for smart medical care and elderly care

Abstract

The present invention provides an intelligent accompanying robot for smart medical care and elderly care, comprising an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring accompanying unit; the intelligent control center controls the visual acquisition device to collect video data of the object to be accompanied; the acquisition unit obtains the target point cloud of the object to be accompanied in the video data; the reconstruction processing unit performs reconstruction processing on the target point cloud to obtain a reconstructed point cloud model; the estimation processing unit performs target frame size estimation processing based on the point cloud model to obtain a target frame of the target size; the trajectory optimization processing unit performs trajectory optimization processing on the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory; the intelligent monitoring accompanying unit performs accompanying tracking on the object to be accompanied based on the optimized target trajectory, and performs social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied. The present invention realizes accurate and effective accompanying.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent accompanying robot for smart medical care and elderly care. Background Art

[0002] With the aging of society, the number of elderly people is increasing, but young people spend less and less time with the elderly. Therefore, intelligent robots have become the mainstream trend. In the process of intelligent care, it is necessary to accurately identify the object to be accompanied and track and accompany in real time. Therefore, higher requirements are put forward for high-precision target detection and tracking optimization technology. However, the existing technology has high limitations in the trajectory optimization scheme of the intelligent care robot for the object to be accompanied, resulting in low accuracy and poor reliability of target tracking and labeling, which makes it impossible to accurately identify the object to be accompanied and track and accompany in real time, resulting in poor effect of intelligent care. Summary of the invention

[0003] In view of the technical problems existing in the prior art, the present invention provides an intelligent accompanying robot for smart medical care and elderly care, so as to improve the effect of intelligent accompanying and realize accurate and effective accompanying for the accompanying object.

[0004] In a first aspect, the present invention provides an intelligent accompanying robot for smart medical care and elderly care, the intelligent accompanying robot comprising an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center is respectively connected to the visual acquisition device, the acquisition unit, the reconstruction processing unit, the estimation processing unit, the trajectory optimization processing unit and the intelligent monitoring and accompanying unit to store and manage the data of each unit;

[0005] The intelligent control center is used to respond to the intelligent accompanying instruction for the object to be accompanied, and control the visual acquisition device to collect video data of the object to be accompanied;

[0006] The acquisition unit is used to acquire a target point cloud of the object to be accompanied in a data frame of the video data;

[0007] The reconstruction processing unit is used to perform reconstruction processing on the target point cloud to obtain a reconstructed point cloud model;

[0008] The estimation processing unit is used to perform target frame size estimation processing based on the point cloud model to obtain a target frame of a target size;

[0009] The trajectory optimization processing unit is used to perform trajectory optimization processing on the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory;

[0010] The intelligent monitoring and accompanying unit is used to carry out accompanying and tracking of the object to be accompanied based on the optimized target trajectory, and to carry out social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied.

[0011] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, the target point cloud is reconstructed to obtain a reconstructed point cloud model, including:

[0012] Constructing a local coordinate system of the object to be accompanied based on different observation positions of the object to be accompanied;

[0013] Based on the local coordinate system of the object to be accompanied, the target point clouds of the object to be accompanied in continuous data frames are superimposed to obtain a reconstructed point cloud model.

[0014] According to an embodiment of the present invention, the intelligent accompanying robot for smart medical care and elderly care constructs a local coordinate system of the object to be accompanied based on different observation positions of the object to be accompanied, including:

[0015] In the case where an edge of the object to be escorted is observed by the visual acquisition device, a center point of the edge is used as the origin of the local coordinate system to construct a first local coordinate system of the object to be escorted;

[0016] In the case where two edges of the object to be escorted are observed by the visual acquisition device, an intersection point formed by the two edges is used as the origin of the local coordinate system to construct a second local coordinate system of the object to be escorted;

[0017] When the three edges of the object to be escorted are observed by the visual acquisition device, the center point of the longest edge among the three edges is used as the origin of the local coordinate system to construct a third local coordinate system of the object to be escorted.

[0018] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, the target point cloud of the object to be accompanied in continuous data frames is superimposed based on the local coordinate system of the object to be accompanied to obtain a reconstructed point cloud model, including:

[0019] Based on the local coordinate system of the first object to be accompanied, superimposing target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model;

[0020] Based on the local coordinate system of the second object to be accompanied, superimposing the target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model;

[0021] Based on the third local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model.

[0022] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, a target frame size estimation process is performed based on the point cloud model to obtain a target frame of a target size, including:

[0023] When an edge of the object to be escorted is observed by the visual acquisition device, obtaining a length observation value corresponding to the edge;

[0024] Counting the lengths of the other unobserved edges in all target frames collected by the visual acquisition device within a preset time period to obtain a statistical result;

[0025] According to the statistical result, the median of the length of another edge that is not observed is used as a length estimation value; according to the length observation value and the length estimation value, a target frame of the target size is obtained;

[0026] The target frame is a rectangular frame composed of edges of different lengths.

[0027] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, a target frame size estimation process is performed based on the point cloud model to obtain a target frame of a target size, including:

[0028] When two edges of the object to be escorted are observed by the visual acquisition device, a maximum absolute value in the x-axis direction and a maximum absolute value in the y-axis direction in the local coordinate system of the second object to be escorted are obtained;

[0029] Obtain the longest value of the edge lengths of all target frames in the x-axis direction and the longest value of the other edge lengths of all target frames in the y-axis direction;

[0030] Determine a first distance difference between a maximum absolute value in the x-axis direction of the local coordinate system of the second object to be accompanied and a longest value of the edge lengths of all target frames in the x-axis direction;

[0031] Determine a second distance difference between the maximum absolute value in the y-axis direction and the longest value of another edge length of all target frames in the y-axis direction;

[0032] When both the first distance difference and the second distance difference are less than a preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation condition, and the maximum absolute value in the x-axis direction and the maximum absolute value in the y-axis direction in the local coordinate system of the second object to be accompanied are respectively used as the length estimation values ​​of the two edges of the target frame, to obtain a target frame of the target size;

[0033] In the case that the first distance difference and the second distance difference are greater than or equal to the preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation condition, and the longest value of the edge length of all target frames in the x-axis direction and the longest value of the other edge length of all target frames in the y-axis direction are respectively used as the length estimation values ​​of the two edges of the target frame, to obtain the target frame of the target size;

[0034] The target frame is a rectangular frame consisting of edges of different lengths.

[0035] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, a target frame size estimation process is performed based on the point cloud model to obtain a target frame of a target size, including:

[0036] When the three edges of the object to be escorted are observed by the visual acquisition device, the maximum absolute value and the minimum absolute value of the x-axis direction in the local coordinate system of the third object to be escorted are obtained, and a first difference between the maximum absolute value and the minimum absolute value in the x-axis direction is determined;

[0037] Obtaining the maximum absolute value and the minimum absolute value of the y-axis direction in the local coordinate system of the third object to be accompanied, and determining a second difference between the maximum absolute value and the minimum absolute value in the x-axis direction;

[0038] Using the first difference and the second difference as estimated values ​​of the lengths of two edges of the target frame, respectively, to obtain a target frame of the target size;

[0039] The target frame is a rectangular frame consisting of edges of different lengths.

[0040] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, the target point cloud of the object to be accompanied in the data frame of the video data is obtained, including:

[0041] Acquire a data frame of the video data collected by the visual acquisition device, and extract a target point cloud in a target frame corresponding to the object to be accompanied from the data frame;

[0042] The target point cloud is motion compensated to obtain a target point cloud after motion compensation, and the target point cloud after motion compensation is used as the target point cloud of the object to be accompanied in the data frame collected by the visual acquisition device.

[0043] According to the intelligent accompanying robot for smart medical care and elderly care provided by an embodiment of the present invention, the target frame based on the target size performs trajectory optimization processing on the object to be accompanied to obtain the optimized target trajectory, including:

[0044] According to the target size of the target frame, the center point of the target frame in the local coordinate system of the object to be escorted is determined, and the center point of the target frame in the local coordinate system of the object to be escorted is projected back to the global coordinate system of the visual acquisition device to obtain the optimized target trajectory.

[0045] In a second aspect, the present invention further provides an intelligent accompanying method for smart medical care for the elderly, which is applied to the intelligent accompanying robot for smart medical care for the elderly as described in the first aspect, and the method comprises:

[0046] In response to the intelligent accompanying instruction for the object to be accompanied, control the visual acquisition device to collect video data of the object to be accompanied;

[0047] Acquire a target point cloud of the object to be accompanied in a data frame of the video data;

[0048] Reconstructing the target point cloud to obtain a reconstructed point cloud model;

[0049] Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size;

[0050] Performing trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory;

[0051] The object to be accompanied is tracked based on the optimized target trajectory, and social interaction is performed with the object to be accompanied according to the emotional state information of the object to be accompanied.

[0052] In a third aspect, the present invention further provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing any of the above-mentioned intelligent care methods for smart medical care and elderly care.

[0053] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium, characterized in that a computer software program is stored in the storage medium, and when the computer software program is executed by a processor, it implements the intelligent care method for smart medical care and elderly care as described in any of the above-mentioned methods.

[0054] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned intelligent care methods for smart medical care and elderly care.

[0055] The intelligent accompanying robot for smart medical care and elderly care provided by the present invention includes an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center controls the visual acquisition device to collect video data of the object to be accompanied; the acquisition unit acquires the target point cloud of the object to be accompanied in the video data; the reconstruction processing unit reconstructs the target point cloud to obtain a reconstructed point cloud model; the estimation processing unit performs target frame size estimation processing based on the point cloud model to obtain a target frame of the target size; the trajectory optimization processing unit optimizes the trajectory of the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory; the intelligent monitoring and accompanying unit performs accompanying and tracking on the object to be accompanied based on the optimized target trajectory, and performs social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied, thereby effectively improving the accuracy and reliability of target tracking and marking, thereby improving the automatic marking effect of the trajectory of the object to be accompanied, and further being able to accurately identify and track the object to be accompanied in real time, thereby improving the effect of intelligent accompanying, and realizing accurate and effective accompanying of the object to be accompanied. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a structural schematic diagram of the intelligent accompanying robot for smart medical care and elderly care provided by the present invention;

[0057] Figure 2 It is a schematic diagram of the position of the local coordinate system of the first object to be accompanied provided by the present invention;

[0058] Figure 3 is a schematic diagram of the position of the local coordinate system of the second object to be accompanied provided by the present invention;

[0059] Figure 4 is a schematic diagram of the position of the local coordinate system of the third object to be accompanied provided by the present invention;

[0060] Figure 5 It is a flow chart of the intelligent accompanying method for smart medical care and elderly care provided by the present invention;

[0061] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0063] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0065] Reference Figure 1 , Figure 1 It is a structural schematic diagram of the intelligent accompanying robot for smart medical care and elderly care provided by the present invention. The intelligent accompanying robot includes an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center is respectively connected to the visual acquisition device, the acquisition unit, the reconstruction processing unit, the estimation processing unit, the trajectory optimization processing unit and the intelligent monitoring and accompanying unit to store and manage the data of each unit.

[0066] Optionally, the intelligent control center responds to the intelligent accompanying instruction for the person to be accompanied, and controls the visual acquisition device to collect video data of the person to be accompanied.

[0067] Among them, when intelligent escort is needed for a certain user, the intelligent escort instruction and the designated object to be escorted can be input on the display interface of the intelligent escort robot, or the intelligent escort instruction and the designated object to be escorted can be informed to the intelligent escort robot through voice interaction. It should be noted that the embodiment of the present invention is applied to smart medical care and elderly care, so the object to be escorted in the embodiment of the present invention is generally an elderly person. The visual acquisition device can be a depth camera, a point cloud camera, a laser radar, etc. installed on the intelligent escort robot. Therefore, the intelligent control center responds to the intelligent escort instruction for the object to be escorted, and controls the visual acquisition device to collect video data of the object to be escorted.

[0068] Optionally, the acquisition unit acquires a target point cloud of the object to be accompanied in a data frame of the video data.

[0069] In an embodiment of the present invention, the input is the point cloud data of each collected data frame and the target frame of the object to be accompanied in each data frame, so as to extract the target point cloud in the target frame corresponding to the object to be accompanied from the data frame through the algorithm of judging whether the point cloud is within the polygonal bounding box in the prior art.

[0070] Furthermore, motion compensation is performed on the target point cloud to obtain a motion-compensated target point cloud, and the motion-compensated target point cloud is used as the target point cloud of the object to be escorted in the data frame acquired by the visual acquisition device. In the process of performing motion compensation on the target point cloud to obtain the motion-compensated target point cloud, motion compensation technology can be applied to the extracted target point cloud to eliminate point cloud distortion caused by the motion of the object to be escorted.

[0071] Since the object to be escorted is still moving during the point cloud scanning of the visual acquisition device, the shape of the point cloud is distorted. The tracking frame (i.e., the target frame) input by the embodiment of the present invention filters the speed, and can be calculated by multiplying the target speed and the time difference between t0-t1 during the point cloud scanning of the visual acquisition device to obtain the offset of the target point cloud. Then, according to the offset of the target point cloud and the timestamp of each point of the target point cloud, each target point cloud is compensated to the position corresponding to the timestamp of the current data frame, thereby obtaining the target point cloud after motion compensation.

[0072] Optionally, the reconstruction processing unit performs reconstruction processing on the target point cloud to obtain a reconstructed point cloud model.

[0073] In an embodiment of the present invention, a local coordinate system of the object to be cared for can be constructed based on different observation positions of the object to be cared for, and based on the local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model.

[0074] Therefore, it can be understood that after motion compensation, the target point clouds of continuous data frames are integrated using point cloud reconstruction technology to construct a stable and continuous point cloud model (i.e., target 3D model). Therefore, the embodiment of the present invention establishes different local coordinate systems of the object to be accompanied according to the observation situation of the object to be accompanied:

[0075] Specifically, in the process of constructing the local coordinate system of the object to be escorted based on different observation positions of the object to be escorted, when the visual acquisition device observes an edge of the object to be escorted, the center point of the edge is used as the origin of the local coordinate system, the sideline where the edge is located is used as the y-axis, and the x-axis is established through the origin of the local coordinate system and perpendicular to the edge to construct the first local coordinate system of the object to be escorted. In one embodiment, the visual acquisition device only observes one edge of the object to be escorted.

[0076] In this case, the embodiment of the present invention selects the center point of the observable edge as the local coordinate center (i.e., the origin of the local coordinate system). Figure 2 As shown in FIG. 1 , this is the local coordinate system of the first object to be escorted when an edge of the object to be escorted is observed. Ego represents the intelligent escort robot, agent represents the object to be escorted, the arrow represents the x-axis of the local coordinate system, and t1, t2, and t3 represent different moments.

[0077] In the case where the visual acquisition device observes two edges of the object to be escorted, the intersection of the two edges is used as the origin of the local coordinate system, the sideline where one of the two edges is located is used as the y-axis, and the sideline where the other of the two edges is located is used as the x-axis to construct a second local coordinate system of the object to be escorted. In one embodiment, the visual acquisition device only observes the two edges of the object to be escorted.

[0078] For this situation, the embodiment of the present invention selects the intersection of the two observable edges as the target local coordinate center point (i.e., the origin of the local coordinate system), such as Figure 3 As shown, it is the local coordinate system of the second object to be escorted when two edges of the object to be escorted are observed.

[0079] In the case where the visual acquisition device observes three edges of the object to be escorted, the center point of the longest edge among the three edges is used as the origin of the local coordinate system, and the x-axis is established through the origin of the local coordinate system and perpendicular to the longest edge to construct the third local coordinate system of the object to be escorted. In one embodiment, the visual acquisition device observes the three edges of the object to be escorted. For this situation, the embodiment of the present invention selects the edge with the most observations (i.e., the longest edge), as follows Figure 4 The center point of the long side in is used as the origin of the local coordinate system to overlay the target point cloud. Figure 4As shown, it is the local coordinate system of the third object to be escorted when the three edges of the object to be escorted are observed.

[0080] Based on the local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model, including: based on the first local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model; based on the second local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model; based on the third local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model.

[0081] Optionally, the estimation processing unit performs target frame size estimation processing based on the point cloud model to obtain a target frame of the target size.

[0082] In an embodiment of the present invention, when an edge of the object to be escorted is observed by the visual acquisition device, the length observation value corresponding to the edge is obtained; the length of another edge that is not observed in all target frames collected by the visual acquisition device within a preset time period is counted to obtain a statistical result; and the median of the length of another edge that is not observed is used as a length estimation value according to the statistical result; and a target frame of the target size is obtained according to the length observation value and the length estimation value. That is, the visual acquisition device only observes one edge of the object to be escorted, so the other edge can only be estimated. The embodiment of the present invention selects the median of this edge (i.e., an edge of the object to be escorted that is not observed) of all tracking frames in all collected data frames as the best estimate (i.e., as the length estimation value of the edge of the object to be escorted that is not observed), and excludes outliers that are too large or too small.

[0083] In addition, when the two edges of the object to be escorted are observed based on the visual acquisition device, the maximum absolute value in the x-axis direction and the maximum absolute value in the y-axis direction in the local coordinate system of the second object to be escorted are obtained; and the longest value of the edge length of all target frames in the x-axis direction and the longest value of the other edge length of all target frames in the y-axis direction are obtained; a first distance difference between the maximum absolute value in the x-axis direction and the longest value of the edge length of all target frames in the x-axis direction in the local coordinate system of the second object to be escorted is determined; a second distance difference between the maximum absolute value in the y-axis direction and the longest value of the other edge length of all target frames in the y-axis direction is determined; when both the first distance difference and the second distance difference are less than a preset distance threshold In the case where the first distance difference and the second distance difference are greater than or equal to the preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation conditions, and the maximum absolute value of the edge length of all target frames in the x-axis direction and the maximum absolute value of the y-axis direction in the local coordinate system of the second object to be accompanied are respectively used as the estimated lengths of the two edges of the target frame, to obtain the target frame of the target size; in the case where the first distance difference and the second distance difference are greater than or equal to the preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation conditions, and the longest value of the edge length of all target frames in the x-axis direction and the longest value of the other edge length of all target frames in the y-axis direction are respectively used as the estimated lengths of the two edges of the target frame, to obtain the target frame of the target size.That is, although the two sides of the object to be escorted are observed, there is still an incomplete observation. Therefore, the embodiment of the present invention calculates the maximum absolute value coordinates max_x (i.e., the maximum absolute value in the x-axis direction in the local coordinate system of the second object to be escorted) and max_y (i.e., the maximum absolute value in the y-axis direction) in the x-axis direction and the y-axis direction in the local coordinate system of the second object to be escorted, as well as the longest value max_l (i.e., obtaining the longest value of the edge length of all target frames in the x-axis direction) and max_w (i.e., the longest value of the other edge length of all target frames in the y-axis direction) in the two directions corresponding to the x-axis direction and the y-axis direction for the reconstructed point cloud model. When the point cloud model is at a distance from the maximum coordinates and the lengths of the two sides in the x-axis direction and the y-axis direction When the deviation values ​​dx (i.e., the first distance difference) and dy (i.e., the second distance difference) are less than the preset distance threshold, the point cloud model observation is considered to be complete, and max_x and max_y are used as the optimal estimated values ​​of the target frame (i.e., the maximum absolute value in the x-axis direction and the maximum absolute value in the y-axis direction in the local coordinate system of the second object to be accompanied are respectively used as the length estimates of the two edges of the target frame to obtain the target frame of the target size); otherwise, it is considered that the reconstructed point cloud model is still incompletely observed, and max_l and max_w are used as the length estimates of the target frame (i.e., the longest value of the edge lengths of all target frames in the x-axis direction and the longest value of the other edge lengths of all target frames in the y-axis direction are respectively used as the length estimates of the two edges of the target frame to obtain the target frame of the target size).

[0084] In addition, when the three edges of the object to be escorted are observed by the visual acquisition device, the maximum absolute value and the minimum absolute value of the x-axis direction in the local coordinate system of the third object to be escorted are obtained, and the first difference between the maximum absolute value and the minimum absolute value in the x-axis direction is determined; and the maximum absolute value and the minimum absolute value of the y-axis direction in the local coordinate system of the third object to be escorted are obtained, and the second difference between the maximum absolute value and the minimum absolute value in the x-axis direction is determined; the first difference and the second difference are respectively used as the estimated lengths of the two edges of the target frame to obtain the target frame of the target size. The target frame is as follows: Figure 2 The rectangular frame shown includes edges of different lengths. That is, the embodiment of the present invention observes the three edges of the object to be escorted, and it can be considered that the point cloud model observation is relatively complete. The reconstructed point cloud model is used to calculate the maximum and minimum values ​​of the x-axis direction and the y-axis direction in the local coordinate system of the object to be escorted to obtain the estimated length of the side length, the first difference max_l = max_x-min_x, and the second difference max_w = max_y-min_y (that is, the first difference and the second difference are used as the length estimates of the two edges of the target frame, respectively, to obtain the target frame of the target size).

[0085] Optionally, the trajectory optimization processing unit performs trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory.

[0086] In an embodiment of the present invention, the center point of the target frame in the local coordinate system of the object to be escorted can be determined according to the target size of the target frame, and the center point of the target frame in the local coordinate system of the object to be escorted can be projected back to the global coordinate system of the visual acquisition device to obtain the optimized target trajectory. Therefore, it can be understood that in the process of optimizing the trajectory of the target frame, the trajectory of the target frame is adjusted to ensure that the position and size of the target frame maintain the best consistency and accuracy during the entire tracking process. In the size optimization step, according to the target frame of the optimized size (i.e., the target frame of the target size), the center point of the target frame in the local coordinate system of the object to be escorted is obtained, and through the global posture of the object to be escorted at different times, the center point of the object to be escorted in the local coordinate system of the object to be escorted can be projected back to the global coordinate system to obtain the optimized target trajectory. Among them, the local coordinate system of the object to be escorted can be the first local coordinate system of the object to be escorted, the second local coordinate system of the object to be escorted, or the third local coordinate system of the object to be escorted. The global coordinate system is a coordinate system with the center of the intelligent escort robot on which the visual acquisition device is installed as the origin.

[0087] The embodiment of the present invention adopts a method of jointly optimizing the distribution of the object to be accompanied and the point cloud, which can accurately track the object to be accompanied. By using the target motion information to perform point cloud compensation, the ghosting phenomenon when the target point cloud is superimposed is reduced, and the accuracy of the tracking frame is improved. The target frame size is estimated based on comprehensive observations to ensure the accuracy of the tracking frame size. This method can accurately estimate the size of the tracking frame and provide a more reliable basis for subsequent target identification. In addition, the consistency of the tracking frame size is maintained. After determining the size of the target frame, the center position of other tracking frames will be dynamically adjusted according to the degree of fit between the target frame and the point cloud model to ensure that the size of the tracking frame remains consistent throughout the tracking process, thereby ensuring the consistency of the tracking frame size, thereby improving the accuracy and reliability of target tracking and labeling.

[0088] Optionally, the intelligent monitoring and accompanying unit tracks the person to be accompanied based on the optimized target trajectory, and performs social interaction with the person to be accompanied according to the emotional state information of the person to be accompanied.

[0089] Specifically, the object to be accompanied is tracked according to the optimized target trajectory, and the emotional state information of the object to be accompanied is obtained, and social interaction is performed with the object to be accompanied according to the emotional state information of the object to be accompanied. In one embodiment, the face of the object to be accompanied is in a smiling state, that is, it can be determined that the object to be accompanied is currently in a happy state, so it is only necessary to follow the object to be accompanied. If the face of the object to be accompanied is in a frowning state, that is, it can be determined that the object to be accompanied is currently in an unhappy state. At this time, the intelligent accompanying robot needs to interact with the object to be accompanied, obtain the reason why the object to be accompanied is unhappy, and give guidance measures in time according to the reason for unhappiness. It should be noted that how to determine the frowning state, the smiling state, etc. in the embodiment of the present invention, and how to determine the happy state according to the smiling state and the unhappy state according to the frowning state can be obtained by a variety of methods, which will not be repeated again.

[0090] The intelligent accompanying robot for smart medical care and elderly care provided by the embodiment of the present invention includes an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center controls the visual acquisition device to collect video data of the object to be accompanied; the acquisition unit obtains the target point cloud of the object to be accompanied in the video data; the reconstruction processing unit reconstructs the target point cloud to obtain a reconstructed point cloud model; the estimation processing unit performs target frame size estimation processing based on the point cloud model to obtain a target frame of the target size; the trajectory optimization processing unit optimizes the trajectory of the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory; the intelligent monitoring and accompanying unit performs accompanying tracking on the object to be accompanied based on the optimized target trajectory, and performs social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied, thereby effectively improving the accuracy and reliability of target tracking and labeling, thereby improving the automatic labeling effect of the trajectory of the object to be accompanied, and further being able to accurately identify and track the object to be accompanied in real time, thereby improving the effect of intelligent accompanying, and realizing accurate and effective accompanying of the object to be accompanied.

[0091] Reference Figure 5 , Figure 5 The figure is a flow chart of the intelligent accompanying method for smart medical care for the elderly provided by the present invention. The intelligent accompanying method for smart medical care for the elderly according to the embodiment of the present invention is applied to the intelligent accompanying robot for smart medical care for the elderly.

[0092] The intelligent accompanying robot includes an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center is respectively connected to the visual acquisition device, the acquisition unit, the reconstruction processing unit, the estimation processing unit, the trajectory optimization processing unit and the intelligent monitoring and accompanying unit to store and manage the data of each unit.

[0093] The intelligent accompanying method for smart medical care and elderly care of the embodiment of the present invention includes:

[0094] Step 10, responding to the intelligent accompanying instruction for the object to be accompanied, controlling the visual acquisition device to collect video data of the object to be accompanied;

[0095] Step 20, obtaining a target point cloud of the object to be accompanied in a data frame of the video data;

[0096] Step 30, reconstructing the target point cloud to obtain a reconstructed point cloud model;

[0097] Step 40, performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size;

[0098] Step 50, performing trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory;

[0099] Step 60: tracking the person to be accompanied based on the optimized target trajectory, and performing social interaction with the person to be accompanied according to the emotional state information of the person to be accompanied.

[0100] The embodiment of the present invention responds to the intelligent accompanying instruction for the object to be accompanied, controls the visual acquisition device to collect video data of the object to be accompanied; obtains the target point cloud of the object to be accompanied in the data frame of the video data; reconstructs the target point cloud to obtain a reconstructed point cloud model; estimates the size of the target frame based on the point cloud model to obtain a target frame of the target size; optimizes the trajectory of the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory; tracks the object to be accompanied based on the optimized target trajectory, and interacts socially with the object to be accompanied according to the emotional state information of the object to be accompanied. Therefore, the accuracy and reliability of target tracking and labeling can be effectively improved, thereby improving the automatic labeling effect of the trajectory of the object to be accompanied, and further can accurately identify and track the object to be accompanied in real time, thereby improving the effect of intelligent accompanying and realizing accurate and effective accompanying of the object to be accompanied.

[0101] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the intelligent accompanying method for smart medical care and elderly care, and the method includes:

[0102] In response to the intelligent accompanying instruction for the object to be accompanied, control the visual acquisition device to collect video data of the object to be accompanied;

[0103] Acquire a target point cloud of the object to be accompanied in a data frame of the video data;

[0104] Reconstructing the target point cloud to obtain a reconstructed point cloud model;

[0105] Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size;

[0106] Performing trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory;

[0107] The object to be accompanied is tracked based on the optimized target trajectory, and social interaction is performed with the object to be accompanied according to the emotional state information of the object to be accompanied.

[0108] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0109] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the intelligent accompanying method for smart medical care and elderly care provided by the above methods, the method comprising:

[0110] In response to the intelligent accompanying instruction for the object to be accompanied, control the visual acquisition device to collect video data of the object to be accompanied;

[0111] Acquire a target point cloud of the object to be accompanied in a data frame of the video data;

[0112] Reconstructing the target point cloud to obtain a reconstructed point cloud model;

[0113] Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size;

[0114] Performing trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory;

[0115] The object to be accompanied is tracked based on the optimized target trajectory, and social interaction is performed with the object to be accompanied according to the emotional state information of the object to be accompanied.

[0116] The above-described embodiment of the intelligent accompanying robot for smart medical care and elderly care is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Ordinary technicians in this field can understand and implement it without creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent accompanying robot for smart medical care and elderly care, characterized in that: The intelligent accompanying robot includes an intelligent control center, a visual acquisition device, an acquisition unit, a reconstruction processing unit, an estimation processing unit, a trajectory optimization processing unit and an intelligent monitoring and accompanying unit; the intelligent control center is respectively connected to the visual acquisition device, the acquisition unit, the reconstruction processing unit, the estimation processing unit, the trajectory optimization processing unit and the intelligent monitoring and accompanying unit to store and manage the data of each unit; The intelligent control center is used to respond to the intelligent accompanying instruction for the object to be accompanied, and control the visual acquisition device to collect video data of the object to be accompanied; The acquisition unit is used to acquire a target point cloud of the object to be accompanied in a data frame of the video data; The reconstruction processing unit is used to perform reconstruction processing on the target point cloud to obtain a reconstructed point cloud model; The estimation processing unit is used to perform target frame size estimation processing based on the point cloud model to obtain a target frame of a target size; The trajectory optimization processing unit is used to perform trajectory optimization processing on the object to be accompanied based on the target frame of the target size to obtain an optimized target trajectory; The intelligent monitoring and accompanying unit is used to carry out accompanying and tracking of the object to be accompanied based on the optimized target trajectory, and to carry out social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied; The step of reconstructing the target point cloud to obtain a reconstructed point cloud model includes: Constructing a local coordinate system of the object to be accompanied based on different observation positions of the object to be accompanied; Based on the local coordinate system of the object to be accompanied, superimposing the target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on different observation positions of the object to be accompanied, a local coordinate system of the object to be accompanied is constructed, including: In the case where an edge of the object to be escorted is observed by the visual acquisition device, a center point of the edge is used as the origin of the local coordinate system to construct a first local coordinate system of the object to be escorted; In the case where two edges of the object to be escorted are observed by the visual acquisition device, an intersection point formed by the two edges is used as the origin of the local coordinate system to construct a second local coordinate system of the object to be escorted; In the case where three edges of the object to be escorted are observed by the visual acquisition device, the center point of the longest edge among the three edges is used as the origin of the local coordinate system to construct a third local coordinate system of the object to be escorted; The method of performing superposition processing on the target point clouds of the object to be accompanied in continuous data frames based on the local coordinate system of the object to be accompanied to obtain a reconstructed point cloud model includes: Based on the local coordinate system of the first object to be accompanied, superimposing target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on the local coordinate system of the second object to be accompanied, superimposing the target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on the third local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model.

2. The intelligent accompanying robot for smart medical care and elderly care according to claim 1 is characterized in that: Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size includes: When an edge of the object to be escorted is observed by the visual acquisition device, obtaining a length observation value corresponding to the edge; Counting the lengths of the other unobserved edges in all target frames collected by the visual acquisition device within a preset time period to obtain a statistical result; According to the statistical result, the median of the length of another edge that is not observed is used as a length estimation value; according to the length observation value and the length estimation value, a target frame of the target size is obtained; The target frame is a rectangular frame consisting of edges of different lengths.

3. The intelligent accompanying robot for smart medical care and elderly care according to claim 1 is characterized in that: Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size includes: When two edges of the object to be escorted are observed by the visual acquisition device, a maximum absolute value in the x-axis direction and a maximum absolute value in the y-axis direction in the local coordinate system of the second object to be escorted are obtained; Obtain the longest value of the edge lengths of all target frames in the x-axis direction and the longest value of the other edge lengths of all target frames in the y-axis direction; Determine a first distance difference between a maximum absolute value in the x-axis direction of the local coordinate system of the second object to be accompanied and a longest value of the edge lengths of all target frames in the x-axis direction; Determine a second distance difference between the maximum absolute value in the y-axis direction and the longest value of another edge length of all target frames in the y-axis direction; When both the first distance difference and the second distance difference are less than a preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation condition, and the maximum absolute value in the x-axis direction and the maximum absolute value in the y-axis direction in the local coordinate system of the second object to be accompanied are respectively used as the length estimation values ​​of the two edges of the target frame, to obtain a target frame of the target size; In the case that the first distance difference and the second distance difference are greater than or equal to the preset distance threshold, it is determined that the point cloud model formed by the currently observed target point cloud meets the preset completeness observation condition, and the longest value of the edge length of all target frames in the x-axis direction and the longest value of the other edge length of all target frames in the y-axis direction are respectively used as the length estimation values ​​of the two edges of the target frame, to obtain the target frame of the target size; The target frame is a rectangular frame consisting of edges of different lengths.

4. The intelligent accompanying robot for smart medical care and elderly care according to claim 1 is characterized in that: Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size includes: When the three edges of the object to be escorted are observed by the visual acquisition device, the maximum absolute value and the minimum absolute value of the x-axis direction in the local coordinate system of the third object to be escorted are obtained, and a first difference between the maximum absolute value and the minimum absolute value in the x-axis direction is determined; Obtaining the maximum absolute value and the minimum absolute value of the y-axis direction in the local coordinate system of the third object to be accompanied, and determining a second difference between the maximum absolute value and the minimum absolute value in the x-axis direction; Using the first difference and the second difference as estimated values ​​of the lengths of two edges of the target frame, respectively, to obtain a target frame of the target size; The target frame is a rectangular frame consisting of edges of different lengths.

5. The intelligent accompanying robot for smart medical care and elderly care according to claim 1 is characterized in that: The step of obtaining a target point cloud of the object to be accompanied in the data frame of the video data comprises: Acquire a data frame of the video data collected by the visual acquisition device, and extract a target point cloud in a target frame corresponding to the object to be accompanied from the data frame; The target point cloud is motion compensated to obtain a target point cloud after motion compensation, and the target point cloud after motion compensation is used as the target point cloud of the object to be accompanied in the data frame collected by the visual acquisition device.

6. The intelligent accompanying robot for smart medical care and elderly care according to claim 1 is characterized in that: The target frame based on the target size performs trajectory optimization processing on the object to be escorted to obtain an optimized target trajectory, including: According to the target size of the target frame, the center point of the target frame in the local coordinate system of the object to be escorted is determined, and the center point of the target frame in the local coordinate system of the object to be escorted is projected back to the global coordinate system of the visual acquisition device to obtain the optimized target trajectory.

7. An intelligent accompanying method for smart medical care for the elderly, applied to the intelligent accompanying robot for smart medical care for the elderly as claimed in claim 1, characterized in that: The method comprises: In response to the intelligent accompanying instruction for the object to be accompanied, control the visual acquisition device to collect video data of the object to be accompanied; Acquire a target point cloud of the object to be accompanied in a data frame of the video data; Reconstructing the target point cloud to obtain a reconstructed point cloud model; Performing target frame size estimation processing based on the point cloud model to obtain a target frame of the target size; Performing trajectory optimization processing on the object to be escorted based on the target frame of the target size to obtain an optimized target trajectory; Tracking the object to be accompanied based on the optimized target trajectory, and performing social interaction with the object to be accompanied according to the emotional state information of the object to be accompanied; The step of reconstructing the target point cloud to obtain a reconstructed point cloud model includes: Constructing a local coordinate system of the object to be accompanied based on different observation positions of the object to be accompanied; Based on the local coordinate system of the object to be accompanied, superimposing the target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on different observation positions of the object to be accompanied, a local coordinate system of the object to be accompanied is constructed, including: In the case where an edge of the object to be escorted is observed by the visual acquisition device, a center point of the edge is used as the origin of the local coordinate system to construct a first local coordinate system of the object to be escorted; In the case where two edges of the object to be escorted are observed by the visual acquisition device, an intersection point formed by the two edges is used as the origin of the local coordinate system to construct a second local coordinate system of the object to be escorted; In the case where three edges of the object to be escorted are observed by the visual acquisition device, the center point of the longest edge among the three edges is used as the origin of the local coordinate system to construct a third local coordinate system of the object to be escorted; The method of performing superposition processing on the target point clouds of the object to be accompanied in continuous data frames based on the local coordinate system of the object to be accompanied to obtain a reconstructed point cloud model includes: Based on the local coordinate system of the first object to be accompanied, superimposing target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on the local coordinate system of the second object to be accompanied, superimposing the target point clouds of the object to be accompanied in continuous data frames to obtain a reconstructed point cloud model; Based on the third local coordinate system of the object to be cared for, the target point clouds of the object to be cared for in continuous data frames are superimposed to obtain a reconstructed point cloud model.

Images