Method for planning a movement trajectory of a robot end tool, robot and storage medium

Abstract

The application relates to a robot end tool motion trajectory planning method, a robot and a storage medium. The method comprises the following steps: acquiring a face image of a sampled person, recognizing key feature points of a lip part of the sampled person according to the face image; judging whether the mouth of the sampled person is open based on the key feature points of the lip part; in the case that it is determined that the mouth of the sampled person is open, fitting an elliptical plane for representing the lip part in the mouth; planning a direction perpendicular to the elliptical plane as a motion direction of the robot end tool, and planning a center position of the elliptical plane as a motion starting position of the robot end tool. The robot end tool motion trajectory planning method in the embodiment of the application can be applied to a throat swab sampling process. When the above motion trajectory planning method is applied to sampling, the robot end tool can accurately reach a sampling position to implement sampling, so that a higher effective sampling rate is achieved.

Description

Technical Field

[0001] This application relates to the field of medical robot technology, and in particular to a method for planning the motion trajectory of a robot end effector, a robot, and a storage medium. Background Technology

[0002] Throat swab testing is a common medical diagnostic method for infectious diseases. In most cases, throat swab sampling is performed manually. Healthcare workers hold a swab and insert it into the patient's mouth to swab the pharynx, tonsils, palatine arches, and other areas, collecting secretions from these sites. This sampling method exposes healthcare workers to the risk of viral infection. Therefore, robots capable of automating throat swab sampling have been proposed, allowing robots to perform the sampling work in place of healthcare workers and eliminating the risk of viral infection for them.

[0003] Currently, most robots used for throat swab sampling are still in the conceptual stage. Before they can be mass-produced and put into use, several technical challenges need to be addressed. One of these challenges is how to plan the motion trajectory of the robot's end effector (such as a throat swab) to ensure that it accurately reaches the sampling site and thus guarantees the effectiveness of the sampling. Summary of the Invention

[0004] Therefore, it is necessary to provide a motion trajectory planning method for a robot end effector, a robot, and a storage medium, with the aim of enabling the robot end effector to accurately reach the sampling site, thereby ensuring the effectiveness of the sampling.

[0005] According to one aspect of this application, a motion trajectory planning method for a robot end effector is proposed, the method comprising:

[0006] Acquire a facial image of the subject and identify key feature points of the subject's lips based on the facial image;

[0007] Determine whether the mouth of the sampled person is open based on the key feature points of the lips;

[0008] Given that the mouth of the sampled person is open, an elliptical plane is fitted to characterize the lips in the mouth.

[0009] The direction perpendicular to the elliptical plane is designated as the motion direction of the robot end effector, and the center position of the elliptical plane is designated as the starting position of the robot end effector's motion.

[0010] The motion trajectory planning method for a robot end effector in this application can be applied to the throat swab sampling process. The robot end effector can be a throat swab. Therefore, the motion trajectory planning method can determine the motion trajectory of the throat swab, enabling the robot to perform throat swab sampling. Extensive experimental verification by the applicant has shown that using the above-mentioned motion trajectory planning method for sampling allows the robot end effector to accurately reach the sampling site and achieve a high effective sampling rate.

[0011] In some embodiments, the step of fitting an elliptical plane to characterize the lips in the oral cavity when the mouth of the sampled person is determined to be open includes:

[0012] When it is determined that the mouth of the sampled person is open, a first virtual plane is determined that is tangent to both the upper and lower lips of the sampled person.

[0013] Obtain the contour lines of the key feature points of the lips;

[0014] An orthographic image is obtained on the first virtual plane based on the contour line, wherein the orthographic image is used as an elliptical plane to characterize the lips in the oral cavity.

[0015] In some embodiments, the step of determining whether the sampled person's mouth is open based on the key feature points of the lips includes:

[0016] The lip distance and mouth opening distance are obtained based on the key feature points of the lips;

[0017] The ratio of the mouth opening distance to the lip distance is compared with a preset value to determine whether the oral cavity is open.

[0018] In some embodiments, the planning method further includes:

[0019] During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the head position of the sampled person;

[0020] When the deviation between the head position and the head position of the sampled person when the robot end effector begins to move exceeds a first preset threshold, the robot end effector is controlled to stop moving.

[0021] In some embodiments, the planning method further includes:

[0022] During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the position of the sampled person's lips;

[0023] When the deviation between the lip position and the lip position of the sampled person when the robot end effector begins to move exceeds a second preset threshold, the robot end effector is controlled to stop moving.

[0024] In some embodiments, the planning method further includes:

[0025] When the deviation between the head position and the head position of the sampled person when the robot end effector begins to move exceeds a first preset threshold, or when the deviation between the lip position and the lip position of the sampled person when the robot end effector begins to move exceeds a second preset threshold, a reminder message is issued to remind the sampled person.

[0026] In some embodiments, the step of acquiring the facial image of the sampled person includes:

[0027] The facial image of the sampled person is acquired by the first image acquisition device;

[0028] Before the step of acquiring the facial image of the sampled subject and identifying key feature points of the sampled subject's lips based on the facial image, the planning method further includes:

[0029] The image of the subject is acquired by a second image acquisition device to determine the facial position of the subject.

[0030] The first image acquisition device is moved to a position near the face of the person being sampled, so that the face of the person being sampled is within the acquisition range of the first image acquisition device.

[0031] In some embodiments, after defining the direction perpendicular to the elliptical plane as the motion direction of the robot end effector and defining the center position of the elliptical plane as the starting position of the robot end effector's motion, the method further includes:

[0032] The sampling is performed based on the direction of motion of the robot end effector and the starting position of the robot end effector.

[0033] According to another aspect of this application, a robot is proposed, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the motion trajectory planning method of any of the above embodiments.

[0034] According to another aspect of this application, a computer-readable storage medium is proposed that stores a computer program thereon, which, when executed by a processor, implements the motion trajectory planning method of any of the above embodiments. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the motion trajectory planning method for a robot end effector according to an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of key feature points of the lips in one embodiment of this application;

[0037] Figure 3 This is a schematic diagram (on a head model) of an elliptical plane representing the lips in the oral cavity, as described in one embodiment of this application.

[0038] Figure 4 This is a schematic diagram (on a human face) of an elliptical plane representing the lips in the oral cavity, as described in one embodiment of this application.

[0039] Figure 5 This is a flowchart illustrating the motion trajectory planning method for a robot end effector according to another embodiment of this application;

[0040] Figure 6 This is a structural block diagram of a motion trajectory planning device according to an embodiment of this application;

[0041] Figure 7 This is a structural block diagram of a robot according to one embodiment of this application;

[0042] Figure 8 This is a structural block diagram of a computer-readable storage medium according to an embodiment of this application;

[0043] Figure 9 This is a flowchart illustrating the operation of the robot used for nucleic acid sampling in one embodiment of this application. Detailed Implementation

[0044] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0045] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0046] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0047] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0048] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0049] It should be noted that when an element is referred to as being "fixed to" or "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0050] Currently, most robots used for throat swab sampling are still in the conceptual stage. Before they can be mass-produced and put into use, several technical challenges need to be addressed. One of these challenges is how to plan the motion trajectory of the robot's end effector (such as a throat swab) to ensure that it accurately reaches the sampling site and thus guarantees the effectiveness of the sampling.

[0051] To address the aforementioned technical problems, according to one aspect of this application, a method for planning the motion trajectory of a robot end effector is proposed, such as... Figure 1 As shown, the motion trajectory planning method includes the following steps:

[0052] Step S101: Obtain the facial image of the sampled person and identify the key feature points of the sampled person's lips based on the facial image.

[0053] In some embodiments, an image acquisition device (such as a camera or video camera) can be used to acquire a facial image of the subject. After acquiring the facial image of the subject, a facial key feature point detection algorithm is used to identify the key feature points of the lips in the facial image.

[0054] In other embodiments, after acquiring the facial image of the sampled person, other algorithms capable of recognizing key facial feature points can also be used, such as shape model algorithms, cascaded deep neural network algorithms, etc. For example, Figure 2 Some key feature points of the lips are shown. Figure 2 In the diagram, part A represents the key feature points of the upper lip, and part B represents the key feature points of the lower lip.

[0055] Step S102: Determine whether the sampled person's mouth is open based on key feature points of the lips.

[0056] When a person's mouth opens and closes, the distance between the upper and lower lips changes significantly. Therefore, in some embodiments, the openness of a sample's mouth can be determined based on the distance between the upper and lower lips. Specifically, key lip feature points include upper lip feature points and lower lip feature points. The current distance between the upper and lower lips can be calculated from these feature points, and then compared with a preset threshold to determine whether the sample's mouth is open. For example, the preset threshold is set to 3cm. When the current distance between the upper and lower lips is greater than or equal to 3cm, the sample's mouth is determined to be open; when the current distance is less than 3cm, the sample's mouth is determined to be closed. Of course, the preset threshold can also be other values, such as 3.1cm, 3.3cm, 3.5cm, etc., and this application embodiment does not impose specific limitations on this.

[0057] In some other embodiments, the openness of the sampled person's mouth can be determined based on the current shape of the lips. It is understood that when a person's mouth is closed, the shape of the lips is approximately elliptical. An ellipse has a major axis and a minor axis. As the mouth opens, the minor axis of this ellipse gradually increases. Therefore, the ratio between the minor and major axes can be used to determine whether the sampled person's mouth is open. Specifically, the outline of the key feature points of the lips can be obtained first, and this outline is approximately elliptical. Then, the ratio of the minor and major axes of this ellipse is compared with a preset threshold to determine whether the sampled person's mouth is open. For example, the preset threshold can be set to 0.7. When the ratio of the minor and major axes of the ellipse is greater than or equal to 0.7, the sampled person's mouth is determined to be open; when the ratio is less than 0.7, the sampled person's mouth is determined to be closed. Of course, the preset threshold can also be other values, such as 0.6, 0.65, 0.75, etc., and this application embodiment does not impose specific limitations on this.

[0058] Step S103: Given that the mouth of the sampled person is open, fit an elliptical plane to characterize the lips in the mouth.

[0059] refer to Figure 3 and Figure 4 As shown in the figure, the elliptical plane K used to represent the lips in the oral cavity is illustrated.

[0060] It is understandable that when the sampled person's mouth is open, the actual lip contour of the sampled person is approximately an "ellipse," which is a spatial geometric shape that is not on a plane. In this embodiment, the fitted elliptical plane is used to replace the actual lip contour of the sampled person. On the one hand, the sampled lip contour is simplified; on the other hand, given the complex spatial geometry of the lip contour, it is difficult to uniquely determine the direction of motion of the robot's end effector. However, by using the fitted elliptical plane to represent the lips in this embodiment, the direction of motion of the robot's end effector can be determined more easily and uniquely.

[0061] Step S104: The direction perpendicular to the elliptical plane is planned as the motion direction of the robot end effector, and the center position of the elliptical plane is planned as the starting position of the robot end effector.

[0062] Once the direction and starting position of the robot's end effector are determined, the trajectory of the robot's end effector is determined, thus completing the planning of the robot's end effector's trajectory.

[0063] The motion trajectory planning method for a robot end effector in this application can be applied to the throat swab sampling process. The robot end effector can be a throat swab. Therefore, the motion trajectory planning method can determine the motion trajectory of the throat swab, enabling the robot to perform throat swab sampling. Extensive experimental verification by the applicant has shown that using the above-mentioned motion trajectory planning method for sampling allows the robot end effector to accurately reach the sampling site and achieve a high effective sampling rate.

[0064] In some embodiments, the step of fitting an elliptical plane to characterize the lips in the oral cavity, given that the mouth of the sampled person is determined to be open, includes:

[0065] Given that the sampler's mouth is open, a first virtual plane is determined that is tangent to both the sampler's upper and lower lips;

[0066] Obtain the outline of key feature points of the lips;

[0067] An orthographic image is obtained on a first virtual plane based on the contour lines, wherein the orthographic image is used as an elliptical plane to represent the lips in the oral cavity.

[0068] Specifically, the key feature points of the lips include the feature points of the upper lip and the feature points of the lower lip. Therefore, the contour lines of the key feature points, i.e., the lip contour lines of the sampled subject, can be obtained based on the key feature points of the lips. Furthermore, based on the feature points of the upper and lower lip of the sampled subject, a first virtual plane tangent to both the upper and lower lips can be determined. Projecting the contour lines of the key feature points onto the first virtual plane yields an orthographic image, which can then be used as the elliptical plane K representing the lips in the oral cavity (see reference). Figure 3 and Figure 4 ).

[0069] In some embodiments, the step of determining whether the sampled person's mouth is open based on key feature points of the lips includes:

[0070] The lip distance and mouth opening distance are obtained based on key feature points of the lips;

[0071] The ratio of mouth opening distance to lip distance is compared with a preset value to determine whether the mouth is open.

[0072] The lip distance can be equal to the average thickness of the upper lip and the lower lip (i.e., the sum of the upper and lower lip thicknesses divided by 2). The following example illustrates how to calculate the thickness of the upper and lower lip:

[0073] For example, in Figure 2 In the diagram, at one location on the upper lip, the upper edge feature point a4 corresponds to the lower edge feature point a8, and the Euclidean distance Li between feature points a4 and a8 is obtained. At another location on the upper lip, the upper edge feature point a3 corresponds to the lower edge feature point a9, and the Euclidean distance Li between feature points a3 and a9 is obtained. This process is repeated to obtain the Euclidean distances Li between each upper edge feature point and its corresponding lower edge feature point, resulting in multiple Li values. The average of these multiple Li values ​​is then calculated to determine the thickness of the upper lip.

[0074] For example, in Figure 2 In the diagram, at one location on the lower lip, the upper edge feature point b10 corresponds to the lower edge feature point b2, and the Euclidean distance Ji between feature points b10 and b2 is obtained. At another location on the lower lip, the upper edge feature point b9 corresponds to the lower edge feature point b3, and the Euclidean distance Ji between feature points b9 and b3 is obtained. This process is repeated to obtain the Euclidean distances Ji between each upper edge feature point and its corresponding lower edge feature point on the lower lip, resulting in multiple Ji values. The average of these multiple Ji values ​​is then calculated to determine the thickness of the lower lip.

[0075] The mouth opening distance refers to the average Euclidean distance Hi between the lower edge feature point of the upper lip and the upper edge feature point of the lower lip. An example is given below:

[0076] For example, at one location on the lips, the lower edge feature point a8 of the upper lip corresponds to the upper edge feature point b10 of the lower lip, and the Euclidean distance Hi between feature points a8 and b10 is obtained. At another location on the lips, the lower edge feature point a9 of the upper lip corresponds to the upper edge feature point b9 of the lower lip, and the Euclidean distance Hi between feature points a9 and b9 is obtained. This process continues, obtaining the Euclidean distances Hi between each lower edge feature point of the upper lip and its corresponding upper edge feature point of the lower lip, resulting in multiple Hi values. The average of these multiple Hi values ​​is then calculated to obtain the mouth opening distance.

[0077] This embodiment uses the distance between the upper and lower lips of the sampled person as the criterion for determining whether the sampled person's mouth is open. For example, the preset value can be 1.5. That is, if the ratio of the mouth opening distance to the lip distance is less than 1.5, it is determined that the sampled person's mouth is not open, or the opening degree does not meet the sampling standard; if the ratio of the mouth opening distance to the lip distance is greater than or equal to 1.5, it is determined that the sampled person's mouth is open, and sampling can be performed. In other embodiments, the preset value can also be 1.3, 1.4, 1.6, 1.7, etc., and can be set according to the actual situation.

[0078] In some embodiments, after designating the direction perpendicular to the elliptical plane as the motion direction of the robot end effector and designating the center position of the elliptical plane as the starting position of the robot end effector's motion, the method further includes:

[0079] Sampling is performed based on the direction of motion of the robot's end effector and the starting position of its motion.

[0080] Specifically, the movement direction and starting position of the robot's end effector are planned, which means the movement trajectory of the robot's end effector is planned. The robot drives the end effector along the above-mentioned movement trajectory to reach the sampling position for sampling. The robot's end effector can be, for example, a throat swab.

[0081] In some embodiments, the planning method further includes:

[0082] During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the head position of the sampled person;

[0083] When the deviation between the head position and the sampled person's head position when the robot end effector starts moving exceeds a first preset threshold, the robot end effector is controlled to stop moving.

[0084] For example, a depth camera or depth camera can be used to obtain a facial image of the subject, and the head position of the subject can be obtained by combining it with a corresponding ranging algorithm.

[0085] After determining the trajectory of the robotic end effector, it moves according to the planned trajectory to perform the sampling operation. Before the end effector reaches the sampling site, the person being sampled may make significant movements, especially if the person is a child. If the person changes their posture, the robot will deviate from the sampling site if it continues sampling, resulting in invalid sampling. Additionally, it may cause injury to the person being sampled.

[0086] To avoid the aforementioned situation, in this embodiment, during the movement of the robot's end effector, a facial image of the sampled person is acquired to track the head position. The current head position is compared in real time with the head position when the robot's end effector begins to move. Once the deviation in head position exceeds a first preset threshold, the robot's end effector is controlled to stop moving, pausing or terminating the sampling. Afterward, the robot's end effector trajectory can be replanned, and sampling can be re-implemented.

[0087] In some embodiments, the planning method further includes:

[0088] During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the position of the sampled person's lips;

[0089] When the deviation between the lip position and the lip position of the sampled person when the robot end effector starts moving exceeds a second preset threshold, the robot end effector is controlled to stop moving.

[0090] In some cases, the subject's head position may not have deviated significantly, but the mouth may have made movements sufficient to affect the sampling validity. In such situations, continuing the sampling operation is also unsuitable. Therefore, this embodiment tracks the subject's lip position simultaneously with the head position. Specifically, the current lip position is compared in real-time with the subject's lip position when the robot's end effector began moving. Once the deviation in lip position exceeds a second preset threshold, the robot's end effector is stopped, pausing or terminating the sampling process. The robot's end effector trajectory can then be replanned, and sampling can be re-implemented. This setup further enhances the safety of the sampling process.

[0091] In some embodiments, the planning method further includes:

[0092] When the deviation between the head position and the head position of the sampled person when the robot end effector starts moving exceeds a first preset threshold, or when the deviation between the lip position and the lip position of the sampled person when the robot end effector starts moving exceeds a second preset threshold, a reminder message is issued to remind the sampled person.

[0093] For example, the reminder message may be a prompt for the sampled subject to maintain their head and lip posture. Alternatively, the reminder message may include information indicating that sampling will be re-implemented, as well as a prompt for the sampled subject to maintain their head and lip posture. This is to ensure successful sampling in the next motion trajectory planning and sampling cycle.

[0094] In some embodiments, the step of acquiring a facial image of a subject includes:

[0095] The facial image of the subject is acquired through the first image acquisition device;

[0096] Before the steps of acquiring facial images of the sampled subject and identifying key feature points of the sampled subject's lips based on the facial images, the planning method also includes:

[0097] The image of the subject is acquired by the second image acquisition device to determine the facial position of the subject.

[0098] The first image acquisition device is moved to a position near the face of the person being sampled, so that the face of the person being sampled is within the acquisition range of the first image acquisition device.

[0099] Specifically, the facial image of the sampled person can be acquired using a first image acquisition device. However, the first image acquisition device has its own acquisition range. If the sampled person's face is located outside the acquisition range of the first image acquisition device, the first image acquisition device cannot acquire the facial image of the sampled person or cannot acquire a clear facial image. In this embodiment, before acquiring the facial image of the sampled person through the first image acquisition device, the image of the sampled person is first acquired through a second image acquisition device to determine the facial position of the sampled person. Then, the first image acquisition device is controlled to move to the vicinity of the facial position of the sampled person, so that the facial position of the sampled person is within the acquisition range of the first image acquisition device. This ensures that the facial image of the sampled person is successfully acquired.

[0100] Understandably, the second image acquisition device can be, for example, a depth camera or depth video camera, such as an inhand camera. Combined with appropriate ranging algorithms and facial key feature point algorithms, it can determine the facial position of the sampled person. Alternatively, the second image acquisition device can be a camera or video camera with lower precision but a wider field of view compared to the first image acquisition device, such as a global camera. It is sufficient to acquire the facial position of the sampled person; high-precision facial images of the sampled person are not required.

[0101] In some embodiments, such as Figure 5 As shown, the motion trajectory planning method for robot end effectors includes:

[0102] Step 201: Acquire an image of the subject using the second image acquisition device to determine the facial position of the subject;

[0103] Step 202: Move the first image acquisition device to the vicinity of the subject's face so that the subject's face is within the acquisition range of the first image acquisition device;

[0104] Step 203: Acquire the facial image of the subject using the first image acquisition device, and identify the key feature points of the subject's lips based on the facial image;

[0105] Step 204: Obtain the lip distance and mouth opening distance based on the key feature points of the lips;

[0106] Step 205: Compare the ratio of mouth opening distance to lip distance with a preset value to determine whether the mouth is open;

[0107] Step 206: With the sampled person's mouth open, determine a first virtual plane that is tangent to both the upper and lower lips of the sampled person;

[0108] Step 207: Obtain the outline of key feature points of the lips;

[0109] Step 208: Obtain an orthographic projection image on the first virtual plane based on the contour lines, wherein the orthographic projection image is used as an elliptical plane to represent the lips in the oral cavity;

[0110] Step 209: Define the direction perpendicular to the elliptical plane as the motion direction of the robot's end effector, and define the center position of the elliptical plane as the starting position of the robot's end effector's motion;

[0111] Step 210: Sample according to the movement direction and starting position of the robot end effector;

[0112] Step 211: During the movement of the robot's end-effector, the second image acquisition device is continuously operated to acquire the facial image of the sampled person in real time to track the head position of the sampled person, and the first image acquisition device is continuously operated to acquire the facial image of the sampled person in real time to track the lip position of the sampled person.

[0113] Step 212: When the deviation between the head position and the head position of the sampled person when the robot end-effector starts moving exceeds the first preset threshold, control the robot end-effector to stop moving. When the deviation between the lip position and the lip position of the sampled person when the robot end-effector starts moving exceeds the second preset threshold, control the robot end-effector to stop moving.

[0114] The above method allows for the determination of the throat swab's trajectory, enabling the robot to perform throat swab sampling. Furthermore, by implementing this method, the robot's end effector can accurately reach the sampling site, thereby achieving a high effective sampling rate.

[0115] According to another aspect of this application, a motion trajectory planning device for a robot end effector is proposed, such as... Figure 6 As shown, the motion trajectory planning device includes:

[0116] The first acquisition and recognition module 100 is used to acquire the facial image of the sampled person and identify the key feature points of the sampled person's lips based on the facial image.

[0117] The judgment module 200 is used to determine whether the mouth of the sampled person is open based on key feature points of the lips;

[0118] Fitting module 300 is used to fit an elliptical plane that characterizes the lips in the oral cavity;

[0119] The planning module 400 is used to plan the direction perpendicular to the elliptical plane as the motion direction of the robot end effector and to plan the center position of the elliptical plane as the starting position of the robot end effector's motion.

[0120] In some embodiments, the fitting module includes:

[0121] The virtual plane determination module is used to determine a first virtual plane that is tangent to both the upper and lower lips of the sampled person when the sampled person's mouth is open.

[0122] The contour line acquisition module is used to acquire the contour lines of key feature points of the lips;

[0123] The projection fitting module is used to obtain an orthographic image on the first virtual plane based on the contour lines, and to use the orthographic image as an elliptical plane to characterize the lips in the oral cavity.

[0124] In some embodiments, the determination module includes:

[0125] The distance acquisition module is used to obtain the lip distance and mouth opening distance based on key feature points of the lips;

[0126] The comparison module is used to compare the ratio of the mouth opening distance to the lip distance with a preset value to determine whether the mouth is open.

[0127] In some embodiments, the motion trajectory planning device further includes:

[0128] The head position tracking module is used to acquire facial images of the sampled person in real time during the movement of the robot's end tool in order to track the head position of the sampled person.

[0129] The first control module is used to control the robot end effector to stop moving when the deviation between the head position and the head position of the sampled person when the robot end effector starts moving exceeds a first preset threshold.

[0130] In some embodiments, the motion trajectory planning device further includes:

[0131] The lip position tracking module is used to acquire facial images of the sampled person in real time during the movement of the robot's end tool in order to track the position of the sampled person's lips.

[0132] The second control module is used to control the robot end-effector to stop moving when the deviation between the lip position and the lip position of the sampled person when the robot end-effector starts moving exceeds a second preset threshold.

[0133] In some embodiments, the motion trajectory planning device further includes:

[0134] The reminder module is used to issue a reminder message to the sampled person when the deviation between the head position and the head position of the sampled person when the robot end tool starts to move exceeds a first preset threshold, or when the deviation between the lip position and the lip position of the sampled person when the robot end tool starts to move exceeds a second preset threshold.

[0135] In some embodiments, the first acquisition and recognition module includes an acquisition submodule, which is used to acquire a facial image of the sampled person through a first image acquisition device;

[0136] The motion trajectory planning device also includes:

[0137] The second acquisition and recognition module is used to acquire an image of the sampled person through the second image acquisition device and determine the facial position of the sampled person.

[0138] The third control module is used to control the first image acquisition device to move to the vicinity of the sampled person's face, so that the sampled person's face is within the acquisition range of the first image acquisition device.

[0139] In some embodiments, the motion trajectory planning device further includes:

[0140] The sampling module is used to sample based on the movement direction and starting position of the robot's end effector.

[0141] According to another aspect of this application, a robot, such as Figure 7 As shown, the robot includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the motion trajectory planning method in any of the above embodiments.

[0142] According to another aspect of this application, a computer-readable storage medium, such as... Figure 8 As shown, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the motion trajectory planning method in any of the above embodiments.

[0143] Further implementation details of the robot and computer-readable storage medium provided in some embodiments of this application can be found in the relevant descriptions of the embodiments of the robot motion trajectory planning method described above, and will not be repeated here.

[0144] A robot used for throat swab sampling is an automated device capable of performing throat swab sampling. The following section uses nucleic acid sampling as an example to introduce this robot.

[0145] This robot can replace manual labor in performing tasks such as unpacking swabs, loading and unloading test tubes, nucleic acid sampling, breaking swabs and placing them into test tubes, and disinfecting after testing. It can effectively reduce labor intensity and, to a certain extent, reduce the risk of infection and cross-infection for medical staff.

[0146] The robot mainly consists of the following modules:

[0147] Swab loading module: Applicable to existing throat swab packaging, it can independently load swabs for each test, and automate the entire process of swab grabbing, unpacking and swab retrieval.

[0148] It is feasible to feed one 500-pack of swabs at a time, and manual replenishment is required after use.

[0149] Face and mouth position recognition detection module: The face detection module identifies the position and posture of the person being tested, and the oropharyngeal recognition module automatically identifies the required detection sites in the oropharynx.

[0150] Detection action execution module: The throat swab sampling robot can be a force-controlled adaptive robot, which completes standardized nucleic acid sampling actions through force control. The entire force control is completed by force-position composite control to complete the nucleic acid detection action at the designated position, ensuring the comfort of the testing personnel and the detection effect; it has an abnormal handling mechanism, which automatically stops / avoids when the force exceeds the limit or is not in place, to prevent damage.

[0151] Test tube reagent loading and storage module: Automatically completes the loading, capping, and insertion of tested swabs for individual test tubes. It has test tube coding function and a separate test tube storage area for convenient loading, unloading, and transportation.

[0152] The test tube storage area has an upper section as a test tube buffer area, which can store up to 18*32=572 test tubes. The lower section can be used as a spare test tube buffer area. If the test tube buffer area in the upper section runs out of material, the maintenance personnel can directly replenish the material from the lower section, or it can be used as a storage area for other items.

[0153] Human-computer interaction module: Enables fully automated interaction with inspection personnel, with process guidance function and simple operation throughout; a camera is installed at the front end of the robotic arm, allowing the inspected personnel to view the recognition results and inspection actions in real time during the inspection.

[0154] Disinfection module: Fully automated disinfection of areas that may cause cross-infection, using different methods and frequencies. This includes: spray disinfection of high-risk areas after a single test, regular spray disinfection of workbenches and potentially high-risk areas, and continuous ultraviolet disinfection of the entire interior area.

[0155] Nucleic acid sampling workstation: The workstation is designed as an independent unit, which facilitates construction and maintenance.

[0156] Based on the aforementioned robot, the procedure for conducting nucleic acid sampling can be referenced. Figure 9 .

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for planning the motion trajectory of a robot end effector, characterized in that, include: The image of the subject is acquired by a second image acquisition device to determine the facial position of the subject. The first image acquisition device is moved to a position near the face of the subject being sampled, so that the face of the subject being sampled is within the acquisition range of the first image acquisition device. The first image acquisition device acquires a facial image of the subject, and identifies key feature points of the subject's lips based on the facial image; the key feature points include upper lip feature points and lower lip feature points. The lip distance and mouth opening distance are obtained based on the key feature points of the lips; the lip distance is the average of the thickness of the upper lip and the thickness of the lower lip. The ratio of the mouth opening distance to the lip distance is compared with a preset value to determine whether the mouth of the sampled person is open. Given that the mouth of the sampled person is open, an elliptical plane is fitted to characterize the lips in the mouth; this includes: given that the mouth of the sampled person is open, determining a first virtual plane that is tangent to both the upper and lower lips of the sampled person; obtaining the contour lines of key feature points of the lips; and obtaining an orthographic projection image on the first virtual plane based on the contour lines, wherein the orthographic projection image is used as the elliptical plane to characterize the lips in the mouth. The direction perpendicular to the elliptical plane is designated as the motion direction of the robot end effector, and the center position of the elliptical plane is designated as the starting position of the robot end effector's motion.

2. The motion trajectory planning method for a robot end effector according to claim 1, characterized in that, The planning method also includes: During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the head position of the sampled person; When the deviation between the head position and the head position of the sampled person when the robot end effector begins to move exceeds a first preset threshold, the robot end effector is controlled to stop moving.

3. The motion trajectory planning method for a robot end effector according to claim 2, characterized in that, The planning method also includes: During the movement of the robot's end effector, facial images of the sampled person are acquired in real time to track the position of the sampled person's lips; When the deviation between the lip position and the lip position of the sampled person when the robot end effector begins to move exceeds a second preset threshold, the robot end effector is controlled to stop moving.

4. The motion trajectory planning method for a robot end effector according to claim 3, characterized in that, The planning method also includes: When the deviation between the head position and the head position of the sampled person when the robot end effector begins to move exceeds a first preset threshold, or when the deviation between the lip position and the lip position of the sampled person when the robot end effector begins to move exceeds a second preset threshold, a reminder message is issued to remind the sampled person.

5. The motion trajectory planning method for a robot end effector according to claim 1, characterized in that, After defining the direction perpendicular to the elliptical plane as the motion direction of the robot end effector and defining the center position of the elliptical plane as the starting position of the robot end effector's motion, the method further includes: The sampling is performed based on the direction of motion of the robot end effector and the starting position of the robot end effector.

6. A robot comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the motion trajectory planning method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the motion trajectory planning method as described in any one of claims 1 to 5.

Images