Throat swab sampling method and device, sampling equipment and computer program product

Abstract

The present application proposes a throat swab sampling method and device, a sampling device and a computer program product. The throat swab sampling method is applied to a sampling device including a robotic arm, and the throat swab sampling method includes: obtaining a target image, which is a three-dimensional image of the inside of the oral cavity; determining multiple sampling areas according to the target image, and determining a sampling point in each sampling area; establishing a three-dimensional coordinate system, and determining the position information of each sampling point and sampling area in the three-dimensional coordinate system; determining sampling operation information according to the target image and the position information, and controlling the robotic arm to perform sampling according to the sampling operation information.

Description

Technical Field

[0001] The present application relates to the field of sampling technology, and in particular to a throat swab sampling method and apparatus, sampling equipment and a computer program product. Background Art

[0002] With the development and impact of the epidemic, routine nucleic acid testing has become an important means of epidemic screening and prevention and control. At present, in order to meet the needs of large-scale sample collection and increase the sampling speed, automated nucleic acid testing equipment based on robotic arms has gradually been put into use. However, the existing automated nucleic acid testing methods based on robotic arms have a relatively simple sampling process, poor adaptability and inaccurate sampling, which reduces the sampling quality and efficiency. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art or related art.

[0004] To this end, the first aspect of the present application is to propose a throat swab sampling method.

[0005] The second aspect of the present application is to propose a throat swab sampling device.

[0006] The third aspect of the present application is to propose a throat swab sampling device.

[0007] A fourth aspect of the present application is to provide a sampling device.

[0008] A fifth aspect of the present application is to provide a readable storage medium.

[0009] A sixth aspect of the present application is to provide a computer program product.

[0010] In view of this, according to one aspect of the present application, a throat swab sampling method is proposed, which is applied to a sampling device including a robotic arm. The throat swab sampling method includes: acquiring a target image, which is a three-dimensional image of the inside of the oral cavity; determining multiple sampling areas according to the target image, and determining a sampling point in each sampling area; establishing a three-dimensional coordinate system, and determining the position information of each sampling point and sampling area in the three-dimensional coordinate system; determining sampling operation information according to the target image and the position information, and controlling the robotic arm to perform sampling according to the sampling operation information.

[0011] The technical solution of the throat swab sampling method provided by the present application can be implemented by a throat swab sampling device, which can also be determined according to actual use requirements and is not specifically limited here. In order to more clearly describe the throat swab sampling method provided by the present application, the following description is based on the implementation of the throat swab sampling method as a throat swab sampling device.

[0012] The throat swab sampling method provided in the present application, in the process of sampling the person to be sampled by a sampling device with a mechanical arm, obtains a three-dimensional image of the inside of the oral cavity of the person to be sampled, that is, a target image, and then determines multiple sampling areas inside the oral cavity of the person to be sampled according to the obtained target image, and then determines a corresponding sampling point from each sampling area. On this basis, a three-dimensional coordinate system is established based on the three-dimensional image of the inside of the oral cavity of the person to be sampled, and the position information of each sampling area and sampling point in the above-mentioned oral cavity in the above-mentioned three-dimensional coordinate system is determined. Further, according to the above-mentioned three-dimensional image, each sampling area and each sampling point in the above-mentioned three-dimensional coordinate system, the sampling operation information of the mechanical arm in the sampling device during the sampling process of the person to be sampled is determined, and then the mechanical arm movement in the above-mentioned sampling device is controlled according to the determined sampling operation information, so as to realize the sample collection work of the person to be sampled. In this way, the mechanical arm is controlled to perform corresponding actions based on the visual information inside the oral cavity of the person to be sampled, so as to sample the person to be sampled, which ensures the flexibility and adaptability of the sampling action of the mechanical arm, thereby ensuring the accuracy of the sampling of the person to be sampled by the mechanical arm, and improving the sampling efficiency.

[0013] The above-mentioned throat swab sampling method according to the present application may also have the following additional technical features:

[0014] In the above technical solution, sampling operation information is determined according to the target image and position information, including: determining the sampling order of the robot arm for each sampling point according to the target image and the position information of each sampling point; determining the entry angle of the robot arm according to the target image and the position information of the sampling point; determining the motion information of the robot arm at each sampling point according to the position information of the target image, the sampling point and the sampling area; and determining the sampling order, the entry angle and the motion information as the sampling operation information.

[0015] In this technical solution, according to the target image and the position coordinates of each sampling point in the above three-dimensional coordinate system, the sampling order of the robotic arm for multiple sampling points inside the oral cavity of the person to be sampled is determined, that is, the motion trajectory of the robotic arm between the various sampling points during the entire sampling process is determined. Further, for the sampling point located at the first position in the above sampling order, according to the three-dimensional image of the oral cavity of the person to be sampled and the position coordinates of the sampling point in the above three-dimensional coordinate system, the entrance angle of the robotic arm when sampling the sampling point is determined, that is, the direction in which the robotic arm clamps the sampling swab to enter the oral cavity of the person to be sampled is determined. Further, for each sampling point inside the oral cavity of the person to be sampled, according to the three-dimensional image of the oral cavity of the person to be sampled, the position coordinate range of each sampling area in the above three-dimensional coordinate system, and the position coordinates of each sampling point in the above three-dimensional coordinate system, the specific motion information of the robotic arm when sampling each sampling point is determined. On this basis, the above-determined sampling order, entrance angle, and motion information are determined as the sampling operation information of the above-mentioned robotic arm during the sampling process of the person to be sampled. In this way, the sampling operation information of the robotic arm is determined based on the visual information inside the mouth of the person to be sampled, which ensures the accuracy, flexibility and adaptability of the robotic arm's motion control, thereby ensuring the accuracy of the robotic arm's sampling of the person to be sampled and improving the sampling efficiency.

[0016] In any of the above technical solutions, a sampling swab is provided on the robotic arm, and the sampling order of each sampling point by the robotic arm is determined according to the target image and the position information of each sampling point, including: determining the position information of the sampling swab head in the three-dimensional coordinate system according to the target image; determining the distance information between every two sampling points and the distance information between each sampling point and the sampling swab head according to the position information of the sampling swab head and each sampling point; determining the sampling order according to the shortest path algorithm and the distance information; controlling the robotic arm to perform sampling according to the sampling operation information, including: controlling the robotic arm to sample each sampling point according to the sampling order.

[0017] In this technical solution, the robot arm samples the person to be sampled by clamping the sampling swab. Before sampling begins, the position coordinates of the head of the sampling swab in the three-dimensional coordinate system are determined according to the target image, and then the distance information between any two sampling points and the distance information from the head of the sampling swab to each sampling point are determined according to the various sampling points inside the oral cavity and the position coordinates of the head of the sampling swab in the three-dimensional coordinate system. On this basis, based on the shortest path algorithm, combined with the above-determined distance information, the sampling order of each sampling point of the robot arm during the entire sampling process is determined, that is, the motion trajectory of the robot arm between each sampling point is determined. Further, the robot arm is controlled to move between each sampling point according to the above-determined motion trajectory, so as to sample each sampling point in turn according to the determined sampling order. In this way, while ensuring the accuracy of the robot arm motion control, the time spent by the robot arm in sampling each sampling point is reduced, and the power resources of the sampling equipment are saved.

[0018] In any of the above technical solutions, the entry angle of the robotic arm is determined according to the target image and the position information of the sampling point, including: determining the sampling port plane and determining the position information of the sampling port plane in the three-dimensional coordinate system; determining the depth information from the sampling point to the sampling port plane according to the position information of the sampling point and the sampling port plane; determining the entry angle of the robotic arm relative to the sampling point according to the target image and the depth information; controlling the robotic arm to perform sampling according to the sampling operation information, including: controlling the robotic arm to rotate to the target position according to the entry angle and then enter the oral cavity to sample the sampling point.

[0019] In the technical scheme, in the process of determining the entrance angle of the robot arm, the sampling port plane when sampling the person to be sampled is first determined, and then the position information of the sampling port plane in the above three-dimensional coordinate system is determined. On this basis, according to the position information of the sampling port plane in the above three-dimensional coordinate system, the plane equation of the sampling port plane in the above three-dimensional coordinate system is determined, and according to the plane equation and the position coordinates of the sampling point in the above three-dimensional coordinate system, the distance information from the sampling point to the sampling port plane is determined. Further, according to the distance information from the sampling point to the sampling port plane, the target angle value of the entrance angle when the robot arm samples the sampling point is determined, and the orientation information of the sampling point inside the oral cavity is determined according to the target image, and then the target rotation direction of the entrance angle when the robot arm samples the sampling point is determined according to the orientation information. Further, when sampling the sampling point, the robot arm is first controlled to rotate to the target position according to the above entrance angle, that is, after the robot arm is controlled to rotate the above target angle value along the above target rotation direction, the robot arm is controlled to clamp the sampling swab into the oral cavity of the person to be sampled to sample the sampling point. In this way, the close contact between the sampling swab head and the sampling point is ensured during the sampling process, thereby ensuring the sampling quality of the sampling point and improving the sampling efficiency.

[0020] In any of the above technical solutions, the motion information includes the rotation angle of the robot arm when sampling the target sampling point. The motion information of the robot arm at each sampling point is determined according to the position information of the target image, the sampling point and the sampling area, including: determining the target sampling area corresponding to the target sampling point according to the position information of the sampling point and the sampling area; determining the rotation angle of the robot arm when sampling the target sampling point according to the target sampling area; controlling the robot arm to sample according to the sampling operation information, including: for each sampling point, controlling the robot arm to swing a preset number of times according to the rotation angle to sample the sampling point.

[0021] In this technical solution, the target sampling area where the target sampling point is located is determined based on the position coordinate range of each sampling area in the above three-dimensional coordinate system and the position coordinate of each sampling point in the above three-dimensional coordinate system, and then the rotation angle of the sampling swab along the inner wall of the oral cavity when the robot arm samples the target sampling point is determined based on the target sampling area. When sampling the target sampling point, after the sampling swab head reaches the target sampling point, the robot arm is controlled to swing back and forth a preset number of times according to the above-determined rotation angle, so that the sampling swab head repeatedly rubs the inner wall of the oral cavity at the target sampling point, thereby achieving sampling of the target sampling point. In this way, the sampling point type uses the corresponding rotation angle to sample the sampling point, which ensures the flexibility and adaptability of the sampling action of the robot arm, thereby ensuring the accuracy of the robot arm in sampling the sampler and improving the sampling efficiency.

[0022] In any of the above technical solutions, the motion information also includes displacement information after the robot arm completes sampling of the target sampling point. The motion information of the robot arm at each sampling point is determined according to the position information of the target image, the sampling point and the sampling area, and also includes: determining obstacle information in the target sampling area according to the target image; determining displacement information after the robot arm completes sampling of the target sampling point according to the obstacle information; controlling the robot arm to perform sampling according to the sampling operation information, including: after sampling the target sampling point, controlling the robot arm to move the target distance according to the displacement information and then sampling the next sampling point.

[0023] In this technical solution, obstacle information in the target sampling area where the target sampling point is located, such as the uvula, teeth, tongue and other obstacle information, is determined based on the three-dimensional image of the inside of the oral cavity of the person to be sampled. On this basis, the displacement information of the robotic arm holding the sampling swab after the robotic arm completes sampling at the target sampling point is determined based on the obstacle information in the target sampling area. When the sampling of the target sampling point is completed, the robotic arm is controlled to move the target distance according to the displacement information determined above, so that after the sampling swab head moves the target distance, the robotic arm is controlled to hold the sampling swab and move it to the next sampling point for sampling. In this way, the sampling swab is prevented from touching sensitive parts such as the uvula, teeth, tongue, etc. when moving inside the oral cavity, meeting the needs of users.

[0024] In any of the above technical solutions, the motion information includes the motion length of the robot arm when sampling each sampling point. The motion information of the robot arm at each sampling point is determined according to the position information of the target image, the sampling point and the sampling area, including: determining the oral cavity size according to the target image; determining multiple auxiliary sampling points in the sampling area corresponding to the sampling point according to the oral cavity size, the sampling point and the position information of the sampling area; determining a sampling arc according to the multiple auxiliary sampling points; determining the length of the sampling arc as the motion length; controlling the robot arm to perform sampling according to the sampling operation information, including: for each sampling point, controlling the robot arm to move along the corresponding sampling arc to perform sampling.

[0025] In this technical solution, the size of the oral cavity of the person to be sampled is determined according to the three-dimensional image of the inside of the oral cavity of the person to be sampled. On this basis, according to the oral cavity size of the person to be sampled, each sampling area and the position coordinates of the target sampling point in the above three-dimensional coordinate system, multiple auxiliary sampling points are determined in the sampling area where the target sampling point is located, that is, the above target sampling area. On this basis, the above target sampling point and multiple auxiliary sampling points are connected in sequence to determine a sampling arc, and the length of the above sampling arc is determined as the movement length of the sampling swab head along the inner wall of the oral cavity when the robot arm samples the target sampling point. In the process of sampling each sampling point, the robot arm is controlled to clamp the sampling swab to move so that the sampling swab head rubs the inner wall of the oral cavity along the corresponding sampling arc, thereby realizing the sampling of the sampling point. In this way, according to the oral cavity size of the person to be sampled, the movement length of the sampling swab head along the inner wall of the oral cavity when sampling the sampling point is determined, so that the movement length of the sampling swab head along the inner wall of the oral cavity matches the oral cavity size of the person to be sampled, ensuring the adaptability and flexibility of the sampling action of the robot arm and meeting the needs of users.

[0026] According to the second aspect of the present application, a throat swab sampling device is proposed, which is applied to a sampling equipment including a robotic arm. The throat swab sampling device includes: an acquisition unit, used to acquire a target image, the target image is a three-dimensional image of the inside of the oral cavity; a processing unit, used to determine multiple sampling areas according to the target image, and determine a sampling point in each sampling area; the processing unit is also used to establish a three-dimensional coordinate system, and determine the position information of each sampling point and sampling area in the three-dimensional coordinate system; the processing unit is also used to determine sampling operation information according to the target image and the position information, and control the robotic arm to perform sampling according to the sampling operation information.

[0027] The throat swab sampling device provided in the present application, in the process of sampling the person to be sampled by the sampling device with a mechanical arm, obtains the three-dimensional image of the inside of the oral cavity of the person to be sampled, that is, the target image, acquired by the distance measuring device on the sampling device through the acquisition unit, and then determines multiple sampling areas inside the oral cavity of the person to be sampled according to the acquired target image through the processing unit, and then determines a corresponding sampling point from each sampling area. On this basis, a three-dimensional coordinate system is established based on the three-dimensional image of the inside of the oral cavity of the person to be sampled acquired by the processing unit, and the position information of each sampling area and sampling point in the above-mentioned oral cavity in the above-mentioned three-dimensional coordinate system is determined. Further, the sampling operation information of the mechanical arm in the sampling device during the sampling process of the person to be sampled is determined by the processing unit according to the above-mentioned three-dimensional image, each sampling area and each sampling point in the above-mentioned three-dimensional coordinate system, and then the mechanical arm movement in the above-mentioned sampling device is controlled according to the determined sampling operation information, thereby realizing the sample collection work of the person to be sampled. In this way, the robotic arm is controlled to perform corresponding actions based on the visual information inside the mouth of the person to be sampled so as to sample the person to be sampled, thereby ensuring the flexibility and adaptability of the robotic arm's sampling action, thereby ensuring the accuracy of the robotic arm's sampling of the person to be sampled and improving the sampling efficiency.

[0028] According to the third aspect of the present application, a throat swab sampling device is proposed, comprising: a memory storing a program or instruction; a processor, which implements the steps of the throat swab sampling method in any of the above technical solutions when executing the program or instruction. Therefore, the throat swab sampling device proposed in the third aspect of the present application has all the beneficial effects of the throat swab sampling method in any of the above technical solutions in the first aspect, which will not be repeated here.

[0029] According to the fourth aspect of the present application, a sampling device is proposed, comprising: a robotic arm; and the throat swab sampling device in the technical solution of the third aspect, wherein the throat swab sampling device controls the robotic arm to perform sampling. Therefore, the sampling device proposed in the fourth aspect of the present application has all the beneficial effects of the throat swab sampling device in the technical solution of the third aspect, which will not be described in detail here.

[0030] In this technical solution, the sampling device also includes: a distance measuring device for obtaining three-dimensional image information of the inside of the oral cavity of the person to be sampled.

[0031] According to the fifth aspect of the present application, a readable storage medium is proposed, on which a program or instruction is stored, and when the program or instruction is executed by a processor, a throat swab sampling method as in any of the above technical solutions is implemented. Therefore, the readable storage medium proposed in the fifth aspect of the present application has all the beneficial effects of the throat swab sampling method in any of the technical solutions in the first aspect above, which will not be repeated here.

[0032] According to the sixth aspect of the present application, a computer program product is proposed, which includes a computer program, and when the computer program is executed by a processor, a throat swab sampling method as in any of the above technical solutions is implemented. Therefore, the computer program product proposed in the sixth aspect of the present application has all the beneficial effects of the throat swab sampling method in any of the technical solutions in the first aspect, which will not be repeated here.

[0033] Additional aspects and advantages of the present application will become apparent in the following description or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0035] Figure 1 One of the schematic flow charts of the throat swab sampling method according to an embodiment of the present application is shown;

[0036] Figure 2 The second schematic diagram shows the process of the throat swab sampling method according to an embodiment of the present application;

[0037] Figure 3 The third flowchart of the throat swab sampling method according to the embodiment of the present application is shown;

[0038] Figure 4 The fourth flowchart of the throat swab sampling method according to an embodiment of the present application is shown;

[0039] Figure 5 The fifth flow chart of the throat swab sampling method according to the embodiment of the present application is shown;

[0040] Figure 6 The sixth flow chart of the throat swab sampling method according to the embodiment of the present application is shown;

[0041] Figure 7 FIG7 shows a flow chart of the throat swab sampling method according to an embodiment of the present application;

[0042] Figure 8 One of the principle schematic diagrams of the throat swab sampling method of an embodiment of the present application is shown;

[0043] Fig. 9 The second schematic diagram shows the principle of the throat swab sampling method according to an embodiment of the present application;

[0044] Fig.10 The third schematic diagram shows the principle of the throat swab sampling method of the embodiment of the present application;

[0045] Fig.11One of the structural block diagrams of the throat swab sampling device of an embodiment of the present application is shown;

[0046] Fig.12 The second structural block diagram of the throat swab sampling device of the embodiment of the present application is shown;

[0047] Fig.13 A structural block diagram of a sampling device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0050] Combine the following Figures 1 to 13 , the throat swab sampling method and apparatus, sampling equipment and computer program product provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0051] Embodiment 1, Figure 1 One of the flow diagrams of the throat swab sampling method of the embodiment of the present application is shown. The sampling method includes the following steps 102 to 110:

[0052] Step 102, acquiring a target image;

[0053] Step 104, determining a plurality of sampling areas according to the target image, and determining a sampling point in each sampling area;

[0054] Step 106, establishing a three-dimensional coordinate system, and determining the position information of each sampling point and sampling area in the three-dimensional coordinate system;

[0055] Step 108, determining sampling operation information according to the target image and position information;

[0056] Step 110: Control the robot arm to perform sampling according to the sampling operation information.

[0057] The technical solution of the throat swab sampling method provided by the present application can be implemented by a throat swab sampling device, which can also be determined according to actual use requirements and is not specifically limited here. In order to more clearly describe the throat swab sampling method provided by the present application, the following description is based on the implementation of the throat swab sampling method as a throat swab sampling device.

[0058] The throat swab sampling method provided in the present application is applied to an automated or semi-automated sampling device including a robotic arm. The sampling device may specifically include one or more robotic arms. In actual application, the sampling device may specifically be an automatic sampling booth with one or more sampling robotic arms, a movable sampling robot with one or more sampling robotic arms, a semi-automatic sampling booth with an automatic sampling robotic arm, a semi-automatic sampling robot with an automatic sampling robotic arm, etc., which is not specifically limited here.

[0059] During the sampling process of the above-mentioned sampling equipment, the movement of the robotic arm thereon is controlled so that the robotic arm performs sampling actions such as acquiring the sampling container, installing the sampling container, collecting samples, storing and operating the samples, thereby realizing the sample collection work for the person to be sampled.

[0060] Specifically, the throat swab sampling method provided in the present application, in the process of sampling the person to be sampled by the above-mentioned sampling device, obtains a three-dimensional image of the inside of the oral cavity of the person to be sampled, that is, a target image, and then determines multiple sampling areas and multiple sampling points inside the oral cavity of the person to be sampled according to the acquired target image. Among them, there is a one-to-one correspondence between multiple sampling areas and multiple sampling points, that is, after determining the multiple sampling areas inside the oral cavity of the person to be sampled, a corresponding sampling point is determined from each sampling area. On this basis, a three-dimensional coordinate system is established based on the three-dimensional image of the inside of the oral cavity of the person to be sampled, and the position information of each sampling area and sampling point in the above-mentioned three-dimensional coordinate system is determined. That is, the position coordinate range of each sampling area in the oral cavity of the person to be sampled in the above-mentioned three-dimensional coordinate system is determined, and the position coordinates of each sampling point in the above-mentioned three-dimensional coordinate system are determined. Furthermore, based on the above-mentioned three-dimensional image, the position information of each sampling area and each sampling point in the above-mentioned three-dimensional coordinate system, the sampling operation information of the robotic arm in the sampling equipment during the sampling process of the person to be sampled is determined, and then the movement of the robotic arm in the above-mentioned sampling equipment is controlled according to the determined sampling operation information, thereby realizing the sample collection work for the person to be sampled.

[0061] That is to say, in the throat swab sampling method proposed in the present application, in the process of sampling the person to be sampled by means of a sampling device with a robotic arm, based on the visual information inside the oral cavity of the person to be sampled, that is, the three-dimensional image information, the sampling operation information of the robotic arm during the sampling process is determined, and then the robotic arm is controlled to move according to the determined sampling operation information to sample the person to be sampled. In this way, the robotic arm is controlled to perform corresponding actions based on the visual information inside the oral cavity of the person to be sampled so as to sample the person to be sampled, which ensures the flexibility and adaptability of the sampling action of the robotic arm, as well as the accuracy of the control of the movement of the robotic arm, thereby ensuring the accuracy of the robotic arm sampling of the person to be sampled and improving the sampling efficiency.

[0062] In the actual application process, the above-mentioned distance measuring device can be a shooting device with a distance measuring function, a sensor with a distance measuring function, or a combination of the two. For example, the above-mentioned distance measuring device can be a RGB-D camera, a combination of an RGB camera and a laser sensor, a combination of an RGB camera and a millimeter wave radar sensor, a combination of an RGB camera and an infrared camera, etc. As for the specific type of the above-mentioned distance measuring device, those skilled in the art can select it according to the actual situation, and no specific limitation is made here.

[0063] Furthermore, the above three-dimensional coordinate system can specifically be a TCP (Tool Center Point) coordinate system. The TCP coordinate system is a coordinate system with the tool center point in the initial state as the origin. In the sampling method proposed in the present application, a certain point on the robot arm, such as the end point of the robot arm's central axis, can be used as the origin to establish a three-dimensional coordinate system, and no specific restrictions are made here. A three-dimensional coordinate system is established based on the tool center point, and then the robot arm movement is controlled to perform sampling according to the position information of each sampling area and each sampling point in the above three-dimensional coordinate system, thereby ensuring the accuracy of the robot arm movement control, thereby further ensuring the accuracy of sampling the person to be sampled.

[0064] Embodiment 2: Figure 2 The second flow chart of the throat swab sampling method of the embodiment of the present application is shown. Based on the embodiment 1, the above step S108 may specifically include the following steps 108a to 108d:

[0065] Step 108a, determining the sampling order of each sampling point by the robot arm according to the target image and the position information of each sampling point;

[0066] Step 108b, determining the entrance angle of the robot arm according to the position information of the sampling point and the target image;

[0067] Step 108c, determining the motion information of the robot arm at each sampling point according to the sampling area, the position information of the sampling point, and the target image;

[0068] Step 108d, determining the motion information, entry angle, and sampling sequence as sampling operation information of the robot arm.

[0069] In this embodiment, on the basis of the first embodiment, in the process of determining the sampling operation information of the mechanical arm in the above-mentioned sampling device during the sampling process of the person to be sampled, specifically, according to the position coordinates of each sampling point in the above-mentioned three-dimensional coordinate system and the target image, the sampling order of multiple sampling points inside the oral cavity of the person to be sampled is determined, that is, the motion trajectory of the mechanical arm between the various sampling points during the entire sampling process is determined. Further, for the sampling point located at the first position in the above-mentioned sampling order, according to the three-dimensional image inside the oral cavity of the person to be sampled and the position coordinates of the sampling point in the above-mentioned three-dimensional coordinate system, the entrance angle of the mechanical arm when sampling the sampling point is determined, that is, the direction in which the mechanical arm clamps the sampling swab to enter the oral cavity of the person to be sampled is determined. Further, for each sampling point inside the oral cavity of the person to be sampled, according to the three-dimensional image inside the oral cavity of the person to be sampled, the position coordinate range of each sampling area in the above-mentioned three-dimensional coordinate system, and the position coordinates of each sampling point in the above-mentioned three-dimensional coordinate system, the specific motion information of the mechanical arm when sampling each sampling point is determined.

[0070] On this basis, the sampling order of the robotic arm for each sampling point in the entire sampling process, the entrance angle when the robotic arm starts sampling, and the motion information of the robotic arm when sampling each sampling point are determined as the sampling operation information of the robotic arm in the process of sampling the person to be sampled, and then the movement of the robotic arm is controlled according to the determined sampling operation information, so as to realize the sample collection work of the person to be sampled. In this way, based on the visual information inside the oral cavity of the person to be sampled, the entrance angle, sampling order and specific motion information of the robotic arm in the sampling process are determined, and then the robotic arm is controlled to sample the person to be sampled, which ensures the flexibility and adaptability of the robotic arm's sampling action, as well as the accuracy of the robotic arm's motion control, thereby ensuring the accuracy of the robotic arm's sampling of the person to be sampled and improving the sampling efficiency.

[0071] Embodiment 3, Figure 3 The third flow chart of the throat swab sampling method of the embodiment of the present application is shown. Based on the above embodiment 2, a sampling swab is provided on the robotic arm, and the above step 108a may specifically include the following steps 108a1 to 108a3, and the above step 110 may specifically include the following step 110a:

[0072] Step 108a1, determining the position information of the sampling swab head in the three-dimensional coordinate system according to the target image;

[0073] Step 108a2, determining the distance information between every two sampling points and the distance information between the sampling swab head and each sampling point according to the position information of each sampling point and the sampling swab head;

[0074] Step 108a3, determining the sampling order of the robot arm according to the distance information and the shortest path algorithm;

[0075] Step 110a, controlling the robot arm to sample each sampling point according to the sampling sequence.

[0076] In this embodiment, based on the second embodiment, in the process of sampling the person to be sampled by the sampling device with the mechanical arm, the sampling swab is clamped by the mechanical arm to sample the person to be sampled.

[0077] On this basis, in the process of determining the sampling order of each sampling point by the robot arm during the whole sampling process according to the position coordinates of each sampling point in the above three-dimensional coordinate system and the target image, specifically, before the sampling starts, the position coordinates of the head of the above sampling swab in the three-dimensional coordinate system are determined according to the target image, and then according to each sampling point inside the oral cavity and the position coordinates of the head of the above sampling swab in the above three-dimensional coordinate system, the distance information between any two sampling points and the distance information from the head of the sampling swab to each sampling point are determined. On this basis, based on the shortest path algorithm, combined with the above determined distance information, the sampling order of the multiple sampling points inside the oral cavity of the person to be sampled is determined by the robot arm, that is, the motion trajectory of the robot arm between the various sampling points during the whole sampling process is determined, so as to ensure that when the robot arm samples each sampling point according to the determined sampling order and motion trajectory, the total distance of the robot arm movement is minimized, thereby reducing the sampling time and saving the power resources of the sampling equipment.

[0078] For example, in the process of sampling the subject to be sampled, the sampling point P on the left side of the oral cavity is determined according to the three-dimensional image of the inside of the oral cavity of the subject to be sampled. L , sampling point P on the right side of the mouth R , sampling point P at the back of the mouth P , sampling point P on the left side of the tongue TL , sampling point P on the right side of the tongue TR There are five sampling points in total, so we get the sampling point set P = {P L , P R , P P , P TL , P TR}, where the above five sampling points are all three-dimensional coordinate points. On this basis, a three-dimensional coordinate system is established, and the above five sampling points and the sampling swab head P are determined. swab The position coordinates in the three-dimensional coordinate system are then calculated based on the sampling points and the sampling swab head P swab The position coordinates of the sampling swab head P swab Further, based on the shortest path algorithm, the sampling order of the five sampling points in the sampling point set P is sorted in combination with the distance information determined above to obtain the sampling swab head P. swab The sampling sequence of each sampling point in the above sampling point set is determined to ensure that the total distance L of the movement of the robot arm is minimized when the robot arm samples each sampling point.

[0079] The total distance L of the robot arm movement can be specifically expressed by the following formula 1:

[0080] Formula 1:

[0081] Where n represents the number of sampling points in the sampling point set P, dist(P i -P i+1 ) represents point P i To point P i+1 The distance between P1 and P n represents each sampling point in the sampling point set P, and P0 represents the sampling swab head P swab At the above point P i The coordinates of (x i ,y i ,z i ), click P i+1 The coordinates of (x i+1 ,y i+1 ,z i+1 ), the above dist(P i -P i+1 ) can be specifically calculated by the following formula 2:

[0082] Formula 2:

[0083] In this embodiment, further, after determining the sampling order of multiple sampling points, that is, after determining the motion trajectory of the robot arm between the sampling points during the entire sampling process, the robot arm is controlled to move between the sampling points according to the above-determined motion trajectory, so as to sample the sampling points in sequence according to the determined sampling order. In this way, while ensuring the accuracy of the robot arm motion control, thereby ensuring the accuracy of the robot arm sampling the object to be sampled, the time consumed by the robot arm in sampling each sampling point is reduced, and the power resources of the sampling device are saved.

[0084] In addition, in actual application, the head of the sampling swab can be used as the origin of the coordinate system to establish the above three-dimensional coordinate system, so as to further ensure the accuracy of the motion control of the robotic arm, thereby further ensuring the accuracy of sampling the person to be sampled.

[0085] Embodiment 4: Figure 4 The fourth flow chart of the throat swab sampling method of the embodiment of the present application is shown. Among them, based on the above-mentioned embodiment 2, the above-mentioned step 108b may specifically include the following steps 108b1 to 108b3, and the above-mentioned step 110 may specifically include the following step 110b:

[0086] Step 108b1, determining the sampling port plane, and determining the position information of the sampling port plane in the three-dimensional coordinate system;

[0087] Step 108b2, determining the depth information between the sampling port plane and the sampling point according to the position information of the sampling point and the sampling port plane;

[0088] Step 108b3, determining the entry angle of the robot arm relative to the sampling point according to the target image and depth information;

[0089] Step 110b, controlling the robot arm to rotate to the target position according to the entrance angle and then enter the oral cavity to sample the sampling point.

[0090] In this embodiment, on the basis of the above embodiment, the process of determining the entrance angle of the sampling point when the robot arm samples the sampling point according to the three-dimensional image of the inside of the oral cavity of the person to be sampled and the position coordinates of the sampling point in the three-dimensional coordinate system is further limited. Specifically, in the process of determining the entrance angle of the robot arm, the sampling port plane when the person to be sampled is sampled is first determined, and the sampling port plane is the plane where the lips of the person to be sampled are located, and then the position information of the sampling port plane in the three-dimensional coordinate system is determined. Among them, when determining the position information of the sampling port plane, the position information of the sampling port plane in the three-dimensional coordinate system can be directly determined according to the position information of the specific structure in the sampling device, such as the position information of the component used to fix the chin of the person to be sampled in the automatic sampling booth. In the actual application process, the three-dimensional image including the lips of the person to be sampled can also be obtained by shooting the distance measuring device on the sampling device, and then the position information of the sampling port plane in the three-dimensional coordinate system can be determined according to the three-dimensional image and the above target image. For the determination method of the position information of the sampling port plane, those skilled in the art can choose according to the actual situation, and no specific limitation is made here.

[0091] On this basis, according to the position information of the sampling port plane in the above three-dimensional coordinate system, the plane equation of the sampling port plane in the above three-dimensional coordinate system is determined, and according to the plane equation and the position coordinates of the sampling point in the above three-dimensional coordinate system, the distance information from the sampling point to the sampling port plane, that is, the depth information from the sampling point to the above sampling port plane is determined. Further, according to the distance information from the sampling point to the sampling port plane and the above target image, the entrance angle of the sampling point when the robot arm samples is determined, that is, the entrance direction of the robot arm holding the sampling swab into the oral cavity of the person to be sampled to sample the sampling point is determined. Specifically, according to the distance information from the sampling point to the sampling port plane, the target angle value of the entrance angle when the robot arm samples the sampling point is determined, and the orientation information of the sampling point inside the oral cavity is determined according to the target image, and then the target rotation direction of the entrance angle when the robot arm samples the sampling point is determined according to the orientation information. For example, when the sampling point is located on the left side of the oral cavity, the target rotation direction is right rotation, and when the sampling point is located on the right side of the oral cavity, the target rotation direction is left rotation.

[0092] Furthermore, when sampling the sampling point, the mechanical arm is first controlled to rotate to the target position according to the above-mentioned entrance angle, that is, after the mechanical arm is controlled to rotate along the above-mentioned target rotation direction by the above-mentioned target angle value, the mechanical arm is then controlled to clamp the sampling swab and enter the oral cavity of the person to be sampled to sample the sampling point. In this way, the entrance angle of the sampling swab is flexibly adjusted according to the position of the sampling point, ensuring the close contact between the head of the sampling swab and the sampling point during the sampling process, thereby ensuring the sampling quality of the sampling point and improving the sampling efficiency.

[0093] It is understandable that in the process of sampling the sampling points with the sampling swab, when the sampling points on the left and right sides of the mouth are too close to the edge of the sampling port, if the robotic arm is controlled to clamp the sampling swab in a direction perpendicular to the plane of the sampling port and enter the mouth of the person to be sampled, the sampling swab head is likely to slide along the inner wall of the mouth, resulting in the sampling swab head not being in close contact with the sampling points on the left and right sides of the mouth, thereby reducing the sampling quality of the sampling point.

[0094] Therefore, in this embodiment, according to the distance information from the sampling point to the sampling port plane and the above-mentioned target image, the entrance angle of the robotic arm when sampling the sampling point is determined. Specifically, when the distance from the sampling point to the edge of the sampling port is large, that is, when the vertical distance from the sampling point to the sampling port center axis is small, it means that the sampling point is located in the middle area of ​​the oral cavity. At this time, the robotic arm is controlled to clamp the sampling swab to enter the oral cavity of the person to be sampled in a direction perpendicular to the sampling port plane to sample the sampling point. Among them, the sampling port center axis is a straight line perpendicular to the sampling port plane and passing through the center point of the area surrounded by the user's lips in the sampling port plane. When the distance from the sampling point to the edge of the sampling port is small, that is, when the vertical distance from the sampling point to the sampling port center axis is large, it means that the sampling point is located in the two side areas of the oral cavity. At this time, the specific orientation of the sampling point in the oral cavity is determined according to the target image, and then the robotic arm is controlled to rotate the target angle value along the target rotation direction corresponding to the orientation of the sampling point, and then the sampling swab is clamped to enter the oral cavity of the person to be sampled to sample the sampling point. In this way, the entrance angle of the sampling swab is flexibly adjusted according to the position of the sampling point, ensuring close contact between the head of the sampling swab and the sampling point during the sampling process, thereby improving the sampling quality of the sampling point.

[0095] For example, Fig. 9 As shown in (a), the sampling point P on the left side of the mouth L Close to the edge of the sampling port, if the sampling point P L The robot arm is controlled to hold the sampling swab and approach the sampling point P in a direction perpendicular to the entrance of the sampling port plane M. L , the sampling swab head will slide along the inner wall of the oral cavity, causing the sampling swab head to be in contact with the sampling point P LCannot be in close contact, thus reducing the sampling point P L The sampling quality of Fig. 9 As shown in (b), at the sampling point P L When sampling at the initial sampling point, the robot arm can be controlled to hold the sampling swab and rotate rightward to the target angle θ, and then the robot arm can be controlled to hold the sampling swab and enter the oral cavity of the person to be sampled to sample the sampling point P. L Take samples.

[0096] The target angle θ ranges from 0 to

[0097] Where w is the distance of the sampling swab around the sampling point P L When rotating, the maximum distance that the sampling swab can move on the sampling port plane M, d p is the sampling point P L The vertical distance to the sampling port plane.

[0098] Embodiment 5, Figure 5 The fifth flow chart of the throat swab sampling method of the embodiment of the present application is shown. Among them, based on the above embodiment 2, the motion information includes the rotation angle of the robot arm when sampling the target sampling point, the above step 108c may specifically include the following steps 108c1 and 108c2, and the above step 110 may specifically include the following step 110c:

[0099] Step 108c1, determining the target sampling area where the target sampling point is located according to the sampling area and the location information of the sampling point;

[0100] Step 108c2, determining the rotation angle of the robot arm when sampling the target sampling point according to the target sampling area;

[0101] Step 110c: for each sampling point, control the robot arm to swing a preset number of times according to the rotation angle to sample the sampling point.

[0102] In this embodiment, on the basis of the above embodiment, the motion information of the above-mentioned robot arm when sampling each sampling point may specifically include the rotation angle of the sampling swab along the inner wall of the oral cavity when the robot arm samples the target sampling point. On this basis, in the process of determining the specific motion information of the robot arm when sampling according to the three-dimensional image of the inside of the oral cavity of the person to be sampled, the position information of each sampling area and each sampling point in the above-mentioned three-dimensional coordinate system, specifically, according to the position coordinate range of each sampling area in the above-mentioned three-dimensional coordinate system and the position coordinate of each sampling point in the above-mentioned three-dimensional coordinate system, the target sampling area where the target sampling point is located is determined, and then according to the target sampling area, the rotation angle of the sampling swab along the inner wall of the oral cavity when the robot arm samples the target sampling point is determined. Among them, the specific value of the above-mentioned rotation angle corresponds to the position of the target sampling area in the oral cavity, and the rotation angles corresponding to the sampling points in different sampling areas are different. In this way, according to the different areas of the sampling points in the oral cavity, that is, according to the different types of sampling points, different rotation angles are used to sample the sampling points, which ensures the flexibility and adaptability of the sampling action of the robot arm, thereby ensuring the accuracy of the robot arm sampling the person to be sampled and improving the sampling efficiency.

[0103] In this embodiment, further, after the rotation angle of the sampling swab along the inner wall of the oral cavity is determined when the robotic arm samples the target sampling point, when sampling the target sampling point, after the sampling swab head reaches the target sampling point, the robotic arm is controlled to swing back and forth a preset number of times according to the above-determined rotation angle, so that the sampling swab head repeatedly rubs the inner wall of the oral cavity at the target sampling point, thereby achieving sampling of the target sampling point. Among them, the specific value of the above-mentioned preset number of times can be set by those skilled in the art according to actual conditions, and no specific limitation is made here.

[0104] Embodiment 6, Figure 6 The sixth flow chart of the throat swab sampling method of the embodiment of the present application is shown. In which, based on the above-mentioned embodiment 5, the motion information also includes the displacement information after the robot arm completes the sampling of the target sampling point, the above-mentioned step 108c may specifically include the following steps 108c3 and 108c4, and the above-mentioned step 110 may specifically include the following step 110d:

[0105] Step 108c3, determining obstacle information in the target sampling area according to the target image;

[0106] Step 108c4, determining the displacement information of the robot arm after completing sampling of the target sampling point according to the obstacle information;

[0107] Step 110d, after completing sampling at the target sampling point, control the robotic arm to move a preset distance according to the displacement information and then sample the next sampling point.

[0108] In this embodiment, on the basis of the above embodiment, the motion information of the above-mentioned mechanical arm when sampling each sampling point may also specifically include the displacement information of the mechanical arm holding the sampling swab after the mechanical arm completes sampling at the target sampling point. On this basis, in the process of determining the specific motion information of the mechanical arm when sampling according to the three-dimensional image of the inside of the oral cavity of the person to be sampled, each sampling area and the position information of each sampling point in the above-mentioned three-dimensional coordinate system, specifically, according to the three-dimensional image of the inside of the oral cavity of the person to be sampled, the obstacle information in the target sampling area where the target sampling point is located, such as the obstacle information of the uvula, teeth, tongue, etc., is determined, and then according to the obstacle information in the target sampling area, the displacement information of the mechanical arm holding the sampling swab after the mechanical arm completes sampling at the target sampling point is determined. Further, when the sampling of the target sampling point is completed, the mechanical arm is controlled to move the target distance according to the above-mentioned determined displacement information, so that after the sampling swab head moves the target distance, the mechanical arm is controlled to hold the sampling swab and move to the next sampling point for sampling. In this way, it is avoided that the sampling swab touches sensitive parts such as the uvula, teeth, tongue, etc. when moving inside the oral cavity, and the user's needs are met.

[0109] Among them, the specific value of the above-mentioned target distance can be set by those skilled in the art according to actual conditions, such as setting the above-mentioned target distance to 1 cm, which is not specifically limited here.

[0110] For example, when the unit length of the three-dimensional coordinate system is 1 cm, the Z axis is perpendicular to the sampling port plane, and the positive direction of the Z axis is toward the inside of the oral cavity, the sampling point P at the back of the oral cavity is measured in a direction perpendicular to the sampling port plane. P When sampling, in order to prevent the head of the sampling swab from touching the uvula during movement, the sampling point P at the back of the mouth P After sampling is completed, the robotic arm is first controlled to hold the sampling swab and move it to move the head of the sampling swab from the coordinate point (x, y, z) to the coordinate point (x, y, z-1), that is, to move the head of the sampling swab 1 cm toward the outside of the mouth, and then the robotic arm is controlled to hold the sampling swab and move it to the next sampling point for sampling.

[0111] Embodiment 7, Figure 7 The seventh flow chart of the throat swab sampling method of the embodiment of the present application is shown. In which, based on the above-mentioned embodiment 2, the motion information includes the motion length of the robot arm when sampling each sampling point, the above-mentioned step 108c may specifically include the following steps 108c5 to 108c8, and the above-mentioned step 110 may specifically include the following step 110e:

[0112] Step 108c5, determining the size of the oral cavity according to the target image;

[0113] Step 108c6, determining a plurality of auxiliary sampling points in the sampling area corresponding to the sampling point according to the sampling area, the location information of the sampling point, and the size of the oral cavity;

[0114] Step 108c7, determining a sampling arc according to a plurality of auxiliary sampling points;

[0115] Step 108c8, determining the length of the sampling arc as the movement length of the robot arm;

[0116] Step 110e: for each sampling point, control the robot arm to move along the corresponding sampling arc to perform sampling.

[0117] In this embodiment, on the basis of the above embodiment, the motion information of the above-mentioned mechanical arm when sampling each sampling point may specifically include the motion length of the sampling swab head along the inner wall of the oral cavity when the mechanical arm samples each sampling point such as the target sampling point. On this basis, in the process of determining the specific motion information of the mechanical arm when sampling according to the three-dimensional image of the inside of the oral cavity of the person to be sampled, the position information of each sampling area and each sampling point in the above-mentioned three-dimensional coordinate system, specifically, according to the three-dimensional image of the inside of the oral cavity of the person to be sampled, the size of the oral cavity of the person to be sampled is determined, that is, the size of the sampling port of the person to be sampled is determined. On this basis, according to the size of the oral cavity of the person to be sampled, the position coordinate range of each sampling area in the above-mentioned three-dimensional coordinate system, and the position coordinate of the target sampling point in the above-mentioned three-dimensional coordinate system, multiple auxiliary sampling points are determined in the sampling area where the target sampling point is located, that is, the above-mentioned target sampling area. Among them, the multiple auxiliary sampling points are respectively located on both sides of the target sampling point, and when there are no obstacles in the target sampling area, the multiple auxiliary sampling points correspond to each other.

[0118] On this basis, the above-mentioned target sampling point and multiple auxiliary sampling points are connected in sequence to determine a sampling arc, and the length of the above-mentioned sampling arc is determined as the movement length of the sampling swab head along the inner wall of the oral cavity when the robot arm samples the target sampling point. Further, in the process of sampling each sampling point, the robot arm is controlled to clamp the sampling swab to move so that the sampling swab head rubs the inner wall of the oral cavity along the corresponding sampling arc, thereby achieving sampling of the sampling point. In this way, according to the oral size of the person to be sampled, the movement length of the sampling swab head along the inner wall of the oral cavity when sampling the sampling point is determined, so that the movement length of the sampling swab head along the inner wall of the oral cavity matches the oral size of the person to be sampled, ensuring the adaptability and flexibility of the sampling action of the robot arm and meeting user needs.

[0119] For example, Fig.10 As shown, for the sampling point P on the left side of the mouth LIn the process of sampling the sampling point, four auxiliary sampling points P are first determined within the sampling area where the sampling point is located. L1 , P L2 , P' L1 , P' L2 Among them, the auxiliary sampling point P L1 and auxiliary sampling point P' L1 About the sampling point P on the left side of the mouth L The axis m is symmetrical, and the auxiliary sampling point P L2 and auxiliary sampling point P' L2 About the sampling point P on the left side of the mouth L On this basis, according to the sampling point P on the left side of the oral cavity L And auxiliary sampling point P L1 , P L2 , P' L1 , P' L2 Determine a sampling arc that can cover the inner wall of the oral cavity corresponding to the solid line a and the dotted line b in the figure. Further, control the robotic arm to hold the sampling swab and move it so that the head of the sampling swab rubs against the inner wall of the oral cavity along the sampling arc, thereby achieving the sampling point P on the left side of the oral cavity. L of sampling.

[0120] It should be noted that in the actual application process, due to the obstruction of obstacles inside the oral cavity, the auxiliary sampling point P L1 , P L2 There may not be a corresponding auxiliary sampling point P' L1 , P' L2 The lengths of the solid line a and the dotted line b may be the same or different.

[0121] Furthermore, in the process of determining the above auxiliary sampling points, the following conditions should be met between the auxiliary sampling points:

[0122] And |P Li-x -P L-x |≥d x .

[0123] Among them, 1≤i≤n, n is half of the number of auxiliary sampling points in the absence of obstacles, k min Indicates point P L The minimum slope of the line connecting any point on the arc covered by the solid line a, k max Indicates point P L The maximum slope of the line connecting any point on the arc covered by the solid line a, P Li-y Indicates point P Li The vertical coordinate value, P L-y Indicates point PL The vertical coordinate value, P Li-x Indicates point P Li The horizontal axis value, P L-x Indicates point P L The horizontal axis value, d x Indicates point P L With point P Li The minimum distance between them in the x-axis direction. Specifically, d x It can take any value between 0.8 cm and 1 cm. In actual application, technicians in this field can also adjust d according to actual conditions. x The specific value of is set, and no specific restriction is made here.

[0124] In summary, the throat swab sampling method proposed in this application is as follows: Figure 8 As shown, in the process of sampling the person to be sampled, firstly, based on the visual information inside the oral cavity of the person to be sampled, the sampling point position information and oral cavity segmentation information inside the oral cavity of the person to be sampled are obtained. On this basis, the sampling point type and oral cavity obstacle information are determined according to the sampling point position information and oral cavity segmentation information, and the trajectory of the sampling point is planned. Furthermore, according to the determined sampling point type and oral cavity obstacle information, the adaptive sampling action of the robot arm in the sampling device is adjusted, and the person to be sampled is planned according to the planned sampling point trajectory.

[0125] In this way, the throat swab sampling method proposed in the present application can specifically achieve the following beneficial effects: planning the sampling point trajectory based on the shortest path algorithm, effectively reducing the sampling operation time required for sampling multiple positions in the oral cavity; judging the types of multiple sampling points, and selecting different sampling operation actions for each type of sampling point, thereby improving the sampling quality; in the process of performing sampling operations on the left and right sides of the oral cavity, the sampling action amplitude can be adaptively adjusted according to the oral size of different sampled persons, thereby effectively ensuring the sampling quality; in the process of performing sampling operations on the left and right sides of the oral cavity, in the face of the situation where the sampling point is too close to the edge of the sampling port, the sampling swab head can be adaptively rotated to effectively alleviate the sliding of the swab head during the sampling process in the oral cavity, thereby improving the sampling efficiency.

[0126] Embodiment 8, Fig.11 The structure block diagram of the throat swab sampling device 1100 of the embodiment of the present application is shown. The sampling device is applied to a sampling device including a robotic arm, and the throat swab sampling device 1100 includes an acquisition unit 1102 and a processing unit 1104:

[0127] An acquisition unit 1102 is used to acquire a target image, where the target image is a three-dimensional image of the interior of the oral cavity;

[0128] A processing unit 1104 is used to determine a plurality of sampling areas according to the target image, and determine a sampling point in each sampling area;

[0129] The processing unit 1104 is further used to establish a three-dimensional coordinate system and determine the position information of each sampling point and sampling area in the three-dimensional coordinate system;

[0130] The processing unit 1104 is further used to determine sampling operation information according to the target image and position information, and control the robot arm to perform sampling according to the sampling operation information.

[0131] The throat swab sampling device 1100 provided in the present application, in the process of sampling the person to be sampled by the sampling device with a mechanical arm, obtains the three-dimensional image of the inside of the oral cavity of the person to be sampled, that is, the target image, collected by the distance measuring device on the sampling device through the acquisition unit 1102, and then determines the multiple sampling areas and multiple sampling points inside the oral cavity of the person to be sampled according to the acquired target image through the processing unit 1104. Among them, there is a one-to-one correspondence between the multiple sampling areas and the multiple sampling points, that is, after determining the multiple sampling areas inside the oral cavity of the person to be sampled, a corresponding sampling point is determined from each sampling area. On this basis, a three-dimensional coordinate system is established based on the three-dimensional image of the inside of the oral cavity of the person to be sampled obtained by the processing unit 1104, and the position information of each sampling area and sampling point in the above-mentioned three-dimensional coordinate system is determined. That is, the position coordinate range of each sampling area in the oral cavity of the person to be sampled in the above-mentioned three-dimensional coordinate system is determined, and the position coordinates of each sampling point in the above-mentioned three-dimensional coordinate system are determined. Furthermore, the processing unit 1104 determines the sampling operation information of the robotic arm in the sampling device during the sampling process of the person to be sampled based on the above-mentioned three-dimensional image, each sampling area and the position information of each sampling point in the above-mentioned three-dimensional coordinate system, and then controls the movement of the robotic arm in the above-mentioned sampling device according to the determined sampling operation information, thereby realizing the sample collection work for the person to be sampled.

[0132] That is to say, in the throat swab sampling device 1100 proposed in the present application, in the process of sampling the person to be sampled by means of a sampling device with a robotic arm, based on the visual information inside the oral cavity of the person to be sampled, that is, the three-dimensional image information, the sampling operation information of the robotic arm during the sampling process is determined, and then the robotic arm is controlled to move according to the determined sampling operation information to sample the person to be sampled. In this way, the robotic arm is controlled to perform corresponding actions based on the visual information inside the oral cavity of the person to be sampled so as to sample the person to be sampled, which ensures the flexibility and adaptability of the sampling action of the robotic arm, as well as the accuracy of the control of the movement of the robotic arm, thereby ensuring the accuracy of the robotic arm sampling of the person to be sampled and improving the sampling efficiency.

[0133] Among them, the above-mentioned sampling equipment is an automated or semi-automated sampling equipment including a robotic arm. The sampling equipment may specifically include one or more robotic arms, which is not specifically limited here.

[0134] Furthermore, in actual application, the distance measuring device may be a photographing device with a distance measuring function, a sensor with a distance measuring function, or a combination of the two, which is not specifically limited herein.

[0135] Furthermore, the above three-dimensional coordinate system can specifically be a TCP coordinate system. The TCP coordinate system is a coordinate system with the tool center point in the initial state as the origin. In the sampling method proposed in the present application, a certain point on the robot arm, such as the end point of the robot arm's central axis, can be used as the origin to establish a three-dimensional coordinate system, and no specific restrictions are made here. A three-dimensional coordinate system is established based on the tool center point, and then the robot arm movement is controlled to perform sampling according to the position information of each sampling area and each sampling point in the above three-dimensional coordinate system, thereby ensuring the accuracy of the robot arm movement control, thereby further ensuring the accuracy of sampling the person to be sampled.

[0136] In this embodiment, further, the processing unit 1104 is specifically used to: determine the sampling order of the robot arm for each sampling point according to the target image and the position information of each sampling point; determine the entry angle of the robot arm according to the target image and the position information of the sampling point; determine the motion information of the robot arm at each sampling point according to the position information of the target image, the sampling point and the sampling area; determine the sampling order, the entry angle and the motion information as the sampling operation information.

[0137] In this embodiment, further, a sampling swab is provided on the robotic arm, and the processing unit 1104 is specifically used to: determine the position information of the sampling swab head in the three-dimensional coordinate system according to the target image; determine the distance information between every two sampling points and the distance information between each sampling point and the sampling swab head according to the position information of the sampling swab head and each sampling point; determine the sampling order according to the shortest path algorithm and the distance information; and control the robotic arm to sample each sampling point in the sampling order.

[0138] In this embodiment, further, the processing unit 1104 is specifically used to: determine the sampling port plane according to the target image; determine the depth information from the sampling point to the sampling port plane according to the position information of the sampling point; determine the entrance angle of the robotic arm relative to the sampling point according to the depth information; control the robotic arm to rotate along the preset direction at the entrance angle and then enter the oral cavity to sample the sampling point.

[0139] In this embodiment, further, the motion information includes a rotation angle when the robot arm samples the target sampling point, and the processing unit 1104 is specifically used to: determine the target sampling area corresponding to the target sampling point according to the position information of the sampling point and the sampling area; determine the rotation angle when the robot arm samples the target sampling point according to the target sampling area; for each sampling point, control the robot arm to swing a preset number of times according to the rotation angle to sample the sampling point.

[0140] In this embodiment, further, the motion information also includes displacement information after the robot arm completes sampling of the target sampling point. The processing unit 1104 is specifically used to: determine obstacle information in the target sampling area according to the target image; determine displacement information after the robot arm completes sampling of the target sampling point according to the obstacle information; after sampling the target sampling point, control the robot arm to move a preset distance according to the displacement information and then sample the next sampling point.

[0141] In this embodiment, further, the motion information includes the motion length of the robot arm when sampling each sampling point, and the processing unit 1104 is specifically used to: determine the size of the oral cavity according to the target image; determine multiple auxiliary sampling points in the sampling area corresponding to the sampling point according to the size of the oral cavity, the sampling point and the position information of the sampling area; determine the sampling arc according to the multiple auxiliary sampling points; determine the length of the sampling arc as the motion length; for each sampling point, control the robot arm to move along the corresponding sampling arc to perform sampling.

[0142] Embodiment 9, Fig.12 The structure block diagram of the throat swab sampling device 1200 provided in the embodiment of the present application is shown. The throat swab sampling device 1200 includes:

[0143] A memory 1202, where programs or instructions are stored;

[0144] Processor 1204, when processor 1204 executes the above program or instructions, implements the steps of the throat swab sampling method in any of the above embodiments.

[0145] The throat swab sampling device 1200 provided in this embodiment includes a memory 1202 and a processor 1204. When the program or instructions in the memory 1202 are executed by the processor 1204, the steps of the throat swab sampling method in any of the above embodiments are implemented. Therefore, the throat swab sampling device 1200 has all the beneficial effects of the throat swab sampling method in any of the above embodiments, which will not be repeated here.

[0146] Specifically, the memory 1202 and the processor 1204 may be connected via a bus or other means. The processor 1204 may include one or more processing units, and the processor 1204 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and other chips.

[0147] Embodiment 10, Fig.13 The structural block diagram of the sampling device 1300 provided in the embodiment of the present application is shown. The sampling device 1300 includes: a mechanical arm 1302; and the throat swab sampling device 1200 in the above embodiment.

[0148] The sampling device 1300 provided in the embodiment of the present application includes the throat swab sampling device 1200 in the above embodiment. Therefore, the sampling device 1300 has all the technical effects of the throat swab sampling device 1200 in the above embodiment, which will not be repeated here.

[0149] Among them, the above-mentioned sampling equipment 1300 includes but is not limited to: an automatic sampling booth with one or more sampling robotic arms, a movable sampling robot with one or more sampling robotic arms, an manned semi-automatic sampling booth with an automatic sampling robotic arm, an manned semi-automatic sampling robot with an automatic sampling robotic arm, etc., and no specific restrictions are made here.

[0150] Specifically, during the operation of the above-mentioned sampling equipment 1300, the throat swab sampling device 1200 controls the movement of the robotic arm 1302 thereon, so that the robotic arm 1302 performs sampling actions such as acquiring the sampling container, installing the sampling container, collecting samples, storing and operating the samples, thereby realizing the sample collection work for the person to be sampled.

[0151] In this embodiment, further, Fig.13 As shown, the sampling device 1300 further includes a distance measuring device 1304, which may be a RGB-D camera, a combination of an RGB camera and a laser sensor, a combination of an RGB camera and a millimeter wave radar sensor, a combination of an RGB camera and an infrared camera, etc. As for the specific type of the distance measuring device 1304, those skilled in the art may select it according to actual conditions, and no specific limitation is made here.

[0152] Specifically, during the operation of the above-mentioned sampling device 1300, the three-dimensional image information of the inside of the oral cavity of the person to be sampled is collected through the ranging device 1304 therein, and then the throat swab sampling device 1200 therein controls the movement of the mechanical arm 1302 thereon according to the three-dimensional image information collected by the ranging device 1304, thereby realizing the sample collection work of the person to be sampled.

[0153] Embodiment 11, an embodiment of the fifth aspect of the present application, provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the throat swab sampling method in any of the above embodiments are implemented.

[0154] The readable storage medium provided in the embodiment of the present application, when the program or instruction stored therein is executed by the processor, can implement the steps of the throat swab sampling method in any of the above embodiments. Therefore, the readable storage medium has all the beneficial effects of the throat swab sampling method in any of the above embodiments, which will not be repeated here.

[0155] Specifically, the above-mentioned readable storage medium may include any medium capable of storing or transmitting information. Examples of readable storage media include electronic circuits, semiconductor memory devices, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), flash memory, erasable ROM (EROM), magnetic tape, floppy disk, optical disk, hard disk, optical fiber medium, radio frequency (RF) link, optical data storage device, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0156] Embodiment 12, the embodiment of the sixth aspect of the present application, proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the throat swab sampling method as in any of the above embodiments. Therefore, the computer program product proposed in the sixth aspect of the present application has all the beneficial effects of the throat swab sampling method in any of the embodiments of the first aspect above, which will not be repeated here.

[0157] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and limited; the terms "connection", "installation", "fixation" and the like should be understood in a broad sense, for example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0158] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0159] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0160] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A throat swab sampling method, characterized in that: Applied to a sampling device including a robotic arm, the throat swab sampling method includes: Acquire a target image, where the target image is a three-dimensional image of the interior of the oral cavity; Determine a plurality of sampling areas according to the target image, and determine a sampling point in each sampling area; Establishing a three-dimensional coordinate system and determining the position information of each sampling point and sampling area in the three-dimensional coordinate system; Determine sampling operation information according to the target image and the position information, and control the robotic arm to perform sampling according to the sampling operation information; The determining of sampling operation information according to the target image and the position information includes: Determining a sampling order of each sampling point by the robotic arm according to the target image and the position information of each sampling point; Determining an entrance angle of the robotic arm according to the target image and the position information of the sampling point; Determine the motion information of the robot arm at each sampling point according to the target image, the sampling point and the position information of the sampling area; Determining the sampling order, the entry angle and the motion information as the sampling operation information; The robotic arm is provided with a sampling swab, and the sampling order of the robotic arm for each sampling point is determined according to the target image and the position information of each sampling point, including: Determine the position information of the sampling swab head in the three-dimensional coordinate system according to the target image; According to the position information of the sampling swab head and each of the sampling points, determine the distance information between every two sampling points, and the distance information between each sampling point and the sampling swab head; Determine the sampling order according to the shortest path algorithm and the distance information; The controlling the mechanical arm to perform sampling according to the sampling operation information includes: Controlling the robotic arm to sample each sampling point according to the sampling order; The motion information includes the motion length of the robot arm when sampling each sampling point, and the determining the motion information of the robot arm at each sampling point according to the target image, the sampling point and the position information of the sampling area includes: determining the size of the oral cavity according to the target image; Determine a plurality of auxiliary sampling points in the sampling area corresponding to the sampling point according to the size of the oral cavity, the sampling point and the position information of the sampling area; Determine a sampling arc according to the plurality of auxiliary sampling points; Determining the length of the sampling arc as the movement length; Controlling the robotic arm to perform sampling according to the sampling operation information includes: For each sampling point, the mechanical arm is controlled to move along the corresponding sampling arc to perform sampling.

2. The throat swab sampling method according to claim 1, characterized in that: The step of determining the entrance angle of the robot arm according to the target image and the position information of the sampling point comprises: Determine a sampling port plane, and determine position information of the sampling port plane in the three-dimensional coordinate system; Determine the depth information from the sampling point to the sampling port plane according to the position information of the sampling point and the sampling port plane; Determining an entrance angle of the robotic arm relative to the sampling point according to the target image and the depth information; Controlling the robotic arm to perform sampling according to the sampling operation information includes: The robot arm is controlled to rotate to a target position according to the entrance angle and then enters the oral cavity to sample the sampling point.

3. The throat swab sampling method according to claim 1, characterized in that: The motion information includes a rotation angle of the robot arm when sampling the target sampling point, and determining the motion information of the robot arm at each sampling point according to the position information of the target image, the sampling point and the sampling area includes: Determine a target sampling area corresponding to the target sampling point according to the location information of the sampling point and the sampling area; Determining the rotation angle of the robot arm when sampling the target sampling point according to the target sampling area; Controlling the robotic arm to perform sampling according to the sampling operation information includes: For each sampling point, the mechanical arm is controlled to swing a preset number of times according to the rotation angle to sample the sampling point.

4. The throat swab sampling method according to claim 3, characterized in that: The motion information also includes displacement information after the robot completes sampling the target sampling point. The determining of the motion information of the robot at each sampling point based on the position information of the target image, the sampling point and the sampling area also includes: Determine obstacle information in the target sampling area according to the target image; Determine the displacement information of the robotic arm after completing sampling of the target sampling point according to the obstacle information; Controlling the robotic arm to perform sampling according to the sampling operation information includes: After sampling the target sampling point, the robot arm is controlled to move the target distance according to the displacement information and then samples the next sampling point.

5. A throat swab sampling device, characterized in that: Applicable to a sampling device including a robotic arm, the pharyngeal swab sampling device includes: An acquisition unit, used for acquiring a target image, wherein the target image is a three-dimensional image of the interior of the oral cavity; A processing unit, configured to determine a plurality of sampling areas according to the target image, and determine a sampling point in each sampling area; The processing unit is further used to establish a three-dimensional coordinate system and determine the position information of each sampling point and sampling area in the three-dimensional coordinate system; The processing unit is further used to determine sampling operation information according to the target image and the position information, and control the mechanical arm to perform sampling according to the sampling operation information; In the step of determining the sampling operation information according to the target image and the position information, the processing unit is used to determine the sampling order of the robot arm for each sampling point according to the target image and the position information of each sampling point; determine the entry angle of the robot arm according to the target image and the position information of the sampling point; determine the motion information of the robot arm at each sampling point according to the target image, the sampling point and the position information of the sampling area; determine the sampling order, the entry angle and the motion information as the sampling operation information; The robotic arm is provided with a sampling swab, and in the step of determining the sampling order of each sampling point by the robotic arm according to the target image and the position information of each sampling point, the processing unit is used to determine the position information of the sampling swab head in the three-dimensional coordinate system according to the target image; determine the distance information between every two sampling points and the distance information between each sampling point and the sampling swab head according to the position information of the sampling swab head and each sampling point; determine the sampling order according to the shortest path algorithm and the distance information; In the step of the processing unit controlling the mechanical arm to perform sampling according to the sampling operation information, the processing unit is used to control the mechanical arm to perform sampling on each sampling point according to the sampling sequence; The motion information includes the motion length of the robot arm when sampling each sampling point. The processing unit determines the motion information of the robot arm at each sampling point according to the target image, the sampling point and the position information of the sampling area, and is specifically used to determine the size of the oral cavity according to the target image; determine multiple auxiliary sampling points in the sampling area corresponding to the sampling point according to the size of the oral cavity, the sampling point and the position information of the sampling area; determine a sampling arc according to the multiple auxiliary sampling points; and determine the length of the sampling arc as the motion length; In the step of controlling the robotic arm to perform sampling according to the sampling operation information, the processing unit is used to control the robotic arm to move along the corresponding sampling arc for each sampling point to perform sampling.

6. A throat swab sampling device, characterized in that: include: Memory, storing programs or instructions; A processor, which implements the steps of the throat swab sampling method as described in any one of claims 1 to 4 when executing the program or instruction.

7. A sampling device, characterized in that: include: Robotic arm; The throat swab sampling device as described in claim 6, wherein the throat swab sampling device controls the robotic arm to perform sampling.

8. The sampling device according to claim 7, characterized in that Also includes: The distance measuring device is used to obtain three-dimensional image information of the inside of the oral cavity of the person to be sampled.

9. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by the processor, the steps of the throat swab sampling method as described in any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the throat swab sampling method according to any one of claims 1 to 4 are implemented.

Images