A throat swab sampling process image acquisition processing method

Abstract

The present application relates to the field of image data processing, in particular to a throat swab sampling process image acquisition processing method, feature points are identified from a mouth depth image when a target object opens the mouth, first and second mouth sub-images are extracted and the spatial relationship of different sub-images is determined; according to the potential obstacle boundary of the second mouth sub-image and the spatial relationship, a reference movement path of the throat swab is determined, and the actual movement path is compared to adjust the moving state of the throat swab; the tissue features of the first mouth sub-image are visually recognized to calibrate the wiping track, and the deformation features of the space in the cavity are recognized from the dynamic wiping image of the throat swab along the wiping track, the contact features of the throat swab and the space tissue in the oral cavity are estimated, and the acting force applied to the throat swab is adjusted. The present application distinguishes and recognizes the movement process and wiping action process of the throat swab from the space outside the oral cavity to the space inside the oral cavity, prevents the throat swab from being contaminated, and ensures sufficient sampling.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and in particular to a method for image acquisition and processing during the throat swab sampling process. Background Technology

[0002] Pharyngeal swab sampling is used for the detection of infectious diseases, especially respiratory infectious diseases. It involves swabbing the throat tissue inside the mouth with a swab, collecting mucus from the throat tissue on the swab surface to provide a sample for testing for disease infection. Considering the specific medical conditions of the population using pharyngeal swabs, sampling personnel need to wear protective clothing and clean their hands before each sampling. Manual pharyngeal swab sampling also presents problems such as the risk of infection for personnel and low efficiency.

[0003] Automated throat swab sampling using robotic arms and other mechanical devices can address the aforementioned problems to some extent. Existing mechanical devices focus on using machine vision to quickly and accurately place the throat swab inside the oral cavity and make contact with the throat tissue to complete the sampling operation. However, they neglect the potential for unintended contact during the swab's movement from the external to the internal oral cavity, as well as the adequacy of the sample collection, easily leading to swab contamination or insufficient sampling. Therefore, accurately identifying and controlling the movement of the throat swab from the external to the internal oral cavity and its contact state with the oral cavity is of great significance for preventing contamination and improving the reliability and stability of the sampling operation. Summary of the Invention

[0004] Considering that existing machine vision-based automated throat swab sampling operations rely on the lips as a single reference element to control the movement of the throat swab, they cannot prevent unintended contact and contamination of the swab, nor can they guarantee sufficient sampling. To improve the accuracy and reliability of visual control during throat swab sampling, this invention provides an image acquisition and processing method for the throat swab sampling process, comprising the following steps:

[0005] S100: Acquire a mouth depth image of the target object when its mouth is open, identify the feature points of the mouth depth image, thereby obtaining a first mouth sub-image and a second mouth sub-image; determine the spatial relationship between the first mouth sub-image and the second mouth sub-image;

[0006] S200: Extract potential obstacle boundaries from the second oral sub-image; determine the reference motion path of the pharyngeal swab based on the potential obstacle boundaries and the spatial relationship; compare the reference motion path with the actual motion path of the pharyngeal swab, and adjust the movement state of the pharyngeal swab.

[0007] S300: Visually recognize the tissue features of the first oral sub-image to mark the swabbing trajectory of the pharyngeal swab, and acquire dynamic swabbing images of the pharyngeal swab along the swabbing trajectory;

[0008] S400: Identify the deformation characteristics of the intraoral space from the dynamic wiping image, thereby estimating the contact characteristics between the pharyngeal swab and the intraoral space tissues, and adjusting the force applied to the pharyngeal swab.

[0009] Preferably, in S100, an image of the mouth depth of the target object when its mouth is open is acquired, specifically:

[0010] Several mouth depth images of a target object with its mouth open are captured, and the distribution of object outlines and shadow elements in each mouth depth image are identified; wherein, the distribution of object outlines refers to the range of object outline distribution in the entire image; and the distribution of shadow elements refers to the range of shadow element coverage in the entire image.

[0011] Based on the overlapping features between the distribution of object outlines and the distribution of shadow elements in the image, one of the mouth depth images is selected.

[0012] Preferably, in S100, feature points of the mouth depth image are identified to obtain a first mouth sub-image and a second mouth sub-image, specifically as follows:

[0013] Extract the pixel texture distribution of the selected oral depth image, and identify several feature points belonging to the corresponding tissue parts within the oral depth image based on the pixel texture distribution;

[0014] Based on the spatial distribution location and spatial distribution clustering of the aforementioned feature points, the boundary line corresponding to the tissue region is determined; based on the boundary line, a first oral sub-image and a second oral sub-image are extracted from the oral depth image; wherein, the first oral sub-image and the second oral sub-image correspond to the intraoral space and the extraoral space, respectively.

[0015] Preferably, in S100, determining the spatial relationship between the first mouth sub-image and the second mouth sub-image specifically involves:

[0016] Based on the selected spatial coordinate system of the mouth depth image and the spatial position of the boundary line, the spatial depth positions of the first mouth sub-image and the second mouth sub-image are calibrated, thereby determining the relative spatial depth relationship between the first mouth sub-image and the second mouth sub-image.

[0017] Preferably, in S200, a potential obstacle boundary is extracted from the second oral sub-image, and a reference motion path of the pharyngeal swab is determined based on the potential obstacle boundary and the spatial relationship, specifically as follows:

[0018] The second oral sub-image is used to perform lip contour recognition to extract the potential obstacle boundary for the pharyngeal swab to enter the oral cavity of the target object; wherein, the potential obstacle boundary refers to the boundary where the lip area meets the space inside the oral cavity;

[0019] Based on the contour curvature of the potential obstacle boundary and the relative spatial depth relationship between the first oral sub-image and the second oral sub-image, the reference motion path of the pharyngeal swab from the external oral space to the internal oral space is determined.

[0020] Preferably, in S200, the reference movement path is compared with the actual movement path of the pharyngeal swab, and the movement state of the pharyngeal swab is adjusted, specifically as follows:

[0021] Visual recognition is used to identify the actual movement path of the pharyngeal swab from the external space of the oral cavity to the internal space of the oral cavity. The reference movement path and the actual movement path are mapped to the world coordinate system for comparison to obtain the spatial deviation between the two.

[0022] Based on the displacement deviation component and posture deviation component of the spatial deviation, a sequence of actions to change the movement of the pharyngeal swab is determined, thereby adjusting the movement state of the mechanical equipment on the pharyngeal swab.

[0023] Preferably, in S300, the tissue features of the first oral sub-image are visually identified to mark the swabbing trajectory of the pharyngeal swab, specifically as follows:

[0024] Visually identify the morphological features of the pharyngeal tissue in the first oral cavity sub-image, and mark several key tissue points based on the morphological features of the pharyngeal tissue; wherein, the key tissue points refer to the locations where mucus accumulates on the surface of the pharyngeal tissue.

[0025] The labels are arranged according to the spatial positions of the key points of the tissues to generate the wiping trajectory of the pharyngeal swab on the pharyngeal tissues; wherein, the wiping trajectory refers to the trajectory corresponding to the unidirectional wiping action on the pharyngeal tissues.

[0026] Preferably, in S300, a dynamic wiping image of the pharyngeal swab along the wiping trajectory is acquired, specifically as follows:

[0027] The free end of the pharyngeal swab is synchronously spatially compared with the wiping trajectory to acquire a dynamic wiping image of the pharyngeal swab wiping trajectory.

[0028] Preferably, in S400, the deformation features of the intraoral space are identified from the dynamic wiping image to estimate the contact features between the pharyngeal swab and the intraoral tissues, specifically as follows:

[0029] The deformation features of the tissue surface contour in the oral cavity are identified from the dynamic wiping image. Based on the structural parameters of the free end of the pharyngeal swab and the deformation features of the tissue surface contour, the changes in the contact area between the pharyngeal swab and the tissue in the oral cavity during wiping are estimated. The free end structural parameters include the shape and structural parameters of the free end of the pharyngeal swab.

[0030] Preferably, in S400, the force applied to the pharyngeal swab is adjusted, specifically as follows:

[0031] Based on the characteristics of the change in contact area, determine whether the pharyngeal swab can fully sample from the intraoral space tissue; if full sampling is not achieved, adjust the magnitude and / or direction of the force applied to the pharyngeal swab by the mechanical equipment.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The image acquisition and processing method for pharyngeal swab sampling of the present invention extracts a first and a second oral cavity sub-image from the feature points of the oral cavity depth image when the target object opens its mouth, and determines the spatial relationship between the different sub-images; based on the potential obstacle boundaries and spatial relationships of the second oral cavity sub-image, the reference movement path of the pharyngeal swab is determined, and compared with the actual movement path to adjust the movement state of the pharyngeal swab; the tissue features of the first oral cavity sub-image are visually identified to mark the wiping trajectory, and the deformation features of the cavity space are identified from the dynamic wiping image of the pharyngeal swab along the wiping trajectory to estimate the contact features between the pharyngeal swab and the oral cavity tissues, thereby adjusting the force applied to the pharyngeal swab. By spatially distinguishing and identifying the movement process of the pharyngeal swab from the external space to the internal space and the wiping action process, the accuracy of the pharyngeal swab's movement from the external space to the internal space and the wiping reliability are improved, preventing pharyngeal swab contamination and ensuring sufficient sampling. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0035] Figure 1 This is a flowchart of an image acquisition and processing method for a throat swab sampling process provided by the present invention.

[0036] Figure 2 It is the distribution of feature points identified in the mouth depth image.

[0037] Figure 3 It represents the relative spatial depth relationship between the first and second mouth sub-images.

[0038] Figure 4 These are the baseline and actual movement paths of the throat swab.

[0039] Figure 5 It refers to the movement and adjustment of the throat swab by mechanical equipment.

[0040] Figure 6 It is the wiping trajectory of the throat swab on the throat tissue.

[0041] Figure 7 It is the mechanical equipment that applies adjustment to the force exerted on the throat swab. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and not for limiting the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all structures. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0043] The terms "comprising" and "having," and any variations thereof, used in this invention are intended to cover non-exclusive inclusion. For example, a process, method, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0044] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0045] Please see Figure 1 As shown, the present invention provides an image acquisition and processing method for the throat swab sampling process, the method comprising the following steps:

[0046] S100: Acquire a mouth depth image of the target object when its mouth is open, identify the feature points of the mouth depth image, and obtain a first mouth sub-image and a second mouth sub-image; determine the spatial relationship between the first mouth sub-image and the second mouth sub-image.

[0047] Furthermore, in S100, an oral depth image of the target object when its mouth is open is acquired, specifically:

[0048] Capture several mouth depth images of the target object when it opens its mouth, and identify the distribution of object outlines and shadow elements in each mouth depth image; where the distribution of object outlines refers to the range of object outline distribution in the entire image; and the distribution of shadow elements refers to the range of shadow elements covered in the entire image.

[0049] Based on the overlapping characteristics between the distribution of object outlines and the distribution of shadow elements in the image, select one of the mouth depth images.

[0050] Pharyngeal swab sampling involves swabbing the throat of a target individual with a swab, collecting mucus from the throat onto the free end of the swab (such as the end with cotton attached), thus collecting a certain amount of mucus. Please refer to [link to relevant documentation]. Figure 5 To achieve automated sampling, robotic arms and other mechanical devices are used to collect throat swabs. These robotic arms are capable of multi-degree-of-freedom (e.g., five or six degrees of freedom) motion. The free end of the robotic arm holds the throat swab, which also moves in multiple degrees of freedom under the influence of the arm's movement. Furthermore, the robotic arm is equipped with a camera to capture images of the target object's mouth, enabling machine vision functionality. This camera can be, but is not limited to, a monocular depth camera or a binocular camera.

[0051] The throat swab, held by a robotic arm, needs to move from the external space of the target's mouth into the internal space, and then contact and swab the throat tissue to ensure the completion of the entire throat swab sampling process. Considering the varying mouth opening sizes of different target subjects, and the need for the throat swab to remain clean before sampling the throat tissue surface, the swab must not come into contact with other objects, such as the target's lips and teeth, during its movement from the external space into the internal space. Otherwise, the swab will be contaminated, reducing the reliability of the sampling. The lips, as the boundary between the internal and external spaces of the mouth, can be identified and located to provide a positional reference for the robotic arm to smoothly move the throat swab into the internal space. Furthermore, the movement of the throat swab by the robotic arm actually involves the robotic arm moving the throat swab along a certain depth direction (from the external space to the internal space), and the target's lips are located at the corresponding position in this depth direction. To accurately distinguish between the external and internal spaces visually, it is necessary to acquire an image of the mouth when the target's mouth is open, and to delineate the external and internal spaces at the depth of field level of the image.

[0052] Due to differences in the oral physiological structure of different target subjects, when a target subject has their mouth open, local shadows may form within the oral cavity due to occlusion, making it impossible to accurately identify the actual physiological structure of the oral cavity. To ensure the recognizability of oral depth images, several oral depth images are captured when the target subject has their mouth open. Each oral depth image can have the same field of view and shooting angle. Then, the distribution range of object contours and the coverage range of shadow elements in the global image of each oral depth image are identified. The distribution range of object contours refers to the contour distribution range formed by the physiological structure of the oral cavity in the corresponding oral depth image, and the coverage range of shadow elements refers to the distribution range of shadow elements formed by the oral cavity due to light occlusion in the corresponding oral depth image. By comparing the distribution of object contours and the distribution of shadow elements in the global oral depth image, the percentage of overlap between the two (i.e., the ratio of the overlap area to the area of ​​the global oral depth image) is obtained. The oral depth image with the smallest percentage of overlap area is selected as the navigation image for subsequent pharyngeal swab sampling and identification control, thereby improving the accuracy of pharyngeal swab sampling navigation.

[0053] Further, in S100, feature points of the mouth depth image are identified to obtain the first mouth sub-image and the second mouth sub-image, specifically:

[0054] Extract the pixel texture distribution of the selected oral depth image, and identify several feature points belonging to the corresponding tissue parts within the oral depth image based on the pixel texture distribution.

[0055] Based on the spatial distribution and clustering of several feature points, the boundary line corresponding to the tissue part is determined; based on the boundary line, a first oral sub-image and a second oral sub-image are extracted from the oral depth image; wherein, the first oral sub-image and the second oral sub-image correspond to the intraoral space and the extraoral space, respectively.

[0056] As discussed above, the lips, acting as the boundary between the intraoral and extraoral spaces, constitute an obstacle for the pharyngeal swab to enter the intraoral space from the extraoral space. Furthermore, the extraoral and intraoral spaces have different spatial structures, leading to different motion control of the pharyngeal swab within these two spaces. To accurately distinguish between the intraoral and extraoral spaces, based on the physiological characteristics of the lip lines on the lip surface, the pixel texture distribution of selected oral depth images is extracted. This extracted pixel texture distribution is then compared with the texture features corresponding to the physiological structure of the lip lines to identify several feature points within the oral depth image belonging to the lip region. Please refer to [link to relevant documentation]. Figure 2 Several feature points are concentrated in the lip area, corresponding to the image area of ​​the mouth depth image.

[0057] Considering the discrete distribution of the aforementioned feature points, in order to accurately determine the continuous boundary line formed by the lips in the oral depth image, a continuous fitting is performed on all feature points based on their spatial distribution location and spatial clustering to form the boundary line corresponding to the lip area. This continuous fitting can be, but is not limited to, fitting a continuous smooth curve based on the coordinates of all feature points. Then, the oral depth image is divided according to the aforementioned boundary line, resulting in a first oral sub-image and a second oral sub-image corresponding to the intraoral and extraoral spaces, respectively. This provides image data for subsequent control of the pharyngeal swab to accurately and without collision enter the intraoral space from the extraoral space.

[0058] Furthermore, in S100, the spatial relationship between the first mouth sub-image and the second mouth sub-image is determined, specifically as follows:

[0059] Based on the selected spatial coordinate system of the mouth depth image and the spatial position of the boundary line, the spatial depth positions of the first mouth sub-image and the second mouth sub-image are calibrated, thereby determining the relative spatial depth relationship between the first mouth sub-image and the second mouth sub-image.

[0060] In the selected oral depth images, the intraoral space has a greater depth of field than the extraoral space. Considering that the target's mouth-opening motion causes a rotational angle between the intraoral and extraoral spaces, it is necessary to accurately characterize the relative spatial depth relationship between the two spaces to accommodate the spatial depth transformation of the pharyngeal swab as it moves from the extraoral to the intraoral space, especially when crossing the boundary line. First, based on the spatial coordinate system of the selected oral depth images and the spatial position of the aforementioned boundary line, the spatial depth positions of the first and second oral sub-images are determined. Then, using these spatial depth positions as a reference, the image coordinate systems of the first and second oral sub-images are determined. (See [link to relevant documentation]). Figure 3 The coordinate systems (O1, X1, Y1, Z1) and (O2, X2, Y2, Z2) correspond to the image coordinate systems of the first and second oral cavity sub-images, respectively. A spatial transformation is then performed on these coordinate systems (O1, X1, Y1, Z1) and (O2, X2, Y2, Z2) to obtain the relative spatial depth relationship between the first and second oral cavity sub-images. This accurately calibrates the spatial depth difference between the extraoral and intraoral spaces, providing a reliable spatial basis for subsequent adjustments to the pharyngeal swab movement.

[0061] S200: Extract potential obstacle boundaries from the second oral cavity sub-image, determine the reference motion path of the pharyngeal swab based on the potential obstacle boundaries and spatial relationships, and adjust the movement state of the pharyngeal swab by comparing the reference motion path with the actual motion path of the pharyngeal swab.

[0062] Furthermore, in S200, potential obstacle boundaries are extracted from the second oral sub-image, and the baseline motion path of the pharyngeal swab is determined based on the potential obstacle boundaries and spatial relationships, specifically as follows:

[0063] The lip contour is identified in the second oral cavity sub-image to extract the potential obstacle boundary of the pharyngeal swab entering the oral cavity of the target object; where the potential obstacle boundary refers to the boundary where the lip area meets the space inside the oral cavity.

[0064] Based on the contour curvature of the potential obstacle boundary and the relative spatial depth relationship between the first and second oral sub-images, the baseline motion path of the pharyngeal swab from the external oral space to the internal oral space is determined.

[0065] The second oral cavity sub-image corresponds to the external oral cavity space. The lips serve as the dividing line between the internal and external oral cavity spaces. When a pharyngeal swab enters the relatively narrow internal oral cavity space from the wide external space, it is first affected by the lips. To enable the pharyngeal swab to adapt to the transition from the external to the internal oral cavity space and to accurately avoid obstacles formed by the lips, lip contour recognition is performed on the second oral cavity sub-image. This identifies the potential obstacle boundary formed by the lips during the pharyngeal swab's entry into the target's internal oral cavity. This potential obstacle boundary can be, but is not limited to, the boundary corresponding to the physiological structure where the lips meet the internal oral cavity space. It accurately represents the size and shape of the outermost opening of the internal oral cavity, helping to provide accurate obstacle avoidance reference during pharyngeal swab movement. Furthermore, this potential obstacle boundary, as the spatial transformation boundary between the external and internal oral cavity spaces, forms a key obstacle to pharyngeal swab movement. Its depth in space directly affects the changes in pharyngeal swab movement. Specifically, based on the contour curvature of the potential obstacle boundary and the relative spatial depth relationship between the first and second oral sub-images, the size and shape of the opening boundary formed by the potential obstacle boundary and the spatial depth position of the potential obstacle boundary are determined. Combining the opening boundary size and the spatial depth position, the reference motion path of the pharyngeal swab from the external space to the internal space of the oral cavity is determined. It can be understood that the reference motion path refers to the ideal reference roadbed corresponding to the potential obstacle boundary, which is continuously and without collision during the process of the pharyngeal swab moving from the external space to the internal space of the oral cavity, providing a reliable reference for controlling the movement of the pharyngeal swab.

[0066] Furthermore, in S200, the reference motion path is compared with the actual motion path of the pharyngeal swab, and the movement state of the pharyngeal swab is adjusted, specifically as follows:

[0067] Visual recognition is used to identify the actual movement path of the pharyngeal swab from the external space to the internal space of the oral cavity. The reference movement path and the actual movement path are mapped to the world coordinate system for comparison, and the spatial deviation between the two is obtained.

[0068] Based on the displacement and posture components of the spatial deviation, the sequence of actions to change the movement of the pharyngeal swab is determined, thereby adjusting the movement state of the mechanical equipment on the pharyngeal swab.

[0069] During the process of the robotic arm gripping and moving the throat swab, a camera simultaneously captures dynamic images of the throat swab. These dynamic images are then processed through frame-by-frame analysis to obtain the actual motion sub-paths of the throat swab across several consecutive time intervals. All these actual motion sub-paths are then connected sequentially to form the actual motion path, allowing for precise spatial calibration of the throat swab's actual motion trajectory. Please refer to [link / reference]. Figure 4 The reference motion path and the actual motion path are mapped to the world coordinate system (O, x, y, z) to obtain the spatial deviation between the two motion paths. This spatial deviation may include, but is not limited to, displacement deviation components in the x, y, and z axes of the two motion paths, and / or attitude deviation components in the three-axis attitude angle dimensions. This is done to adjust the trajectory of the pharyngeal swab and prevent unintended contact and contamination during movement (especially during entry into the oral cavity). Please also refer to [further details omitted]. Figure 5 Using a multi-degree-of-freedom robotic arm to hold a throat swab, the movement sequence of the throat swab is determined based on the displacement deviation component and the posture deviation component included in the aforementioned spatial deviation. The aforementioned movement sequence refers to the sequence of movement parameters corresponding to the spatial state change of the throat swab that the robotic arm needs to make within a preset time interval, thereby adjusting the movement state of the throat swab to ensure that the throat swab moves along the reference movement path and improves the accuracy of movement.

[0070] S300: Visually recognizes the tissue features of the first oral sub-image to mark the swab trajectory of the pharyngeal swab and acquires dynamic swab images of the pharyngeal swab along the swab trajectory.

[0071] Furthermore, in S300, the tissue features of the first oral cavity sub-image are visually recognized to mark the swabbing trajectory of the pharyngeal swab, specifically as follows:

[0072] Visually identify the morphological features of the pharyngeal tissue in the first sub-image of the mouth, and mark several key tissue points based on the morphological features of the pharyngeal tissue; among them, key tissue points refer to the locations where mucus is abundant on the surface of the pharyngeal tissue.

[0073] The labels are arranged according to the spatial location of several key tissue points to generate the wiping trajectory of the pharyngeal swab on the pharyngeal tissue; where the wiping trajectory refers to the trajectory corresponding to the unidirectional wiping action on the pharyngeal tissue.

[0074] The first sub-image of the mouth corresponds to the intraoral space. The purpose of pharyngeal swab sampling is to obtain mucus samples from the pharyngeal tissue. Considering the complex shape and structure of the pharyngeal tissue and its soft tissue nature, precise control of the contact between the pharyngeal swab and the surface of the pharyngeal tissue (especially areas of adhesive accumulation) is required during the sampling process. To this end, the morphological features of the pharyngeal tissue in the first sub-image of the mouth are identified. These morphological features may include, but are not limited to, the physiological structural features of the pharyngeal tissue itself and the morphological features of the mucus adhering to the surface of the pharyngeal tissue. The above-mentioned morphological features are screened, and the locations where the surface mucus is rich are marked as tissue key points. It is ensured that the above-mentioned tissue key points and the distribution of mucus on the surface of the pharyngeal tissue meet a specific correlation. The above-mentioned specific correlation means that only the locations where the amount of mucus accumulation exceeds a preset threshold are considered tissue key points. Then, spatial position arrangement labels are obtained by transforming the spatial coordinates of all tissue key points. These labels can be, but are not limited to, the sequential labels of the arrangement of all tissue key points along a certain direction. Using these spatial position arrangement labels as a reference, all tissue key points are arranged continuously to obtain the trajectory corresponding to the unidirectional wiping action of the pharyngeal swab on the throat tissue. Please refer to [link / reference needed]. Figure 6 As shown, by constructing the above-mentioned swabbing trajectory, it is ensured that the throat swab collects a sufficient amount of sample during a single unidirectional swabbing process, thereby improving sampling efficiency.

[0075] Furthermore, in S300, dynamic swabbing images of the throat swab along the swabbing trajectory are acquired, specifically:

[0076] The free end of the pharyngeal swab is spatially compared with the swabbing trajectory to acquire dynamic swabbing images of the pharyngeal swab along the swabbing trajectory.

[0077] In practice, a camera is used to continuously track and locate the free end of the pharyngeal swab, and a synchronous spatial comparison is made with the swabbing trajectory. Dynamic swabbing images of the pharyngeal swab along the swabbing trajectory are collected. These dynamic swabbing images contain the changes in the shape and structure of the pharyngeal tissue during the swabbing process, providing a reliable basis for determining the changes in the contact state between the pharyngeal swab and the pharyngeal tissue.

[0078] S400: Identifies the deformation characteristics of the intraoral space from dynamic swabbing images, thereby estimating the contact characteristics between the pharyngeal swab and the intraoral tissues, and adjusting the force applied to the pharyngeal swab.

[0079] Furthermore, in S400, deformation features of the intraoral space are identified from the dynamic swabbing image to estimate the contact features between the pharyngeal swab and the intraoral tissues, specifically:

[0080] The deformation features of the tissue surface in the oral cavity are identified from dynamic wiping images. Based on the structural parameters of the free end of the pharyngeal swab and the deformation features of the tissue surface, the changes in the contact area between the pharyngeal swab and the tissue in the oral cavity during wiping are estimated. The structural parameters of the free end include the shape and structural parameters of the free end of the pharyngeal swab.

[0081] The pharyngeal tissue is a soft structure. During the swabbing process, the pharyngeal swab comes into contact with the tissue, causing deformation. The closer the contact, the more mucus sample the swab collects, but the greater the irritation to the pharyngeal tissue. To balance sufficient sample collection with minimizing irritation, this study estimates the change in the contact area between the pharyngeal swab and the pharyngeal tissue during swabbing based on the structural parameters of the pharyngeal swab's free end (such as the radius of curvature) and the tissue surface deformation characteristics. This quantitatively characterizes the physical contact state between the pharyngeal swab and the pharyngeal tissue, providing a basis for subsequent adjustments to the robotic arm's action on the pharyngeal swab.

[0082] Furthermore, in S400, the force applied to the throat swab is adjusted as follows:

[0083] Based on the characteristics of changes in contact area, determine whether the pharyngeal swab can achieve sufficient sampling from the intraoral space tissue; if sufficient sampling cannot be achieved, adjust the magnitude and / or direction of the force applied to the pharyngeal swab by the mechanical equipment.

[0084] The current contact area between the pharyngeal swab and pharyngeal tissue is extracted based on the changes in the contact area between the pharyngeal swab and the pharyngeal tissue during swabbing. If the contact area is less than a preset area threshold, it is determined that the pharyngeal swab has failed to adequately sample the intraoral tissue. In this case, the force applied by the robotic arm to the pharyngeal swab can be increased or the direction of the force applied by the robotic arm can be changed to ensure that the pharyngeal swab can make sufficient contact with the pharyngeal tissue and improve the sample collection efficiency from the pharyngeal tissue. Please refer to [link to relevant documentation]. Figure 7 When insufficient sampling is not achieved, the force applied by the robotic arm to the pharyngeal swab is gradually increased as the pharyngeal swab is swabbed in the pharyngeal tissue over time to ensure sufficient pharyngeal swab sampling.

[0085] In one embodiment of the present invention, the present invention also provides a computer-readable storage medium on which a computer program is stored, the computer program implementing the method described above when executed by a processor.

[0086] In one embodiment of the present invention, the present invention also provides a computer device, the computer device including at least a memory and a processor, wherein a computer program is stored in the memory, and the computer program, when executed by the processor, implements the method described above.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image acquisition and processing during a throat swab sampling process, characterized in that, The method includes the following steps: S100: Acquire a mouth depth image of the target object when its mouth is open, identify the feature points of the mouth depth image, thereby obtaining a first mouth sub-image and a second mouth sub-image; determine the spatial relationship between the first mouth sub-image and the second mouth sub-image; S200: Extract potential obstacle boundaries from the second oral sub-image; determine the reference motion path of the pharyngeal swab based on the potential obstacle boundaries and the spatial relationship; compare the reference motion path with the actual motion path of the pharyngeal swab, and adjust the movement state of the pharyngeal swab. S300: Visually recognize the tissue features of the first oral sub-image to mark the swabbing trajectory of the pharyngeal swab, and acquire dynamic swabbing images of the pharyngeal swab along the swabbing trajectory; S400: Identify the deformation characteristics of the intraoral space from the dynamic wiping image, thereby estimating the contact characteristics between the pharyngeal swab and the intraoral space tissues, and adjusting the force applied to the pharyngeal swab.

2. The method according to claim 1, characterized in that, In S100, a mouth depth image is acquired when the target object opens its mouth, specifically: Several mouth depth images of a target object with its mouth open are captured, and the distribution of object outlines and shadow elements in each mouth depth image are identified; wherein, the distribution of object outlines refers to the range of object outline distribution in the entire image; and the distribution of shadow elements refers to the range of shadow element coverage in the entire image. Based on the overlapping features between the distribution of object outlines and the distribution of shadow elements in the image, one of the mouth depth images is selected.

3. The method according to claim 2, characterized in that, In S100, feature points of the mouth depth image are identified to obtain a first mouth sub-image and a second mouth sub-image, specifically as follows: Extract the pixel texture distribution of the selected oral depth image, and identify several feature points belonging to the corresponding tissue parts within the oral depth image based on the pixel texture distribution; Based on the spatial distribution location and spatial clustering of the aforementioned feature points, the boundary line corresponding to the tissue region is determined; Based on the boundary line, a first oral sub-image and a second oral sub-image are extracted from the oral depth image; wherein the first oral sub-image and the second oral sub-image correspond to the intraoral space and the extraoral space, respectively.

4. The method according to claim 3, characterized in that, In S100, the spatial relationship between the first mouth sub-image and the second mouth sub-image is determined, specifically as follows: Based on the selected spatial coordinate system of the mouth depth image and the spatial position of the boundary line, the spatial depth positions of the first mouth sub-image and the second mouth sub-image are calibrated, thereby determining the relative spatial depth relationship between the first mouth sub-image and the second mouth sub-image.

5. The method according to claim 1, characterized in that, In S200, potential obstacle boundaries are extracted from the second oral sub-image, and the reference motion path of the pharyngeal swab is determined based on the potential obstacle boundaries and the spatial relationship, specifically as follows: The second oral sub-image is used to perform lip contour recognition to extract the potential obstacle boundary for the pharyngeal swab to enter the oral cavity of the target object; wherein, the potential obstacle boundary refers to the boundary where the lip area meets the space inside the oral cavity; Based on the contour curvature of the potential obstacle boundary and the relative spatial depth relationship between the first oral sub-image and the second oral sub-image, the reference motion path of the pharyngeal swab from the external oral space to the internal oral space is determined.

6. The method according to claim 5, characterized in that, In S200, the reference movement path is compared with the actual movement path of the pharyngeal swab, and the movement state of the pharyngeal swab is adjusted, specifically as follows: Visual recognition is used to identify the actual movement path of the pharyngeal swab from the external space of the oral cavity to the internal space of the oral cavity. The reference movement path and the actual movement path are mapped to the world coordinate system for comparison to obtain the spatial deviation between the two. Based on the displacement deviation component and posture deviation component of the spatial deviation, a sequence of actions to change the movement of the pharyngeal swab is determined, thereby adjusting the movement state of the mechanical equipment on the pharyngeal swab.

7. The method according to claim 1, characterized in that, In S300, the tissue features of the first oral sub-image are visually identified to mark the swabbing trajectory of the pharyngeal swab, specifically as follows: Visually identify the morphological features of the pharyngeal tissue in the first oral cavity sub-image, and mark several key tissue points based on the morphological features of the pharyngeal tissue; wherein, the key tissue points refer to the locations where mucus accumulates on the surface of the pharyngeal tissue. The labels are arranged according to the spatial positions of the key points of the tissues to generate the wiping trajectory of the pharyngeal swab on the pharyngeal tissues; wherein, the wiping trajectory refers to the trajectory corresponding to the unidirectional wiping action on the pharyngeal tissues.

8. The method according to claim 7, characterized in that, In S300, a dynamic wiping image of the pharyngeal swab along the wiping trajectory is acquired, specifically: The free end of the pharyngeal swab is synchronously spatially compared with the wiping trajectory to acquire a dynamic wiping image of the pharyngeal swab wiping trajectory.

9. The method according to claim 1, characterized in that, In S400, deformation features of the intraoral space are identified from the dynamic wiping image to estimate the contact features between the pharyngeal swab and the intraoral tissues, specifically: The deformation features of the tissue surface contour in the oral cavity are identified from the dynamic wiping image. Based on the structural parameters of the free end of the pharyngeal swab and the deformation features of the tissue surface contour, the changes in the contact area between the pharyngeal swab and the tissue in the oral cavity during wiping are estimated. The free end structural parameters include the shape and structural parameters of the free end of the pharyngeal swab.

10. The method according to claim 9, characterized in that, In S400, the force applied to the throat swab is adjusted as follows: Based on the characteristics of the change in contact area, determine whether the pharyngeal swab can fully sample from the intraoral space tissue; if full sampling is not achieved, adjust the magnitude and / or direction of the force applied to the pharyngeal swab by the mechanical equipment.

Images