A method, device, equipment and storage medium for taking a dental plaque

Abstract

The present application provides a dental plaque shooting method, device, equipment and storage medium, the method comprises the following steps: after starting the endoscope with the purple light, the first pose information of the target dental plaque is determined according to the red imaging area of the real-time video stream of the endoscope; the second pose information is determined according to the target reference image of the target dental plaque; the target movement information from the first pose information to the second pose information is determined by comparing the pose feature difference; in response to the endoscope completing the movement based on the target movement information, the current video frame of the real-time video stream is determined as the target shooting image of the target dental plaque. According to the technical scheme of the embodiment of the present application, the pose difference between the shooting pose and the reference pose of the target dental plaque is used to determine the target movement information to prompt the user to move the endoscope, so that the endoscope can be aligned with the target dental plaque at the standard pose and then shot, and accurate dental plaque images are provided for subsequent identification.

Description

Technical Field

[0001] This invention relates to the field of intelligent endoscopy technology, and in particular to a method, apparatus, device, and storage medium for photographing dental plaque. Background Technology

[0002] Currently, some endoscopic systems integrate LED lights and cameras into the endoscope, enabling the display of real-time video streams transmitted from the endoscope on a display device, facilitating relevant diagnoses by doctors. In the field of oral examinations, plaque detection is a crucial step. Related technologies primarily utilize LED lights emitting special colors, taking advantage of the color rendering properties of plaque under these specific lights to display it in the video stream. Medical staff can then visually observe and locate individual plaques, taking photographs as needed.

[0003] However, the angle and distance of the camera can change the surface color and shape of dental plaque in the image to some extent. The relevant technology can only be used by medical staff to adjust the position of the endoscope for taking pictures based on their experience. When making subsequent identification based on the shape and color of dental plaque, deviations are prone to occur, affecting the auxiliary diagnosis effect based on dental plaque images. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a method, apparatus, device, and storage medium for photographing dental plaque, which can guide the movement direction of the endoscope based on the detected dental plaque, ensuring that the dental plaque is photographed in the correct shooting posture.

[0005] In a first aspect, embodiments of the present invention provide a method for photographing dental plaque, applied to an intelligent terminal of an endoscope system. The endoscope system further includes an endoscope communicatively connected to the intelligent terminal. The intelligent terminal has multiple preset reference images, each preset reference image uniquely corresponding to a type of dental plaque. The preset reference images are images of dental plaque photographed at a standard shooting angle. The method includes:

[0006] When a red imaging area is detected from the real-time video stream of the endoscope, the pose feature of the red imaging area is extracted to obtain the first pose information of the target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is violet light.

[0007] Based on the shape of the target dental plaque, a target reference image is determined from multiple preset reference images, and pose features are extracted from the reference region of the dental plaque in the target reference image to obtain second pose information;

[0008] The target movement information is determined by comparing the differences in pose features, and the target movement information is then displayed.

[0009] In response to the endoscope completing its movement based on the target movement information, the current video frame of the real-time video stream is determined as the target image of the target dental plaque.

[0010] According to some embodiments of the present invention, the smart terminal is pre-equipped with an ORB detector, and the pose feature extraction includes:

[0011] Determine the target area, wherein the target area is the red imaging area or the reference area;

[0012] When the target area is the red imaging area, remove the gingival area in the target area where the imaging saturation is lower than the preset saturation;

[0013] The ORB detector determines the ORB descriptors corresponding to multiple target feature points in the target region, and the ORB descriptors are determined as target pose information, wherein the target pose information is either the first pose information or the second pose information.

[0014] According to some embodiments of the present invention, determining target movement information adjusted from the first pose information to the second pose information by comparing pose feature differences includes:

[0015] The ORB descriptor based on the red imaging region is determined as the first descriptor, and the ORB descriptor obtained based on the reference region is determined as the second descriptor;

[0016] Determine the target Hamming distance between each first descriptor and each second descriptor, and identify two target feature points whose target Hamming distance is less than a preset distance threshold as feature point pairs;

[0017] A target transformation matrix is ​​constructed based on multiple sets of the aforementioned feature point pairs. The target transformation matrix is ​​used to indicate the perspective transformation relationship from the red imaging region to the reference region.

[0018] Based on the camera intrinsic parameter matrix of the endoscope, the target transformation matrix is ​​decomposed to obtain the target rotation matrix and the target translation vector, wherein the target rotation matrix is ​​used to indicate the rotation angle, and the target translation vector is used to indicate the translation direction and translation distance;

[0019] The target movement information is generated based on the target rotation matrix and the target translation vector.

[0020] According to some embodiments of the present invention, constructing a target transformation matrix based on multiple sets of said feature point pairs includes:

[0021] Construct multiple point pair sets, wherein each point pair set includes multiple randomly selected sets of feature point pairs, and the number of feature point pairs in each point pair set is equal;

[0022] Based on any of the point pair sets, construct the corresponding homography matrix, and test the feature point pairs that are not selected into the current point pair set based on the homography matrix. The points that pass the test are determined as target in-place point pairs.

[0023] The set of point pairs with the most points within the target is determined as the target set, and the homography matrix corresponding to the target set is determined as the target transformation matrix.

[0024] According to some embodiments of the present invention, the target transformation matrix is ​​decomposed based on the camera intrinsic parameter matrix of the endoscope to obtain the target rotation matrix and the target translation vector, including:

[0025] The target transformation matrix is ​​normalized based on the camera intrinsic parameter matrix;

[0026] The target transformation matrix is ​​decomposed by SVD to obtain the initial rotation matrix and the target translation vector;

[0027] The target rotation matrix is ​​obtained by performing SVD decomposition on the initial rotation matrix and then orthogonalizing it.

[0028] According to some embodiments of the present invention, after displaying the target movement information, the method further includes:

[0029] When the endoscope is in motion, the target movement information is updated in real time;

[0030] First motion information and second motion information are determined from multiple target motion information, wherein the frame number difference between the video frames corresponding to the first motion information and the second motion information is a preset difference value;

[0031] The movement trend information of the endoscope is determined based on the second movement information and the first movement information;

[0032] When the movement trend information is used to indicate an increase in location difference, a prompt message is generated based on the second movement information.

[0033] According to some embodiments of the present invention, after determining the current video frame of the real-time video stream as the target image of the target dental plaque, the method further includes:

[0034] Mark the red imaging area corresponding to the target dental plaque;

[0035] The marked red imaging area is ignored in the real-time video stream based on the endoscope.

[0036] In a second aspect, embodiments of the present invention provide a dental plaque imaging device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the dental plaque imaging method as described in the first aspect above.

[0037] Thirdly, embodiments of the present invention provide an electronic device including a dental plaque imaging device as described in the second aspect above.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for performing the dental plaque imaging method as described in the first aspect above.

[0039] The dental plaque imaging method according to embodiments of the present invention has at least the following beneficial effects: when a red imaging area is detected from the real-time video stream of the endoscope, pose feature extraction is performed on the red imaging area to obtain the first pose information of the target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the endoscope light is ultraviolet light; a target reference image is determined from multiple preset reference images based on the shape of the target dental plaque, and pose feature extraction is performed on the reference area of ​​the dental plaque in the target reference image to obtain the second pose information; target movement information from the first pose information to the second pose information is determined by comparing the pose feature differences, and the target movement information is displayed; in response to the endoscope completing movement based on the target movement information, the current video frame of the real-time video stream is determined as the target image of the target dental plaque. According to the technical solution of the present invention, the pose difference between the imaging pose and the reference pose of the target dental plaque can be used to determine the target movement information to prompt the user to move the endoscope, ensuring that the endoscope can align the target dental plaque with a standard pose before imaging, providing an accurate dental plaque image for subsequent identification. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an endoscope system provided in one embodiment of the present invention;

[0041] Figure 2 This is a flowchart of a dental plaque imaging method provided in another embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of a dental plaque imaging device provided in another embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0045] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0046] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0047] This invention provides a method, apparatus, device, and storage medium for capturing dental plaque. The method includes: when a red imaging region is detected in the real-time video stream of an endoscope, performing pose feature extraction on the red imaging region to obtain first pose information of the target dental plaque, wherein the number of target dental plaques is greater than or equal to 1, and the endoscope uses ultraviolet light; determining a target reference image from multiple preset reference images based on the shape of the target dental plaque, and performing pose feature extraction on the reference region of the dental plaque in the target reference image to obtain second pose information; determining target movement information from the first pose information to the second pose information by comparing pose feature differences, and displaying the target movement information; and, in response to the endoscope completing movement based on the target movement information, determining the current video frame of the real-time video stream as the target captured image of the target dental plaque. According to the technical solution of the present invention, the target movement information can be determined by the difference between the imaging pose and the reference pose of the target dental plaque, prompting the user to move the endoscope, ensuring that the endoscope can be aligned with the target dental plaque in a standard pose before taking a picture, and providing accurate dental plaque images for subsequent identification.

[0048] First, refer to Figure 1 , Figure 1 This is a schematic diagram of an endoscope system provided in an embodiment of the present invention. The endoscope system in this embodiment includes a smart terminal 20 and an endoscope 10. The endoscope 10 is equipped with a camera and an LED light. The smart terminal 20 has multiple preset reference images. Each preset reference image uniquely corresponds to a type of dental plaque. The preset reference image is an image of dental plaque taken at a standard shooting angle.

[0049] It should be noted that the smart terminal 20 can be a computer or a tablet computer, as long as it has computing power and display function.

[0050] It should be noted that the preset reference images can be captured and manually labeled. Each preset reference image corresponds to a type of dental plaque, for example... Figure 1 As shown, one preset reference image corresponds to a circular dental plaque, one corresponds to a square dental plaque, and one corresponds to a dental plaque of the opposite sex. Figure 1 The images shown are for illustrative purposes only. Specific preset reference images can be taken in advance according to actual needs, ensuring that each preset reference image corresponds to a standard shooting pose.

[0051] The following is based on the appendix Figure 1 The endoscopic system shown further illustrates the technical solution of the embodiments of the present invention.

[0052] Reference Figure 2 , Figure 2 This is a flowchart of a dental plaque imaging method provided in an embodiment of the present invention. The dental plaque imaging method includes, but is not limited to, the following steps:

[0053] S10, when a red imaging area is detected from the real-time video stream of the endoscope, the pose feature of the red imaging area is extracted to obtain the first pose information of the target dental plaque, wherein the number of target dental plaques is greater than or equal to 1, and the light of the endoscope is violet light.

[0054] It should be noted that the endoscope in this embodiment uses 405nm ultraviolet light. Dental plaque will appear red under 405nm ultraviolet light. In the real-time video stream of the endoscope, it is possible to determine whether a red imaging area appears by simple color recognition.

[0055] It should be noted that the red imaging area does not necessarily have to be an area with a clear closed edge; it can be any color. Of course, in this embodiment, a range of RGB values ​​can be preset, and pixels within that range are determined as the red imaging area.

[0056] It should be noted that in this embodiment, the current video frame is determined as the initial image. The red imaging region is obtained from the initial image through an edge recognition algorithm, and the first pose information of the red imaging region is further determined. For example, the descriptor of the Oriented Fast and Rotated BRIEF (ORB) algorithm can be used as the representation of the pose information, or the Accelerated-KAZE (AKAZE) algorithm can be used to describe the pose information. The ORB algorithm and the AKAZE algorithm are well known to those skilled in the art, and will not be described in detail here.

[0057] It should be noted that there can be multiple dental plaques in the oral cavity. Therefore, the number of target dental plaques in this embodiment is at least one. In the subsequent processing, the processing process for one dental plaque will be described. In the case of multiple target dental plaques, the technical solution of this embodiment can be executed one by one to determine the corresponding target movement information. The multiple target movement information is then fused according to the path order, so that the operator can complete the imaging of the target dental plaques one by one with the prompts of the target movement information.

[0058] S20, a target reference image is determined from multiple preset reference images based on the shape of the target dental plaque, and pose features are extracted from the reference region of the dental plaque in the target reference image to obtain second pose information.

[0059] It should be noted that after extracting the red imaging region, a target reference image can be determined based on the shape of the target dental plaque. This can be done through simple shape similarity. Since the pose of the target dental plaque in the real-time video stream is likely not standard, if the shape of the target dental plaque is the same as a preset reference image, that preset reference image can be excluded. Then, the target reference image with the highest similarity is determined from the remaining preset reference images. For example, if there are multiple elliptical and circular dental plaques in the preset reference images, and the target dental plaque in the real-time video stream is elliptical A, then the preset reference image of elliptical plaque A can be excluded, and the target reference image is determined from the remaining elliptical or circular preset reference images.

[0060] It should be noted that after determining the target reference image, since the dental plaque in the target reference image is a dental plaque in a standard shooting pose, the reference area of ​​the dental plaque can be directly extracted through simple image recognition, and the second pose information can be extracted in the same way as the first pose information. This will not be repeated here.

[0061] S30: By comparing the differences in pose features, determine the target movement information from the first pose information to the second pose information, and display the target movement information.

[0062] It should be noted that the second pose information corresponds to the standard shooting pose. As long as the pose of the target dental plaque meets the second pose information after the endoscope is moved, the target movement information can be determined by comparing the pose feature differences between the first pose information and the second pose information. The target movement information is used to adjust from the first pose information to the second pose information.

[0063] For example, such as Figure 1 As shown, after extracting the red imaging area (the solid area in the figure) from the video frame, the target reference image corresponding to the circular dental plaque is determined. The target movement information determined based on the first pose information and the second pose information is "move 2 cm to the left, move 1 cm down, and rotate 5 degrees clockwise forward". After the operator moves the endoscope according to the target movement information, the target dental plaque in the second pose information can be captured.

[0064] S40, in response to the endoscope completing its movement based on the target movement information, determines the current video frame of the real-time video stream as the target image of the target dental plaque.

[0065] It should be noted that the movement information of the endoscope can be determined through image tracking, which will not be elaborated on here. After confirming that the movement path of the endoscope meets the target movement information, the target dental plaque can be determined to be in the standard shooting pose, and the current video frame in the real-time video stream can be used as the target image for shooting. Alternatively, a shooting button can be set in the endoscope, and the operator can trigger the shooting after judging the pose to be accurate from the display interface of the smart terminal.

[0066] The technical solution of this embodiment can utilize the characteristic that dental plaque appears red under ultraviolet light to extract the red imaging area from the real-time video stream and determine the first pose information. The target reference image is determined by shape similarity, thereby determining the target movement information of the second pose information adjusted to the standard shooting pose. Based on the target movement information, the operator is guided to move the endoscope. After the movement is completed, the target dental plaque at the standard angle is automatically shot, ensuring that the target dental plaque is shot at the standard angle, thereby improving the accuracy and reliability of subsequent identification.

[0067] In another embodiment, the smart terminal is pre-equipped with an ORB detector, and in steps S10 and S20, the pose feature extraction includes:

[0068] S01, Determine the target area, where the target area is the red imaging area or the reference area;

[0069] S02, when the target area is a red imaging area, remove the gingival area whose imaging saturation is lower than the preset saturation in the target area;

[0070] S03, the ORB descriptor corresponding to each of the multiple target feature points in the target region is determined by the ORB detector, and the ORB descriptor is determined as the target pose information, wherein the target pose information is either the first pose information or the second pose information.

[0071] It should be noted that the pose feature extraction in this embodiment is applied in both steps S10 and S20. When applied to step S10, the target area is the red imaging area, and when applied to step S20, the target area is the reference area.

[0072] It is worth noting that when the target area is a red imaging area, since the image is taken under a special ultraviolet light, and there are many types of tissues in the oral cavity, they will also produce different characteristics due to the ultraviolet light. Healthy tooth hard tissue usually shows a bluish, white or light purple fluorescence, while gingival tissue, because it is rich in blood, usually absorbs a lot of ultraviolet light and appears dark red, deep purple or even close to black, and has low saturation. Based on this, tooth hard tissue can be removed by simple color screening. However, if the gingival area is a dark red image, it has a certain similarity in color to dental plaque. This embodiment takes advantage of the low saturation of gingival tissue, sets a preset saturation in the smart terminal, and the part with the imaging saturation lower than the preset saturation is identified as the gingival area, thereby removing the gingival area from the red imaging area and avoiding interference from gingival tissue to the subsequent matching target reference image.

[0073] It should be noted that the ORB detector is a common feature detector. The ORB detector has a fast computation speed, sufficient to meet the requirements of real-time video processing, and is not excessively affected by changes in light uniformity, making it suitable for the complex lighting environment inside the oral cavity. The ORB detector can extract multiple target feature points in the target area and describe each target feature point using ORB descriptors. In this embodiment, the ORB descriptor is defined as the target pose information. When the target area is the red imaging area, the target pose information is the first pose information; when the target area is the reference area, the target pose information is the second pose information.

[0074] In another embodiment, in step S30, the target movement information adjusted from the first pose information to the second pose information is determined by comparing the differences in pose features. This specifically includes, but is not limited to, the following steps:

[0075] S31, the ORB descriptor based on the red imaging region is determined as the first descriptor, and the ORB descriptor obtained based on the reference region is determined as the second descriptor;

[0076] S32, determine the target Hamming distance between each first descriptor and each second descriptor, and determine two target feature points whose target Hamming distance is less than a preset distance threshold as feature point pairs;

[0077] S33, construct the target transformation matrix based on multiple sets of feature point pairs. The target transformation matrix is ​​used to indicate the perspective transformation relationship from the red imaging area to the reference area.

[0078] S34, based on the camera intrinsic parameter matrix decomposition of the endoscope, the target transformation matrix is ​​obtained to obtain the target rotation matrix and the target translation vector, wherein the target rotation matrix is ​​used to indicate the rotation angle, and the target translation vector is used to indicate the translation direction and translation distance;

[0079] S35 generates target movement information based on the target rotation matrix and the target translation vector.

[0080] It should be noted that after determining the first descriptor and the second descriptor, this embodiment uses the Hamming distance as the similarity metric for ORB descriptors. After determining the target Hamming distance for each first descriptor and the second descriptor, since each descriptor corresponds to a feature point, two target feature points whose target Hamming distance is less than the business threshold are determined as feature point pairs. That is, a feature point pair includes a feature point from the red imaging region and a feature point from the reference region, and the two target feature points of the same feature point pair have the highest similarity.

[0081] It should be noted that this embodiment constructs a target transformation matrix based on multiple sets of feature point pairs. The target transformation matrix can be a homography matrix, which represents the perspective transformation relationship from the red imaging area to the target reference area. After obtaining the perspective transformation relationship, given the intrinsic parameter matrix of the endoscope's camera, the target transformation matrix can be decomposed into the camera's target rotation matrix and target translation vector. The target translation vector can be directly the movement vector along the x, y, and z axes, with the direction of the vector being the direction of movement and the magnitude of the vector being the distance of movement. The target rotation matrix can be converted into Euler angles using a formula, thereby obtaining the specific pitch, yaw, and roll angles that need to be moved. By determining the target rotation matrix and target translation vector as the target movement vector, the endoscope can be moved according to the target rotation matrix and target translation vector, transforming the current shooting angle into the standard angle of the reference area.

[0082] In another embodiment, in step S33, the target transformation matrix is ​​constructed based on multiple sets of feature point pairs, which specifically includes, but is not limited to, the following steps:

[0083] S331, construct multiple point pair sets, wherein each point pair set includes multiple randomly selected feature point pairs, and the number of feature point pairs in each point pair set is equal;

[0084] S332, based on any set of point pairs, construct the corresponding homography matrix, test the feature point pairs that are not selected in the current set of point pairs based on the homography matrix, and determine the points that pass the test as the target point pairs;

[0085] S333, the set of point pairs with the most points within the target is determined as the target set, and the homography matrix corresponding to the target set is determined as the target transformation matrix.

[0086] It should be noted that when multiple feature point pairs are initially matched, there are usually some incorrect matches, such as matching similar but different enamel spots. In this embodiment, the feature point pairs are filtered out by the Random Sample Consensus (RANSAC) algorithm. The smallest set is randomly selected from multiple sets of feature point pairs to obtain the point pair set. The number of feature point pairs in a point pair set is 3, and the number of feature point pairs in multiple point pair sets is the same.

[0087] It should be noted that after obtaining multiple point pair sets, this embodiment purchases a homography matrix based on each point pair set. Each homography matrix can describe the perspective transformation relationship from the current frame to the template image, that is, the perspective transformation relationship from the red imaging area to the reference area. In this embodiment, the homography matrix is ​​substituted into each unselected feature point pair. When the feature point pair satisfies the homography matrix, it can be determined as the target in-place point pair of the homography matrix. The more target in-place point pairs there are, the more accurate the perspective transformation relationship represented by the corresponding homography matrix is. Therefore, in this embodiment, the point pair set with the most target in-place point pairs is determined as the target set, and the corresponding homography matrix is ​​determined as the target transformation matrix.

[0088] In another embodiment, in step S34, the target transformation matrix is ​​decomposed based on the camera intrinsic parameter matrix of the endoscope to obtain the target rotation matrix and the target translation vector, which specifically includes, but is not limited to, the following steps:

[0089] S341, Normalize the target transformation matrix based on the camera intrinsic parameter matrix;

[0090] S342, Perform SVD decomposition on the target transformation matrix to obtain the initial rotation matrix and the target translation vector;

[0091] S343, after performing SVD decomposition on the initial rotation matrix, orthogonalization is performed to obtain the target rotation matrix.

[0092] It should be noted that after obtaining the target transformation matrix, decomposing the homography matrix is ​​a crucial step in computer vision for deriving 3D camera motion (rotation and translation) from 2D image mapping. Therefore, in this embodiment, the target transformation matrix is ​​decomposed from the camera intrinsic parameter matrix of an endoscope. The homography matrix has 3 feature point pairs, so the homography matrix in this embodiment is a 3×3 matrix, describing the projection mapping relationship when two different camera viewpoints image the same plane. In the scenario of this embodiment, the plane is the surface of a tooth.

[0093] It should be noted that the camera intrinsic parameter matrix is ​​also a 3×3 matrix, describing the camera's internal parameters, including focal length (f_x, f_y) and principal point coordinates (c_x, c_y). These can be obtained in advance through camera calibration. The expression for the camera intrinsic parameter matrix is ​​as follows: K=[[f_x,0,c_x],[0,f_y,c_y],[0,0,1]], where K is the camera intrinsic parameter matrix.

[0094] It should be noted that during the decomposition process, the homography matrix is ​​first normalized using the camera intrinsic parameter matrix to eliminate the influence of the camera intrinsic parameters, resulting in a homography matrix based on normalized coordinates. This homography matrix can describe the relationship between the poses of the two cameras, thus achieving a unified information dimension for the subsequent determination of the target movement information of the endoscope.

[0095] It should be noted that the expression for the normalized target transformation matrix is ​​as follows: H_norm =λ×(R+(t*n) ) / d), where R is a 3×3 rotation matrix, t is a 3×1 translation vector, n is a 3×1 plane normal vector, d is the distance from the camera to the plane, and λ is a preset scaling factor. After solving for R, t, n, and d, the target rotation matrix and target translation vector of this embodiment can be obtained. In this embodiment, singular value decomposition (SVD) is performed on H_norm to obtain the target rotation matrix and target translation vector. The specific process is as follows:

[0096] With H_norm=[ , , ] indicates that, among them , , Let them be three different column vectors. =1 / || × ||and =1 / || × ||, take λ=( + ) / 2 as an estimate of the scale factor, define = λ × × , define = λ × × , define = × (vector cross product to obtain the third orthogonal basis), define t = λ × × , define n = (in this case, the initial assumption is that the normal vector of the plane is roughly parallel to the optical axis), based on this, an initial rotation matrix = , , is obtained. Perform SVD decomposition on the 3x3 initial rotation matrix obtained above: = U × S / × , where is the rotation performed first, S represents scaling, and U represents the second rotation. After obtaining the initial rotation matrix , orthogonalize the initial rotation matrix to the optimal matrix. The obtained optimal matrix is the target rotation matrix of this embodiment. That is, the target rotation matrix represents the specific operations of rotation and scaling. The specific orthogonalization process is a technique well-known to those skilled in the art and will not be elaborated here.

[0097] It should be noted that after obtaining the target rotation matrix and the target translation vector, the target rotation matrix can be converted into Euler angles to generate accurate guiding instructions as target movement information.

[0098] Exemplarily, taking the target translation vector as (t_x, t_y, t_z) as an example, when t_x > the preset x-axis threshold, the corresponding movement information is "Please move the lens to the right"; when t_x < the x-axis threshold, the corresponding movement information is "Please move the lens to the left"; when t_y > the preset y-axis threshold, the corresponding movement information is "Please move the lens downwards", / ; when t_y < the y-axis threshold, the corresponding movement information is "Please move the lens upwards"; when t_z > the preset z-axis threshold, the corresponding movement information is "You are too close, please step back slightly", when t_z < the z-axis threshold, the corresponding movement information is "You are too far away, please move forward closer". The Euler angle deviation can be used to indicate clockwise rotation or counterclockwise rotation, etc.

[0099] Additionally, in one embodiment, after performing step S30, it further includes but is not limited to the following steps:

[0100] S36, When the endoscope is in motion, the target movement information is updated in real time;

[0101] S37, determine the first motion information and the second motion information from multiple target motion information, wherein the frame number difference between the video frames corresponding to the first motion information and the second motion information is a preset difference value;

[0102] S38, determine the endoscope's movement trend information based on the second movement information and the first movement information;

[0103] S39, when the movement trend information is used to indicate an expansion of the positional difference, a prompt message is generated based on the second movement information.

[0104] It should be noted that after determining the target movement information, this embodiment uses the movement state of the endoscope as the trigger to update the target movement information in real time. If the endoscope does not move, the target movement information is not updated. The target movement information obtained based on the red imaging area of ​​each frame can be referenced... Figure 2 The descriptions of the embodiments shown will not be repeated here.

[0105] It should be noted that after obtaining multiple consecutive target movement information, this embodiment selects the first and second movement information corresponding to two video frames at intervals of several frames. For example, after obtaining the first movement information, the second movement information is obtained 10 frames later. Calculating at intervals of several frames can effectively reduce the consumption of computing resources. Furthermore, this embodiment determines movement trend information based on the first and second movement information. When the movement trend information indicates an increasing positional difference, i.e., the endoscope has not been adjusted to the standard shooting pose of the target dental plaque, a prompt message is generated based on the second movement information to prompt the operator to perform the operation according to the prompt.

[0106] In another embodiment, after step S40 is performed, the following steps are included, but are not limited to:

[0107] S41, mark the red imaging area corresponding to the target dental plaque;

[0108] S42 ignores the marked red imaging area in the endoscope-based real-time video stream.

[0109] It should be noted that this embodiment can include multiple target dental plaques. After capturing an image of one target dental plaque, the corresponding target dental plaque needs to be masked to avoid generating further target movement information that could mislead the operator. This embodiment marks the red imaging area. For example, the position of the red imaging area relative to the hard tissue of the tooth can be recorded so that the endoscope can move away and then move to the corresponding target dental plaque. Alternatively, the marking can be triggered using the hard tissue of the tooth as a reference, thereby ignoring the marked red imaging area in the real-time video stream and avoiding repeated capturing of the same target dental plaque.

[0110] like Figure 3 As shown, Figure 3 This is a structural diagram of a dental plaque imaging device provided in one embodiment of the present invention. The present invention also provides a dental plaque imaging device, comprising:

[0111] The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0112] The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and called and executed by the processor 401 using the dental plaque imaging method of the embodiments of this application.

[0113] Input / output interface 403 is used to implement information input and output;

[0114] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0115] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0116] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0117] This application also provides an electronic device, including the dental plaque imaging device described above.

[0118] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described dental plaque imaging method.

[0119] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0121] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for photographing dental plaque, characterized in that, A smart terminal for use in an endoscope system, the endoscope system further including an endoscope communicatively connected to the smart terminal, the smart terminal having multiple preset reference images, each preset reference image uniquely corresponding to a type of dental plaque, the preset reference images being images of dental plaque taken at a standard shooting angle, the method comprising: When a red imaging area is detected from the real-time video stream of the endoscope, the pose feature of the red imaging area is extracted to obtain the first pose information of the target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is violet light. Based on the shape of the target dental plaque, a target reference image is determined from multiple preset reference images, and pose features are extracted from the reference region of the dental plaque in the target reference image to obtain second pose information; The target movement information is determined by comparing the differences in pose features, and the target movement information is then displayed. In response to the endoscope completing its movement based on the target movement information, the current video frame of the real-time video stream is determined as the target image of the target dental plaque; wherein, the second pose information corresponds to the standard shooting pose; After displaying the target movement information, the method further includes: When the endoscope is in motion, the target movement information is updated in real time; First motion information and second motion information are determined from multiple target motion information, wherein the frame number difference between the video frames corresponding to the first motion information and the second motion information is a preset difference value; The movement trend information of the endoscope is determined based on the second movement information and the first movement information; When the movement trend information is used to indicate an increase in location difference, a prompt message is generated based on the second movement information.

2. The method for photographing dental plaque according to claim 1, characterized in that, The smart terminal is pre-installed with an ORB detector, and the pose feature extraction includes: Determine the target area, wherein the target area is the red imaging area or the reference area; When the target area is the red imaging area, remove the gingival area in the target area where the imaging saturation is lower than the preset saturation; The ORB detector determines the ORB descriptors corresponding to multiple target feature points in the target region, and the ORB descriptors are determined as target pose information, wherein the target pose information is either the first pose information or the second pose information.

3. The method for photographing dental plaque according to claim 2, characterized in that, Determining target movement information by comparing pose feature differences from the first pose information to the second pose information includes: The ORB descriptor based on the red imaging region is determined as the first descriptor, and the ORB descriptor obtained based on the reference region is determined as the second descriptor; Determine the target Hamming distance between each first descriptor and each second descriptor, and identify two target feature points whose target Hamming distance is less than a preset distance threshold as feature point pairs; A target transformation matrix is ​​constructed based on multiple sets of the aforementioned feature point pairs. The target transformation matrix is ​​used to indicate the perspective transformation relationship from the red imaging region to the dormitory reference region. Based on the camera intrinsic parameter matrix of the endoscope, the target transformation matrix is ​​decomposed to obtain the target rotation matrix and the target translation vector, wherein the target rotation matrix is ​​used to indicate the rotation angle, and the target translation vector is used to indicate the translation direction and translation distance; The target movement information is generated based on the target rotation matrix and the target translation vector.

4. The method for photographing dental plaque according to claim 3, characterized in that, Constructing a target transformation matrix based on multiple sets of feature point pairs, including: Construct multiple point pair sets, wherein each point pair set includes multiple randomly selected sets of feature point pairs, and the number of feature point pairs in each point pair set is equal; Based on any of the point pair sets, construct the corresponding homography matrix, and test the feature point pairs that are not selected into the current point pair set based on the homography matrix. The points that pass the test are determined as target in-place point pairs. The set of point pairs with the most points within the target is determined as the target set, and the homography matrix corresponding to the target set is determined as the target transformation matrix.

5. The method for photographing dental plaque according to claim 3, characterized in that, Based on the camera intrinsic parameter matrix of the endoscope, the target transformation matrix is ​​decomposed to obtain the target rotation matrix and the target translation vector, including: The target transformation matrix is ​​normalized based on the camera intrinsic parameter matrix; The target transformation matrix is ​​decomposed by SVD to obtain the initial rotation matrix and the target translation vector; The target rotation matrix is ​​obtained by performing SVD decomposition on the initial rotation matrix and then orthogonalizing it.

6. The method for photographing dental plaque according to claim 2, characterized in that, After determining the current video frame of the real-time video stream as the target image of the target dental plaque, the method further includes: Mark the red imaging area corresponding to the target dental plaque; The marked red imaging area is ignored in the real-time video stream based on the endoscope.

7. A dental plaque imaging device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the dental plaque imaging method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, Includes the dental plaque imaging device as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the dental plaque imaging method as described in any one of claims 1 to 6.

Images