Dental plaque shooting method and device, equipment and storage medium

Abstract

The invention provides a dental plaque shooting method and device, equipment and a storage medium, and the method comprises the steps: starting an endoscope through an ultraviolet lamp, and determining the first pose information of a target dental plaque according to a red imaging region of a real-time video stream of the endoscope; determining second pose information according to the target reference image of the target dental plaque; determining target movement information adjusted from the first pose information to the second pose information by comparing the pose feature differences; and in response to completion of movement of the endoscope based on the target movement information, determining a current video frame of the real-time video stream as a target shot image of the target dental plaque. According to the technical scheme provided by the embodiment of the invention, the target movement information can be determined by utilizing the pose difference between the shooting pose of the target dental plaque and the reference pose to prompt the user to move the endoscope, so that the endoscope can be ensured to shoot after being aligned with the target dental plaque in the standard pose, and an accurate dental plaque image is provided for subsequent recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent endoscopes, and in particular to a dental plaque shooting method, device, equipment and storage medium. BACKGROUND

[0002] At present, some endoscope systems have integrated LED lights and cameras in endoscopes, which can display the video stream returned by the endoscope in real time in a display device, facilitating doctors to make relevant diagnoses. In the field of oral examination, the detection of dental plaque is an important part. In related technologies, a special color light is emitted by an LED light, and the color characteristics of dental plaque under the special light are reflected in the video stream. Medical personnel locate each dental plaque and take photos by observing with the naked eye.

[0003] However, the angle and distance of the camera will change the surface color and shape of the dental plaque in the image to some extent. The related technology can only adjust the position of the endoscope for shooting by the experience of medical personnel, which may deviate when identifying based on the shape and color of the dental plaque, affecting the auxiliary diagnosis effect based on the dental plaque image. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a dental plaque shooting method, device, equipment and storage medium, which can prompt the moving direction of the endoscope according to the detected dental plaque, and ensure that the dental plaque is shot at the correct shooting pose.

[0005] In a first aspect, the present application provides a dental plaque shooting method applied to a smart terminal of an endoscope system, wherein the endoscope system further comprises an endoscope in communication connection with the smart terminal, the smart terminal is pre-provided with a plurality of preset reference images, each of the preset reference images corresponds to a type of dental plaque, the preset reference image is an image of dental plaque shot at a standard shooting angle, and the method comprises: detecting a red imaging area from a real-time video stream of the endoscope, performing pose feature extraction on the red imaging area to obtain first pose information of a target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is purple light; determining a target reference image from the plurality of preset reference images based on the shape of the target dental plaque, performing pose feature extraction on a reference area of the dental plaque of the target reference image to obtain second pose information; determining target movement information from the first pose information to the second pose information by comparing the pose feature difference, and displaying the target movement information; In response to the endoscope completing the movement based on the target movement information, a current video frame of the real-time video stream is determined as a target shooting image of the target dental plaque.

[0006] According to some embodiments of the present application, the intelligent terminal is preconfigured with an ORB detector, and the pose feature extraction includes: determining a target region, wherein the target region is the red imaging region or the reference region; when the target region is the red imaging region, removing a gingival region with an imaging saturation lower than a preset saturation from the target region; determining, by the ORB detector, a plurality of target feature points each corresponding to an ORB descriptor in the target region, and determining the ORB descriptor as target pose information, wherein the target pose information is the first pose information or the second pose information.

[0007] According to some embodiments of the present application, the target movement information is determined by comparing the pose feature difference from the first pose information to the second pose information, including: determining the ORB descriptor based on the red imaging region as a first descriptor, and determining the ORB descriptor based on the reference region as a second descriptor; determining a target Hamming distance between each of the first descriptors and each of the second descriptors, and determining two of the target feature points with the target Hamming distance less than a preset distance threshold as a feature point pair; constructing a target transformation matrix based on a plurality of the feature point pairs, wherein the target transformation matrix is used to indicate a perspective transformation relationship from the red imaging region to the reference region; decomposing the target transformation matrix based on an intrinsic matrix of a camera of the endoscope to obtain a target rotation matrix and a target translation vector, wherein the target rotation matrix is used to indicate a rotation angle, and the target translation vector is used to indicate a translation direction and a translation distance; generating the target movement information based on the target rotation matrix and the target translation vector.

[0008] According to some embodiments of the present application, the target transformation matrix is constructed based on a plurality of the feature point pairs, including: constructing a plurality of point pair sets, wherein one of the point pair sets includes a plurality of the feature point pairs randomly selected, and the number of the feature point pairs in each of the point pair sets is equal; constructing a corresponding homography matrix based on any of the point pair sets, testing the feature point pairs not selected into the current point pair set based on the homography matrix, and determining a target inlier pair by testing. The point pair set with the most target in-point pairs is determined as a target set, and a homography matrix corresponding to the target set is determined as the target transformation matrix.

[0009] According to some embodiments of the present application, the target transformation matrix is decomposed based on an intrinsic camera matrix of the endoscope to obtain a target rotation matrix and a target translation vector, including: The target transformation matrix is normalized based on the intrinsic camera matrix; The target transformation matrix is SVD decomposed to obtain an initial rotation matrix and the target translation vector; The target rotation matrix is obtained by orthogonalizing the initial rotation matrix after SVD decomposition.

[0010] According to some embodiments of the present application, after the target movement information is displayed, the method further includes: When the endoscope is in a moving state, the target movement information is updated in real time; First movement information and second movement information are determined from a plurality of target movement information, wherein the frame number difference between the video frames corresponding to the first movement information and the second movement information is a preset difference value; Movement trend information of the endoscope is determined based on the second movement information and the first movement information; When the movement trend information is used to indicate that the position difference is expanding, prompt information is generated based on the second movement information.

[0011] According to some embodiments of the present application, after the current video frame of the real-time video stream is determined as the target photographing image of the target dental plaque, the method further includes: The red imaging area corresponding to the target dental plaque is marked; The marked red imaging area is ignored in the real-time video stream based on the endoscope.

[0012] In a second aspect, embodiments of the present application provide a dental plaque photographing device, including at least one control processor and a memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the dental plaque photographing method as described in the first aspect above.

[0013] In a third aspect, embodiments of the present application provide an electronic device including the dental plaque photographing device as described in the second aspect above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium storing computer executable instructions for performing the dental plaque shooting method according to the first aspect.

[0015] According to the dental plaque shooting method provided by the embodiment of the present application, at least the following beneficial effects are achieved: when a red imaging area is detected from the real-time video stream of the endoscope, first pose information of a target dental plaque is extracted from the red imaging area, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is purple light; a target reference image is determined from a plurality of preset reference images based on the shape of the target dental plaque, second pose information is extracted from a reference area of dental plaque of the target reference image; target movement information from the first pose information to the second pose information is determined by comparing the pose feature difference, and the target movement information is displayed; in response to the endoscope completing movement based on the target movement information, a current video frame of the real-time video stream is determined as a target shooting image of the target dental plaque. According to the technical solution of the embodiment of the present application, the pose difference between the shooting pose of the target dental plaque and the reference pose is used to determine the target movement information to prompt the user to move the endoscope, so as to ensure that the endoscope can be aligned with the target dental plaque in a standard pose for shooting, and provide an accurate dental plaque image for subsequent identification. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a schematic diagram of an endoscope system provided by an embodiment of the present application; Figure 2 is a flowchart of a dental plaque shooting method provided by another embodiment of the present application; Figure 3 is a structural diagram of a dental plaque shooting device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0017] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0018] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as a limitation of the present application, which indicates or implies that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation.

[0019] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described that the first, second is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0020] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical scheme.

[0021] The embodiment of the present application provides a dental plaque shooting method, device, equipment and storage medium, wherein the dental plaque shooting method comprises: when a red imaging area is detected from a real-time video stream of the endoscope, performing pose feature extraction on the red imaging area to obtain first pose information of a target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is purple light; determining a target reference image from a plurality of preset reference images based on the shape of the target dental plaque, performing pose feature extraction on a reference area of dental plaque of the target reference image to obtain second pose information; determining target movement information from the first pose information to the second pose information by comparing the pose feature difference, displaying the target movement information; in response to the endoscope completing movement based on the target movement information, determining a current video frame of the real-time video stream as a target shooting image of the target dental plaque. According to the technical scheme of the embodiment of the present application, the pose difference between the shooting pose of the target dental plaque and the reference pose is used to determine the target movement information to prompt the user to move the endoscope, so that the endoscope can be aligned with the target dental plaque at a standard pose for shooting, and accurate dental plaque images are provided for subsequent identification.

[0022] Firstly, refer to Figure 1 , Figure 1 The present embodiment provides a schematic diagram of an endoscope system, the endoscope system of the present embodiment comprises a smart terminal 20 and an endoscope 10, the endoscope 10 is provided with a camera and an LED lamp, the smart terminal 20 is pre-provided with a plurality of preset reference images, each preset reference image corresponds to a type of dental plaque uniquely, and the preset reference image is an image of dental plaque shot at a standard shooting angle.

[0023] It should be noted that the smart terminal 20 can be a computer or a tablet computer, which can have computing processing capability and display function.

[0024] It should be noted that the preset reference image can be realized by shooting and manually labeling, and each preset reference image corresponds to a type of dental plaque, for example Figure 1 As shown in the figure, one preset reference image corresponds to a circular dental plaque, one corresponds to a square dental plaque, and one corresponds to an irregular dental plaque. Figure 1 The specific preset reference image can be pre-shot according to actual needs, and it is only necessary to ensure that each preset reference image corresponds to a standard shooting pose.

[0025] The technical solutions of the embodiments of the present application will be further described below based on the endoscope system shown in the figure. Figure 1

[0026] Referring to the figure Figure 2 , Figure 2 A flowchart of a dental plaque shooting method provided by the embodiment of the present application, which includes but is not limited to the following steps: S10, when a red imaging area is detected from the real-time video stream of the endoscope, the first pose information of the target dental plaque is obtained by extracting the pose features of the red imaging area, wherein the number of target dental plaque is greater than or equal to 1, and the light of the endoscope is purple light.

[0027] It should be noted that the endoscope of the present embodiment uses purple light with a wavelength of 405nm, and the dental plaque will show red under the irradiation of 405nm purple light. In the real-time video stream of the endoscope, whether a red imaging area appears can be determined by simple color recognition.

[0028] It should be noted that the red imaging area is not necessarily an area with obvious closed edges, but only the corresponding color, of course, the present embodiment can pre-set the interval of RGB value, and the pixels within the interval are determined as the red imaging area.

[0029] It should be noted that the present embodiment determines the current video frame as the initial image, obtains the red imaging area from the initial image through the edge recognition algorithm, and further determines the first pose information of the red imaging area. For example, the descriptor of the Oriented FAST and Rotated BRIEF (ORB) algorithm can be used as the representation of the pose information, or the Accelerated-KAZE (AKAZE) algorithm can be used to describe the pose information. The ORB algorithm and the AKAZE algorithm are well known to those skilled in the art, and will not be described here.

[0030] ​It should be noted that the number of dental plaque in the oral cavity can be multiple, and the number of target dental plaque in the embodiment is at least one. In the subsequent processing process, the processing process for one dental plaque is described. In the case of multiple target dental plaques, the technical solution of the embodiment can be executed one by one to determine the corresponding target movement information, and the multiple target movement information is path fused according to the path order, so that the operator can complete the shooting of the target dental plaque one by one under the prompt of the target movement information.

[0031] S20, determining a target reference image from the shape of the target dental plaque from multiple preset reference images, and extracting pose features based on the reference area of the dental plaque of the target reference image to obtain second pose information.

[0032] It should be noted that after the red imaging area is extracted, the target reference image can be determined according to the shape of the target dental plaque. The target reference image can be determined by simple shape similarity. Since the pose of the target dental plaque in the real-time video stream is most likely not standard, if the shape of the target dental plaque is the same as a preset reference image, the preset reference image can be excluded, and the target reference image with the highest similarity can be determined from the remaining preset reference images. For example, there are multiple elliptical and circular dental plaques in the preset reference image. When the target dental plaque in the real-time video stream is an ellipse A, the preset reference image of the elliptical dental plaque can be excluded, and the target reference image can be determined from the remaining elliptical or circular preset reference images.

[0033] It should be noted that after the target reference image is determined, since the dental plaque of the target reference image is a dental plaque in a standard shooting pose, the reference area of the dental plaque can be directly extracted by simple pattern recognition, and the second pose information can be extracted in the same way as the first pose information. This will not be repeated here.

[0034] S30, determining the target movement information from the first pose information to the second pose information by comparing the pose feature difference, and displaying the target movement information.

[0035] It should be noted that the second pose information corresponds to a standard shooting pose. As long as the pose of the target dental plaque meets the second pose information after moving the endoscope, the target movement information can be determined by comparing the pose feature difference corresponding to the first pose information and the second pose information. The target movement information is used to adjust from the first pose information to the second pose information.

[0036] For example, as Figure 1As shown, after the red imaging area (solid area in the figure) is extracted from the video frame, the target reference image corresponding to the circular plaque is determined, and the target movement information determined according to the first pose information and the second pose information is "2 cm left, 1 cm down, and 5 degrees clockwise forward". After the operator moves the endoscope according to the target movement information, the target plaque in the second pose information can be photographed.

[0037] S40, in response to the endoscope completing the movement based on the target movement information, determining the current video frame of the real-time video stream as the target shooting image of the target plaque.

[0038] It should be noted that the movement information of the endoscope can be determined by image tracking, which will not be described in detail here. After determining that the movement path of the endoscope meets the target movement information, it can be determined that the target plaque is in the standard shooting pose, and the current video frame in the real-time video stream can be taken as the target shooting image. Of course, a shooting button can also be set in the endoscope, and the pose can be manually determined from the display interface of the intelligent terminal to trigger shooting.

[0039] Through the technical scheme of the embodiment, the characteristics of the plaque imaging as red under the purple light can be used to extract the red imaging area from the real-time video stream and determine the first pose information, to determine the target reference image by shape similarity, to determine the target movement information of the second pose information adjusted to the standard shooting pose, to guide the operator to move the endoscope according to the target movement information, to automatically shoot the target plaque at the standard angle after moving, to ensure that the target plaque is shot at the standard angle, and to improve the accuracy and reliability of subsequent identification.

[0040] In addition, in an embodiment, the intelligent terminal is pre-provided with an ORB detector, and in steps S10 and S20, the pose feature extraction includes: S01, determining a target area, wherein the target area is a red imaging area or a reference area; S02, when the target area is a red imaging area, removing the gingival area with an imaging saturation lower than a preset saturation from the target area; S03, determining a plurality of target feature points each corresponding to an ORB descriptor in the target area through the ORB detector, and determining the ORB descriptor as the target pose information, wherein the target pose information is the first pose information or the second pose information.

[0041] It should be noted that the pose feature extraction of the embodiment is applied in steps S10 and S20. When applied to step S10, the target area is a red imaging area, and when applied to step S20, the target area is a reference area.

[0042] It is worth noting that when the target region is the red imaging region, since the image is taken under a special violet light, there are many types of tissues in the oral cavity, and different characteristics will be generated due to the illumination of the violet light. Healthy tooth hard tissues usually present blue, white or light purple fluorescence, while gingival tissues, which are rich in blood, usually absorb a large amount of violet light and present dark red, deep purple or even close to black, and have low saturation. Based on this, the tooth hard tissues can be removed by simple color screening, and if the gingival region is dark red imaging, the color has a certain similarity with dental plaque. In this embodiment, the low saturation of the gingival tissue is used to set a preset saturation in the intelligent terminal. The part with a saturation lower than the preset saturation is determined as the gingival region, so as to remove the gingival region from the red imaging region and avoid the interference of the gingival tissue on the subsequent matching target reference image.

[0043] It should be noted that the ORB detector is a common feature detector. The ORB detector has a fast calculation speed and can meet the requirements of real-time video processing, and is not excessively affected by the uniformity change of light, and is suitable for the complex illumination environment in the oral cavity. A plurality of target feature points can be extracted from the target region by the ORB detector, and each target feature point is described in the form of an ORB descriptor. In this embodiment, the ORB descriptor is determined as the target pose information. When the target region is the red imaging region, the target pose information is the first pose information, and when the target region is the reference region, the target pose information is the second pose information.

[0044] In addition, in an embodiment, in step S30, the target movement information from the first pose information to the second pose information is determined by comparing the pose feature difference, and specifically includes but is not limited to the following steps: S31, determining the first descriptor based on the ORB descriptor of the red imaging region as the first descriptor, and determining the second descriptor based on the ORB descriptor obtained from the reference region as the second descriptor; S32, determining the target Hamming distance between each first descriptor and each second descriptor, and determining two target feature points with a target Hamming distance less than a preset distance threshold as a feature point pair; S33, constructing a target transformation matrix based on a plurality of feature point pairs, the target transformation matrix being used to indicate the perspective transformation relationship from the red imaging region to the reference region; S34, decomposing the target transformation matrix based on the camera intrinsic matrix of the endoscope to obtain a target rotation matrix and a target translation vector, wherein the target rotation matrix is used to indicate the rotation angle, and the target translation vector is used to indicate the translation direction and the translation distance; S35, generating the target movement information based on the target rotation matrix and the target translation vector.

[0045] It should be noted that after the first descriptor and the second descriptor are determined, the embodiment takes the Hamming distance as the similarity measurement standard of the ORB descriptor, determines the target Hamming distance of each first descriptor and second descriptor, and since each descriptor corresponds to a feature point, two target feature points with a target Hamming distance less than a commercial threshold are determined as a feature point pair, that is, a feature point pair includes one feature point from the red imaging area and one feature point from the reference area, and the similarity of the two target feature points of the same feature point pair is the highest.

[0046] It should be noted that the embodiment constructs a target transformation matrix based on multiple sets of feature point pairs, and the target transformation matrix can be a homography matrix, so that the target transformation matrix represents the perspective transformation relationship from the red imaging area to the target reference area. After obtaining the perspective transformation relationship, in the case where the camera intrinsic matrix of the endoscope is known, the target transformation matrix can be decomposed into a target rotation matrix and a target translation vector of the camera, the target translation vector can be directly the movement vectors of the x-axis, y-axis and z-axis, the direction of the vector is the movement direction, and the modulus of the vector is the movement distance. The target rotation matrix can be converted into Euler angles by the formula, so as to obtain the specific pitch angle, yaw angle and roll angle that need to be moved, and the target rotation matrix and the target translation vector are determined as the target movement vector, so that the endoscope can be moved according to the target rotation matrix and the target translation vector, and can be converted from the current shooting angle to the standard angle of the reference area.

[0047] In addition, in an embodiment, in step S33, a target transformation matrix is constructed based on multiple sets of feature point pairs, and specifically includes but is not limited to the following steps: S331, a plurality of point pair sets are constructed, wherein a point pair set includes a plurality of sets of feature point pairs selected at random, and the number of feature point pairs in each point pair set is equal; S332, based on any point pair set, a corresponding homography matrix is constructed, and based on the homography matrix, a feature point pair not selected into the current point pair set is tested, and a target in-point pair determined by the test is determined; S333, a point pair set with the most target in-point pairs is determined as a target set, and a homography matrix corresponding to the target set is determined as a target transformation matrix.

[0048] It should be noted that when a plurality of feature point pairs are initially matched, some incorrect matches are usually included, such as matching similar but different enamel spots. The embodiment filters the feature point pairs by using a random sample consensus (RANSAC) algorithm, randomly selects a minimum set from a plurality of sets of feature point pairs to obtain a point pair set, the number of feature point pairs in a point pair set is 3 pairs, and the number of feature point pairs in a plurality of point pair sets is the same.

[0049] It should be noted that after obtaining a plurality of point pair sets, the embodiment purchases a homography matrix based on each point pair set, and each homography matrix can describe the perspective transformation relationship from the current frame to the template image, that is, the perspective transformation relationship from the red imaging area to the reference area. The embodiment substitutes the homography matrix into each unselected feature point pair. When the feature point pair satisfies the homography matrix, it can be determined as a target in-point pair of the homography matrix. When the target in-point pair is more, the perspective transformation relationship represented by the corresponding homography matrix is more accurate. Therefore, the embodiment determines the point pair set with the most target in-point pairs as the target set, and determines the corresponding homography matrix as the target transformation matrix.

[0050] In addition, in an embodiment, in step S34, the target transformation matrix is decomposed based on the camera intrinsic matrix of the endoscope to obtain a target rotation matrix and a target translation vector, and specifically includes but is not limited to the following steps: S341, normalizing the target transformation matrix based on the camera intrinsic matrix; S342, performing SVD decomposition on the target transformation matrix to obtain an initial rotation matrix and a target translation vector; S343, performing orthogonalization processing on the initial rotation matrix after SVD decomposition to obtain the target rotation matrix.

[0051] It should be noted that after obtaining the target transformation matrix, decomposing the homography matrix is a key step in computer vision for deriving 3D camera motion (rotation and translation) from 2D image mapping. Therefore, the embodiment decomposes the target transformation matrix based on the camera intrinsic matrix of the endoscope. The number of feature point pairs of the homography matrix is 3, so the homography matrix of the embodiment is a 3x3 matrix, which describes the projection mapping relationship when the same plane is imaged from two different camera perspectives. In the scenario of the embodiment, the plane is the surface of the tooth.

[0052] It should be noted that the camera intrinsic matrix is also a 3x3 matrix, which describes the internal parameters of the camera, including focal length (f_x, f_y), principal point coordinates (c_x, c_y). The camera intrinsic matrix can be obtained in advance through camera calibration. The expression of the camera intrinsic matrix is as follows: K = [[f_x, 0, c_x], [0, f_y, c_y], [0, 0, 1]], and K is the camera intrinsic matrix.

[0053] It should be noted that in the decomposition process, the homography matrix is first normalized by the camera intrinsic matrix to eliminate the influence of the camera intrinsic parameters, and a homography matrix based on normalized coordinates is obtained, so that the homography matrix can describe the relationship between the two camera poses, and the information dimension is unified for subsequent determination of the target movement information of the endoscope.

[0054] It should be noted that the expression of the normalized target transformation matrix is as follows: H norm = λ × (R + (t * n ) / d), wherein R is a 3x3 rotation matrix, t is a 3x1 translation vector, n is a 3x1 plane normal vector, d is a distance from the camera to the plane, and λ is a preset scaling factor. After R, t, n, and d are solved, the target rotation matrix and the target translation vector of the embodiment can be obtained. The embodiment performs singular value decomposition (SVD) on H norm to obtain the target rotation matrix and the target translation vector, and the specific process is as follows: H norm is expressed as follows: , , , , , are three different column vectors, let =1 / || × || and =1 / || × ||, take λ=( + ) / 2 as an estimate of the scale factor, define =λ× × , define =λ× × , define = × (the vector cross product, to obtain a third orthogonal basis), define t=λ× × , and define n= (in this case, it is initially assumed that the plane normal vector is approximately parallel to the optical axis), based on which an initial rotation matrix =[ , , ] is obtained. The 3x3 initial rotation matrix obtained above is subjected to SVD decomposition: =U×S / × , wherein is the first rotation, S represents scaling, and U represents the second rotation. After the initial rotation matrix is obtained, the initial rotation matrix The orthogonalization of the rotation matrix is the optimal matrix, and the optimal matrix is the target rotation matrix of the embodiment, i.e., the target rotation matrix represents the specific operations of rotation and scaling. The specific orthogonalization process is a technique familiar to those skilled in the art, and will not be described here.

[0055] It should be noted that after obtaining the target rotation matrix and the target translation vector, the target rotation matrix can be converted into Euler angles to generate accurate guide instructions as target movement information.

[0056] Exemplarily, taking the target translation vector (t_x, t_y, t_z) as an example, when t_x> the preset x-axis threshold, the corresponding movement information is "please move the lens to the right"; when t_x< the x-axis threshold, the corresponding movement information is "please move the lens to the left"; when t_y> the preset y-axis threshold, the corresponding movement information is "please move the lens downward"; when t_y< the y-axis threshold, the corresponding movement information is "please move the lens upward"; when t_z> the preset z-axis threshold, the corresponding movement information is "you are too close, please move back a little"; when t_z< the z-axis threshold, the corresponding movement information is "you are too far away, please move forward to get closer". The Euler angle deviation can be used to indicate clockwise rotation or counterclockwise rotation, etc.

[0057] In addition, in an embodiment, after step S30 is performed, the following steps are further included, but are not limited to: S36, when the endoscope is in a moving state, updating the target movement information in real time; S37, determining first movement information and second movement information from the plurality of target movement information, wherein the frame number difference between the video frames corresponding to the first movement information and the second movement information is a preset difference; S38, determining movement trend information of the endoscope based on the second movement information and the first movement information; S39, when the movement trend information is used to indicate that the position difference is expanding, generating prompt information based on the second movement information.

[0058] It should be noted that after determining the target movement information, the embodiment updates the target movement information in real time based on the moving state of the endoscope. When the endoscope does not move, the target movement information is not updated. The target movement information obtained based on each frame of red imaging area can refer to the embodiment described in Figure 2 The descriptions of the embodiments described in the above are not repeated here.

[0059] It should be noted that after obtaining multiple consecutive target movement information, this embodiment selects the first and second movement information corresponding to two video frames at intervals of several frames. For example, after obtaining the first movement information, the second movement information is obtained 10 frames later. Calculating at intervals of several frames can effectively reduce the consumption of computing resources. Furthermore, this embodiment determines movement trend information based on the first and second movement information. When the movement trend information indicates an increasing positional difference, i.e., the endoscope has not been adjusted to the standard shooting pose of the target dental plaque, a prompt message is generated based on the second movement information to prompt the operator to perform the operation according to the prompt.

[0060] In another embodiment, after step S40 is performed, the following steps are included, but are not limited to: S41, mark the red imaging area corresponding to the target dental plaque; S42 ignores the marked red imaging area in the endoscope-based real-time video stream.

[0061] It should be noted that this embodiment can include multiple target dental plaques. After capturing an image of one target dental plaque, the corresponding target dental plaque needs to be masked to avoid generating further target movement information that could mislead the operator. This embodiment marks the red imaging area. For example, the position of the red imaging area relative to the hard tissue of the tooth can be recorded so that the endoscope can move away and then move to the corresponding target dental plaque. Alternatively, the marking can be triggered using the hard tissue of the tooth as a reference, thereby ignoring the marked red imaging area in the real-time video stream and avoiding repeated capturing of the same target dental plaque.

[0062] like Figure 3 As shown, Figure 3 This is a structural diagram of a dental plaque imaging device provided in one embodiment of the present invention. The present invention also provides a dental plaque imaging device, comprising: The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 402 and are called and executed by the processor 401 to implement the dental plaque shooting method of the embodiments of the present application; The input / output interface 403 is configured to realize information input and output. The communication interface 404 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 405 is configured to transmit information between various components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device. The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other through the bus 405 to realize the communication connection between the device.

[0063] The embodiments of the present application also provide an electronic device comprising the dental plaque shooting device as described above.

[0064] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium. The storage medium stores a computer program. The computer program is executed by a processor to implement the dental plaque shooting method described above.

[0065] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The above-described device embodiments are only schematic, and units described as separate components can or can not be physically separated, and can be implemented in one place or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0066] As will be appreciated by one of ordinary skill in the art, all or some of the steps, systems, etc. in the above-disclosed methods can be embodied in software, firmware, hardware, and / or suitable combinations thereof. Some or all of the physical components can be implemented with software executed by a processor, such as a central processing unit, a digital signal processor, or microprocessor, or can be implemented as hardware, or as an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media), and communication media (or transitory media). As is well known to those of ordinary skill in the art, computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves or other transport mechanisms, and includes any information delivery media.

[0067] The above description is that of the preferred embodiments of the application. Various modifications and changes can be made thereto without departing from the spirit and scope of the application, which is to be given the broadest interpretation of the laws.

Claims

1. A method for photographing dental plaque, characterized in that, A smart terminal for use in an endoscope system, the endoscope system further comprising an endoscope communicatively connected to the smart terminal, the smart terminal having multiple preset reference images, each preset reference image uniquely corresponding to a type of dental plaque [CY1], the preset reference images being images of dental plaque taken at a standard shooting angle, the method comprising: When a red imaging area is detected from the real-time video stream of the endoscope, the pose feature of the red imaging area is extracted to obtain the first pose information of the target dental plaque, wherein the number of the target dental plaque is greater than or equal to 1, and the light of the endoscope is violet light. Based on the shape of the target dental plaque, a target reference image is determined from multiple preset reference images, and pose features are extracted from the reference region of the dental plaque in the target reference image to obtain second pose information; The target movement information is determined by comparing the differences in pose features, and the target movement information is then displayed. In response to the endoscope completing its movement based on the target movement information, the current video frame of the real-time video stream is determined as the target image of the target dental plaque.

2. The method for photographing dental plaque according to claim 1, characterized in that, The smart terminal is pre-installed with an ORB detector, and the pose feature extraction includes: Determine the target area, wherein the target area is the red imaging area or the reference area; When the target area is the red imaging area, remove the gingival area in the target area where the imaging saturation is lower than the preset saturation; The ORB detector determines the ORB descriptors corresponding to multiple target feature points in the target region, and the ORB descriptors are determined as target pose information, wherein the target pose information is either the first pose information or the second pose information.

3. The method for photographing dental plaque according to claim 2, characterized in that, Determining target movement information by comparing pose feature differences from the first pose information to the second pose information includes: The ORB descriptor based on the red imaging region is determined as the first descriptor, and the ORB descriptor obtained based on the reference region is determined as the second descriptor; Determine the target Hamming distance between each first descriptor and each second descriptor, and identify two target feature points whose target Hamming distance is less than a preset distance threshold as feature point pairs; A target transformation matrix is ​​constructed based on multiple sets of the aforementioned feature point pairs. The target transformation matrix is ​​used to indicate the perspective transformation relationship from the red imaging region to the reference region. Based on the camera intrinsic parameter matrix of the endoscope, the target transformation matrix is ​​decomposed to obtain the target rotation matrix and the target translation vector, wherein the target rotation matrix is ​​used to indicate the rotation angle, and the target translation vector is used to indicate the translation direction and translation distance; The target movement information is generated based on the target rotation matrix and the target translation vector.

4. The method for photographing dental plaque according to claim 3, characterized in that, Constructing a target transformation matrix based on multiple sets of feature point pairs, including: Construct multiple point pair sets, wherein each point pair set includes multiple randomly selected sets of feature point pairs, and the number of feature point pairs in each point pair set is equal; Based on any of the point pair sets, construct the corresponding homography matrix, and test the feature point pairs that are not selected into the current point pair set based on the homography matrix. The points that pass the test are determined as target in-place point pairs. The set of point pairs with the most points within the target is determined as the target set, and the homography matrix corresponding to the target set is determined as the target transformation matrix.

5. The method for photographing dental plaque according to claim 3, characterized in that, Based on the camera intrinsic parameter matrix of the endoscope, the target transformation matrix is ​​decomposed to obtain the target rotation matrix and the target translation vector, including: The target transformation matrix is ​​normalized based on the camera intrinsic parameter matrix; The target transformation matrix is ​​decomposed by SVD to obtain the initial rotation matrix and the target translation vector; The target rotation matrix is ​​obtained by performing SVD decomposition on the initial rotation matrix and then orthogonalizing it.

6. The method for photographing dental plaque according to any one of claims 1 to 5, characterized in that, After displaying the target movement information, the method further includes: When the endoscope is in motion, the target movement information is updated in real time; First motion information and second motion information are determined from multiple target motion information, wherein the frame number difference between the video frames corresponding to the first motion information and the second motion information is a preset difference value; The movement trend information of the endoscope is determined based on the second movement information and the first movement information; When the movement trend information is used to indicate an increase in location difference, a prompt message is generated based on the second movement information.

7. The method for photographing dental plaque according to claim 2, characterized in that, After determining the current video frame of the real-time video stream as the target image of the target dental plaque, the method further includes: Mark the red imaging area corresponding to the target dental plaque; The marked red imaging area is ignored in the real-time video stream based on the endoscope.

8. A dental plaque imaging device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the dental plaque imaging method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, Includes the dental plaque imaging device as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the dental plaque imaging method as described in any one of claims 1 to 7.

Images