Oral three-dimensional image generation method, system and electronic device

Abstract

The present application provides a method, system and electronic device for generating an oral three-dimensional image, which relates to the field of image processing; the method for generating an oral three-dimensional image comprises: extracting feature points from multiple oral images using a preset feature extraction network; inputting the multiple oral images and the feature points into a matching network to obtain a matching relationship between a reference frame and a non-reference frame; wherein the non-reference frame comprises other frames in the multiple oral images except the reference frame; calculating the position and posture relationship between the reference frame and the non-reference frame according to the feature points and the matching relationship; generating an oral three-dimensional image according to the multiple oral images and the position and posture relationship. Using the oral three-dimensional image generation method provided in the embodiment of the present application, it is possible to overcome the problem of image matching difficulties caused by repeated tooth textures and reconstruct a more accurate oral three-dimensional image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an oral three-dimensional image method, system and electronic device. Background Art

[0002] Three-dimensional visualization technology of medical images is increasingly used in modern medicine. In order to include more human information in the three-dimensional structure, three-dimensional image reconstruction is often performed based on the collected human information. Accurate and high-quality three-dimensional reconstructed images can make a great contribution to medical services.

[0003] Oral 3D image generation or oral image reconstruction refers to the process of generating / reconstructing the 3D structure of teeth from photographed teeth images. With the continuous development of deep learning algorithms, 3D reconstruction algorithms and neural network rendering algorithms have made breakthrough progress and have shown great advantages over traditional algorithms. However, due to the complexity of oral conditions and the large differences in the poses of collected oral images, oral 3D image generation / reconstruction can easily fail. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, system and electronic device for generating an oral 3D image, which uses a feature extraction network with added position coding and global information of the image to greatly solve the matching difficulty caused by the complex oral situation (for example, repeated texture of teeth); in addition, the scale transformation is performed on the posture information of different images so that images with large posture differences can be used for 3D image reconstruction. It can overcome the problem of image matching difficulty caused by repeated texture of teeth and reconstruct a more accurate oral 3D image.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an oral three-dimensional image, which includes: extracting feature points from multiple oral images using a preset feature extraction network; inputting multiple oral images and feature points into a matching network to obtain a matching relationship between a reference frame and a non-reference frame; wherein the non-reference frame includes other frames in the multiple oral images except the reference frame; calculating the pose relationship between the reference frame and the non-reference frame based on the feature points and the matching relationship; and generating an oral three-dimensional image based on the multiple oral images and the pose relationship.

[0006] In the above implementation process, in order to generate an oral three-dimensional image, the oral three-dimensional image generation method provided in the embodiment of the present application first extracts feature points from a number of oral images; after extracting the feature points, the feature points are matched; further, the posture relationship between different images is obtained, and the image is reconstructed based on the posture relationship and the oral image. It can be seen that the use of the oral three-dimensional image generation method provided in the embodiment of the present application can not only overcome the matching difficulty caused by repeated tooth textures, but also use images with large posture differences for image reconstruction, and on this basis establish a three-dimensional image with higher accuracy.

[0007] Optionally, in an embodiment of the present application, extracting feature points from multiple oral images using a preset feature extraction network includes: obtaining position codes and global feature vectors corresponding to the multiple oral images based on the multiple oral images; adding the position codes and global feature vectors to the network structure of the feature extraction network to obtain a preset feature extraction network; and inputting the multiple oral images into the preset feature extraction network to extract feature points from the multiple oral images.

[0008] In the above implementation process, in order to extract feature points from multiple oral images, the embodiment of the present application adds position coding and image global information to the feature extraction network to obtain a preset feature extraction network; and then extracts features from multiple oral images through the feature extraction network. It can be seen that the embodiment of the present application adds global information and position information to the feature extraction network, so that the feature extraction network obtains the specific position of each tooth in the entire oral cavity, thereby avoiding tooth matching errors, such as the problem of left teeth being mistakenly matched to right teeth.

[0009] Optionally, in an embodiment of the present application, calculating the pose relationship between the reference frame and the non-reference frame based on the feature points and the matching relationship includes: calculating the initial pose relationship between the reference frame and each non-reference frame respectively based on the matching relationship; and correcting the scale information of the initial pose relationship based on the matching relationship and the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame.

[0010] In the above implementation process, there is an initial pose relationship between the reference frame and each non-reference frame. Such a pose relationship can be represented by a matrix and a vector. Furthermore, the scale information of the initial pose relationship calculated by correction is calculated based on the matching relationship and the initial pose relationship, so that the scale information is unified, thereby overcoming the problem that pictures with large pose differences cannot be used for image reconstruction.

[0011] Optionally, in an embodiment of the present application, the matching relationship includes a first matching relationship between the target frame and the reference frame and a plurality of second matching relationships between the plurality of remaining frames and the reference frame; wherein the remaining frames include the remaining frames of the oral image other than the target frame and the reference frame. Based on the matching relationship and the initial pose relationship, correcting the scale information of the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame includes: obtaining a first matching relationship, and obtaining a target frame matching point between the target frame and the reference frame according to the first matching relationship; obtaining a plurality of second matching relationships, and obtaining a remaining frame matching point between the reference frame and each remaining frame according to the plurality of second matching relationships; correcting the scale information of the initial pose relationship according to the target frame matching point and the remaining frame matching point to obtain the pose relationship between the reference frame and the non-reference frame.

[0012] In the above implementation process, in order to correct the initial posture relationship, the oral 3D image generation method provided in the embodiment of the present application needs to obtain the residual frame matching points between the reference frame and each residual frame according to multiple second matching relationships; according to multiple second matching relationships between the multiple residual frames and the reference frame, the same feature points between each residual frame and the reference frame, i.e., the residual frame matching points, are obtained. Based on the target matching points and the residual matching points, the scale information of the initial posture relationship is corrected.

[0013] Optionally, in an embodiment of the present application, based on the target frame matching points and the remaining frame matching points, correcting the scale information of the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame includes: calculating the basic three-dimensional coordinates of the target frame matching points in the coordinate system of the reference frame; obtaining matching points in the remaining frame matching points that are consistent with the target frame matching points, and calculating multiple target three-dimensional coordinates of the consistent matching points in the coordinate system of the reference frame; based on the basic three-dimensional coordinates and the multiple target three-dimensional coordinates, correcting the scale information of the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame.

[0014] In the above implementation process, in order to correct the scale information of a series of posture relationships, the basic three-dimensional coordinates and the target three-dimensional coordinates are obtained; it should be understood that the above-mentioned basic three-dimensional coordinates and the target three-dimensional coordinates are already in the same coordinate system, so that the scale information of the initial posture relationship can be smoothly corrected.

[0015] Optionally, in an embodiment of the present application, based on the basic three-dimensional coordinates and multiple target three-dimensional coordinates, correcting the scale information of the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame includes: calculating the basic distance between the basic three-dimensional coordinates and the coordinate origin of the reference frame; wherein the basic distance represents the scale information of the initial pose relationship of the target frame; based on the multiple target three-dimensional coordinates, obtaining multiple target distances between the multiple target three-dimensional coordinates and the coordinate origin of the reference frame in the coordinate system of the reference frame; based on the basic distance and the multiple target distances, correcting the scale information of the initial pose relationship of the remaining frames to obtain the pose relationship between the reference frame and the non-reference frame.

[0016] In the above implementation process, the oral three-dimensional image generation method provided in the embodiment of the present application, after obtaining the basic three-dimensional coordinates and multiple target three-dimensional coordinates, respectively calculates the distance between the basic three-dimensional coordinates and the multiple target three-dimensional coordinates and the origin of the reference frame coordinate system in the coordinate system of the reference frame, that is, the basic distance and the multiple target distances respectively; based on the basic distance and the target distance, multiple initial posture relationships are corrected so that the image can match the scale of the entire three-dimensional scene.

[0017] Optionally, in an embodiment of the present application, correcting the scale information of the initial pose relationship of the remaining frames according to the base distance and multiple target distances includes: calculating the average value of multiple target distances; correcting the average value of the target distance according to the base distance to correct the scale information of the initial pose relationship of the remaining frames.

[0018] In the above implementation process, in order to correct the scale information of the initial position relationship, it is necessary to calculate the average value of multiple target distances, and correct the average value of the target distance according to the basic distance, so as to achieve the purpose of correcting the scale information of the initial posture relationship.

[0019] In a second aspect, an embodiment of the present application provides an oral three-dimensional image generation system, the system comprising: a feature extraction module, a matching module, a posture relationship acquisition module and a three-dimensional reconstruction module; the feature extraction module is used to extract feature points from multiple oral images using a preset feature extraction network; the matching module is used to input multiple oral images and feature points into the matching network to obtain a matching relationship between a reference frame and a non-reference frame; wherein the non-reference frame includes other frames in the multiple oral images except the reference frame; the posture relationship acquisition module is used to calculate the posture relationship between the reference frame and the non-reference frame based on the feature points and the matching relationship; and the three-dimensional reconstruction module is used to generate an oral three-dimensional image based on multiple oral images and the posture relationship.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0021] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are read and executed by a processor, the steps in any of the above-mentioned implementation methods are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flow chart of generating an oral three-dimensional image provided in an embodiment of the present application;

[0024] Figure 2 A flow chart of feature point extraction provided in an embodiment of the present application;

[0025] Figure 3 A first flow chart for obtaining a posture relationship provided in an embodiment of the present application;

[0026] Figure 4 A second flow chart for obtaining a posture relationship provided in an embodiment of the present application;

[0027] Figure 5 A third flow chart for obtaining a posture relationship provided in an embodiment of the present application;

[0028] Figure 6 A fourth flow chart for obtaining a posture relationship provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of correcting the initial posture relationship provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the modules of the oral 3D image generation system provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0033] During the research process, the applicant found that with the continuous development of deep learning algorithms, in the field of oral image reconstruction, three-dimensional reconstruction algorithms and neural network rendering algorithms have made breakthrough progress, showing great advantages over traditional algorithms. In the problem of tooth reconstruction, it is necessary to accurately estimate the pose of the teeth in order to achieve reconstruction. The pose estimation method mainly consists of the traditional pose estimation method (COLMAP) and the deep learning pose estimation method (DF-VO); among them, the traditional pose estimation method will fail in pose estimation on images such as teeth that contain repeated textures; the deep learning pose estimation algorithm cannot be used between images with large pose differences.

[0034] Based on this, an embodiment of the present application provides a method for generating oral three-dimensional images, which uses a feature extraction network that adds position coding and global image information to greatly solve the matching difficulty problem caused by repeated tooth textures; in addition, the posture information of different images is scaled so that images with large posture differences can be used for three-dimensional image reconstruction.

[0035] Before introducing the specific solution of the present application, a brief introduction to image reconstruction is given here.

[0036] Image matching is the process of identifying the same point between two or more images through a certain matching algorithm, such as identifying the same point in two oral images; in computer vision, 3D reconstruction refers to the process of reconstructing the 3D information of an object based on a single view or multiple views. Since the information of a single view is incomplete, 3D reconstruction requires the use of empirical knowledge. Multi-view 3D reconstruction, such as binocular positioning, is relatively easy. The method is to first calibrate the camera, that is, calculate the position of each camera, and then use the information in multiple 2D images to reconstruct the 3D information.

[0037] Please see Figure 1 , Figure 1 A flowchart of generating an oral 3D image provided in an embodiment of the present application; a method for generating an oral 3D image provided in an embodiment of the present application comprises the following steps:

[0038] Step S100: extracting feature points from a plurality of oral cavity images using a preset feature extraction network.

[0039] In the above step S100, feature points in multiple oral images are extracted using a preset feature extraction network; illustratively, a SuperPoint network is used to extract feature points, and multiple oral images are input into the SuperPoint network; wherein the number of oral images is variable, and 20 to 50 oral images, or more oral images, may be input.

[0040] Step S101: multiple oral cavity images and feature points are input into a matching network to obtain a matching relationship between a reference frame and a non-reference frame.

[0041] In the above step S101, after extracting feature points from a plurality of oral cavity images, the plurality of oral cavity images from which features are extracted and the corresponding feature points are input into a matching network to obtain a matching relationship between a plurality of reference frames and non-reference frames.

[0042] It should be noted that the reference frame is any one of the multiple oral images, and a relatively clear and complete image can be selected; then, the other images among the multiple oral images except the reference frame are non-reference frames.

[0043] Exemplarily, multiple oral images and feature points are input into a matching network, such as a SuperGlue network, to obtain a matching relationship between a reference frame and each non-reference frame; it should be understood that the matching relationship can be understood as how many identical feature points exist between the reference frame and the non-reference frame.

[0044] Step S102: Calculate the pose relationship between the reference frame and the non-reference frame based on the feature points and the matching relationship.

[0045] In the above step S102, the pose relationship between the reference frame and the non-reference frame is calculated based on the feature points and the matching relationship.

[0046] In order to better understand the solution for obtaining the posture relationship, the posture is introduced here.

[0047] In visual SLAM, pose is the transformation from the world coordinate system to the camera coordinate system, including rotation and translation. Usually the world coordinate system is defined by oneself, and once defined, it cannot be changed. Usually the coordinates of the points used for composition are the coordinates in the world coordinate system. Then, the pose relationship between images is to unify the coordinates so that images with large pose differences can also be used, thereby realizing the reconstruction of the three-dimensional image of the oral cavity.

[0048] Step S103: Generate a three-dimensional oral cavity image based on the multiple oral cavity images and the position and posture relationship.

[0049] In the above step S103, multiple oral images and posture relationships are input into an image reconstruction network, such as an SRGAN network and a NeuS network, to generate an oral 3D image. For example, adding BARF (Bundle-Adjusting Neural Radiance Fields) to the NeuS network can obtain a more accurate camera posture.

[0050] pass Figure 1 It can be seen that in order to generate an oral three-dimensional image, the oral three-dimensional image generation method provided in the embodiment of the present application first extracts feature points from a number of oral images; after extracting the feature points, the feature points are matched; further, the posture relationship between different images is obtained, and the image is reconstructed based on the posture relationship and the oral image. It can be seen that the use of the oral three-dimensional image generation method provided in the embodiment of the present application can not only overcome the matching difficulty caused by repeated tooth textures, but also use images with large posture differences for image reconstruction, and on this basis establish a three-dimensional image with higher accuracy.

[0051] Please see Figure 2 , Figure 2 A feature point extraction flow chart provided for an embodiment of the present application; in an optional implementation of an embodiment of the present application, extracting feature points from multiple oral images using a preset feature extraction network may include the following steps:

[0052] Step S200: acquiring position codes and global feature vectors corresponding to the multiple oral cavity images according to the multiple oral cavity images.

[0053] In the above step S200, position codes and global feature vectors corresponding to the oral cavity images are obtained through multiple oral cavity images.

[0054] It should be noted that position coding can be understood as a type of coding that captures the order of features; the mainstream position coding methods of the target are mainly divided into two categories: absolute position coding and relative position coding. Among them, the absolute position coding works by informing the network architecture of the position of each element in the input sequence, which is similar to marking each element of the input sequence with a "position label" to indicate its absolute position; while the relative position coding works on the self-attention mechanism to inform the network architecture of the distance between each pair of elements; the embodiment of the present application does not make specific restrictions on the method of generating position coding, and the method of generating position coding should not be a limitation on the scope of protection of the oral three-dimensional image generation method in the embodiment of the present application.

[0055] In addition, global features can be understood as feature extraction of global information of each image, expressed in the form of vector rows.

[0056] Step S201: adding the position encoding and the global feature vector to the network structure of the feature extraction network to obtain a preset feature extraction network.

[0057] In the above step S201, after the position code and the global feature vector are added to the network structure of the feature extraction network, the preset feature extraction network used for extracting features mentioned above is formed.

[0058] The preset feature extraction network structure is explained in more detail. In the 256-dimensional descriptor, the first 128 dimensions are descriptors with the same meaning as the original descriptor, which are obtained by adding a layer of fully connected network to the original descriptor, and the original descriptor uses pre-trained weights; the next 28 dimensions are positional encoding, and the next 100 dimensions are global information of the image, which comes from shufflenet pre-trained on imagenet (the output dimension is 512 dimensions), followed by a layer of fully connected network to obtain a 100-dimensional global feature vector.

[0059] Step S202: inputting the plurality of oral cavity images into a preset feature extraction network to extract feature points from the plurality of oral cavity images.

[0060] In the above step S202, the multiple oral cavity images are input into a preset feature extraction network, so as to extract feature points in the multiple oral cavity images.

[0061] pass Figure 2It can be seen that in order to extract feature points from multiple oral images, the embodiment of the present application adds position coding and image global information to the feature extraction network to obtain a preset feature extraction network; and then extracts features from multiple oral images through the feature extraction network. It can be seen that the embodiment of the present application adds global information and position information to the feature extraction network, so that the feature extraction network obtains the specific position of each tooth in the entire oral cavity, thereby avoiding tooth matching errors, such as the problem of left teeth being mistakenly matched to right teeth.

[0062] Please see Figure 3 , Figure 3 The first flow chart for obtaining the pose relationship provided in the embodiment of the present application; in an optional implementation of the embodiment of the present application, the pose relationship between the reference frame and the non-reference frame is calculated according to the feature points and the matching relationship, which can be achieved by the following steps:

[0063] Step S300: Calculate the initial pose relationship between the reference frame and each non-reference frame according to the matching relationship.

[0064] In the above step S300, in order to obtain the pose relationship between the reference frame and the non-reference frame, it is first necessary to calculate the initial pose relationship between the reference frame and each non-reference frame respectively.

[0065] In the embodiment of the present application, the e-tracker in the deep learning pose estimation method (DF-VO) can be used to calculate the pose relationship between each non-reference frame and the reference frame. Specifically, the e-tracker uses epipolar geometry to calculate the initial pose relationship between the two images based on 2d-2d feature point matching; wherein the initial pose relationship includes a 3×3 rotation matrix and a 1×3 translation vector.

[0066] Optionally, when calculating the initial pose relationship, a GRIC score is obtained. If it is higher than a preset threshold, a reasonable pose relationship cannot be obtained from these feature points, and the calculation is considered to have failed. It can be understood that if all n oral images are calculated successfully, then n-1 initial pose relationships will be obtained for n images.

[0067] Step S301: Based on the matching relationship and the initial pose relationship, correct the scale information of the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame.

[0068] In the above step S301, the matching relationship, i.e., the matching relationship between the reference frame and each non-reference frame, can be understood as the number of identical feature points in the reference frame and each non-reference frame. Based on the matching relationship between the reference frame and each non-reference frame, and the initial pose relationship, the scale information of the initial pose relationship is corrected to obtain the pose relationship between the reference frame and the non-reference frame.

[0069] pass Figure 3 It can be seen that there is an initial pose relationship between the reference frame and each non-reference frame. Such a pose relationship can be represented by a matrix and a vector. Furthermore, the scale information of the initial pose relationship calculated by correction is calculated based on the matching relationship and the initial pose relationship, so that the scale information is unified, overcoming the problem that pictures with large pose differences cannot be used for image reconstruction.

[0070] Please see Figure 4 , Figure 4 A second flow chart for obtaining a pose relationship provided in an embodiment of the present application; the matching relationship includes a first matching relationship between a target frame and a reference frame and multiple second matching relationships between multiple remaining frames and the reference frame; wherein the remaining frames include the remaining frames of the oral image other than the target frame and the reference frame. In an optional implementation of the embodiment of the present application, based on the matching relationship and the initial pose relationship, the scale information of the initial pose relationship is corrected to obtain the pose relationship between the reference frame and the non-reference frame, including the following steps:

[0071] Step S400: obtaining a first matching relationship, and obtaining target frame matching points between a target frame and a reference frame according to the first matching relationship.

[0072] In the above step S400, according to the matching relationship between the target frame and the reference frame, the target frame matching points between the target frame and the reference frame are obtained; wherein the target frame matching points can be understood as the number of consistent feature points in the target frame and the reference frame. It should be noted that the target frame can select the first frame of several oral images.

[0073] Step S401: obtaining a plurality of second matching relationships, and obtaining a remaining frame matching point between a reference frame and each remaining frame according to the plurality of second matching relationships.

[0074] In the above step S401, according to the plurality of second matching relationships between the plurality of remaining frames and the reference frame, the same feature points between each remaining frame and the reference frame, ie, the remaining frame matching points, are obtained.

[0075] Step S402: Correcting the scale information of the initial pose relationship according to the target frame matching points and the remaining frame matching points to obtain the pose relationship between the reference frame and the non-reference frame.

[0076] In the above step S402, after obtaining the target frame matching points and the remaining frame matching points, the initial pose relationship is corrected according to the target frame matching points and the remaining frame matching points, thereby obtaining the pose relationship between the reference frame and the non-reference frame.

[0077] pass Figure 4It can be seen that in order to correct the initial posture relationship, the oral 3D image generation method provided in the embodiment of the present application needs to obtain the residual frame matching points between the reference frame and each residual frame according to multiple second matching relationships; according to multiple second matching relationships between the multiple residual frames and the reference frame, the same feature points between each residual frame and the reference frame, i.e., the residual frame matching points, are obtained. Based on the target matching points and the residual matching points, the scale information of the initial posture relationship is corrected.

[0078] Please see Figure 5 , Figure 5 A third flow chart for obtaining a pose relationship provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, according to the target frame matching points and the remaining frame matching points, the scale information of the initial pose relationship is corrected to obtain the pose relationship between the reference frame and the non-reference frame, including the following steps:

[0079] Step S500: Calculate the basic three-dimensional coordinates of the target frame matching point in the coordinate system of the reference frame.

[0080] In the above step S500, the basic three-dimensional coordinates of the target frame matching point in the reference frame coordinate system are calculated; it should be noted that the target frame matching point is actually two points on two images, and the above basic three-dimensional coordinates can be obtained through triangulation.

[0081] Step S501: obtaining matching points in the remaining frames that are consistent with the matching points in the target frame, and calculating a plurality of target three-dimensional coordinates of the consistent matching points in the coordinate system of the reference frame.

[0082] In the above step S501, the matching points in the remaining frames that are consistent with the matching points in the target frame are obtained, and multiple target three-dimensional coordinates of these consistent matching points in the reference frame coordinate system are calculated. Similarly, the target three-dimensional coordinates can also be calculated using a triangulation method.

[0083] Step S502: Correcting the scale information of the initial pose relationship according to the basic three-dimensional coordinates and the multiple target three-dimensional coordinates to obtain the pose relationship between the reference frame and the non-reference frame.

[0084] In the above step S502, after the basic three-dimensional coordinates and the multiple target three-dimensional coordinates are obtained, the pose relationship between the reference frame and the non-reference frame is obtained based on the scale information of the initial pose relationship between the two.

[0085] pass Figure 5 It can be seen that in order to correct the scale information of a series of posture relationships, the basic three-dimensional coordinates and the target three-dimensional coordinates are obtained; it should be understood that the above-mentioned basic three-dimensional coordinates and the target three-dimensional coordinates are already in the same coordinate system, so that the scale information of the initial posture relationship can be smoothly corrected.

[0086] Please see Figure 6 , Figure 6 A fourth flow chart for obtaining a pose relationship provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, correcting scale information of an initial pose relationship according to a basic three-dimensional coordinate and a plurality of target three-dimensional coordinates to obtain a pose relationship between a reference frame and a non-reference frame includes the following steps:

[0087] Step S600: Calculate the basic distance between the basic three-dimensional coordinates and the origin of the reference frame coordinates.

[0088] Step S601: according to the multiple target three-dimensional coordinates, obtain multiple target distances between the multiple target three-dimensional coordinates in the coordinate system of the reference frame and the coordinate origin of the reference frame.

[0089] In the above steps S600-S601, the basic three-dimensional coordinates and the multiple target three-dimensional coordinates are respectively calculated, and the distances between them and the origin of the reference frame coordinate system in the reference frame coordinate system are respectively the basic distance and the multiple target distances.

[0090] Step S602: Correct the scale information of the initial position relationship of the remaining frames according to the base distance and the multiple target distances to obtain the pose relationship between the reference frame and the non-reference frame.

[0091] pass Figure 6 It can be seen that the oral three-dimensional image generation method provided in the embodiment of the present application calculates the basic three-dimensional coordinates and the multiple target three-dimensional coordinates, respectively, after obtaining the basic three-dimensional coordinates and the multiple target three-dimensional coordinates, and the distances between the basic three-dimensional coordinates and the origin of the reference frame coordinate system in the coordinate system of the reference frame, that is, the basic distance and the multiple target distances, respectively; based on the basic distance and the target distance, multiple initial posture relationships are corrected so that the image can match the scale of the entire three-dimensional scene.

[0092] Please see Figure 7 , Figure 7 A flowchart of correcting the initial pose relationship provided in an embodiment of the present application; in an optional implementation of the embodiment of the present application, correcting the scale information of the initial pose relationship of the remaining frames according to the base distance and multiple target distances includes the following steps:

[0093] Step S700: Calculate the average value of multiple target distances.

[0094] In the above step S700, in order to correct the initial posture relationship, the average value of the obtained multiple target distances is calculated.

[0095] Step S701: Correct the average value of the target distance according to the base distance to correct the scale information of the initial pose relationship of the remaining frames.

[0096] In the above step S701, the base distance is used to correct the average value of the target distance, thereby correcting the scale information of the initial posture relationship of the remaining frames. It can be understood that each remaining frame has an average value of the target distance. During correction, there is a conversion coefficient for converting the average value of the target distance of each remaining frame into the base distance, thereby realizing the correction of the initial posture scale information of the remaining frames.

[0097] pass Figure 7 It can be seen that in order to correct the scale information of the initial position relationship, it is necessary to calculate the average value of multiple target distances, and correct the average value of the target distance according to the basic distance, so as to achieve the purpose of correcting the scale information of the initial posture relationship.

[0098] In an optional implementation, 2D-2D matching cannot obtain the scale of the entire 3D scene, that is, after calculating the initial pose information, the actual translation vector is αt; where α is unknown, and the modulus of t is 1 (||t||=1). In the above initial pose calculation process, vector t can be obtained. Then, the initial pose relationship between each two images will have an unknown α, which is the scale of the pose relationship.

[0099] Here is a specific implementation example of pose scale correction. If there are n oral images, the ref frame is the reference frame and the first frame is the target frame. If the first frame and the ref frame have m matching point pairs, then the pose scale of the scene is based on the pose relationship between the two frames (the modulus of the translation part of the two frames is 1).

[0100] Furthermore, the 3D coordinates of these m points are obtained by triangulation, and the 3D coordinates of these m points in the camera coordinates of the ref frame are recorded. Starting from the second frame, based on the pose relationship obtained by the e-tracker of DFVO, the scale information of each pose relationship is calculated. There are m k There are m matching point pairs, among which k ′ belongs to the previous m matching points (the 1st frame and the ref frame), for these m k ′ points for triangulation and calculate the m k The average distance of the points ′ from the origin in the camera coordinate system of the ref frame is used to restore the pose scale of the kth frame.

[0101] Please see Figure 8 , Figure 8 A module schematic diagram of an oral 3D image generation system provided in an embodiment of the present application; the oral 3D image generation system 100 includes: a feature extraction module 110, a matching module 120, a posture relationship acquisition module 130 and a 3D reconstruction module 140.

[0102] The feature extraction module 110 is used to extract feature points from a plurality of oral cavity images using a preset feature extraction network.

[0103] The matching module 120 is used to input the multiple oral images and feature points into the matching network to obtain the matching relationship between the reference frame and the non-reference frame; wherein the non-reference frame includes other frames in the multiple oral images except the reference frame;

[0104] The posture relationship acquisition module 130 is used to calculate the posture relationship between the reference frame and the non-reference frame according to the feature points and the matching relationship; and

[0105] The three-dimensional reconstruction module 140 is used to generate a three-dimensional image of the oral cavity according to a plurality of oral cavity images and posture relationships.

[0106] In an optional embodiment, the feature extraction module 110 uses a preset feature extraction network to extract feature points from multiple oral images, including: the feature extraction module 110 obtains position codes and global feature vectors corresponding to the multiple oral images based on the multiple oral images; the position codes and global feature vectors are added to the network structure of the feature extraction network to obtain a preset feature extraction network; the feature extraction module 110 inputs the multiple oral images into the preset feature extraction network to extract feature points from the multiple oral images.

[0107] In an optional embodiment, the pose relationship acquisition module 130 includes an initial pose acquisition module 131; the pose relationship acquisition module 130 calculates the pose relationship between the reference frame and the non-reference frame based on the feature points and the matching relationship, including: the initial pose acquisition module 131 calculates the initial pose relationship between the reference frame and each non-reference frame respectively according to the matching relationship; the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship based on the matching relationship and the initial pose relationship to obtain the pose relationship between the reference frame and the non-reference frame.

[0108] In an optional embodiment, the matching relationship includes a first matching relationship between the target frame and the reference frame and a plurality of second matching relationships between the plurality of remaining frames and the reference frame; wherein the remaining frames include the remaining frames of the oral image other than the target frame and the reference frame. The posture relationship acquisition module 130 also includes a matching point acquisition module 132; the posture relationship acquisition module 130 corrects the scale information of the initial posture relationship based on the matching relationship and the initial posture relationship to obtain the posture relationship between the reference frame and the non-reference frame, including: the matching point acquisition module 132 obtains the first matching relationship, and obtains the target frame matching point between the target frame and the reference frame according to the first matching relationship; the matching point acquisition module 132 obtains a plurality of second matching relationships, and the posture relationship acquisition module 130 obtains the remaining frame matching points between the reference frame and each remaining frame according to the plurality of second matching relationships; according to the target frame matching points and the remaining frame matching points, the scale information of the initial posture relationship is corrected to obtain the posture relationship between the reference frame and the non-reference frame.

[0109] In an optional embodiment, the pose relationship acquisition module 130 also includes a coordinate calculation module 133; the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship according to the target frame matching points and the remaining frame matching points to obtain the pose relationship between the reference frame and the non-reference frame, including: the coordinate calculation module 133 calculates the basic three-dimensional coordinates of the target frame matching points in the coordinate system of the reference frame; the matching point acquisition module 132 obtains the matching points in the remaining frame matching points that are consistent with the target frame matching points, and the coordinate calculation module 133 calculates multiple target three-dimensional coordinates of the consistent matching points in the coordinate system of the reference frame; the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship according to the basic three-dimensional coordinates and the multiple target three-dimensional coordinates to obtain the pose relationship between the reference frame and the non-reference frame.

[0110] In an optional embodiment, the pose relationship acquisition module 130 also includes a distance calculation module 134; the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship based on the basic three-dimensional coordinates and multiple target three-dimensional coordinates to obtain the pose relationship between the reference frame and the non-reference frame, including: the distance calculation module 134 calculates the basic distance between the basic three-dimensional coordinates and the reference frame coordinate origin; wherein the basic distance represents the scale information of the initial pose relationship of the target frame; the distance calculation module 134 obtains multiple target distances between the multiple target three-dimensional coordinates and the coordinate origin of the reference frame in the coordinate system of the reference frame based on multiple target three-dimensional coordinates; the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship of the remaining frames based on the basic distance and multiple target distances to obtain the pose relationship between the reference frame and the non-reference frame.

[0111] In an optional embodiment, the pose relationship acquisition module 130 corrects the scale information of the initial pose relationship of the remaining frames according to the base distance and multiple target distances, including: the pose relationship acquisition module 130 calculates the average value of multiple target distances; corrects the average value of the target distance according to the base distance to correct the scale information of the initial pose relationship of the remaining frames.

[0112] See also Fig. 9 , Fig. 9 The electronic device 300 provided in the embodiment of the present application includes: a processor 301 and a memory 302, wherein the memory 302 stores machine-readable instructions executable by the processor 301, and when the machine-readable instructions are executed by the processor 301, the above method is executed.

[0113] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the steps in any of the above implementation methods are executed.

[0114] The computer-readable storage medium may be a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or any other medium that can store program codes. The storage medium is used to store the program, and the processor executes the program after receiving the execution instruction. The method executed by the electronic terminal of the process definition disclosed in any embodiment of the present invention can be applied to the processor or implemented by the processor.

[0115] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0116] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0118] Alternatively, the present invention may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the present invention may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part.

[0119] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0120] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0121] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating an oral three-dimensional image, characterized in that: The method comprises: Extract feature points from multiple oral images using a preset feature extraction network; Input the multiple oral images and the feature points into a matching network to obtain a matching relationship between a reference frame and a non-reference frame; wherein the non-reference frame includes other frames in the multiple oral images except the reference frame; the matching relationship includes a first matching relationship between a target frame and the reference frame and a plurality of second matching relationships between a plurality of remaining frames and the reference frame; the remaining frames include the remaining frames of the oral images except the target frame and the reference frame; According to the matching relationship, respectively calculating the initial posture relationship between the reference frame and each of the non-reference frames; Acquire the first matching relationship, and acquire a target frame matching point between the target frame and the reference frame according to the first matching relationship; acquire a plurality of the second matching relationships, and acquire a remaining frame matching point between the reference frame and each of the remaining frames according to the plurality of the second matching relationships; correct the scale information of the initial pose relationship according to the target frame matching point and the remaining frame matching point to obtain a pose relationship between the reference frame and the non-reference frame; A three-dimensional image of the oral cavity is generated according to the multiple oral cavity images and the posture relationship.

2. The method according to claim 1, characterized in that: The method of extracting feature points from multiple oral images using a preset feature extraction network includes: According to the multiple oral images, obtaining position codes and global feature vectors corresponding to the multiple oral images; Adding the position encoding and the global feature vector to the network structure of the feature extraction network to obtain the preset feature extraction network; The plurality of oral cavity images are input into the preset feature extraction network to extract feature points from the plurality of oral cavity images.

3. The method according to claim 1, characterized in that The step of correcting the scale information of the initial pose relationship according to the target frame matching points and the remaining frame matching points to obtain the pose relationship between the reference frame and the non-reference frame includes: Calculating the basic three-dimensional coordinates of the target frame matching point in the coordinate system of the reference frame; Acquire matching points in the remaining frame matching points that are consistent with matching points in the target frame, and calculate multiple target three-dimensional coordinates of the consistent matching points in the coordinate system of the reference frame; The scale information of the initial pose relationship is corrected according to the basic three-dimensional coordinates and the plurality of target three-dimensional coordinates to obtain the pose relationship between the reference frame and the non-reference frame.

4. The method according to claim 3, characterized in that The step of correcting the scale information of the initial pose relationship according to the basic three-dimensional coordinates and the plurality of target three-dimensional coordinates to obtain the pose relationship between the reference frame and the non-reference frame includes: Calculating a basic distance between the basic three-dimensional coordinates and the reference frame coordinate origin; wherein the basic distance represents scale information of the initial posture relationship of the target frame; According to the plurality of target three-dimensional coordinates, obtaining a plurality of target distances between the plurality of target three-dimensional coordinates and the coordinate origin of the reference frame in the coordinate system of the reference frame; According to the base distance and the plurality of target distances, scale information of the initial pose relationship of the remaining frames is corrected to obtain a pose relationship between the reference frame and the non-reference frame.

5. The method according to claim 4, characterized in that The step of correcting the scale information of the initial pose relationship of the remaining frames according to the base distance and the plurality of target distances includes: Calculating an average of a plurality of said target distances; The average value of the target distance is corrected according to the base distance to correct the scale information of the initial pose relationship of the remaining frames.

6. An oral three-dimensional image generation system, characterized in that: The system comprises: a feature extraction module, a matching module, a posture relationship acquisition module and a three-dimensional reconstruction module; The feature extraction module is used to extract feature points from multiple oral images using a preset feature extraction network; The matching module is used to input the multiple oral images and the feature points into a matching network to obtain a matching relationship between a reference frame and a non-reference frame; wherein the non-reference frame includes other frames in the multiple oral images except the reference frame; the matching relationship includes a first matching relationship between a target frame and the reference frame and a plurality of second matching relationships between a plurality of remaining frames and the reference frame; wherein the remaining frames include the remaining frames of the oral images except the target frame and the reference frame; The pose relationship acquisition module is used to calculate the initial pose relationship between the reference frame and each of the non-reference frames according to the matching relationship; obtain the first matching relationship, and obtain the target frame matching point between the target frame and the reference frame according to the first matching relationship; obtain a plurality of the second matching relationships, and obtain the remaining frame matching points between the reference frame and each of the remaining frames according to the plurality of the second matching relationships; correct the scale information of the initial pose relationship according to the target frame matching point and the remaining frame matching point to obtain the pose relationship between the reference frame and the non-reference frame; and The three-dimensional reconstruction module is used to generate a three-dimensional image of the oral cavity based on the multiple oral cavity images and the posture relationship.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the method according to any one of claims 1 to 5 are executed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps in the method according to any one of claims 1 to 5 are executed.

Images