Oral health detection system, related methods, devices and equipment

Abstract

The present application discloses an oral health detection system, method and device, an oral image registration model construction method and device, and an electronic device. The system collects the user's oral natural image data through the client, and sends an oral image data fusion display request for the oral natural image data to the server; the server generates the user's oral medical image data and the natural image data fused with each other through the oral image registration model; the client displays the fused image data sent back by the server, so that the user can observe the oral health information according to the fused image data. This processing method allows the user's oral natural image taken by the client and the user's professional medical image to be fused and displayed, and the user can intuitively understand the information about his or her dental health changes through the fused image; therefore, the timeliness of the user's oral health detection can be effectively improved, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to an oral health detection system, method and device, an oral image registration model construction method and device, and an electronic device. Background Art

[0002] Oral diseases (such as caries, gum disease, etc.) are among the most common human diseases today. However, due to the severe shortage and uneven distribution of oral medical resources, many patients cannot receive timely diagnosis and treatment. In order to solve this problem, the integration of the Internet and oral health care has become a research hotspot. This technology is committed to innovating the screening and diagnosis of oral diseases to improve the efficiency of medical services.

[0003] At present, a typical oral image-assisted analysis system uses artificial intelligence (AI) technology to analyze professional medical X-ray images, and outputs oral disease auxiliary diagnosis reports based on oral medical images through oral image-assisted analysis models. The system can predict a variety of common oral diseases and provide lesion area locations, including apical periodontitis, periodontitis, caries, root resorption, root fracture trauma, impacted teeth, etc. It has a high average diagnostic accuracy rate, close to the diagnostic level of junior dentists.

[0004] However, in the process of implementing the present invention, the inventors found that the existing technical solutions have at least the following problems: the solution only performs auxiliary diagnosis of oral diseases based on professional medical X-ray images and gives a diagnosis report, but it cannot meet the needs of patients who want to observe / diagnose dental health problems by themselves based on natural oral images anytime and anywhere. Patients still need to go to the hospital to take oral X-rays before they can use the oral image auxiliary analysis system for auxiliary diagnosis. This will cause patients to be unable to perform oral health tests in a timely manner, so the user experience is poor. In summary, how to track and observe the development and changes of oral diseases based on natural oral images of patients has become an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] The present application provides an oral health detection system to solve the problem that the prior art cannot perform self-service dental health problem observation based on natural oral images of patients. The present application also provides an oral health detection method and device, an oral image registration model construction method and device, and an electronic device.

[0006] The present application provides an oral health detection system, comprising:

[0007] The client is used to collect the user's oral image data as oral natural image data; send an oral image fusion display request for the oral natural image data to the server; and display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server;

[0008] The server is used to receive the request; generate the fused image data according to the oral medical image data and the natural image data of the user through an oral image registration model; and send the fused image data back to the client.

[0009] The present application also provides an oral health detection method, comprising:

[0010] Collecting oral natural image data of the user as oral natural image data;

[0011] Sending an oral image fusion display request for the oral natural image data to a server;

[0012] The display server displays the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user, so that the user can observe the oral health information based on the fused image data.

[0013] Optionally, the natural image data includes local tooth image data;

[0014] The medical image data includes global dental image data;

[0015] The fused image data uses the medical image data as an oral background model and displays the local tooth image data at a position corresponding to the medical image data.

[0016] Optionally, the fused image data includes information on tooth changes at different stages.

[0017] The present application also provides an oral health detection method, comprising:

[0018] receiving an oral image data fusion display request for oral natural image data of a target user;

[0019] Generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model;

[0020] The fused image data is sent back to the requesting party.

[0021] Optionally, the first tooth feature data in the oral natural image data and the second tooth feature data in the user's oral medical image data are extracted through a tooth feature extraction network included in the oral image registration model;

[0022] Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data;

[0023] Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model;

[0024] Performing image transformation processing on the natural image data according to the registration transformation parameters through an oral image transformation network included in the oral image registration model;

[0025] The fused image data is generated according to the transformed natural image data and the medical image data through the oral image fusion network included in the oral image registration model.

[0026] Optionally, the first tooth feature data is extracted from the natural image data by a first tooth feature extraction subnetwork included in the tooth feature extraction network;

[0027] The second tooth feature data is extracted from the medical image data through the second tooth feature extraction subnetwork included in the image feature extraction subnetwork.

[0028] Optionally, the registration transformation parameters include: rotation parameters, translation parameters, and scaling parameters.

[0029] Optionally, the natural image data includes local dental image data, and the medical image data includes global dental image data.

[0030] Optionally, also include:

[0031] The medical image data of the target user is queried from an oral image database.

[0032] Optionally, also include:

[0033] Receiving an image data submission request for oral medical image data of a target user;

[0034] The corresponding records between the target user and the medical image data are stored in an oral image database.

[0035] The present application also provides an oral health detection device, comprising:

[0036] A natural image acquisition unit, used to acquire oral natural image data of a user as oral natural image data;

[0037] A request sending unit, used for sending an oral image fusion display request for the oral natural image data to a server;

[0038] The fused image display unit is used to display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server, so that the user can observe the oral health information based on the fused image data.

[0039] The present application also provides an electronic device, comprising:

[0040] Processor; and

[0041] The memory is used to store a program for implementing an oral health detection method. After the device is powered on and runs the program of the method through the processor, the following steps are performed: collecting the user's oral natural image data as the oral natural image data; sending an oral image fusion display request for the oral natural image data to the server; and displaying the fused image data corresponding to the teeth between the user's oral medical image data and the natural image data sent back by the server, so that the user can observe oral health information based on the fused image data.

[0042] The present application also provides an oral health detection device, comprising:

[0043] A request receiving unit, used for receiving an oral image data fusion display request for oral natural image data of a target user;

[0044] An image fusion unit, configured to generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model;

[0045] The fused image sending back unit is used to send back the fused image data to the requesting party.

[0046] The present application also provides an electronic device, comprising:

[0047] Processor; and

[0048] The memory is used to store a program for implementing the oral health detection method. After the device is powered on and runs the program of the method through the processor, the following steps are performed: receiving an oral image data fusion display request for the oral natural image data of a target user; generating fused image data corresponding to the teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model; and sending the fused image data back to the requesting party.

[0049] Optionally, the device includes: a smart phone, a smart speaker, a tablet computer, and a personal computer.

[0050] The present application also provides a method for constructing an oral image registration model, which is characterized by comprising:

[0051] Determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth;

[0052] Construct the network structure of the oral image registration model;

[0053] The network parameters of the model are trained according to the training data set to obtain the model.

[0054] Optionally, the oral image registration model includes a tooth feature extraction network for extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data;

[0055] The oral image registration model includes a tooth feature fusion network, which is used to determine tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data;

[0056] The oral image registration model includes a registration transformation parameter determination network, which is used to determine the oral image registration transformation parameters according to the fused tooth feature data;

[0057] The oral image registration model includes an oral image transformation network, which is used to perform image transformation processing on the natural image data according to the registration transformation parameters;

[0058] The oral image registration model includes an oral image fusion network, which is used to generate the fused image data based on the transformed natural image data and the medical image data.

[0059] Optionally, the tooth feature extraction network includes a first tooth feature extraction sub-network, which is used to extract the first tooth feature data from the natural image data;

[0060] The image feature extraction subnetwork includes a second tooth feature extraction subnetwork, which is used to extract the second tooth feature data from the medical image data.

[0061] The present application also provides an oral image registration model construction device, comprising:

[0062] A training data determination unit, used to determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth;

[0063] A network construction unit, used to construct a network structure of an oral image registration model;

[0064] A network training unit is used to train the network parameters of the model according to the training data set to obtain the model.

[0065] The present application also provides an electronic device, comprising:

[0066] Processor; and

[0067] The memory is used to store a program for implementing a method for building an oral image registration model. After the device is powered on and the program of the method is run through the processor, the following steps are performed: determining a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth; building a network structure of the oral image registration model; training the network parameters of the model according to the training data set to obtain the model.

[0068] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the computer-readable storage medium is run on a computer, the computer executes the above-mentioned various methods.

[0069] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned various methods.

[0070] Compared with the prior art, this application has the following advantages:

[0071] The oral health monitoring system provided in the embodiment of the present application collects the user's oral natural image data through the client, and sends an oral image data fusion display request for the oral natural image data to the server; the server generates image data in which the user's oral medical image data and the natural image data are fused through the oral image registration model; the client displays the fused image data sent back by the server, so that the user can observe the oral health information based on the fused image data; this processing method enables the natural oral image taken by the user through the client (such as a smart phone, smart speaker, etc.) and the professional medical image of the user to be fused and displayed, and the user can intuitively understand the information on changes in his or her dental health through the fused image; therefore, the timeliness of the user's oral health detection can be effectively improved, thereby improving the user experience.

[0072] The method for constructing an oral image registration model provided in an embodiment of the present application determines a training data set; the training data includes oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth; constructs a network structure of the oral image registration model; trains the network parameters of the model according to the training data set to obtain the model; this processing method enables the oral image registration model to be learned from the training data, and the model can generate a fused image of the teeth corresponding to the natural oral image taken by the user and the professional medical image, and the user can intuitively understand the information on the health changes of his teeth through the fused image; therefore, the accuracy and efficiency of oral image registration can be effectively improved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A structural diagram of an embodiment of an oral health detection system provided by the present application;

[0074] Figure 2 A schematic diagram of an application scenario of an embodiment of an oral health detection system provided by the present application;

[0075] Figure 3 A schematic diagram of device interaction of an embodiment of an oral health detection system provided by the present application;

[0076] Figure 4 A schematic diagram of an oral image registration model of an embodiment of an oral health detection system provided in the present application;

[0077] Figure 5 A schematic diagram of a user interface of an embodiment of an oral health monitoring system provided in the present application. DETAILED DESCRIPTION

[0078] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0079] In the present application, an oral health detection system, method and device, an oral image registration model construction method and device, and an electronic device are provided. In the following embodiments, various solutions are described in detail one by one.

[0080] First embodiment

[0081] Please refer to Figure 1 , which is a structural diagram of an embodiment of an oral health detection system of the present application. The system comprises: a server 1 and a client 2.

[0082] The server 1 may be a server deployed on a cloud server, or a server dedicated to oral health management, which may be deployed in a data center. The server may be a cluster server or a single server.

[0083] The client 2 includes but is not limited to mobile communication devices, namely: commonly known as mobile phones or smart phones, and also includes clients such as smart toothbrushes, smart speakers, personal computers, PADs, iPads, etc.

[0084] Please refer to Figure 2 , which is a scene diagram of the oral health detection system of the present application. The server 1 and the client 2 can be connected through a network, such as the client 2 can be connected to the Internet through WIFI, etc. Oral disease patients can use the image acquisition device equipped with clients such as smart speakers and smart phones to aim at the user's oral area and collect the user's oral natural image data. The client sends the collected image data to the server. The server generates fused image data corresponding to the teeth between the oral natural image and the oral medical image data through the oral image registration model, and sends the fused image data back to the client for display, so that the user can view it, etc.

[0085] Please refer to Figure 3 , which is a device interaction diagram of an embodiment of the oral health detection system of the present application. In this embodiment, the client collects the user's oral natural image data, sends an oral image fusion display request for the oral natural image data to the server; and displays the oral medical image data of the user and the fused image data corresponding to the teeth between the natural image data sent back by the server; the server receives the request; generates the fused image data according to the user's oral medical image data and the natural image data through the oral image registration model; and sends the fused image data back to the client.

[0086] The oral image fusion display request includes at least the oral natural image data, and may also include information such as a user identification. The oral natural image data may be multiple frames of image data in an oral video. By shooting at multiple angles of the oral part, the acquired multiple frames of natural image data may include more comprehensive oral information, such as images of all teeth. The oral natural image data may also be data of an oral image obtained by shooting a certain part of the oral cavity.

[0087] In one example, the client is a terminal device such as a smartphone or a smart speaker, which can be loaded with an oral health management application APP. After the user opens the APP, he can start the collection and processing of oral natural image data through the operation options provided by the APP (such as the "oral modeling detection" option). For example, the CMOS / CCD camera equipped with a smartphone or smart speaker can be used to capture N frames of continuous images of the oral area to perform oral modeling detection.

[0088] In another example, the client is a terminal device such as a smart phone or a smart speaker, which is connected to a smart toothbrush (the two can be connected wirelessly or by wire). The user can turn on the image acquisition function of the smart toothbrush while brushing his teeth. If bleeding teeth are found, the image of the area can be taken. The natural image data of the oral cavity can be collected by a camera set on the surface of the smart toothbrush, and the image data can be sent to the client, and the client sends the data to the server.

[0089] In specific implementation, the user may register the oral health management service provided by the server in advance through the client and become a registered user of the service. The oral image fusion display request sent by the client to the server may include the oral natural image data and the user identification.

[0090] It should be noted that if Figure 2 As shown, since the image acquisition device of the client is usually a camera, any frame of the oral natural image data captured by it is usually local image data in the oral cavity, and the user can usually aim at the part of the tooth that he thinks has problems and shoot it.

[0091] After receiving the request, the server can generate the fused image data according to the oral medical image data of the user and the natural image data through an oral image registration model, and send the fused image data back to the client.

[0092] The oral medical image data may be oral image data taken by professional oral medical testing equipment, such as oral X-rays, etc. The server may query the medical image data of the user from an oral image database according to the user identifier carried in the request.

[0093] In specific implementation, the server can also be used to receive image data submission requests for the user's oral medical image data; store the corresponding records between the user and the medical image data in the oral image database for use in the oral modeling stage.

[0094] The oral image registration model may be a simple model for image registration based on tooth shape. The oral image registration model may also be an artificial intelligence model that uses a human intelligence-like method to make oral image registration responses. The oral image registration model may have a network structure that may include common neural network structures such as DNN, CNN, and RNN. The model may include a feature data extraction network, an image registration network, and the like.

[0095] In one example, the oral image registration model uses the following method to fuse and register the two input modal data. First, an optimization algorithm (such as downhill, SGD, etc.) can be used to optimize the image registration through an artificially defined similarity metric to obtain a rigid body transformation matrix and generate a transformed image; then, the transformed image is used to fuse the image with the X-ray image to obtain the final fused image. This method does not use an artificial intelligence algorithm and cannot form an end-to-end solution, and is relatively limited in terms of registration efficiency and registration accuracy.

[0096] Please refer to Figure 4 , which is a schematic diagram of an oral image registration model of an embodiment of the oral health detection system of the present application. In this embodiment, the oral image registration model includes: a tooth feature extraction network, a tooth feature fusion network, a registration transformation parameter determination network, an oral image transformation network, and an oral image fusion network. The model can perform feature extraction for two images separately, perform feature fusion in a deep layer, and then use regression to obtain registration transformation parameters.

[0097] Depend on Figure 4 It can be seen that due to the limitations of the instant-capture equipment, the instant-capture image (i.e., the natural image of the oral cavity) is an image produced under a macro lens, which mainly captures the details of the teeth, so usually there are only a few or even one tooth. Therefore, the requirements for image registration are high. The tooth position corresponding to the global image (i.e., oral medical image, such as X-ray) can be found based on the characteristics of the teeth in the instant-capture image, such as shape, lesions, etc.

[0098] The server can extract the first tooth feature data in the oral natural image data and the second tooth feature data in the user's oral medical image data through a tooth feature extraction network. The first tooth feature data includes the tooth feature data extracted from the oral natural image data. The second tooth feature data includes the tooth feature data extracted from the medical natural image data.

[0099] In one example, the first tooth feature data may be extracted from the natural image data by a first tooth feature extraction subnetwork included in the tooth feature extraction network; and the second tooth feature data may be extracted from the medical image data by a second tooth feature extraction subnetwork included in the image feature extraction subnetwork. The first tooth feature extraction subnetwork and the second tooth feature extraction subnetwork extract tooth feature data for oral images of two different modalities.

[0100] In this embodiment, the first tooth feature extraction sub-network and the second tooth feature extraction sub-network adopt a residual network (Residual Network, ResNet) structure, and perform CNN convolution on the input image data through the residual mechanism to avoid the disappearance of deep gradients. For example, for two input images, they enter the corresponding feature extraction network respectively, and this network can be a common feature extraction CNN network model such as resnet50, resenet101, densenet121, etc., to extract deep features.

[0101] In specific implementation, the first tooth feature extraction sub-network and the second tooth feature extraction sub-network may also adopt other neural network structures, such as a multi-layer CNN convolutional network or a neural network based on an attention mechanism. In addition, the input oral medical images and natural images may be spliced ​​together to extract tooth feature data through a tooth feature extraction network.

[0102] After the server extracts the first tooth feature data and the second tooth feature data, it can determine the tooth feature data fused between the first tooth feature data and the second tooth feature data based on the first tooth feature data and the second tooth feature data through the tooth feature fusion network included in the oral image registration model.

[0103] The tooth feature fusion network can be processed in the following manner: concatenate the feature weights respectively output by the CNN convolutional layers of the first tooth feature extraction subnetwork and the second tooth feature extraction subnetwork to obtain fused tooth feature data. In specific implementation, a deep fusion method can also be used to fuse the first tooth feature data and the second tooth feature data. The fused tooth feature data may include feature data that reflects the tooth correspondence between oral medical image data and natural image data.

[0104] The server can determine the registration transformation parameters of the oral image registration model through the registration transformation parameter determination network, and determine the oral image registration transformation parameters according to the fused tooth feature data. The registration transformation parameters include but are not limited to: rotation parameters, translation parameters, and scaling parameters.

[0105] In specific implementation, the registration transformation parameter determination network can be processed as follows: for the fused tooth feature data, it is passed through 3 layers of CNN convolutional layers and pooling layers, and then connected to the final fully connected layer to regress 7 parameters, namely the rotation angle in three-dimensional space (rotate x, y, z), the scaling in two-dimensional space (scale x, y) and the translation in two-dimensional space (shift x, y), thereby constructing the image transformation relationship of the affine matrix (affine matrix), which is then input into the image transformation network.

[0106] It should be noted that the oral image registration model can be an unsupervised network model. For the unsupervised oral image registration model, the NGF loss function can be selected, which can better cope with multimodal data.

[0107] The server can perform image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model, and obtain the transformed position relationship of the random image by spatial transformation of the image registration parameters. The specific processing process of the oral image transformation network can be: using the constructed affine matrix to transform the random image, the method uses bilinear interpolation to generate a new transformed random image. The oral image transformation network inputs the transformed random image to the image fusion network.

[0108] Image transformation is the process of interpolating an image based on the affine matrix. The following formula can be used: RS*|image|+T=|image'|. Among them, R can be a 3*3 rotation matrix, S can be a 3*3 scale diagonal matrix. Since it is a 2D projection transformation, the z direction is always 1; T can be a 3*1 translation matrix. Since it is a 2D projection transformation, the z direction is always 0. image is the image coordinate before registration, and image' is the image coordinate after transformation. The transformation relationship of the image position is obtained through the formula, and then the color value or grayscale value of the corresponding position of the newly generated image is obtained by image interpolation.

[0109] The server can generate the fused image data according to the transformed natural image data and the medical image data through the oral image fusion network included in the oral image registration model. Image fusion is to paste the transformed natural image onto the standard medical image. In specific implementation, the steps of the image fusion network are as follows: the position information can be obtained from the affine matrix, and the transformed random image is aligned to the position on the medical image. In addition, the edge of the transformed image can be blurred and pasted on the medical image to make it look smooth, so as to obtain the final result, such as Figure 5 shown.

[0110] In specific implementation, the fused image data may include information on tooth changes at different stages. Tooth changes may be changes in tooth growth or damage, and users perceive changes by observing different images of the same tooth at different stages. It should be noted that the system uses medical images as standard images for modeling templates, and mainly allows users to perceive tooth changes through random images.

[0111] The user can view the fused impact data through the client and feel the dental problems more intuitively. At the same time, a standard model can be established based on the oral medical image data, and all subsequent oral natural images can be matched to the established model to facilitate the user to observe changes. In addition, in the subsequent stage, the fused impact data can also be used as historical data to perform oral health follow-up and prediction based on these historical data.

[0112] In this embodiment, the server can fuse the medical image data and the oral natural image data through the oral image registration model, and mark the local teeth in the natural image in the X-ray film, so that the user can compare and observe the changes of the local teeth in the natural image relative to the corresponding teeth in the X-ray film. The user can directly perceive the changes in the current dental condition relative to the previous dental condition, which can prompt the user to go to the hospital in time or conduct further dental health testing through the system provided in the embodiment of the present application; therefore, the user experience can be effectively improved.

[0113] It can be seen from the above embodiments that the oral health monitoring system provided in the embodiments of the present application collects the user's oral natural image data through the client, and sends an oral image data fusion display request for the oral natural image data to the server; the server generates image data in which the user's oral medical image data and the natural image data are fused through the oral image registration model; the client displays the fused image data sent back by the server, so that the user can observe the oral health information based on the fused image data; this processing method enables the natural oral image taken by the user through the client (such as a smart phone, smart speaker, etc.) and the professional medical image of the user to be fused and displayed, and the user can intuitively understand the information on changes in his or her dental health through the fused image; therefore, the timeliness of the user's oral health detection can be effectively improved, thereby improving the user experience.

[0114] Second embodiment

[0115] Corresponding to the above-mentioned oral health detection system, the present application also provides an oral health detection method, the execution subject of which includes but is not limited to terminal devices, such as smart phones, smart toothbrushes, smart speakers, tablet computers, desktop computers, wearable devices, etc., which can be implemented by installing an application (APP) on the terminal device or installing software on the desktop computer. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in Example 1.

[0116] An oral health detection method provided by the present application comprises:

[0117] Step 1: Collecting oral natural image data of the user as oral natural image data;

[0118] Step 2: Sending an oral image fusion display request for the oral natural image data to the server;

[0119] Step 3: Display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server, so that the user can observe the oral health information based on the fused image data.

[0120] The natural image data includes local tooth image data; the medical image data includes global tooth image data; the fused image data uses the medical image data as an oral background model and displays the local tooth image data at the position corresponding to the medical image data.

[0121] In one example, the fused image data includes information on tooth changes at different stages.

[0122] It can be seen from the above embodiments that the oral health detection method provided in the embodiments of the present application collects the user's oral natural image data through the client, and sends an oral image data fusion display request for the oral natural image data to the server; the server generates image data in which the user's oral medical image data and the natural image data are fused through the oral image registration model; the client displays the fused image data sent back by the server, so that the user can observe the oral health information based on the fused image data; this processing method enables the natural oral image taken by the user through the client (such as a smart phone, smart speaker, etc.) and the professional medical image of the user to be fused and displayed, and the user can intuitively understand the information on changes in his or her dental health through the fused image; therefore, the timeliness of the user's oral health detection can be effectively improved, thereby improving the user experience.

[0123] Third embodiment

[0124] In the above embodiment, an oral health detection method is provided, and correspondingly, the present application also provides an oral health detection device, which corresponds to the embodiment of the above method.

[0125] The parts of this embodiment that are the same as those of the first embodiment are not described in detail. Please refer to the corresponding parts in the first embodiment. The oral health detection device provided by the present application includes:

[0126] A natural image acquisition unit, used to acquire oral natural image data of a user as oral natural image data;

[0127] A request sending unit, used for sending an oral image fusion display request for the oral natural image data to a server;

[0128] The fused image display unit is used to display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server, so that the user can observe the oral health information based on the fused image data.

[0129] Fourth embodiment

[0130] The present application also provides an electronic device. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0131] An electronic device of the present embodiment includes: a processor and a memory; the memory is used to store a program for implementing an oral health detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: collecting the user's oral natural image data as oral natural image data; sending an oral image fusion display request for the oral natural image data to a server; and displaying the fused image data of the user's oral medical image data and the natural image data corresponding to the teeth sent back by the server, so that the user can observe oral health information based on the fused image data.

[0132] The devices include, but are not limited to: smartphones, tablet computers, and personal computers.

[0133] Fifth embodiment

[0134] Corresponding to the above-mentioned oral health detection system, the present application also provides an oral health detection method, the execution subject of the method includes but is not limited to the server, and can also be any device that can implement the method. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in the first embodiment.

[0135] In this embodiment, the method comprises the following steps:

[0136] Step 1: receiving an oral image data fusion display request for oral natural image data of a target user;

[0137] Step 2: Generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model;

[0138] Step 3: Send the fused image data back to the requesting party.

[0139] In one example, the first tooth feature data in the oral natural image data and the second tooth feature data in the user's oral medical image data are extracted through the tooth feature extraction network included in the oral image registration model; the tooth feature data after the first tooth feature data and the second tooth feature data are fused is determined based on the first tooth feature data and the second tooth feature data through the tooth feature fusion network included in the oral image registration model; the oral image registration transformation parameters are determined based on the fused tooth feature data through the registration transformation parameter determination network included in the oral image registration model; the natural image data is subjected to image transformation processing based on the registration transformation parameters through the oral image transformation network included in the oral image registration model; the fused image data is generated based on the transformed natural image data and the medical image data through the oral image fusion network included in the oral image registration model.

[0140] In one example, the first tooth feature data is extracted from the natural image data by a first tooth feature extraction subnetwork included in the tooth feature extraction network; and the second tooth feature data is extracted from the medical image data by a second tooth feature extraction subnetwork included in the image feature extraction subnetwork.

[0141] The registration transformation parameters include, but are not limited to: rotation parameters, translation parameters, and scaling parameters.

[0142] In one example, the natural image data includes local dental image data, and the medical image data includes global dental image data.

[0143] In one example, the method may further include the following step: querying the medical image data of the target user from an oral image database.

[0144] In one example, the method may further include the following steps: receiving an image data submission request for oral medical image data of a target user; and storing a corresponding record between the target user and the medical image data in an oral image database.

[0145] It can be seen from the above embodiments that the oral health detection method provided in the embodiments of the present application collects the user's oral natural image data through the client, and sends an oral image data fusion display request for the oral natural image data to the server; the server generates image data in which the user's oral medical image data and the natural image data are fused through the oral image registration model; the client displays the fused image data sent back by the server, so that the user can observe the oral health information based on the fused image data; this processing method enables the natural oral image taken by the user through the client (such as a smart phone, smart speaker, etc.) and the professional medical image of the user to be fused and displayed, and the user can intuitively understand the information on changes in his or her dental health through the fused image; therefore, the timeliness of the user's oral health detection can be effectively improved, thereby improving the user experience.

[0146] Sixth embodiment

[0147] In the above embodiment, an oral health detection method is provided, and correspondingly, the present application also provides an oral health detection device, which corresponds to the embodiment of the above method.

[0148] The parts of this embodiment that are the same as those of the first embodiment are not described in detail. Please refer to the corresponding parts in the first embodiment. The oral health detection device provided by the present application includes:

[0149] A request receiving unit, used for receiving an oral image data fusion display request for oral natural image data of a target user;

[0150] An image fusion unit, configured to generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model;

[0151] The fused image sending back unit is used to send back the fused image data to the requesting party.

[0152] Seventh embodiment

[0153] The present application also provides an electronic device embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0154] An electronic device of this embodiment includes: a processor and a memory; the memory is used to store a program for implementing an oral health detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: receiving an oral image data fusion display request for oral natural image data of a target user; generating fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data of the target user and the natural image data through an oral image registration model; and sending the fused image data back to the requesting party.

[0155] Eighth embodiment

[0156] Corresponding to the above-mentioned oral health detection system, the present application also provides a method for constructing an oral image registration model, the execution subject of the method includes but is not limited to a server, and can also be any device capable of implementing the method. The parts of this embodiment that are the same as those of the first embodiment are not repeated here, please refer to the corresponding parts in the first embodiment.

[0157] In this embodiment, the method for constructing an oral image registration model includes the following steps:

[0158] Step 1: Determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth;

[0159] Step 2: Construct the network structure of the oral image registration model;

[0160] Step 3: Train the network parameters of the model according to the training data set to obtain the model.

[0161] In one example, the oral image registration model includes a tooth feature extraction network, which is used to extract the first tooth feature data in the oral natural image data and the second tooth feature data in the user's oral medical image data; the oral image registration model includes a tooth feature fusion network, which is used to determine the tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; the oral image registration model includes a registration transformation parameter determination network, which is used to determine the oral image registration transformation parameters according to the fused tooth feature data; the oral image registration model includes an oral image transformation network, which is used to perform image transformation processing on the natural image data according to the registration transformation parameters; the oral image registration model includes an oral image fusion network, which is used to generate the fused image data according to the transformed natural image data and the medical image data.

[0162] In one example, the tooth feature extraction network includes a first tooth feature extraction subnetwork for extracting the first tooth feature data from the natural image data; the image feature extraction subnetwork includes a second tooth feature extraction subnetwork for extracting the second tooth feature data from the medical image data.

[0163] It can be seen from the above embodiments that the method for constructing an oral image registration model provided in the embodiments of the present application is by determining a training data set; the training data includes oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth; constructing a network structure of the oral image registration model; training the network parameters of the model according to the training data set to obtain the model; this processing method enables the oral image registration model to be learned from the training data, and the model can generate a fused image of the teeth corresponding to the natural oral image taken by the user and the professional medical image, and the user can intuitively understand the information on changes in their dental health through the fused image; therefore, the accuracy and efficiency of oral image registration can be effectively improved, thereby improving the user experience.

[0164] Ninth embodiment

[0165] In the above-mentioned embodiment, a method for constructing an oral image registration model is provided, and correspondingly, the present application also provides a device for constructing an oral image registration model. The device corresponds to the embodiment of the above-mentioned method.

[0166] The parts of this embodiment that are the same as those of the first embodiment are not described in detail. Please refer to the corresponding parts in the first embodiment. The present application provides an oral image registration model construction device comprising:

[0167] A training data determination unit, used to determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth;

[0168] A network construction unit, used to construct a network structure of an oral image registration model;

[0169] A network training unit is used to train the network parameters of the model according to the training data set to obtain the model.

[0170] Tenth embodiment

[0171] The present application also provides an electronic device embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0172] An electronic device of the present embodiment includes: a processor and a memory; the memory is used to store a program for implementing a method for building an oral image registration model. After the device is powered on and the program of the method is run through the processor, the following steps are performed: determining a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth; building a network structure of an oral image registration model; training the network parameters of the model according to the training data set to obtain the model.

[0173] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0177] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. An oral health detection system, characterized in that: include: The client is used to collect the user's oral image data as oral natural image data; send an oral image fusion display request for the oral natural image data to the server; and display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server; The server is used to receive the request; generate the fused image data according to the oral medical image data of the user and the natural image data through the oral image registration model; and send the fused image data back to the client; wherein, generating the fused image data according to the oral medical image data of the user and the natural image data through the oral image registration model includes: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

2. An oral health detection method, characterized in that: include: Collecting oral natural image data of the user as oral natural image data; Sending an oral image fusion display request for the oral natural image data to a server; Display the fused image data corresponding to the teeth between the oral medical image data and the natural image data of the user sent back by the server, so that the user can observe the oral health information according to the fused image data; The fused image data is generated in the following manner: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

3. The method according to claim 2, characterized in that The natural image data includes local tooth image data; The medical image data includes global dental image data; The fused image data uses the medical image data as an oral background model and displays the local tooth image data at a position corresponding to the medical image data.

4. The method according to claim 2, characterized in that: The fused image data includes information on tooth changes at different stages.

5. An oral health detection method, characterized in that: include: receiving an oral image data fusion display request for oral natural image data of a target user; Generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model; Sending back fused image data to the requesting party; The step of generating fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through the oral image registration model includes: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

6. The method according to claim 5, characterized in that Extracting the first tooth feature data from the natural image data through a first tooth feature extraction subnetwork included in the tooth feature extraction network; The second tooth feature data is extracted from the medical image data through the second tooth feature extraction subnetwork included in the image feature extraction subnetwork.

7. The method according to claim 5, characterized in that The registration transformation parameters include: rotation parameters, translation parameters, and scaling parameters.

8. The method according to claim 5, characterized in that The natural image data includes local tooth image data, and the medical image data includes global tooth image data.

9. The method according to claim 5, characterized in that Also includes: The medical image data of the target user is queried from an oral image database.

10. The method according to claim 5, characterized in that Also includes: Receiving an image data submission request for oral medical image data of a target user; The corresponding records between the target user and the medical image data are stored in an oral image database.

11. An oral health detection device, characterized in that: include: A natural image acquisition unit, used to acquire oral natural image data of a user as oral natural image data; A request sending unit, used for sending an oral image fusion display request for the oral natural image data to a server; A fused image display unit, used to display fused image data corresponding to teeth between the oral medical image data and the natural image data of the user sent back by the server, so that the user can observe oral health information according to the fused image data; The fused image data is generated in the following manner: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

12. An electronic device, characterized in that: include: Processor; and The memory is used to store a program for implementing an oral health detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: collecting the user's oral natural image data as the oral natural image data; sending an oral image fusion display request for the oral natural image data to the server; displaying the fused image data of the user's oral medical image data and the natural image data corresponding to the teeth sent back by the server, so that the user can observe the oral health information according to the fused image data, wherein the fused image data is generated in the following manner: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

13. An oral health detection device, characterized in that: include: A request receiving unit, used for receiving an oral image data fusion display request for oral natural image data of a target user; An image fusion unit, configured to generate fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through an oral image registration model; A fused image sending back unit, used for sending back fused image data to the requesting party; The step of generating fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data and the natural image data of the target user through the oral image registration model includes: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

14. An electronic device, characterized in that: include: processor; as well as A memory for storing a program for implementing an oral health detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: receiving an oral image data fusion display request for oral natural image data of a target user; generating fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data of the target user and the natural image data through an oral image registration model; and sending the fused image data back to the requesting party; wherein, generating fused image data corresponding to teeth between the oral medical image data and the natural image data according to the oral medical image data of the target user and the natural image data through an oral image registration model includes: Extracting first tooth feature data in the oral natural image data and second tooth feature data in the user's oral medical image data through a tooth feature extraction network included in the oral image registration model; Determine, by a tooth feature fusion network included in the oral image registration model, tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; Determining the oral image registration transformation parameters according to the fused tooth feature data through a registration transformation parameter determination network included in the oral image registration model; Performing image transformation processing on the natural image data according to the registration transformation parameters through the oral image transformation network included in the oral image registration model; Generate the fused image data according to the transformed natural image data and the medical image data through an oral image fusion network included in the oral image registration model; The oral image registration model is learned from oral medical image data, oral natural image data, and labeled data of the correspondence between natural image teeth and medical image teeth.

15. A method for constructing an oral image registration model, characterized in that: include: Determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth; Construct the network structure of the oral image registration model; Training the network parameters of the model according to the training data set to obtain the model; The oral image registration model includes a tooth feature extraction network for extracting first tooth feature data from the oral natural image data and second tooth feature data from the user's oral medical image data; The oral image registration model includes a tooth feature fusion network, which is used to determine tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; The oral image registration model includes a registration transformation parameter determination network, which is used to determine the oral image registration transformation parameters according to the fused tooth feature data; The oral image registration model includes an oral image transformation network, which is used to perform image transformation processing on the natural image data according to the registration transformation parameters; The oral image registration model includes an oral image fusion network, which is used to generate fused image data based on the transformed natural image data and the medical image data.

16. The method according to claim 15, characterized in that The tooth feature extraction network includes a first tooth feature extraction sub-network, which is used to extract the first tooth feature data from the natural image data; The image feature extraction subnetwork includes a second tooth feature extraction subnetwork, which is used to extract the second tooth feature data from the medical image data.

17. An oral image registration model construction device, characterized in that: include: A training data determination unit, used to determine a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth; A network construction unit, used to construct a network structure of an oral image registration model; A network training unit, used to train the network parameters of the model according to the training data set to obtain the model; The oral image registration model includes a tooth feature extraction network for extracting first tooth feature data from the oral natural image data and second tooth feature data from the user's oral medical image data; The oral image registration model includes a tooth feature fusion network, which is used to determine tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; The oral image registration model includes a registration transformation parameter determination network, which is used to determine the oral image registration transformation parameters according to the fused tooth feature data; The oral image registration model includes an oral image transformation network, which is used to perform image transformation processing on the natural image data according to the registration transformation parameters; The oral image registration model includes an oral image fusion network, which is used to generate fused image data based on the transformed natural image data and the medical image data.

18. An electronic device, characterized in that: include: processor; as well as The memory is used to store a program for implementing a method for building an oral image registration model. After the device is powered on and the program of the method is run through the processor, the following steps are performed: determining a training data set; the training data includes oral medical image data, oral natural image data, and annotated data of the correspondence between natural image teeth and medical image teeth; building a network structure of an oral image registration model; training the network parameters of the model according to the training data set to obtain the model; The oral image registration model includes a tooth feature extraction network for extracting first tooth feature data from the oral natural image data and second tooth feature data from the user's oral medical image data; The oral image registration model includes a tooth feature fusion network, which is used to determine tooth feature data after the first tooth feature data and the second tooth feature data are fused according to the first tooth feature data and the second tooth feature data; The oral image registration model includes a registration transformation parameter determination network, which is used to determine the oral image registration transformation parameters according to the fused tooth feature data; The oral image registration model includes an oral image transformation network, which is used to perform image transformation processing on the natural image data according to the registration transformation parameters; The oral image registration model includes an oral image fusion network, which is used to generate fused image data based on the transformed natural image data and the medical image data.

Images