Oral panoramic radiograph feature recognition method, system and terminal based on neural network

Abstract

The application provides a neural network-based oral panoramic radiograph feature recognition method, system and terminal, and particularly relates to the technical fields of artificial intelligence deep learning and oral panoramic radiograph, and the scheme collects an oral panoramic radiograph and an oral state description text corresponding to the oral panoramic radiograph; an image encoder is used to extract image features from the oral panoramic radiograph, a text encoder is used to extract text features from the oral state description text, then the image features and the text features are fused to obtain fused features; based on the fused features, an explanation text and a picture segmentation result of the fused features are obtained; and based on the explanation text and the picture segmentation result, a feature recognition result of the oral panoramic radiograph is obtained. The scheme is based on the synergistic effect of the explanation text and the picture segmentation result, and can improve the accuracy, effectiveness and comprehensiveness of the feature recognition result of the extracted oral panoramic radiograph.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence deep learning and oral panoramic radiography, and in particular to an oral panoramic radiography feature recognition method, system and terminal based on a neural network. BACKGROUND

[0002] Oral panoramic radiography is a kind of imaging technology that can obtain the total image of all teeth and surrounding tissues in the oral cavity through one imaging by applying the principle of narrow slit and circular arc tomography. At present, oral panoramic radiography has been widely used to assist in oral diagnosis and treatment, and deep neural network has become an auxiliary means for oral disease diagnosis, which can further realize caries depth recognition and root tip lesion detection.

[0003] Although deep neural network has been used to extract features of oral panoramic radiography, the existing technology has poor effect on feature region recognition and feature extraction under various complex conditions such as tooth loss and artificial fillers due to the complexity of oral panoramic radiography, resulting in low accuracy of oral health detection results. SUMMARY

[0004] In view of the above shortcomings of the prior art, the present application aims to provide an oral panoramic radiography feature recognition method, system and terminal based on a neural network, which aims to solve the problem of low accuracy of oral health detection results in the prior art.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides an oral panoramic radiography feature recognition method based on a neural network, comprising:

[0006] Collecting an oral panoramic radiography and an oral state description text corresponding to the oral panoramic radiography;

[0007] Extracting image features from the oral panoramic radiography using a trained image encoder, and extracting text features from the oral state description text using a trained text encoder;

[0008] Fusing the image features and the text features to obtain fused features;

[0009] Based on the fused features, obtaining an explanation text and a picture segmentation result of the fused features; and based on the explanation text and the picture segmentation result, obtaining a feature recognition result of the oral panoramic radiography.

[0010] Optionally, the training process of the trained image encoder comprises:

[0011] Collecting a plurality of oral panoramic radiographies to construct an oral panoramic radiography sample set;

[0012] Constructing an image encoder model;

[0013] inputting the oral panoramic sample set into the image encoder model, extracting image features of the oral panoramic in the oral panoramic sample set, calculating an image encoder loss based on a preset image encoder loss function according to the image features, optimizing the image encoder model according to the image encoder loss, and obtaining a trained image encoder.

[0014] Optionally, the fusing the image features and the text features to obtain fused features comprises:

[0015] fusing the image features and the text features by using a trained fusion neural network to obtain fused features, wherein a training process of the trained fusion neural network comprises:

[0016] collecting a plurality of oral panoramic images and a plurality of oral state description texts to construct an oral feature sample set;

[0017] constructing a fusion neural network model;

[0018] inputting the oral feature sample set into the fusion neural network model, extracting oral features in the oral feature sample set, calculating a fusion neural network loss based on a preset fusion neural network loss function according to the oral features, optimizing the fusion neural network model according to the fusion neural network loss, and obtaining a trained fusion neural network.

[0019] Optionally, the obtaining an explanation text and an image segmentation result of the fused features based on the fused features comprises:

[0020] decoding the fused features by using a trained text decoder to obtain an explanation text of the fused features;

[0021] segmenting the fused features by using a trained image segmentation network to obtain an image segmentation result.

[0022] Optionally, the training process of the text decoder comprises:

[0023] constructing a text decoder model;

[0024] inputting the fused features into the text decoder model, extracting an explanation text in the fused features, calculating a text decoder loss based on a preset text decoder loss function according to the explanation text, optimizing the text decoder model according to the text decoder loss, and obtaining a trained text decoder.

[0025] Optionally, the method further comprises:

[0026] The feature recognition result of the oral panoramic film is used to calculate an oral feature index parameter.

[0027] The second aspect of the present application provides a neural network-based oral panoramic film feature recognition system, which comprises:

[0028] A data acquisition module is configured to acquire an oral panoramic film and an oral state description text corresponding to the oral panoramic film.

[0029] A feature extraction module is configured to extract image features from the oral panoramic film by using a trained image encoder and extract text features from the oral state description text by using a trained text encoder.

[0030] A feature fusion module is configured to fuse the image features and the text features to obtain fused features.

[0031] A feature recognition module is configured to obtain an explanation text and a picture segmentation result of the fused features based on the fused features, and obtain a feature recognition result of the oral panoramic film based on the explanation text and the picture segmentation result.

[0032] Optionally, the system further comprises:

[0033] An oral feature index extraction module is configured to calculate an oral feature index parameter by using the feature recognition result of the oral panoramic film.

[0034] The third aspect of the present application provides an intelligent terminal, which comprises a memory, a processor, and a neural network-based oral panoramic film feature recognition program stored in the memory and executable on the processor.

[0035] The fourth aspect of the present application provides a computer readable storage medium, which stores a neural network-based oral panoramic film feature recognition program.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] The present application firstly extracts image features of the collected oral panoramic film by adopting the trained image encoder, extracts text features of the collected oral state description text by adopting the trained text encoder, so as to take the image features and the text features as data features of the oral panoramic film feature extraction; then the image features and the text features are fused to obtain the fusion features, so as to realize the supplement and improvement of the text features to the image features; and then based on the fusion features, the explanation text and the picture segmentation result of the fusion features are obtained, so as to verify and supplement each other, so as to take the explanation text as a strong explanation and supplement of the feature recognition result of the oral panoramic film, and the accuracy, effectiveness and comprehensiveness of the feature recognition result of the oral panoramic film can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0039] Figure 1 Flow chart of the oral panoramic film feature recognition method based on neural network of the present application;

[0040] Figure 2 Example schematic diagram of the oral panoramic film feature recognition method based on neural network of the present application;

[0041] Figure 3 Structure schematic diagram of the oral panoramic film feature recognition system based on neural network of the present application;

[0042] Figure 4 Structure schematic diagram of the intelligent terminal of the present application. DETAILED DESCRIPTION

[0043] In the following description, specific details are set forth such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it should be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0044] It should be understood that when used in the specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0046] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0047] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0049] The oral panoramic radiograph feature recognition method based on the multi-modal deep neural network provided by the present application mainly adopts a multi-modal deep neural network to fuse text and image data types to form a multi-modal data set composed of fused features, and then adopts the multi-modal deep neural network to recognize various image feature data and text feature data related to the oral cavity included in the fused features, wherein the text feature data is mainly used to assist in recognizing the image feature data included in the oral panoramic radiograph, so as to improve the accuracy and comprehensiveness of the effective features extracted from the oral panoramic radiograph, so that the oral health condition detection result is more accurate and comprehensive.

[0050] Exemplary method

[0051] The embodiment of the present application provides an oral panoramic radiograph feature recognition method based on a neural network, which is deployed on electronic devices such as computers and servers, and is applied to the diagnosis of oral diseases. It is aimed at the diagnosis and treatment of diseases related to the oral cavity, such as abnormal oral structure (e.g., missing teeth, presence of artificial fillings, etc.), abnormal periodontal tissue (e.g., osteoporosis, infection, fracture, etc.), tooth correction, dental implantation, or other oral surgery planning. Specifically, as shown in Figure 1 and Figure 2 The steps of the method of the present embodiment include:

[0052] Step S100: Collecting an oral panoramic film and an oral state description text corresponding to the oral panoramic film.

[0053] Specifically, a plurality of oral panoramic films and oral state description texts corresponding to the oral panoramic films are collected, and it is ensured that each oral panoramic film and the corresponding oral state description text have the same dimension.

[0054] Further, as other preferred embodiments, the oral panoramic film can also be preprocessed, such as filtering and denoising, adjusting the image to a fixed size, normalizing the pixel value range, adjusting the contrast and brightness of the image, etc., to ensure that each image has the same size and resolution, and to obtain the preprocessed oral panoramic film, so as to improve the accuracy of subsequent analysis.

[0055] Step S200: Extracting image features from the oral panoramic film using the trained image encoder, and extracting text features from the oral state description text using the trained text encoder.

[0056] Specifically, a plurality of oral panoramic films are collected to construct an oral panoramic film sample set; an image encoder model is constructed, such as a convolutional neural network (CNN) and a deep learning model, such as VGG, ResNet or Inception, etc. The oral panoramic film sample set is input into the image encoder model to extract the image features of the oral panoramic films in the oral panoramic film sample set. Based on a preset image encoder loss function, the image encoder loss is calculated according to the image features, and the image encoder model is optimized according to the image encoder loss to obtain a trained image encoder. Then the oral panoramic film is input into the trained image encoder to extract the image features, wherein the image features of the oral panoramic film refer to known image features included in the oral panoramic film in the sample set. This embodiment uses an image feature vector F1 to represent.

[0057] The selection and parameter setting of the image encoder may have an impact on the effect of image feature extraction. Specifically, the best image feature representation can be obtained by selection and adjustment according to actual task requirements. Available image encoder models include but are not limited to convolutional neural networks (CNN) and deep learning models, such as VGG, ResNet or Inception, etc.

[0058] Similarly, a plurality of oral state description texts are collected to construct an oral state description text sample set; a text encoder model is constructed, the oral state description text sample set is input into the text encoder model, the oral state description texts in the oral state description text sample set are extracted, a text encoder loss is calculated according to the oral state description texts based on a preset text encoder loss function, the text encoder model is optimized according to the text encoder loss, and a trained text encoder is obtained. Then the oral state description text is input into the trained text encoder, and the text features are extracted, wherein the oral state description text refers to known text features included in the oral state description text sample set. The text feature vector F2 is used to represent the text features in this embodiment, and the text feature vector F2 is converted into the same dimension as the image feature vector F1.

[0059] The selection and parameter setting of the text encoder can affect the effect of text feature extraction. Specifically, the selection and adjustment can be performed according to actual task requirements to obtain the best text feature representation. The available text encoder models include but are not limited to BERT, RoBERTa, ALBERT, NEZHA, XLNET, ERNIE, etc.

[0060] Further, as other preferred embodiments, dimension reduction techniques such as principal component analysis, independent component analysis, high correlation filtering, forward feature selection, etc. can be used to reduce the dimension of the extracted image features and text features, or clustering algorithms can be used to classify the features.

[0061] Step S300: The image features and the text features are fused to obtain fused features.

[0062] Specifically, the fusion of the image features and the text features is realized by training a fusion neural network, and the training process of the fusion neural network is specifically as follows:

[0063] A plurality of panoramic dental films and a plurality of oral state description texts are collected to construct an oral feature sample set; a fusion neural network model is constructed, the oral feature sample set is input into the fusion neural network model, the oral features in the oral feature sample set are extracted, a fusion neural network loss is calculated according to the oral features based on a preset fusion neural network loss function, the fusion neural network model is optimized according to the fusion neural network loss, and a trained fusion neural network is obtained. Then the extracted image features and text features are input into the trained fusion neural network to generate fused features. The fused features referred to in this embodiment represent a fused feature vector or a fused feature matrix.

[0064] Further, as other preferred embodiments, appropriate evaluation indicators such as accuracy, mean square error, etc. can be selected according to the complexity of the task and the size of the data set, etc. actual application conditions, and the trained fusion neural network is evaluated using the validation set or test set, and optimized according to the evaluation result.

[0065] In this embodiment, the fusion of features of different modalities is realized by constructing a fusion neural network. As other preferred embodiments, linear superposition, splicing, attention mechanism, etc. can be selected to realize the fusion of features of different modalities according to the actual task requirements. In the method of realizing the fusion of text features and image features by using the attention mechanism, the self-attention module in the transformer is used to exchange information between the image features and the text features in the middle layer. For example, a 4-dimensional hidden vector is used to perform self-attention with the feature vectors of the image features and the text features of the two modalities to realize information exchange.

[0066] It should be noted that the way of feature fusion and the design of network structure may affect the expression ability of the fused features. According to the requirements of the specific task, some experiments and adjustments may be needed to obtain the best fused feature representation.

[0067] In this embodiment, the oral panoramic radiographs and the oral status description texts used to train the model are complete oral detection or diagnosis data obtained from existing clinical experiments.

[0068] For example, a panoramic X-ray machine or other equipment is used to generate a set of oral panoramic radiographs as an oral panoramic radiograph data set, and it is ensured that each image has the same size and resolution. The oral panoramic radiograph includes rich information of the oral cavity and the jawbone, such as the structure and positional relationship information of the full set of teeth and the jawbone (such as tumor cells, missing teeth, artificial fillings, the height, density and shape of the jawbone, etc.), the condition of the periodontal tissue (such as osteoporosis, infection, fracture, etc.), so as to provide various reference information for oral disease diagnosis, tooth analysis or other oral surgery planning, etc.

[0069] Based on the existing diagnosis medical records and other texts, a set of oral status description texts is generated as an oral status description sample set.

[0070] In step S400, based on the fused features, the explanation text and the picture segmentation result of the fused features are obtained; and based on the explanation text and the picture segmentation result, the feature recognition result of the oral panoramic radiograph is obtained.

[0071] Specifically, the step S410 for obtaining the explanation text of the fusion feature, the step S420 for obtaining the picture segmentation result, and the step S430 for obtaining the feature recognition result of the panoramic oral film are included, wherein the step S410, the step S420 and the step S430 are as follows respectively:

[0072] Step S410: decoding the fusion feature by using the trained text decoder to obtain the explanation text of the fusion feature.

[0073] Specifically, a text decoder model is constructed, the fusion feature is input into the text decoder model, the explanation text in the fusion feature is extracted, the text decoder loss is calculated based on the preset text decoder loss function according to the explanation text, the text decoder model is optimized according to the text decoder loss, and the trained text decoder is obtained. Then the generated fusion feature is input into the trained text decoder to generate the explanation text related to the fusion feature.

[0074] Since the selection and parameter setting of the text decoder may affect the accuracy of the text explanation. Specifically, the selection and adjustment can be made according to the actual task requirements to obtain the best explanation text related to the fusion feature. Common text encoder models are generally available for selection, such as Bert, GRU, and even ChatGPT.

[0075] Step S420: segmenting the fusion feature by using the trained image segmentation network to obtain the picture segmentation result.

[0076] Specifically, an image segmentation network model is constructed, such as U-Net, FCN (Fully Convolutional Network) or DeepLab, and the constructed image segmentation network model is trained by using the fusion feature. Specifically, the fusion feature is input into the image segmentation network model, the image feature in the fusion feature is extracted, the image segmentation network loss is calculated based on the preset image segmentation network loss function according to the extracted image feature in the fusion feature, and the image segmentation network model is optimized according to the image segmentation network loss to obtain the trained image segmentation network.

[0077] During the training process, a segmentation loss function (such as a cross-entropy loss function) is used to measure the difference between the network output and the label, and a gradient descent optimization algorithm is used to update the network parameters, so that the image segmentation network learns to train the segmentation result of the existing panoramic oral film sample that realizes accurate segmentation as the target, obtains the trained image segmentation network, and then maps the input image feature to the original image size and generates the segmentation result by using the trained image segmentation network.

[0078] It should be noted that in actual application, a large amount of training data and computing resources are often needed to support training to ensure the accuracy of the segmentation result.

[0079] Step S430: obtaining the feature recognition result of the panoramic oral film based on the interpretation text and the image segmentation result.

[0080] Specifically, the image feature recognition result is obtained according to the image segmentation result.

[0081] The feature recognition result of the panoramic oral film is obtained based on the interpretation text and the image feature recognition result, so as to further improve the image feature recognition result by the interpretation text, and improve the accuracy, effectiveness and comprehensiveness of the feature recognition result of the panoramic oral film.

[0082] In the process of training each neural network model in the embodiment, the loss function that can be selected includes cross-entropy loss function, 0-1 loss function, absolute value loss function, logarithmic loss function, square loss function, etc., which can be selected according to actual application needs.

[0083] Further, as other preferred embodiments, the feature recognition result of the panoramic oral film can be used to calculate the oral feature index parameters. For example, the distance, angle and symmetry between teeth are calculated by using the image segmentation result, so as to evaluate whether the number of teeth and the arrangement of teeth are normal; by analyzing the morphological features of teeth and periodontal tissues, tooth lesions such as dental caries, periodontitis and facial tumors are detected; by analyzing the bone features (such as bone density, fracture, etc.) in the segmented image, corresponding medical indexes such as bone density and fracture are calculated.

[0084] For example, as shown in the panoramic oral film Figure 2 and the corresponding oral state description text of the panoramic oral film is that the maxilla lacks one tooth, after the processing of steps S100-S400, the interpretation text related to the fusion features is that the maxilla lacks one tooth on the left side, which verifies the effectiveness of the method of the present application.

[0085] For another example, a panoramic oral film (not shown in the figure) is known, and the corresponding oral state description text of the panoramic oral film is that the gum atrophy and the oral cyst, after the processing of steps S100-S400, the interpretation text related to the fusion features is that the gum atrophy appears on the right side of the maxilla, and the oral cyst of 0.5 millimeter appears on the left side of the mandible, which also verifies the effectiveness of the method of the present application.

[0086] The embodiment extracts image features from the collected panoramic oral film by using the trained image encoder, extracts text features from the collected oral state description text by using the trained text encoder, and takes the image features and the text features as the data set for feature extraction of the panoramic oral film; then the trained neural network is used to fuse the image features and the text features to obtain the fused features, so as to realize the supplement and improvement of the text features to the image features; and then the explanation text and the picture segmentation result of the fused features are obtained based on the fused features respectively, so as to verify each other to obtain the feature recognition result of the panoramic oral film, and the obtained feature recognition result of the panoramic oral film can be a strong explanation of the feature recognition result of the panoramic oral film, and the accuracy, effectiveness and comprehensiveness of the feature recognition result of the panoramic oral film can be improved.

[0087] Exemplary system

[0088] As shown in Figure 3 corresponding to the above-mentioned neural network-based panoramic oral film feature recognition method, the embodiment of the present application also provides a neural network-based panoramic oral film feature recognition system, which comprises:

[0089] The data acquisition module 310 is used to collect the panoramic oral film and the oral state description text corresponding to the panoramic oral film.

[0090] The feature extraction module 320 is used to extract image features from the panoramic oral film by using the trained image encoder, and extract text features from the oral state description text by using the trained text encoder.

[0091] The feature fusion module 330 is used to fuse the image features and the text features to obtain the fused features.

[0092] The feature recognition module 340 is used to obtain the explanation text and the picture segmentation result of the fused features based on the fused features, and obtain the feature recognition result of the panoramic oral film based on the explanation text and the picture segmentation result.

[0093] Further, it further comprises an oral feature index extraction module for calculating the oral feature index parameters by using the feature recognition result of the panoramic oral film.

[0094] Specifically, in the embodiment, the specific functions of the above-mentioned neural network-based panoramic oral film feature recognition system can also be referred to the corresponding description in the above-mentioned neural network-based panoramic oral film feature recognition method, which will not be repeated here.

[0095] Based on the above-mentioned embodiment, the present application also provides an intelligent terminal, and the principle block diagram thereof can be as shown in Figure 4The intelligent terminal shown in the figure includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a neural network-based oral panoramic radiograph feature recognition program. The internal memory provides an environment for the operating system and the neural network-based oral panoramic radiograph feature recognition program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with external terminals through network connection. The neural network-based oral panoramic radiograph feature recognition program, when executed by the processor, implements the steps of any one of the neural network-based oral panoramic radiograph feature recognition methods described above. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0096] Those skilled in the art can understand that, Figure 4 The block diagram shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the intelligent terminal to which the present application is applied. The specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0097] In one embodiment, an intelligent terminal is provided, which includes a memory, a processor, and a neural network-based oral panoramic radiograph feature recognition program stored in the memory and executable on the processor. The neural network-based oral panoramic radiograph feature recognition program, when executed by the processor, implements the steps of any one of the neural network-based oral panoramic radiograph feature recognition methods provided by the present application.

[0098] The present application also provides a computer-readable storage medium having a neural network-based oral panoramic radiograph feature recognition program stored thereon. The neural network-based oral panoramic radiograph feature recognition program, when executed by a processor, implements the steps of any one of the neural network-based oral panoramic radiograph feature recognition methods provided by the present application.

[0099] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0101] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0102] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different ways to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0103] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented by other ways. For example, the above-mentioned device / terminal equipment embodiments are only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0104] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand; the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not deviate from the spirit and scope of the corresponding technical solutions, and should be included in the protection scope of the present application.

Claims

1. A method for recognizing features of an oral panoramic radiograph based on a neural network, characterized by, The method comprises the following steps: collecting an oral panoramic radiograph and an oral state description text corresponding to the oral panoramic radiograph; extracting image features from the oral panoramic radiograph by using a trained image encoder and extracting text features from the oral state description text by using a trained text encoder; fusing the image features and the text features to obtain fused features; obtaining an explanation text and a picture segmentation result of the fused features based on the fused features, and obtaining a feature recognition result of the oral panoramic radiograph based on the explanation text and the picture segmentation result; the fusing the image features and the text features to obtain fused features comprises: fusing the image features and the text features by using a trained fusion neural network to obtain fused features, wherein the training process of the trained fusion neural network comprises: collecting a plurality of oral panoramic radiographs and a plurality of oral state description texts to construct an oral feature sample set; constructing a fusion neural network model; inputting the oral feature sample set into the fusion neural network model to extract oral features in the oral feature sample set, calculating a fusion neural network loss based on a preset fusion neural network loss function according to the oral features, optimizing the fusion neural network model according to the fusion neural network loss, and obtaining a trained fusion neural network.

2. The neural network-based dental panoramic radiograph feature recognition method of claim 1, wherein, the training process of the trained image encoder comprises: collecting a plurality of oral panoramic radiographs to construct an oral panoramic radiograph sample set; constructing an image encoder model; inputting the oral panoramic radiograph sample set into the image encoder model to extract image features of the oral panoramic radiographs in the oral panoramic radiograph sample set, calculating an image encoder loss based on a preset image encoder loss function according to the image features, optimizing the image encoder model according to the image encoder loss, and obtaining a trained image encoder. 3.The neural network-based dental panoramic radiograph feature recognition method of claim 1, wherein, the obtaining an explanation text and a picture segmentation result of the fused features based on the fused features comprises: decoding the fused features by using a trained text decoder to obtain an explanation text of the fused features; segmenting the fused features by using a trained image segmentation network to obtain a picture segmentation result.

4. The neural network-based dental panoramic radiograph feature recognition method of claim 3, wherein, the training process of the text decoder comprises: constructing a text decoder model; inputting the fused features into the text decoder model to extract an explanation text in the fused features, calculating a text decoder loss based on a preset text decoder loss function according to the explanation text, optimizing the text decoder model according to the text decoder loss, and obtaining a trained text decoder.

5. The neural network-based dental panoramic radiograph feature recognition method according to any one of claims 1-4, characterized in that, Further comprising: calculating an oral feature index parameter by using the feature recognition result of the oral panoramic radiograph.

6. A neural network-based oral panoramic radiograph feature recognition system, characterized by, The system comprises: a data collection module configured to collect an oral panoramic radiograph and an oral state description text corresponding to the oral panoramic radiograph; a feature extraction module configured to extract image features from the oral panoramic radiograph by using a trained image encoder and extract text features from the oral state description text by using a trained text encoder; The feature fusion module is configured to fuse the image features and the text features to obtain fused features. The feature recognition module is configured to obtain an explanation text and a picture segmentation result of the fused features based on the fused features, and obtain a feature recognition result of the panoramic oral film based on the explanation text and the picture segmentation result. The feature fusion module is configured to fuse the image features and the text features to obtain fused features. The image features and the text features are fused by using the trained fusion neural network to obtain fused features. A plurality of panoramic oral films and a plurality of oral state description texts are collected to construct an oral feature sample set. A fusion neural network model is constructed. The oral feature sample set is input into the fusion neural network model to extract oral features in the oral feature sample set, and a fusion neural network loss is calculated based on a preset fusion neural network loss function and the oral features.

7. The neural network-based dental panoramic radiograph feature identification system of claim 6, wherein, The method further includes: An oral feature index extraction module is configured to calculate an oral feature index parameter based on the feature recognition result of the panoramic oral film.

8. The intelligent terminal, characterized by The intelligent terminal includes a memory, a processor, and a neural network-based panoramic oral film feature recognition program stored in the memory and executable on the processor, and the neural network-based panoramic oral film feature recognition program, when executed by the processor, implements the steps of the neural network-based panoramic oral film feature recognition method according to any one of claims 1-5.

9. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a neural network-based panoramic oral film feature recognition program, and the neural network-based panoramic oral film feature recognition program, when executed by the processor, implements the steps of the neural network-based panoramic oral film feature recognition method according to any one of claims 1-5.

Images