Caries identification method, device, equipment and medium based on deep learning

Abstract

The present invention relates to the field of deep learning technology, and in particular to a method, device, equipment and medium for dental caries recognition based on deep learning. The method optimizes the deep learning network, that is, utilizes the feature fusion network to fuse the difference features, so as to integrate the complementary advantages of these difference features and improve the recognition performance of dental caries; utilizes the semantic transfer network to restore the fused difference features to the shallow position, so as to improve the semantic cognition ability of dental caries; utilizes the shared knowledge learning network to interactively learn the shared knowledge between the dental caries position task and the dental caries depth task in the channel dimension and the spatial dimension, so as to achieve high-precision and high-efficiency dental caries position and dental caries depth. Therefore, the above technical scheme can effectively improve the recognition accuracy of dental caries.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a caries identification method, device, equipment and medium based on deep learning. Background Art

[0002] Early identification of oral health problems, especially caries, is crucial to preventing further tooth damage and providing timely treatment. Early caries are difficult to distinguish with the naked eye, especially caries that occur on adjacent surfaces of teeth. Deep learning technology has become an auxiliary means of diagnosing oral diseases, and deep learning technology can be used to further realize the feature recognition of caries.

[0003] However, the dental caries recognition model in the related art has a technical problem of poor recognition accuracy when performing dental caries feature recognition.

[0004] Based on this, the present invention proposes a caries identification method, device, equipment and medium based on deep learning to solve the above technical problems. Summary of the invention

[0005] The present invention describes a method, device, equipment and medium for identifying dental caries based on deep learning, which can effectively improve the recognition accuracy of dental caries.

[0006] According to a first aspect, the present invention provides a method for identifying dental caries based on deep learning, comprising:

[0007] Acquire a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein the imaging field of the first oral CBCT image is no greater than 8×8, and the imaging field of the second oral CBCT image is greater than 8×8 but not greater than 15×15;

[0008] Inputting the first oral CBCT image and the second oral CBCT image into a trained caries recognition model to obtain a caries recognition result of the target patient; wherein the caries recognition model is obtained by training a preset deep learning network using labeled oral CBCT images;

[0009] The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes a plurality of sequentially connected encoding blocks for extracting difference features under different imaging fields of view.

[0010] The input end of the feature fusion network is respectively connected to the output end of the last encoding block of each sub-encoder, so as to perform feature fusion on the difference features;

[0011] The input end of the semantic transfer network is connected to the output end of the feature fusion network, so as to restore the fused difference features to a shallow position;

[0012] The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result.

[0013] According to a second aspect, the present invention provides a caries identification device based on deep learning, comprising:

[0014] An acquisition unit is configured to acquire a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein an imaging field of view of the first oral CBCT image is no greater than 8×8, and an imaging field of view of the second oral CBCT image is greater than 8×8 but not greater than 15×15;

[0015] A recognition unit is configured to input the first oral CBCT image and the second oral CBCT image into a trained caries recognition model to obtain a caries recognition result of the target patient; wherein the caries recognition model is obtained by training a preset deep learning network using labeled oral CBCT images;

[0016] The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes a plurality of sequentially connected encoding blocks for extracting difference features under different imaging fields of view.

[0017] The input end of the feature fusion network is respectively connected to the output end of the last encoding block of each sub-encoder, so as to perform feature fusion on the difference features;

[0018] The input end of the semantic transfer network is connected to the output end of the feature fusion network, so as to restore the fused difference features to a shallow position;

[0019] The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result.

[0020] According to a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method of the first aspect is implemented.

[0021] According to a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method of the first aspect.

[0022] According to the deep learning-based caries identification method, device, equipment and medium provided by the present invention, by optimizing the deep learning network, that is, using the feature fusion network to fuse the difference features, so as to integrate the complementary advantages of these difference features and improve the recognition performance of caries; using the semantic transfer network to restore the fused difference features to the shallow position, so as to improve the semantic cognition ability of caries; using the shared knowledge learning network to interactively learn the shared knowledge between the caries position task and the caries depth task in the channel dimension and the spatial dimension, so as to achieve high-precision and high-efficiency caries position and caries depth. Therefore, the above technical solution can effectively improve the recognition accuracy of caries. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a process of a caries identification method based on deep learning according to an embodiment is shown;

[0025] Figure 2 A schematic block diagram of a caries identification device based on deep learning according to an embodiment is shown;

[0026] Figure 3 A schematic diagram of the structure of a caries recognition model according to an embodiment is shown;

[0027] Figure 4 for Figure 3 The schematic diagram of the structure of the feature fusion network in the caries recognition model shown;

[0028] Figure 5 for Figure 3 A schematic diagram of a portion of the structure of the semantic transfer network in the caries recognition model shown;

[0029] Figure 6 for Figure 3 Another schematic diagram of the structure of the semantic transfer network in the caries recognition model shown;

[0030] Figure 7 for Figure 3 Schematic diagram of the structure of the shared knowledge learning network in the caries recognition model shown. DETAILED DESCRIPTION

[0031] The solution provided by the present invention is described below in conjunction with the accompanying drawings.

[0032] Figure 1 FIG. 1 is a flow chart of a method for identifying caries based on deep learning according to an embodiment. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 As shown, the method includes:

[0033] Step 101, obtaining a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein the imaging field of the first oral CBCT image is no larger than 8×8, and the imaging field of the second oral CBCT image is larger than 8×8 but not larger than 15×15;

[0034] Step 102: Input the first oral CBCT image and the second oral CBCT image into a trained caries recognition model (such as Figure 3 As shown), the caries recognition result of the target patient is obtained; wherein the caries recognition model is obtained by training a preset deep learning network using labeled oral CBCT images;

[0035] The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network, and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes multiple sequentially connected encoding blocks to extract differential features under different imaging fields of view.

[0036] The input end of the feature fusion network is connected to the output end of the last encoding block of each sub-encoder to perform feature fusion on the difference features;

[0037] The input end of the semantic transfer network is connected to the output end of the feature fusion network to restore the fused difference features to the shallow position;

[0038] The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result.

[0039] In this embodiment, by optimizing the deep learning network, that is, using the feature fusion network to fuse the difference features, so as to integrate the complementary advantages of these difference features, the recognition performance of caries is improved; using the semantic transfer network to restore the fused difference features to the shallow position, so as to improve the semantic cognition ability of caries; using the shared knowledge learning network to interactively learn the shared knowledge between the caries position task and the caries depth task in the channel dimension and the spatial dimension, so as to achieve high-precision and high-efficiency caries position and caries depth. Therefore, the above technical solution can effectively improve the recognition accuracy of caries.

[0040] It should be noted that CBCT, the full name of which is "Cone-Beam Computed Tomography (CBCT)", consists of a tube and a flat-panel detector. Unlike the closed-loop circular hole design of general CT, digital oral CBCT adopts an open structure to maintain its flexibility. For oral diseases, dental films and panoramic films have a high diagnostic effect; but for dental pulp diseases, 2D imaging is prone to image overlap, and it is impossible to further clarify the three-dimensional information of the root canal and other surrounding tissues, which is prone to missed diagnosis or misdiagnosis of root fractures. This requires three-dimensional tomographic imaging, namely digital oral CBCT. Digital oral CBCT displays diseased tissue and normal tissue structure from a three-dimensional perspective, namely sagittal, coronal and transverse positions, effectively avoiding the inherent defects of 2D images, such as image overlap of normal tissues and organs, image distortion and deformation caused by changes in projection angles, etc., greatly improving diagnostic capabilities. Digital oral CBCT can complete the patient's oral scan in more than ten seconds, and then generate high-precision three-dimensional images, which completely solves the problem of image overlap and distortion of traditional oral X-ray equipment such as dental film machines and panoramic machines. It can provide 360° all-round stereoscopic observation of the bone density of the gums, the height and width of the alveolar bone itself and other data. It can not only reconstruct the structure of the entire craniofacial area and measure the maxillofacial bone structure, but also reconstruct fine anatomical details.

[0041] However, the imaging field of view (FOV), i.e., the cylindrical stereoscopic data obtained by the CBCT rotating around the human body, is equal to the diameter of the cylinder × height (D × H), which is generally determined by factors such as the detector size and shape, X-ray geometry and calibration. The imaging field of view is one of the important parameter indicators of oral CBCT, which determines how much oral anatomical information of clinical significance will be contained in a single image (without any splicing). During the research and development process, the inventors found that considering the possibility of missed detection of caries, it is necessary to conduct a comprehensive analysis of oral CBCT images with different imaging fields, so as to reduce the missed detection rate of caries, that is, to improve the recognition accuracy of caries.

[0042] In some embodiments, the features of oral CBCT images with different imaging fields are different, and the deep features have rich semantic information of caries, which will not be elaborated here. The imaging field of the first oral CBCT image is ≤ (unit: cm, the same below), which is mainly used for imaging of local dentition, local jaw and unilateral joints, and can meet the needs of root canal treatment and restoration. The disadvantage is that it cannot present full-mouth images, so it is also called root canal CBCT; the imaging field of the second oral CBCT image is >8×8 but ≤15×15. Its biggest advantage is that the full-mouth dentition can be fully displayed in one scan, which meets the clinical applications of orthodontics, implants, etc. for most patients. Therefore, the inventors conducted a comprehensive analysis of oral CBCT images with different imaging fields to improve the recognition accuracy of caries.

[0043] It should be noted that Figure 3 The first oral CBCT image and the second oral CBCT image shown are not real CBCT images, but different shooting angles displayed to reflect the different fields of view of the two images, and the real images are the output images corresponding to the two fields of view.

[0044] like Figure 4 As shown in FIG. 1 , as a preferred implementation, the feature fusion network completes the feature fusion of the difference features by using the following formula:

[0045]

[0046] Z i =f 3×3 [U i +F U,i ,E i +F E,i ]

[0047] Where, 1≤i≤4; E i The feature map generated by the i-th encoding block of one of the sub-encoders; U i The feature map generated by the i-th encoding block of another sub-encoder; f 1×1 is a 1×1 convolution operation to convert E i and U i The number of channels is expanded from iC to 3×iC, and the dimension change operation Split is used to convert E i and U i The number of channels is evenly divided into three parts, and the final feature vector Q is generated i , K i 、V i , and C, H and W represent the number of channels, height and width respectively; the Softmax function is applied to Qi and After transposition and Multiply, and the Softmax function is applied to and K i , K i After transposition and V i Multiply, F E,i and F U,i They represent the results of the interaction between the two types of image information, respectively, and f 3×3 is a 3×3 convolution operation, and Zi represents the final i-th fused feature map of the feature map.

[0048] In this embodiment, the feature map E generated by the encoder i and U i The feature fusion network is used to achieve efficient fusion of hierarchical features and complete the information complementation of different image features. Specifically, a 1×1 convolution is used on E i and U i On the top, generate a feature vector Q of equal dimension i , K i 、V i , and To maximize the expression of the feature information contained in different images. In order to realize the interaction of semantic information of different image feature maps, E i Q i As a query set, U i of As a "key-value" set, it can interact with the semantic information of different images, so that each element in the feature vector of different images can be globally semantically associated, and the degree of global semantic association between different images is measured. Softmax is used to normalize the similarity matrix, evaluate the contribution of each feature point, and compare it with By performing matrix multiplication, the features of the second oral CBCT image can be integrated with the features of the first oral CBCT image. When O(i 2 C 2 N i +iCN i 2 ) complexity, and increases rapidly with the increase of the spatial resolution of the feature map, which greatly limits the efficiency of image information fusion. Based on this, this application applies the Softmax function to Q i iC dimensions and N i Dimension, so we get This satisfies the normalization of the mathematical form and can also reduce the complexity of the calculation method required for the calculation. i of As a query set, E i K i 、V i As a "key-value" set, and perform a fusion operation similar to the above. Finally, the residual idea and channel stacking operation are used to obtain the final feature map fusion result and generate the final Z i It fully realizes the efficient integration of different image information.

[0049] like Figure 5 and Figure 6 As shown in FIG. 1 , as a preferred implementation, the semantic transfer network restores the fused difference features to the shallow position by using the following formula:

[0050]

[0051] T=Softmax(f 1×1 (D2))Softmax(UP(D3)) T UP(D4)

[0052] D1=Concat(f 3×3 (T), Z1)

[0053] Where D i represents the fast decoding block f ED Output decoding feature map, when i = 4, D4 is equivalent to Z4 after f 1×1 The feature map after the function dimension reduction; f ED It includes a first 3×3 convolution block and a second 3×3 convolution block and a third 3×3 convolution block respectively connected to the first 3×3 convolution block, and then adds the output results of the second 3×3 convolution block and the third 3×3 convolution block; UP means upsampling; Softmax function is applied to f respectively 1×1 (Z2) and UP(Z3), UP(Z3) is transposed and then multiplied with UP(Z4), T represents the intermediate feature map; Concat represents the fusion operation; D1 represents the semantic transfer network generating the final decoding feature map.

[0054] In this embodiment, the semantic transfer network is used to enrich the semantic information of shallow features, so that spatial details and semantic information are efficiently integrated. The semantic transfer network uses 1×1 convolution to transform Z i The number of channels iC is uniformly reduced to 64 (iC ≥ 64) to reduce the computational complexity, and then 3×3 convolution is used to extract spatial detail information. Although the semantic transfer network achieves efficient fusion of cross-layer features and effectively integrates deep features and shallow features, this dimensionality reduction method has the risk of semantic attenuation. To this end, this application uses Figure 5The structure quickly transfers the semantic information of deep features to shallow positions, improving the model's full utilization of the semantic information of deep features. Figure 5 and Figure 6 It can be seen that the intermediate feature map T not only has the channel semantic information 4C of the deep feature map D4, but also contains the spatial detail information of the shallow feature map D2 In this way, the semantic transfer network completes the transition from deep feature semantic information to shallow features. Then, the intermediate feature map T reduces the number of channels 4C to 64 through 3×3 convolution, and generates the final decoded feature map through Concat operation with Z1

[0055] like Figure 7 As shown, as a preferred implementation, the shared knowledge learning network uses the following formula to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension:

[0056]

[0057] Where Avgpool represents average pooling, Maxpool represents maximum pooling, t1 represents the caries position feature map generated by D1 using 3×3 convolution, and t2 represents the caries depth feature map generated by D1 using 3×3 convolution; I C are two eigenvectors I C1 and I C2 Channel stacking result; I S are two eigenvectors I S1 and I S2 The spatial stacking result of represents the one-dimensional feature vector after convolution modeling; f 5×5 is a 5×5 convolution operation, f 7×7 is a 7×7 convolution operation, f 9×9 is a 9×9 convolution operation; || 2 represents a split function; and They represent the feature vectors of different tasks t1 and t2 after semantic interaction; and They represent the spatial feature matrices corresponding to tasks t1 and t2 respectively; P1 and P2 represent the final prediction results corresponding to tasks t1 and t2 respectively.

[0058] In this embodiment, the shared knowledge learning network aims to share knowledge between different tasks through channel dimension and spatial dimension, so as to promote the complementary and differential learning between tasks. The multi-task learning mechanism first uses 3×3 convolution to convert the feature map D1 into a caries position feature map and caries depth feature map Then, the three units of information compression, interactive learning and feature activation are used to exchange useful information between different tasks. The information compression unit includes two parts: channel compression and spatial compression. Channel compression uses global adaptive average pooling to convert the spatial dimensions of t1 and t2 from becomes 1×1, and a total of two channel semantic vectors I are obtained C1 and I C ∈R 2C×1×1 are two eigenvectors I C1 and I C2 The channel stacking result highly summarizes the semantic information of two different tasks. In addition, spatial compression uses maximum pooling and average pooling respectively, changing the spatial dimensions of t1 and t2 from C to 1, and obtaining two spatial feature matrices I S1 and Contains the spatial detail information of the two tasks respectively. So far, the semantic information feature map I C and spatial feature map I S They include the channel information and spatial detail information of different task feature maps respectively. Interactive learning aims to learn the shared knowledge of channel semantics and spatial details of different tasks through multiple convolutions, that is, to learn semantic information I C and space details I S , thereby realizing the information interaction of different task feature graphs. Among them, the effect on the feature vector I C The 1×1 convolution is essentially a fully connected neural network, which establishes the connection between the dimensions of different task channels. 1×1 The semantic shared knowledge of the convolutional learning task, the ReLu activation function performs nonlinear mapping, and the obtained It fully characterizes the semantic dependency pattern across tasks. In addition, in order to process and realize local perception of caries at multiple scales, this application uses three convolution kernels of different sizes, 5×5, 7×7 and 9×9, to perceive neighborhood information, thereby realizing local spatial relationship modeling under different task mechanisms. Feature activation is to normalize the generated semantic interaction information and spatial interaction information through the Sigmoid activation function, and perform element-wise multiplication operations with the feature maps of different tasks, so as to realize the potential association of shared knowledge among different tasks and improve the feature interaction capability between multiple tasks. For the channel dimension, Sigmoid is used to Normalize and then split into two feature vectors and Similarly, in the spatial dimension, after using the Sigmoid normalization function, the spatial matrix is ​​converted to and Split them into spatial attention feature maps corresponding to tasks t1 and t2 one by one, and superimpose them in the channel dimension. Finally, use matrix dot multiplication to multiply the original feature map to get the final prediction result. In summary, the shared knowledge learning network learns the shared knowledge of different tasks in the channel dimension and spatial dimension, and uses the Sigmoid activation function to feed back this complementary information to the original task feature map, so as to capture the commonalities and differences between tasks and realize efficient interaction of spatial semantic information of different tasks.

[0059] In summary, the above-mentioned caries recognition model introduces a differential feature fusion mechanism (i.e., feature fusion network) to dynamically associate the global relationship between features and achieve efficient fusion of two types of image information; uses a deep feature semantic transfer scheme (i.e., semantic transfer network) to alleviate the information attenuation problem of deep semantic information in the decoding network and improve the semantic recognition ability of caries features; samples a multi-task shared knowledge learning strategy (i.e., shared knowledge learning network) to interactively perceive the spatial semantic similarity between multiple tasks and improve the caries extraction performance and caries depth accuracy.

[0060] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] According to another embodiment, the present invention provides a caries identification device based on deep learning. Figure 2 A schematic block diagram of a caries identification device based on deep learning according to an embodiment is shown. It can be understood that the device can be implemented by any device, equipment, platform and device cluster with computing and processing capabilities. Figure 2 As shown, the device includes: an acquisition unit 201 and an extraction unit 202. The main functions of each component unit are as follows:

[0062] An acquisition unit 201 is configured to acquire a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein the imaging field of the first oral CBCT image is no greater than 8×8, and the imaging field of the second oral CBCT image is greater than 8×8 but not greater than 15×15;

[0063] The recognition unit 202 is configured to input the first oral CBCT image and the second oral CBCT image into a trained caries recognition model to obtain a caries recognition result of a target patient; wherein the caries recognition model is obtained by training a preset deep learning network using the labeled oral CBCT image;

[0064] The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network, and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes multiple sequentially connected encoding blocks to extract differential features under different imaging fields of view.

[0065] The input end of the feature fusion network is connected to the output end of the last encoding block of each sub-encoder to perform feature fusion on the difference features;

[0066] The input end of the semantic transfer network is connected to the output end of the feature fusion network to restore the fused difference features to the shallow position;

[0067] The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result.

[0068] As a preferred implementation, the feature fusion network completes the feature fusion of the difference features by using the following formula:

[0069]

[0070] Z i =f 3×3 [U i +F U,i ,E i +F E,i ]

[0071] Where, 1≤i≤4; E i The feature map generated by the i-th encoding block of one of the sub-encoders; U i The feature map generated by the i-th encoding block of another sub-encoder; f 1×1 is a 1×1 convolution operation to convert E i and U i The number of channels is expanded from iC to 3×iC, and the dimension change operation Split is used to convert E i and U i The number of channels is evenly divided into three parts, and the final feature vector Q is generated i , K i 、V i , and C, H and W represent the number of channels, height and width respectively; the Softmax function is applied to Q i and After transposition and Multiply, and the Softmax function is applied to and K i , K i After transposition and V i Multiply, F E,i and F U,i They represent the results of the interaction between the two types of image information, respectively, and f 3×3 is a 3×3 convolution operation, Z i Represents the final i-th fused feature map of the feature map.

[0072] As a preferred implementation, the semantic transfer network restores the fused difference features to a shallow position by using the following formula:

[0073]

[0074] T=Softmax(f 1×1 (D2))Softmax(UP(D3)) T UP(D4)

[0075] D1=Concat(f 3×3 (T), Z1)

[0076] Where D i represents the fast decoding block f ED Output decoding feature map, when i = 4, D4 is equivalent to Z4 after f 1×1 The feature map after the function dimension reduction; f ED It includes a first 3×3 convolution block and a second 3×3 convolution block and a third 3×3 convolution block respectively connected to the first 3×3 convolution block, and then performs an addition operation on the output results of the second 3×3 convolution block and the third 3×3 convolution block; UP represents upsampling; Softmax function is applied to f respectively 1×1 (Z2) and UP(Z3), UP(Z3) is transposed and then multiplied with UP(Z4), T represents the intermediate feature map; Concat represents the fusion operation; D1 represents that the semantic transfer network generates the final decoding feature map.

[0077] As a preferred implementation, the shared knowledge learning network interactively learns the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension by using the following formula:

[0078]

[0079] Where Avgpool represents average pooling, Maxpool represents maximum pooling, t1 represents the caries position feature map generated by D1 using 3×3 convolution, and t2 represents the caries depth feature map generated by D1 using 3×3 convolution; I C are two eigenvectors I C1 and I C2 Channel stacking result; I S are two eigenvectors I S1 and I S2 The spatial stacking result of represents the one-dimensional feature vector after convolution modeling; f 5×5 is a 5×5 convolution operation, f 7×7 is a 7×7 convolution operation, f 9×9 is a 9×9 convolution operation; || 2 represents a split function; and They represent the feature vectors of different tasks t1 and t2 after semantic interaction; and They represent the spatial feature matrices corresponding to tasks t1 and t2 respectively; P1 and P2 represent the final prediction results corresponding to tasks t1 and t2 respectively.

[0080] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method described.

[0081] According to another embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the Figure 1 method.

[0082] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0083] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0084] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying dental caries based on deep learning, characterized in that: include: Acquire a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein the imaging field of the first oral CBCT image is no greater than 8×8, and the imaging field of the second oral CBCT image is greater than 8×8 but not greater than 15×15; Inputting the first oral CBCT image and the second oral CBCT image into a trained caries recognition model to obtain a caries recognition result of the target patient; wherein the caries recognition model is obtained by training a preset deep learning network using labeled oral CBCT images; The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes a plurality of sequentially connected encoding blocks for extracting difference features under different imaging fields of view. The input end of the feature fusion network is respectively connected to the output end of the last encoding block of each sub-encoder, so as to perform feature fusion on the difference features; The input end of the semantic transfer network is connected to the output end of the feature fusion network, so as to restore the fused difference features to a shallow position; The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result; The feature fusion network completes the feature fusion of the difference features by using the following formula: Z i =f 3×3 [U i +F U,i ,E i +F E,i ] Where, 1≤i≤4; E i The feature map generated by the i-th encoding block of one of the sub-encoders; U i The feature map generated by the i-th encoding block of another sub-encoder; f 1×1 is a 1×1 convolution operation to convert E i and U i The number of channels is expanded from iC to 3×iC, and the dimension change operation Split is used to convert E i and U i The number of channels is evenly divided into three parts, and the final feature vector Q is generated i , K i 、V i , and C, H and W represent the number of channels, height and width respectively; the Softmax function is applied to Q i and After transposition and Multiply, and the Softmax function is applied to and K i , K i After transposition and V i Multiply, F E,i and F U,i They represent the results of the interaction between the two types of image information, and f 3×3 is a 3×3 convolution operation, Z i Represents the final i-th fused feature map of the feature map; the feature map E generated by the encoder i and U i The feature fusion network is used to achieve efficient fusion of hierarchical features and complete the information complementarity of different image features. A 1×1 convolution is used on E i and U i On the top, generate a feature vector Q of equal dimension i , K i 、V i , and To express the feature information contained in different images; to achieve the interaction of semantic information of different image feature maps, E i Q i As a query set, U i of As a "key-value" set, it can interact with the semantic information of different images, so that each element in the feature vectors of different images can be globally semantically associated, and the degree of global semantic association between different images is measured; the similarity matrix is ​​normalized using Softmax to evaluate the contribution of each feature point and compared with Matrix multiplication is performed to achieve the fusion of the second oral CBCT image feature and the first oral CBCT image feature; the Softmax function is applied to Q i iC dimensions and N i Dimension, so we get It satisfies the normalization of mathematical form and can also reduce the complexity of the calculation method required for calculation; then U i of As a query set, E i K i 、V i As a "key-value" set, and perform a fusion operation similar to the above; finally, use the residual idea and channel stacking operation to obtain the final feature map fusion result and generate the final Z i Fully realize the efficient integration of different image information; The semantic transfer network restores the fused difference features to the shallow position by using the following formula: T=Softmax(f 1×1 (D2))Softmax(UP(D3)) T UP(D4) D1=Concat(f 3×3 (T),Z1) Where D i represents the fast decoding block f ED Output decoding feature map, when i = 4, D4 is equivalent to Z4 after f 1×1 Feature map after function dimensionality reduction; UP means upsampling; f ED The first 3×3 convolution block and the second 3×3 convolution block and the third 3×3 convolution block are connected to the first 3×3 convolution block respectively. 1×1 (Z i ) and UP(D i+1 ), after performing the addition operation, the first 3×3 convolution block and the second 3×3 convolution block are used to perform a convolution operation, and the first 3×3 convolution block and the third 3×3 convolution block are used to perform a convolution operation, and then the results of the two convolution operations are added to obtain D i ; Softmax function is applied to f 1×1 (Z2) and UP(Z3), UP(Z3) is transposed and multiplied with UP(Z4) to obtain T, where T represents the intermediate feature map; Concat represents the fusion operation; D1 represents the semantic transfer network generating the final decoding feature map; the semantic transfer network is used to enrich the semantic information of the shallow features, so that the spatial details and semantic information are efficiently fused; the semantic transfer network uses 1×1 convolution to transform Z i The number of channels iC is uniformly reduced to 64, iC ≥ 64, to reduce the computational complexity, and then the spatial detail information is extracted through 3×3 convolution; the intermediate feature map T not only has the channel semantic information 4C of the deep feature map D4, but also contains the spatial detail information of the shallow feature map D2 In this way, the semantic transfer network completes the transition from deep feature semantic information to shallow features; then, the intermediate feature map T reduces the number of channels 4C to 64 through 3×3 convolution, and generates the final decoding feature map through Concat operation with Z1 The shared knowledge learning network interactively learns the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension by using the following formula: Where Avgpool represents average pooling, Maxpool represents maximum pooling, t1 represents the caries position feature map generated by convolving D1 with 3×3 convolution, and t2 represents the caries depth feature map generated by convolving D1 with 3×3 convolution; I C are two eigenvectors I C1 and I C2 Channel stacking result; I S are two eigenvectors I S1 and I S2 The spatial stacking result of represents the one-dimensional feature vector after convolution modeling; f 5×5 is a 5×5 convolution operation, f 7×7 is a 7×7 convolution operation, f 9×9 is a 9×9 convolution operation; || 2 represents a split function; and They represent the feature vectors of different tasks t1 and t2 after semantic interaction; and Represent the spatial feature matrices corresponding to tasks t1 and t2 respectively; P1 and P2 represent the final prediction results corresponding to tasks t1 and t2 respectively; the shared knowledge learning network aims to interact the shared knowledge between different tasks through channel dimensions and spatial dimensions, so as to promote the complementary and differential learning between tasks; first, a 3×3 convolution is used to convert the feature map D1 into a caries position feature map and caries depth feature map Then, the three units of information compression, interactive learning and feature activation are used to exchange useful information between different tasks. The information compression unit includes channel compression and spatial compression. Channel compression uses global adaptive average pooling to convert the spatial dimensions of t1 and t2 from becomes 1×1, and a total of two channel semantic vectors I are obtained C1 and I C ∈R 2C ×1×1 are two eigenvectors I C1 and I C2 The channel stacking results summarize the semantic information of two different tasks; spatial compression uses maximum pooling and average pooling respectively, changing the spatial dimensions of t1 and t2 from C to 1, and obtaining two spatial feature matrices I S1 and Contains the spatial detail information of the two tasks respectively; so far, the semantic information feature map I C and spatial feature map I S They include the channel information and spatial detail information of different task feature maps respectively; interactive learning aims to learn the shared knowledge of channel semantics and spatial details of different tasks through multiple convolutions, that is, to learn semantic information I C and space details I S , thus realizing the information interaction of different task feature graphs; among them, acting on the feature vector I C The 1×1 convolution is essentially a fully connected neural network, which establishes the connection between the dimensions of different task channels; the two f 1×1 The semantic shared knowledge of the convolutional learning task, the ReLu activation function performs nonlinear mapping, and the obtained It characterizes the semantic dependency pattern across tasks; uses three different sizes of convolution kernels, 5×5, 7×7, and 9×9, to perceive neighborhood information, thereby realizing local spatial relationship modeling under different task mechanisms; feature activation is to normalize the generated semantic interaction information and spatial interaction information through the Sigmoid activation function, and perform element-wise multiplication operations with the feature maps of different tasks, thereby realizing the potential association of shared knowledge among different tasks and improving the feature interaction capability between multiple tasks; for the channel dimension, Sigmoid is used to Normalize and then split into two feature vectors and Similarly, in the spatial dimension, after using the Sigmoid normalization function, the spatial matrix is ​​converted to and They are split into spatial attention feature maps corresponding to tasks t1 and t2 one by one and superimposed in the channel dimension; finally, they are multiplied with the original feature map using matrix dot multiplication to obtain the final prediction result.

2. A caries identification device based on deep learning, characterized in that: include: An acquisition unit is configured to acquire a first oral CBCT image and a second oral CBCT image taken for a target patient; wherein an imaging field of view of the first oral CBCT image is no greater than 8×8, and an imaging field of view of the second oral CBCT image is greater than 8×8 but not greater than 15×15; A recognition unit is configured to input the first oral CBCT image and the second oral CBCT image into a trained caries recognition model to obtain a caries recognition result of the target patient; wherein the caries recognition model is obtained by training a preset deep learning network using labeled oral CBCT images; The deep learning network includes an encoding network, a feature fusion network, a semantic transfer network and a shared knowledge learning network. The encoding network includes two sub-encoders, each of which includes a plurality of sequentially connected encoding blocks for extracting difference features under different imaging fields of view. The input end of the feature fusion network is respectively connected to the output end of the last encoding block of each sub-encoder, so as to perform feature fusion on the difference features; The input end of the semantic transfer network is connected to the output end of the feature fusion network, so as to restore the fused difference features to a shallow position; The input end of the shared knowledge learning network is connected to the output end of the semantic transfer network, and is used to interactively learn the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension to obtain the caries recognition result; The feature fusion network completes the feature fusion of the difference features by using the following formula: Z i =f 3×3 [U i +F U,i ,E i +F E,i ] Where, 1≤i≤4; E i The feature map generated by the i-th encoding block of one of the sub-encoders; U i The feature map generated by the i-th encoding block of another sub-encoder; f 1×1 is a 1×1 convolution operation to convert E i and U i The number of channels is expanded from iC to 3×iC, and the dimension change operation Split is used to convert E i and U i The number of channels is evenly divided into three parts, and the final feature vector Q is generated i , K i 、V i , and C, H and W represent the number of channels, height and width respectively; the Softmax function is applied to Q i and After transposition and Multiply, and the Softmax function is applied to and K i , K i After transposition and V i Multiply, F E,i and F U,i They represent the results of the interaction between the two types of image information, and f 3×3 is a 3×3 convolution operation, Z i Represents the final i-th fused feature map of the feature map; the feature map E generated by the encoder i and U i The feature fusion network is used to achieve efficient fusion of hierarchical features and complete the information complementarity of different image features. A 1×1 convolution is used on E i and U i On the top, generate a feature vector Q of equal dimension i , K i 、V i , and To express the feature information contained in different images; to achieve the interaction of semantic information of different image feature maps, E i Q i As a query set, U i of As a "key-value" set, it can interact with the semantic information of different images, so that each element in the feature vectors of different images can be globally semantically associated, and the degree of global semantic association between different images is measured; the similarity matrix is ​​normalized using Softmax to evaluate the contribution of each feature point and compared with Matrix multiplication is performed to achieve the fusion of the second oral CBCT image feature and the first oral CBCT image feature; the Softmax function is applied to Q i iC dimensions and N i Dimension, so we get It satisfies the normalization of mathematical form and can also reduce the complexity of the calculation method required for calculation; then U i of As a query set, E i K i 、V i As a "key-value" set, and perform a fusion operation similar to the above; finally, use the residual idea and channel stacking operation to obtain the final feature map fusion result and generate the final Z i Fully realize the efficient integration of different image information; The semantic transfer network restores the fused difference features to the shallow position by using the following formula: T=Softmax(f 1×1 (D2))Softmax(UP(D3)) T UP(D4) D1=Concat(f 3×3 (T),Z1) Where D i represents the fast decoding block f ED Output decoding feature map, when i = 4, D4 is equivalent to Z4 after f 1×1 Feature map after function dimensionality reduction; UP means upsampling; f ED The first 3×3 convolution block and the second 3×3 convolution block and the third 3×3 convolution block are connected to the first 3×3 convolution block respectively. 1×1 (Z i ) and UP(D i+1 ), after performing the addition operation, the first 3×3 convolution block and the second 3×3 convolution block are used to perform a convolution operation, and the first 3×3 convolution block and the third 3×3 convolution block are used to perform a convolution operation, and then the results of the two convolution operations are added to obtain D i ; Softmax function is applied to f 1×1 (Z2) and UP(Z3), UP(Z3) is transposed and multiplied with UP(Z4) to obtain T, where T represents the intermediate feature map; Concat represents the fusion operation; D1 represents the semantic transfer network generating the final decoding feature map; the semantic transfer network is used to enrich the semantic information of the shallow features, so that the spatial details and semantic information are efficiently fused; the semantic transfer network uses 1×1 convolution to transform Z i The number of channels iC is uniformly reduced to 64, iC ≥ 64, to reduce the computational complexity, and then the spatial detail information is extracted through 3×3 convolution; the intermediate feature map T not only has the channel semantic information 4C of the deep feature map D4, but also contains the spatial detail information of the shallow feature map D2 In this way, the semantic transfer network completes the transition from deep feature semantic information to shallow features; then, the intermediate feature map T reduces the number of channels 4C to 64 through 3×3 convolution, and generates the final decoding feature map through Concat operation with Z1 The shared knowledge learning network interactively learns the shared knowledge between the caries location task and the caries depth task in the channel dimension and the spatial dimension by using the following formula: Where Avgpool represents average pooling, Maxpool represents maximum pooling, t1 represents the caries position feature map generated by convolving D1 with 3×3 convolution, and t2 represents the caries depth feature map generated by convolving D1 with 3×3 convolution; I C are two eigenvectors I C1 and I C2 Channel stacking result; I S are two eigenvectors I S1 and I S2 The spatial stacking result of represents the one-dimensional feature vector after convolution modeling; f 5×5 is a 5×5 convolution operation, f 7×7 is a 7×7 convolution operation, f 9×9 is a 9×9 convolution operation; || 2 represents a split function; and They represent the feature vectors of different tasks t1 and t2 after semantic interaction; and Represent the spatial feature matrices corresponding to tasks t1 and t2 respectively; P1 and P2 represent the final prediction results corresponding to tasks t1 and t2 respectively; the shared knowledge learning network aims to interact the shared knowledge between different tasks through channel dimensions and spatial dimensions, so as to promote the complementary and differential learning between tasks; first, a 3×3 convolution is used to convert the feature map D1 into a caries position feature map and caries depth feature map Then, the three units of information compression, interactive learning and feature activation are used to exchange useful information between different tasks. The information compression unit includes channel compression and spatial compression. Channel compression uses global adaptive average pooling to convert the spatial dimensions of t1 and t2 from becomes 1×1, and a total of two channel semantic vectors I are obtained C1 and I C ∈R 2C ×1×1 are two eigenvectors I C1 and I C2 The channel stacking results summarize the semantic information of two different tasks; spatial compression uses maximum pooling and average pooling respectively, changing the spatial dimensions of t1 and t2 from C to 1, and obtaining two spatial feature matrices I S1 and Contains the spatial detail information of the two tasks respectively; so far, the semantic information feature map I C and spatial feature map I S They include the channel information and spatial detail information of different task feature maps respectively; interactive learning aims to learn the shared knowledge of channel semantics and spatial details of different tasks through multiple convolutions, that is, to learn semantic information I C and space details I S , thus realizing the information interaction of different task feature graphs; among them, acting on the feature vector I C The 1×1 convolution is essentially a fully connected neural network, which establishes the connection between the dimensions of different task channels; the two f 1×1 The semantic shared knowledge of the convolutional learning task, the ReLu activation function performs nonlinear mapping, and the obtained It characterizes the semantic dependency pattern across tasks; uses three different sizes of convolution kernels, 5×5, 7×7, and 9×9, to perceive neighborhood information, thereby realizing local spatial relationship modeling under different task mechanisms; feature activation is to normalize the generated semantic interaction information and spatial interaction information through the Sigmoid activation function, and perform element-wise multiplication operations with the feature maps of different tasks, thereby realizing the potential association of shared knowledge among different tasks and improving the feature interaction capability between multiple tasks; for the channel dimension, Sigmoid is used to Normalize and then split into two feature vectors and Similarly, in the spatial dimension, after using the Sigmoid normalization function, the spatial matrix is ​​converted to and They are split into spatial attention feature maps corresponding to tasks t1 and t2 one by one and superimposed in the channel dimension; finally, they are multiplied with the original feature map using matrix dot multiplication to obtain the final prediction result.

3. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method of claim 1.

Images