Postoperative nursing condition tracking auxiliary system for orthodontic patient

Abstract

The invention belongs to the field of nursing assistance, and particularly relates to an orthodontic patient postoperative nursing condition tracking assistance system which comprises a tooth image acquisition module, a tooth intelligent detection module, a data storage module, a doctor interface module, a patient interface module and a central control module. According to the method, the computer vision technology based on image segmentation is adopted to assist in accurately and intuitively judging the condition of the teeth of the patient; according to the method, a boundary feature extraction method based on feature cross fusion is creatively adopted, overall boundary information is obtained in a global mapping mode, local feature extraction is performed on the basis of the overall boundary information, multi-level feature cross fusion is performed, and accurate boundary extraction of teeth is realized; according to the invention, a multi-dimensional attention-based feature removal adjustment method is creatively adopted to detect the displacement and necrosis of the teeth, and a mask mechanism is introduced to increase the recognition capability of image details, so that the accurate detection of the displacement and necrosis of the teeth is realized.

Description

Technical Field

[0001] The invention belongs to the field of nursing assistance, and in particular is a postoperative nursing situation tracking assistance system for orthodontic patients. Background Art

[0002] Orthodontic treatment is a medical method to improve oral function and aesthetics by correcting problems such as malocclusion, bite problems and jaw abnormalities. Postoperative care is crucial to the effectiveness of orthodontic treatment, because even if the orthodontic process is successfully completed, the teeth may relapse or shift if postoperative care is not in place; however, manual inspection of tooth displacement is inefficient and the timeliness of care is poor. When existing computer vision technology is used to detect tooth displacement, the complexity of tooth tissues such as enamel, dentin and pulp is high, and the detection accuracy is poor. Summary of the invention

[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a system for tracking the postoperative care of orthodontic patients. In view of the low efficiency and poor timeliness of the existing manual method of checking tooth displacement, the present invention adopts computer vision technology based on image segmentation to detect the postoperative tooth condition of orthodontic patients, realize rapid displacement inspection, and assist nurses and doctors to make more accurate and intuitive judgments on the condition of patients' teeth. In view of the problem that the existing computer vision technology has poor detection accuracy when detecting tooth displacement due to the complexity of tooth tissues such as enamel, dentin and pulp, the present invention creatively adopts a computer vision technology based on image segmentation to detect the postoperative tooth condition of orthodontic patients, realize rapid displacement inspection, and assist nurses and doctors to make more accurate and intuitive judgments on the condition of patients' teeth. The boundary feature extraction method based on feature cross-fusion obtains the overall boundary information in a global mapping manner, and then performs local feature extraction based on the overall boundary information, and performs multi-level feature cross-fusion of the boundary information with global features and local features to achieve accurate boundary extraction of teeth; on the basis of accurate boundary extraction, the present invention creatively adopts a feature excision adjustment method based on multi-dimensional attention to perform displacement detection and tooth necrosis detection on the spatial level and the color channel level, and introduces a mask mechanism adjustment to improve the generalization ability of the attention module and increase its recognition ability of image details, thereby achieving accurate detection of tooth displacement and necrosis.

[0004] The technical solution adopted by the present invention is as follows: the present invention provides an auxiliary system for tracking the postoperative care of orthodontic patients, including a tooth image acquisition module, a tooth intelligent detection module, a data storage module, a doctor interface module, a patient interface module and a central control module;

[0005] The tooth image acquisition module acquires CT images and RGB images of teeth and transmits the images to the tooth intelligent detection module;

[0006] The tooth intelligent detection module includes a boundary intelligent extraction module and a tooth condition intelligent detection module;

[0007] The boundary intelligent extraction module adopts a boundary feature extraction method based on feature cross fusion to extract the tooth boundary according to the tooth CT image to obtain the tooth CT segmentation result;

[0008] The tooth condition intelligent detection module adopts a feature removal adjustment method based on multi-dimensional attention to perform displacement detection and necrosis detection on the tooth RGB image based on the tooth CT segmentation result, obtain the detection results of tooth displacement and necrosis, and transmit the detection results to the data storage module;

[0009] The data storage module stores the detection results of tooth displacement and necrosis;

[0010] The doctor interface module allows doctors to view the patient's daily care status and the test results of tooth displacement and necrosis. The doctor formulates postoperative care guidance for the patient based on the test results and sends it to the patient interface module;

[0011] The patient interface module provides postoperative nursing guidance and scheduled nursing reminders for patients, allowing them to record their daily nursing status;

[0012] The central control module directly controls all modules.

[0013] Furthermore, in the boundary intelligent extraction module, a boundary feature extraction method based on feature cross fusion specifically includes the following steps:

[0014] Step S1: Feature extraction: ResNet network is used as the backbone network to extract five layers of features from dental CT images to obtain features. ( =1,2,...,5);

[0015] Step S2: Global mapping feature extraction, using the partial decoder PD to extract the five-layer features , and Encode and obtain global mapping features ;

[0016] Step S3: Boundary feature extraction, , and Extract the boundary features based on reverse attention to obtain the boundary features. The formula used is as follows: ; ; ;

[0017] In the formula, represents the reverse attention weight, Represents the boundary features, represents element-wise multiplication operation, represents element-wise addition operation, represents the downsampling operation, represents the features obtained by offline sampling, is an all-one matrix, represents the Sigmoid activation function, represents an upsampling operation;

[0018] Step S4: Multi-level feature connection, Level boundary characteristics and +1 level boundary features are connected to obtain connection features: ;

[0019] In the formula, Respectively represent Level boundary characteristics and +1 level border features, represents the feature connection operation, Represents the connection characteristics;

[0020] Step S5: Multi-level cross fusion of features: multi-level cross fusion of connection features to obtain boundary cross fusion features: ; ; ;

[0021] In the formula, represents a 3×3 convolution operation, represents the local attention weight, and They represent pointwise convolution operations with kernel dimensions of 1×1 and 2×2 respectively. represents the Sigmoid activation function, represents the ReLU activation function, is the intermediate variable, Represents boundary cross-fusion features;

[0022] Step S6: Final output, activate and output the boundary cross fusion features to obtain the tooth CT segmentation result.

[0023] Furthermore, in the tooth condition intelligent detection module, a feature excision adjustment method based on multi-dimensional attention specifically comprises the following steps:

[0024] Step Q1: RGB image segmentation: segment the tooth RGB image according to the tooth CT segmentation result to obtain the RGB segmentation map of each tooth;

[0025] Step Q2: Spatial attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling respectively, and the void spatial convolution pooling ASPP, feature connection and dimensionality reduction operations are performed to obtain the spatial attention feature map: ;

[0026] In the formula, represents a simple convolution feature, and Represent average pooling and maximum pooling respectively. Represents the ASPP processing of void space convolution pooling, represents feature connection, Represents the operation of restoring the feature dimension using a 7×7 convolution kernel. represents the spatial attention feature map;

[0027] Step Q3: Channel attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling respectively, and multi-layer perceptron MLP processing, feature connection and dimension reduction operations are performed to obtain the channel attention feature map: ;

[0028] In the formula, represents a simple convolution feature, and Represent average pooling and maximum pooling respectively. Represents multi-layer perceptron MLP processing, represents feature connection, Represents the operation of restoring the feature dimension using a 1×1 convolution kernel. represents the channel attention feature map;

[0029] Step Q4: Mask generation, average pooling of spatial attention feature map and channel attention feature map, upsampling operation to adjust the average pooling feature to the original image size, and according to the preset threshold Generate a mask; ; ;

[0030] In the formula, Represents the spatial attention feature map and the channel attention feature map, is the intermediate variable, is the preset threshold, represents a mask;

[0031] Step Q5: feature cutting adjustment and activation, cover the mask on the spatial attention feature map and the channel attention feature map to obtain the spatial attention feature map and the channel attention feature map after cutting adjustment, and perform feature activation operation at the same time to obtain the detection results of tooth displacement and necrosis.

[0032] The beneficial results achieved by the present invention using the above scheme are as follows:

[0033] (1) In view of the low efficiency and poor timeliness of nursing in the existing manual method of checking tooth displacement, the present invention uses computer vision technology based on image segmentation to detect the tooth condition of orthodontic patients after surgery, realizes rapid displacement inspection, and assists nurses and doctors to make more accurate and intuitive judgments on the condition of patients' teeth;

[0034] (2) In view of the problem that the existing computer vision technology has poor detection accuracy when performing displacement detection on teeth due to the complexity of tooth tissues such as enamel, dentin and pulp, the present invention creatively adopts a boundary feature extraction method based on feature cross-fusion, obtains the overall boundary information in a global mapping manner, and then performs local feature extraction based on the overall boundary information, and performs multi-level feature cross-fusion of the boundary information with global features and local features to achieve accurate boundary extraction of teeth;

[0035] (3) On the basis of precise boundary extraction, the present invention creatively adopts a feature excision adjustment method based on multi-dimensional attention to perform tooth displacement detection and tooth necrosis detection from the spatial level and color channel level, and introduces a mask mechanism adjustment to improve the generalization ability of the attention module and increase its ability to recognize image details, thereby achieving accurate detection of tooth displacement and necrosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A module diagram of a postoperative nursing tracking auxiliary system for orthodontic patients provided by the present invention;

[0037] Figure 2 It is a data processing flow diagram of a boundary feature extraction method based on feature cross fusion;

[0038] Figure 3 Schematic diagram of the data processing flow of a feature excision adjustment method based on multi-dimensional attention.

[0039] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] Example 1, see Figure 1 , the present invention provides an auxiliary system for tracking the postoperative care of orthodontic patients, including a tooth image acquisition module, a tooth intelligent detection module, a data storage module, a doctor interface module, a patient interface module and a central control module;

[0042] The tooth image acquisition module acquires CT images and RGB images of teeth and transmits the images to the tooth intelligent detection module;

[0043] The tooth intelligent detection module includes a boundary intelligent extraction module and a tooth condition intelligent detection module;

[0044] The boundary intelligent extraction module adopts a boundary feature extraction method based on feature cross fusion to extract the tooth boundary according to the tooth CT image to obtain the tooth CT segmentation result;

[0045] The tooth condition intelligent detection module adopts a feature removal adjustment method based on multi-dimensional attention to perform displacement detection and necrosis detection on the tooth RGB image based on the tooth CT segmentation result, obtain the detection results of tooth displacement and necrosis, and transmit the detection results to the data storage module;

[0046] The data storage module stores the detection results of tooth displacement and necrosis;

[0047] The doctor interface module allows doctors to view the patient's daily care status and the test results of tooth displacement and necrosis. The doctor formulates postoperative care guidance for the patient based on the test results and sends it to the patient interface module;

[0048] The patient interface module provides postoperative nursing guidance and scheduled nursing reminders for patients, allowing them to record their daily nursing status;

[0049] The central control module directly controls all modules.

[0050] By performing the above operations, in order to address the problems of low efficiency and poor timeliness of care in the existing manual method of checking tooth displacement, the present invention adopts computer vision technology based on image segmentation to detect the tooth condition of orthodontic patients after surgery, realize rapid displacement inspection, and assist nurses and doctors to make more accurate and intuitive judgments on the condition of patients' teeth.

[0051] Example 2, see Figure 2 This embodiment is based on the above embodiment. In the boundary intelligent extraction module, a boundary feature extraction method based on feature cross fusion specifically includes the following steps:

[0052] Step S1: Feature extraction: ResNet network is used as the backbone network to extract five layers of features from dental CT images to obtain features. ( =1,2,...,5);

[0053] Step S2: Global mapping feature extraction, using the partial decoder PD to extract the five-layer features , and Encode and obtain global mapping features ;

[0054] Step S3: Boundary feature extraction, , and Extract the boundary features based on reverse attention to obtain the boundary features. The formula used is as follows: ; ; ;

[0055] In the formula, represents the reverse attention weight, Represents the boundary features, represents element-wise multiplication operation, represents element-wise addition operation, represents the downsampling operation, represents the features obtained by offline sampling, is an all-one matrix, represents the Sigmoid activation function, represents an upsampling operation;

[0056] Step S4: Multi-level feature connection, Level boundary characteristics and +1 level boundary features are connected to obtain connection features: ;

[0057] In the formula, Respectively represent Level boundary characteristics and +1 level border features, represents the feature connection operation, Represents the connection characteristics;

[0058] Step S5: Multi-level cross fusion of features: multi-level cross fusion of connection features to obtain boundary cross fusion features: ; ; ;

[0059] In the formula, represents a 3×3 convolution operation, represents the local attention weight, and They represent pointwise convolution operations with kernel dimensions of 1×1 and 2×2 respectively. represents the Sigmoid activation function, represents the ReLU activation function, is the intermediate variable, Represents boundary cross-fusion features;

[0060] Step S6: Final output, activate and output the boundary cross fusion features to obtain the tooth CT segmentation result.

[0061] By performing the above operations, in order to address the problem that the existing computer vision technology has poor detection accuracy when performing displacement detection on teeth due to the complexity of tooth tissues such as enamel, dentin and pulp, the present invention creatively adopts a boundary feature extraction method based on feature cross-fusion, obtains the overall boundary information in a global mapping manner, and then performs local feature extraction based on the overall boundary information, and performs multi-level feature cross-fusion of the boundary information with global features and local features to achieve accurate boundary extraction of teeth.

[0062] Example 3, see Figure 3 This embodiment is based on the above embodiment. In the tooth condition intelligent detection module, a feature removal adjustment method based on multi-dimensional attention specifically includes the following steps:

[0063] Step Q1: RGB image segmentation: segment the tooth RGB image according to the tooth CT segmentation result to obtain the RGB segmentation map of each tooth;

[0064] Step Q2: Spatial attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling respectively, and the void spatial convolution pooling ASPP, feature connection and dimensionality reduction operations are performed to obtain the spatial attention feature map: ;

[0065] In the formula, represents a simple convolution feature, and Represent average pooling and maximum pooling respectively. Represents the ASPP processing of void space convolution pooling, represents feature connection, Represents the operation of restoring the feature dimension using a 7×7 convolution kernel. represents the spatial attention feature map;

[0066] Step Q3: Channel attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling respectively, and multi-layer perceptron MLP processing, feature connection and dimension reduction operations are performed to obtain the channel attention feature map: ;

[0067] In the formula, represents a simple convolution feature, and Represent average pooling and maximum pooling respectively. Represents multi-layer perceptron MLP processing, represents feature connection, Represents the operation of restoring the feature dimension using a 1×1 convolution kernel. represents the channel attention feature map;

[0068] Step Q4: Mask generation, average pooling processing is performed on the spatial attention feature map and the channel attention feature map, and the average pooling feature is adjusted to the original image size by upsampling operation, and the preset threshold is used. Generate a mask; ; ;

[0069] In the formula, Represents the spatial attention feature map and the channel attention feature map, is the intermediate variable, is the preset threshold, represents a mask;

[0070] Step Q5: feature cutting adjustment and activation, cover the mask on the spatial attention feature map and the channel attention feature map to obtain the spatial attention feature map and the channel attention feature map after cutting adjustment, and perform feature activation operation at the same time to obtain the detection results of tooth displacement and necrosis.

[0071] By performing the above operations, on the basis of precise boundary extraction, the present invention creatively adopts a feature excision adjustment method based on multi-dimensional attention to perform tooth displacement detection and tooth necrosis detection from the spatial level and the color channel level, and introduces a mask mechanism adjustment to improve the generalization ability of the attention module and increase its ability to recognize image details, thereby achieving accurate detection of tooth displacement and necrosis.

[0072] Embodiment 4: This embodiment is based on the above embodiment. In step Q1, the tooth RGB image is segmented according to the tooth CT segmentation result. Specifically, the following steps are included:

[0073] Step Q11: obtaining the data correspondence between the tooth CT image and the tooth RGB image, and performing feature point-based registration on the tooth CT image and the tooth RGB image through geometric transformation to obtain a registration result;

[0074] Step Q12: Image cropping and scaling, cropping or scaling the tooth RGB image to ensure that its size is consistent with the CT image;

[0075] Step Q13: Image conversion: according to the registration result, the warp() method in OpenCV is used to convert the coordinate system of the CT image into the coordinate system of the RGB image to obtain the mapping result;

[0076] Step Q14: RGB image segmentation: segment the RGB image according to the mapping result.

[0077] Embodiment 5, based on the above embodiment, this embodiment performs ASPP processing using dilated spatial convolution pooling, and specifically includes the following steps:

[0078] Step Q21: Input a feature map, the size of the input feature map is the feature map after average pooling and maximum pooling of H×W×C;

[0079] Step Q22: Extraction of local features: Perform a 1x1 convolution operation on the input feature map, keep the spatial resolution of the feature map unchanged, and transform the number of channels to obtain local features;

[0080] Step Q23: Context information extraction, apply three 3x3 dilated convolutions to the input feature map in parallel, each dilated convolution uses a different expansion rate to capture context information under different receptive fields, and obtain three dilated convolution feature maps:

[0081] Dilation rate of dilated convolution:

[0082] Dilation rate is 6: captures more local context information;

[0083] Dilation rate is 12: capturing medium-scale contextual information;

[0084] The dilation rate is 18: capturing global context information;

[0085] Step Q24: Global average pooling, performing a global average pooling operation on the input feature map to obtain a feature vector, and using a 1x1 convolution to map the feature vector to the same number of channels as the input feature map to obtain a global feature with global context information of the entire image;

[0086] Step Q25: Feature concatenation and channel dimensionality reduction. Concatenate the local features, the three hole convolution feature maps, and the global features, and restore the number of channels to C to obtain the final feature map.

[0087] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0089] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A tracking and auxiliary system for postoperative care of orthodontic patients, characterized in that: It includes a tooth image acquisition module, a tooth intelligent detection module, a data storage module, a doctor interface module, a patient interface module and a central control module; The tooth image acquisition module acquires CT images and RGB images of teeth and transmits the images to the tooth intelligent detection module; The tooth intelligent detection module includes a boundary intelligent extraction module and a tooth condition intelligent detection module; The boundary intelligent extraction module adopts a boundary feature extraction method based on feature cross fusion to extract the tooth boundary according to the tooth CT image to obtain the tooth CT segmentation result; The tooth condition intelligent detection module adopts a feature removal adjustment method based on multi-dimensional attention to perform displacement detection and necrosis detection on the tooth RGB image based on the tooth CT segmentation result, obtain the detection results of tooth displacement and necrosis, and transmit the detection results to the data storage module; The data storage module stores the detection results of tooth displacement and necrosis; The doctor interface module allows doctors to view the patient's daily care status and the test results of tooth displacement and necrosis. The doctor formulates postoperative care guidance for the patient based on the test results and sends it to the patient interface module; The patient interface module provides postoperative nursing guidance and scheduled nursing reminders for patients, allowing them to record their daily nursing status; The patient interface module provides postoperative nursing guidance and scheduled nursing reminders for patients, allowing them to record their daily nursing status; The central control module directly controls all modules.

2. The system for tracking and assisting postoperative care of orthodontic patients according to claim 1, characterized in that: In the boundary intelligent extraction module, a boundary feature extraction method based on feature cross fusion specifically includes the following steps: Step S1: Feature extraction: ResNet network is used as the backbone network to extract five layers of features from dental CT images to obtain features. ( =1,2,...,5); Step S2: Global mapping feature extraction, using the partial decoder PD to extract the five-layer features , and Encode and obtain global mapping features ; Step S3: Boundary feature extraction, , and Extract the boundary features based on reverse attention to obtain the boundary features. The formula used is as follows: ； ； ； In the formula, represents the reverse attention weight, Represents the boundary features, represents element-wise multiplication operation, represents element-wise addition operation, represents the downsampling operation, represents the features obtained by offline sampling, is an all-one matrix, represents the Sigmoid activation function, represents an upsampling operation; Step S4: Multi-level feature connection, Level boundary characteristics and +1 level boundary features are connected to obtain connection features; Step S5: Multi-level cross fusion of features: multi-level cross fusion of connection features to obtain boundary cross fusion features: ； ； ； In the formula, represents a 3×3 convolution operation, represents the local attention weight, and They represent pointwise convolution operations with kernel dimensions of 1×1 and 2×2 respectively. represents the Sigmoid activation function, represents the ReLU activation function, is the intermediate variable, Represents boundary cross-fusion features; Step S6: Final output, activate and output the boundary cross fusion features to obtain the tooth CT segmentation result.

3. The system for tracking and assisting postoperative care of orthodontic patients according to claim 2, characterized in that: In the tooth condition intelligent detection module, a feature excision adjustment method based on multi-dimensional attention specifically comprises the following steps: Step Q1: RGB image segmentation: segment the tooth RGB image according to the tooth CT segmentation result to obtain the RGB segmentation map of each tooth; Step Q2: Spatial attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling respectively, and the void spatial convolution pooling ASPP, feature connection and dimensionality reduction operations are performed to obtain the spatial attention feature map: ； In the formula, represents a simple convolution feature, and Represent average pooling and maximum pooling respectively. Represents the ASPP processing of void space convolution pooling, represents feature connection, Represents the operation of restoring the feature dimension using a 7×7 convolution kernel. represents the spatial attention feature map; Step Q3: Channel attention feature extraction. After performing simple convolution feature extraction on the RGB segmentation map of each tooth, the convolution features are subjected to dimensionality reduction average pooling and maximum pooling processing, and multi-layer perceptron MLP processing, feature connection and dimensionality reduction operations are performed to obtain the channel attention feature map. Step Q4: Mask generation, average pooling processing is performed on the spatial attention feature map and the channel attention feature map, and the average pooling feature is adjusted to the original image size by upsampling operation, and the preset threshold is used. Generate a mask; Step Q5: feature cutting adjustment and activation, cover the mask on the spatial attention feature map and the channel attention feature map to obtain the spatial attention feature map and the channel attention feature map after cutting adjustment, and perform feature activation operation at the same time to obtain the detection results of tooth displacement and necrosis.

Images