Oral imaging device and imaging method thereof

Abstract

The application discloses an imaging method of an oral imaging device, comprising the following steps: S1, acquiring an image sequence of not less than one view angle collected by the oral imaging device when imaging the upper jaw and / or the lower jaw; S2, extracting useful video clips of the upper jaw and the lower jaw from the image sequence, and arranging the video clips into at least two video segments corresponding to the upper jaw and the lower jaw respectively; S3, recognizing the shooting direction of the video segments by combining motion detection data with a deep learning model, and unifying the video sequence; S4, dynamically adjusting the frame extraction method of the video segments according to the motion detection data, retaining key motion node images, reconstructing the calibrated video segments into a standard sequence in a unified direction, and outputting the standard sequence, so as to perform oral three-dimensional scene reconstruction and oral disease analysis. The application also provides an oral imaging device. The application reduces the operation burden of a user, significantly reduces the operation amount of a visual algorithm, and improves the reliability of three-dimensional modeling and the accuracy of oral disease identification.

Description

Technical Field

[0001] This invention relates to the field of medical imaging technology, and in particular to an oral imaging device and its imaging method. Background Technology

[0002] Digital oral diagnosis and treatment is one of the important development directions in the field of oral healthcare in recent years. Imaging equipment such as oral endoscopes and oral scanners have been widely used in oral examinations, caries screening, periodontal condition assessment, treatment process recording, and 3D modeling and treatment planning related to orthodontics. Existing oral imaging equipment can usually continuously acquire intraoral video or image sequences and display, store, and play them back in real time through computer software, providing doctors or operators with intuitive observation data.

[0003] The 3D reconstruction algorithm of the oral scanner mainly relies on image feature matching to achieve video stitching and 3D reconstruction. It requires that the scanning process start from the teeth and there should be no unrelated entry or exit movements. The scanning of the upper and lower jaws also needs to be separated.

[0004] Similarly, in intraoral endoscopic imaging, computer software typically saves the captured video sequence as a "whole segment recording." Unless the user strictly adheres to starting and ending the recording at the entrance and exit, and taking separate shots of the upper and lower jaws, a large amount of irrelevant footage will be included. Subsequent acquisition of the effective "along the dental arch" scan often relies on manual playback and editing, or on simple timestamp marking for rough cropping. However, manual editing is not only time-consuming and labor-intensive, but also heavily influenced by the operator's experience, making it difficult to guarantee the consistency and repeatability of the cropping boundaries. The cost of manual processing increases further, especially in scenarios involving batch acquisition, teaching demonstrations, remote consultations, or long-term follow-ups. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned defects in the prior art and provide an oral imaging method that reduces the user's operational burden and improves the user's experience of using the product, while significantly reducing the computational load of the visual algorithm and improving the reliability of 3D modeling and the accuracy of oral disease identification.

[0006] To achieve the above objectives, the present invention provides an imaging method for an oral imaging device, comprising the following steps:

[0007] S1. Acquire an image sequence of at least one viewpoint captured by an oral imaging device when imaging the maxilla and / or mandible, and simultaneously acquire motion detection data generated by a motion measurement module that is time-aligned with the multi-view image sequence; the motion sensing module includes, but is not limited to, one or more combinations of: accelerometer, gyroscope, magnetometer, attitude sensor, odometer, and visual odometry (VIO) module. The motion detection data generated by the motion sensing module includes at least one or more of: acceleration, angular velocity, attitude angle, and displacement estimate.

[0008] S2. Extract useful video clips of the maxilla and mandible from the image sequence and organize them into at least two video segments corresponding to the maxilla and mandible respectively;

[0009] S3. Identify the shooting direction of video segments by combining motion detection data with a deep learning model, and unify the video order;

[0010] S4. Based on motion detection data, dynamically adjust the frame extraction method of video segments, retain key motion node images, and reconstruct the six calibrated video segments into a standard sequence output in a unified direction, thereby performing oral cavity 3D scene reconstruction and oral disease analysis.

[0011] Furthermore, in step S2, before segmentation, the original motion detection data in each video segment is denoised and standardized.

[0012] Furthermore, step S2 includes the following steps:

[0013] S201. Use a fixed-length sliding window on the time axis of each video segment, and count the point-by-point labels inside each sliding window. Based on the proportion of positive samples in the point-by-point labels, mark the window as a dental arch scanning window or a non-dental arch scanning window.

[0014] S202, Training a deep learning model;

[0015] S203. Using a deep learning model, sliding window inference is performed on the motion detection data to obtain the probability that each window belongs to "dental arch scanning behavior". The window-level probability is mapped back to the original sampling point to form a point-by-point probability curve. Then, multiple dental arch scanning candidate segments are extracted. Combined with the minimum duration constraint, short-term interference is filtered out, and the effective scanning interval that conforms to the clinical operation rules is retained. Based on the effective scanning interval, the corresponding video segments are obtained through image reconstruction algorithm.

[0016] Furthermore, the deep learning model is a temporal model, and the deep learning module includes 1D-CNN, LSTM or TCN, or it can be based on Transformer and various variants of Transformer.

[0017] Furthermore, step S3 includes the following steps:

[0018] S301. Obtain motion detection data corresponding to each video segment and form an independent motion detection data file;

[0019] S302. These motion detection data files are unified to fixed sampling values ​​by resampling.

[0020] S303. Input the resampled motion detection data into the trained temporal model, extract temporal features and determine the scanning direction of each video segment;

[0021] S304. Based on the direction labels output by the model, rearrange all video segments into a standard order from left to right or right to left.

[0022] S305. Align the transition frames at the boundaries of each segment to ensure that the image at the stitching point is continuous without any jumps.

[0023] Furthermore, the frame extraction method in step S4 specifically involves obtaining the acceleration of the device movement corresponding to the video segment through the motion measurement module. When the acceleration of some frames in the video segment is less than a threshold, it is determined that the device is stationary or moving slowly. Any frame of that segment is taken as the image of that segment, and the remaining frames are deleted. If the acceleration is greater than the threshold, it is determined to be a fast-moving frame segment. For that frame segment, sufficient frames are retained according to the ratio of acceleration for three-dimensional reconstruction.

[0024] Furthermore, step S4 also includes visual detection-assisted similarity detection of similar frames.

[0025] Furthermore, if the acceleration is maintained for at least two frames for a duration less than a threshold, the device is determined to be in a stationary or slow-moving state.

[0026] Furthermore, in step S1, when the device is detected to be in a stable, stationary state, it automatically switches to high-resolution imaging mode and enhances the image stream. Simultaneously, it triggers the computer to perform automated analysis of local areas and compare them with oral diseases, thereby achieving the identification of oral diseases. The automated analysis includes one or more combinations of rule-based image analysis, statistical learning-based classification analysis, and deep neural network-based target detection / segmentation analysis. Specifically, the identification of oral diseases may include performing lesion identification / risk warning / category discrimination / similar case retrieval, and outputting corresponding result labels, bounding boxes, or segmentation masks.

[0027] A dental imaging device is also provided, comprising a handle and a probe, at least one set of camera modules disposed on the probe, a motion measurement module located inside the handle, and a computer connected to the handle. The camera modules and motion measurement module are used to acquire video data of the maxilla and mandible and corresponding motion detection data in the imaging method of the dental imaging device according to claims 1-9. The computer can be connected to the handle wirelessly or via a wired connection to achieve data transmission.

[0028] This invention integrates real-time attitude and motion detection data from a motion measurement module to construct a data acquisition and control mechanism centered on device dynamic perception. Without increasing the user's operational burden, it achieves intelligent guidance and image quality optimization during the shooting process. By using acceleration and angular velocity information to collaboratively determine the device's motion state, it accurately identifies stationary periods and triggers high-resolution enhanced imaging. Combined with multi-camera temporal calibration, it significantly reduces the visual algorithm's dependence on feature point matching, improving the stability of 3D modeling and the accuracy of oral disease recognition. Attached Figure Description

[0029] To more clearly illustrate the technology in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the imaging method of the oral imaging device of the present invention;

[0031] Figure 2 This is a schematic diagram of the possible scanning curves of the dental arch curve according to the present invention;

[0032] Figure 3 This is the user interface for comparing historical images and current images in this invention. Detailed Implementation

[0033] The technology of this embodiment of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiment is one embodiment of the present invention, and not all embodiments thereof. Based on this embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0035] Furthermore, if the embodiments of the present invention involve descriptions such as "first" or "second", such descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0036] This invention provides an imaging method for an oral imaging device, such as... Figure 1 As shown, it includes the following steps:

[0037] S1. Acquire an image sequence of at least one viewpoint collected by an oral imaging device when imaging the maxilla and / or mandible, and simultaneously acquire motion detection data generated by a motion measurement module (preferably an IMU module, i.e., an inertial measurement unit module) that is time-aligned with the multi-view image sequence; the motion sensing module includes, but is not limited to, one or more combinations of accelerometers, gyroscopes, magnetometers, attitude sensors, odometers, and visual odometer (VIO) modules. The motion detection data generated by the motion sensing module includes at least one or more of the following: acceleration, angular velocity, attitude angle, and displacement estimate;

[0038] Preferably, the endoscope probe used in this invention is equipped with at least three camera modules, which are fixedly mounted at different angles on the front end of the handle to ensure that the field of view covers the mandible or various areas of the mandible. In use, first click "Start Shooting" on the interactive interface. The device's supplementary light will automatically turn on and begin pushing a real-time video stream. Place the device probe in the mouth, align it, and gently clamp the first molar to be photographed. Smoothly move the handle so that the probe follows the natural curve of the dental arch, slowly and continuously moving from one side of the molar to the other. Place the probe at the starting position of the molar on the other jaw surface and scan again. Finally, gently remove the probe from the mouth to complete the entire scanning process. One scan is completed for each jaw surface (maxilla and mandible), forming a set of image sequences. This image sequence includes videos taken from different directions by the three camera modules, namely three videos of the maxilla and three videos of the mandible, totaling six raw video data segments. The motion measurement module is used during the shooting process. The system collects motion detection data such as acceleration, angular velocity, attitude angle, and displacement of the device in real time, and timestamps the motion detection data stream with the video frames to ensure accurate matching between motion state and image content.

[0039] In some preferred embodiments, when a user is taking a scan, the user may discover lesions (such as tooth decay, gingivitis, tartar, tooth cracks, etc.). At this time, the user may stop to carefully observe the area. After the motion measurement module detects the user's stationary behavior, the system will automatically improve the image clarity, which will help the user to observe the image better.

[0040] The method for determining this static behavior is described as follows:

[0041] The motion measurement module at the acquisition end continuously outputs triaxial acceleration. , , With triaxial angular velocity , , The host computer, acquisition terminal, or computer filters and assesses the stability of this motion detection data, for example, by using sliding window statistics and threshold determination methods.

[0042] Within a time window T_w (e.g., 0.3s to 1.0s), calculate the average angular velocity modulus:

[0043]

[0044] Simultaneously calculate the change in acceleration (variance or difference amplitude after removing the gravitational component):

[0045]

[0046] When satisfied ( Indicates the angular velocity threshold. When the acceleration threshold is exceeded and the duration exceeds T_hold, the acquisition end is determined to be in the "observation stationary state". The system enters the Observe / Inspection state of the state machine; when the motion amplitude recovers and exceeds the threshold and continues for T_exit, it exits the observation state and resumes the scanning state, thereby avoiding frequent jitter switching.

[0047] Optionally, to avoid misjudgment caused by slight hand tremors, a "hysteresis threshold" mechanism can be introduced, that is, the threshold for entering the static state is lower than the threshold for exiting the static state (ωin<ωout), thereby improving state stability.

[0048] This embodiment employs multiple combined technical measures to enhance image clarity. Individual measures can be selected as needed, or multiple measures can be used in combination. Specifically:

[0049] I. By adjusting the high-resolution imaging mode and exposure strategy

[0050] Once the system determines that the observation is static, the acquisition end automatically switches to high-resolution imaging mode, including but not limited to one or more of the following adjustments:

[0051] (1) Enhance the clarity by increasing the resolution, such as switching from the preview resolution (e.g., 640×480, 1280×720) to a high resolution (e.g., 1920×1080, 2560×1440 or the full resolution of the sensor).

[0052] (2) Enhance the sharpness by reducing the frame rate strategy. That is, when observing at rest, the frame rate can be reduced to obtain a longer exposure time and a higher signal-to-noise ratio, or the frame rate can be maintained to reduce the perceived delay.

[0053] (3) Enhance sharpness by controlling exposure and gain, i.e., trigger a "steady exposure mode" in a static state, and improve image quality by extending the exposure time and reducing the gain (ISO). For example, increase the exposure time from 1 / 100 second to 1 / 10 second, while limiting the upper limit of analog gain / digital gain, thereby improving the signal-to-noise ratio, reducing noise and enhancing detail in static observation scenes; and can be combined with frame alignment and temporal noise reduction strategies to further suppress blur and motion blur caused by slight jitter.

[0054] 4. Enhance sharpness by increasing ISP, i.e., improve sharpening / local contrast, stabilize white balance, and enable detail enhancement or noise reduction intensity in the observation file parameters (independent of scan state parameters).

[0055] The aforementioned imaging settings can be set locally at the acquisition terminal, or closed-loop adjustment can be achieved by issuing control parameters from a host computer / computer based on the image quality assessment results.

[0056] II. Employing a strategy of enhanced bitrate and low compression for display

[0057] Once the observation is complete, the encoding and transmission module automatically switches to a "high bitrate / low compression" strategy, allowing users to obtain clearer texture details and edge information when observing lesions. Specifically, this includes the following steps:

[0058] (1) Increase the encoding bitrate, for example, increase the target bitrate from R1 to R2 (R2>R1, for example, 4Mbps→12Mbps). (2) Optimize the quantization parameters and reduce the encoding QP range (for example, QP=30→QP=22) to reduce compression loss. (3) Trigger keyframes and send IDR / I-frames to refresh, avoiding blurring of details due to P-frame prediction errors. (5) Display low-compression rendering on the host computer. On the host computer, use a higher quality decoding and rendering path for the observation frame, reduce secondary scaling, and reduce post-processing compression and frame dropping strategies.

[0059] In some implementations, ROIs can be encoded to achieve an optimal balance between bandwidth efficiency and diagnostic effectiveness. When the user focuses on a localized tooth surface area, a higher bitrate weight is assigned to the ROI area, while non-ROI areas maintain a normal bitrate to reduce overall bandwidth pressure. If the link bandwidth is insufficient, the system prioritizes the bitrate quality of the "ROI local details," ensuring that the lesion area remains identifiable even if the overall frame rate or global clarity is reduced.

[0060] III. Super-resolution and temporal noise reduction

[0061] When observing a static state, the inter-frame displacement is small, making it suitable for superimposing multiple frames of information for enhancement. The system can trigger image enhancement algorithms on the host computer or edge side, including: (1) Super-resolution (SR): 2× / 4× magnification and reconstruction of the current frame or ROI region to enhance the recognizability of micro-cracks and caries textures. (2) Temporal noise reduction (TNR): Multi-frame fusion noise reduction using continuous frame alignment (e.g., based on optical flow or feature point registration) to suppress low-light noise and color spots. (3) Sharpening and contrast optimization: Improve the visibility of tooth surface structures without excessively enhancing artifacts.

[0062] To ensure stable enhancement results, this implementation method allows setting a minimum number of consecutive frames N_min in the static state before outputting the enhancement results; when the static state ends or the image displacement is too large, the temporal enhancement will automatically stop to avoid ghosting and image loss.

[0063] Then, the computer is simultaneously triggered to perform automated analysis of the local area and compare it with oral diseases to identify oral diseases. The analysis results are then loaded and overlaid on the screen, and historical images of the dental area are pushed to users for direct comparison.

[0064] Specifically as follows:

[0065] After entering the observation static state, the system automatically triggers local automated analysis. The triggering conditions may include, but are not limited to: (1) the static state is established (the motion measurement module determines that it is stable); (2) the image clarity reaches the threshold (e.g., the focus score / edge gradient score reaches the threshold); (3) the user actively marks (touch point selection of ROI, voice command, foot switch, etc.).

[0066] The system can use local regions as inference input, reducing computational load and improving timeliness. For example, it can crop ROIs (which can be rectangles, polygons, or masked regions based on segmentation results) from the original frame.

[0067] The system identifies dental caries, gingivitis, tartar, cracks, etc. by comparing the data of the selected area with the preset data. The output includes, but is not limited to: lesion category (Class); location (ROI box / segmentation mask / key point); confidence score (Score); severity level (optional, such as mild / moderate / severe or numerical level); and suggested markings (e.g., "suggest follow-up examination / suggest taking supplementary photos / suggest saving for comparison").

[0068] The system can set thresholds for comparison to push historical comparison charts. For example, when Score > S_th (confidence threshold S_th = 0.6), a prompt is displayed; when Score > S_th2 (second confidence threshold S_th2 = 0.8), a "strong prompt" is entered and historical comparison charts are automatically pushed.

[0069] In this embodiment, the historical image management module in the host computer maintains the mapping relationship between dental regions and images. That is, dental regions can adopt a tooth position numbering system (such as two-digit FDI numbering, quadrant numbering, etc.) or a continuous positioning method of "acquisition path coordinates + dental arch curve position" to generate a dental region index, which facilitates subsequent retrieval.

[0070] The tooth position numbering system described above is as follows: when a lesion is identified or the user manually selects an ROI, the system generates a tooth region index key, such as: ToothID: tooth position number (e.g., 11, 12, 36, etc.); SessionID: this scanning session; ROI features: local texture feature vector or key point summary.

[0071] Then, historical images of the same dental area are retrieved from the local database or cloud data warehouse (which can be keyframes from the last visit, the last scan, or historical images). The filtering rules are as follows: most recent images are preferred; the highest clarity score is preferred; and images from the same or similar angles are preferred.

[0072] The host computer pushes historical images using "side-by-side comparison / overlay flashing / timeline sliding" modes to help users quickly assess lesion progression. Side-by-side comparison mode: The user interface displays historical and current images simultaneously, for example, the historical image on the left and the current image on the right, facilitating intuitive side-by-side comparison of lesions. Overlay flashing mode: The current and historical images are displayed overlaid, with the historical image's transparency adjusted and flashing to highlight areas of change in the current image. This effect helps users quickly identify lesion progression in overlapping areas. Timeline sliding mode: Users can slide the timeline to dynamically view the transition between historical and current images, visually presenting the lesion's change from the past to the present. This mode supports precise selection of data from specific time points for comparison using a slider. Figure 3 As shown, this is a side-by-side comparison pattern.

[0073] The overall page layout could be, for example: Left side: Historical images (last time). Right side: Current image (this time). Bottom: Shooting date, tooth position number, suspected lesion type and confidence level.

[0074] Optionally, the system allows users to save "high-definition observation frames + AI analysis results + historical comparison charts" into a structured report item with one click, which can be used for subsequent diagnosis and treatment records or follow-up examinations.

[0075] In some embodiments, the acquired video group also needs to be quality evaluated. Specifically, a score and threshold decision on the video shooting quality are given by combining peak angular velocity, motion blur and spatial high frequency retention rate. Step S2 is executed only when the quality evaluation meets the requirements; otherwise, the video needs to be reshot.

[0076] Specifically, to ensure the identification and reconstruction effects of subsequent steps, the system performs quality assessment and threshold determination on the acquired video group before it enters step S2. Specifically, the system first aligns the video frame sequence with the synchronously acquired motion data in time, and calculates motion stability indicators reflecting the intensity of shooting jitter (e.g., peak angular velocity or statistics obtained from the angular velocity modulus) based on the motion data; simultaneously, it calculates motion blur indicators reflecting the image sharpness (e.g., sharpness scores obtained based on edge gradients or Laplacian response) and spatial high-frequency retention indicators reflecting the preservation of detail and texture (e.g., statistical high-frequency energy ratio or high-pass response intensity) for the video frames. Then, the system normalizes and weights these indicators to obtain a comprehensive quality score, which is compared with a preset quality threshold: when the comprehensive quality score meets the threshold requirements, the video group is deemed to have acceptable shooting quality and proceeds to step S2; when the comprehensive quality score does not meet the threshold requirements, the video group is deemed to have excessive jitter, severe blur, or loss of detail, triggering a reshoot or prompting the operator to reduce movement speed, stabilize posture, and re-acquire data, thereby filtering low-quality video data at the source and improving the stability and consistency of subsequent analysis and processing.

[0077] S2. Extract useful video clips of the maxilla and mandible from the image sequence and organize them into at least two video segments corresponding to the maxilla and mandible respectively;

[0078] As described in S1, the preferred embodiment is to perform a scan on each jaw (maxilla and mandible) to form a set of image sequences. This set of image sequences includes videos taken from different directions by three camera modules, namely three videos of the maxilla and three videos of the mandible, for a total of six raw video data segments. If the endoscope contains videos taken from two directions by two camera modules, then there will be a total of four raw video data segments, and so on.

[0079] Data preprocessing involves denoising the raw motion detection data for each video segment, for example, by using a Butterworth low-pass filter to remove high-frequency noise and power frequency interference; then, standardization is performed to convert the data of each axis into a distribution with a mean of 0 and a standard deviation of 1, in order to eliminate individual sensor bias and dimensional effects.

[0080] In some implementations, the input not only uses the original sensor signal, but can also add derived features as needed: resultant acceleration modulus, angular velocity modulus; first-order difference, energy, spectral features (FFT); moving statistics (mean / variance / kurtosis, etc.).

[0081] The preprocessed temporal motion detection data is segmented on the time axis, specifically including the following steps:

[0082] S201. Use a fixed-length sliding window to extract each video segment on the timeline, and count the point-by-point labels inside each sliding window. Based on the proportion of positive samples in the point-by-point labels, mark the window as a dental arch scanning window or a non-dental arch scanning window.

[0083] Preferably, more refined labels can be set for the station labels, such as entrance, exit, scanning the upper jaw (left -> right / right -> left), scanning the lower jaw, picking up, putting down, and other random actions, to improve the precision of recognition.

[0084] Preferably, the length of the sliding window is 1.5 to 3 seconds, corresponding to 150 to 300 sampling points at a sampling frequency of 100Hz. This duration is sufficient to cover a typical sub-action unit, such as a single dental arch scan. The sliding window slides in steps of 0.25 to 1 second. This overlapping sampling strategy can ensure both sensitivity to action boundaries and the generation of a sufficient number of training samples.

[0085] There are at least two implementation methods for labeling sampling points. In one implementation, sampling points are pre-labeled manually based on the video, and windows are segmented according to time periods in the video. In another implementation, an automated labeling method is used to form point-by-point labels. For each sliding window, the label categories of all sampling points within it are counted. Depending on the task requirements, an aggregation rule can be set to transform it into a window-level label. For example, in a binary classification task, to identify whether it is a "dental arch scan", if the proportion of points belonging to the "scan maxilla" or "scan mandible" category in the window exceeds a preset threshold, the window is labeled as a "dental arch scan window"; otherwise, it is labeled as a "non-dental arch scan window".

[0086] Alternatively, instead of using a fixed percentage threshold, window labels can employ: a maximum value strategy (any positive point appearing in the window is considered a positive point), a weighted ratio strategy (the center frame has a greater weight than the edge frames), or a continuous labeling strategy based on time period coverage.

[0087] S202, Training a deep learning model;

[0088] The deep learning model described in this embodiment is preferably a temporal model. The deep learning module includes 1D-CNN, LSTM or TCN, etc., or it can be based on Transformer and various variants of Transformer, which can be selected according to actual needs.

[0089] Using motion detection data and the corresponding window-level labels from step S201 above as the training set, a deep learning model is trained. In this embodiment, the deep learning model is a temporal action classification model (such as a classifier based on Long Short-Term Memory Network (LSTM), One-Dimensional Convolutional Neural Network (1D-CNN), Transformer, and its variants). After training, the model can perform real-time or offline window-level action recognition on new, unlabeled motion detection data streams, thereby achieving automatic segmentation of the entire endoscopic operation video and marking the start and end times of different action stages.

[0090] In a preferred embodiment, a 1D convolutional network (1D-CNN) is employed, comprising: multiple one-dimensional convolutional feature extraction layers (Conv1D); normalization layers (such as BatchNorm) and non-linear activations (such as ReLU); pooling layers (MaxPool or AvgPool); adaptive pooling layers (AdaptiveAvgPool); and fully connected classification layers (FC) outputting scanned / non-scanned categories. The training objective function may employ cross-entropy loss.

[0091]

[0092] in For real labels, This is the probabilistic model for model prediction; the optimizer can be Adam, SGD, etc., to iteratively update the model parameters until convergence.

[0093] S203. The system performs sliding window inference on the motion detection data to obtain the probability that each window belongs to "dental arch scanning behavior", and maps the window-level probability back to the original sampling point to form a point-by-point probability curve. Then, it extracts multiple dental arch scanning candidate segments, filters short-term interference by combining minimum duration constraints, retains the effective scanning interval that conforms to clinical operation rules, and obtains the corresponding video segments through image reconstruction algorithm based on the effective scanning interval.

[0094] Specifically, the steps include the following:

[0095] Since the same sampling point may be covered by multiple sliding windows, the system backfills the probability of each window to the sampling points it covers, and normalizes the overlap contribution to obtain the point-by-point dental arch scan probability curve. }

[0096] Specifically, for each sampling point t, count the set of the number of windows covering it. ,in Let L be the start time of the i-th window.

[0097] Pointwise probability is defined as:

[0098]

[0099] Where |Ω(t)| represents the number of windows covering point t. This represents the probability value that the i-th sliding window is classified as "dental arch scanning behavior". The above formula is equivalent to averaging the probabilities of overlapping windows, thus achieving a point-by-point probability generation method that "maps back to the original sampling points and normalizes". Optionally, the above normalization method can also be replaced by a weighted average (e.g., assigning higher weight to the window center point), taking the maximum value, taking the median, or using an exponential moving average to enhance robustness to noise and edge jitter. To reduce the interference of spikes caused by short-term jitter, the point-by-point probability curve can be smoothed, for example, by using moving average filtering, low-pass filtering, or morphological closing operations to reduce missegmentation caused by discontinuous jumps.

[0100] Point-by-point probability curve A binary mask is generated by comparing it with a preset threshold τ. :

[0101]

[0102] when =1 =1 =1 indicates that the sampling point belongs to the candidate state of dental arch scanning.

[0103] In one specific embodiment, the threshold τ can be taken as 0.6 (e.g., Figure 2 (as shown); however, the present invention is not limited thereto, and τ can be set to 0.4 to 0.8 according to the recall priority or precision priority strategy.

[0104] For binary masks Scan the intervals where the values ​​are consecutively 1 to extract all candidate segments [t]. start , t endThe average scan probability of the candidate segment is calculated as the confidence level.

[0105]

[0106] This yielded multiple candidate segments for dental arch scanning and their confidence indices.

[0107] Considering the continuity of the dental arch scanning process, to avoid fragmented segments caused by momentary misjudgments, the system applies a minimum duration constraint to candidate segments. That is, only retain those that satisfy the condition.

[0108]

[0109] The fragment. This refers to the IMU sampling rate.

[0110] For the final retained valid scan intervals, the system outputs the sampling index range and time range corresponding to each interval, and assigns a segment number (segment_id) to the original long sequence for subsequent processing and tracing. Further, based on the time synchronization relationship between the IMU module and the video stream, the system divides the scan interval [t] into segments. start ,t end ) Mapped to video frame interval [F start ,F end The system performs video cropping or reconstruction on the frame interval to obtain the corresponding dental arch scan video segments for applications such as image reconstruction, diagnostic analysis, or historical comparison.

[0111] S3. Identify the shooting direction of video segments by combining motion detection data with a deep learning model, and unify the video order;

[0112] Specifically, the steps include the following:

[0113] S301. Obtain motion detection data corresponding to each video segment and form an independent motion detection data file;

[0114] S302. These motion detection data files are unified to fixed sampling values ​​by resampling.

[0115] S303. Input the resampled motion detection data into the trained deep learning model, extract temporal features and determine the scanning direction of each video segment;

[0116] S304. Based on the direction labels output by the model, rearrange all video segments into a standard order from left to right or right to left.

[0117] S305. Align the transition frames at the boundaries of each segment to ensure that the image at the stitching point is continuous without any jumps.

[0118] S4. Dynamically adjust the frame extraction method of video segments based on motion detection data, retain key motion node images, and reconstruct the calibrated video segments into a standard sequence output in a unified direction, thereby enabling efficient and clear reconstruction of the oral cavity 3D scene and outputting oral disease reports based on disease comparison in step S1.

[0119] Preferably, the frame extraction method in step S4 specifically involves obtaining the acceleration of the device movement corresponding to the video segment through a motion measurement module. When the acceleration of some frames in the video segment is less than a threshold, it is determined that the device is stationary or moving slowly. Any frame of that segment is taken as the image of that segment, and the remaining frames are deleted. It should be noted that, in order to avoid the influence of factors such as jitter, a minimum duration threshold needs to be set. Only when the duration of acceleration below the threshold exceeds the time threshold is it considered a valid stationary segment and frame extraction is performed.

[0120] If the acceleration is greater than the threshold, it is identified as a fast-moving frame segment. For this frame segment, a sufficient number of frames are retained according to the proportion of acceleration for 3D reconstruction, as follows:

[0121] For fast-moving candidate segments (or segments that do not meet the minimum static duration), to ensure the inter-frame overlap and viewpoint continuity required for 3D reconstruction, a "frame extraction based on acceleration intensity" strategy is adopted to retain a sufficient number of frames. (Average acceleration value) calculate:

[0122]

[0123] and : These represent the end and start positions of the interval, respectively.

[0124] : The acceleration value in the k-th frame.

[0125] The number of frames within the interval, used for weighted averaging.

[0126] That is, the greater the acceleration (the faster the movement), the smaller the step size, and the more frames are preserved. The denser:

[0127]

[0128] Where α is a preset scaling factor used to control the effect of acceleration on step size.

[0129] : Prevents small values ​​when divided by zero, used for numerical stability.

[0130] and These are the minimum and maximum step size limits, respectively, ensuring that the step size will not be less than [a certain value]. or greater than .

[0131] Frame indexes preserved during fast motion segments gather:

[0132]

[0133] Alternatively, a ratio can be used to extract frames, as shown in the following formula:

[0134]

[0135] These are preset parameters, where and These are the minimum and maximum values ​​for the percentage of frames to be retained. These two values ​​ensure that the number of frames retained will not be lower than the minimum or higher than the maximum, thus avoiding an unreasonable number of frames.

[0136] The overlap can be set according to the device frame rate and reconstruction algorithm requirements.

[0137] By using this intelligent frame extraction method, we can avoid the problems caused by traditional fixed-interval frame extraction: when the number of frames extracted is too small during periods of fast motion, the effective visual overlap between the extracted images is small or even non-existent, causing the 3D reconstruction algorithm to fail due to missing, broken, or even overall failure; when the number of frames extracted is too large during periods of slow motion, a large number of repetitive images with very little difference are extracted, which will increase the amount of computation and the computation time, consuming unnecessary resources.

[0138] Preferably, step S4 further includes visual detection-assisted similarity detection of similar frames. Pure visual detection, pure IMU detection, or a combination of both can also be used.

[0139] A dental imaging device is also provided, including a handle and a probe, at least one set of camera modules disposed on the probe, a motion measurement module located inside the handle, and a computer connected to the handle. The camera modules and the motion measurement module are used to collect video of the maxilla and mandible and the motion detection data corresponding to the video in the imaging method of the dental imaging device described above.

[0140] Preferably, the oral imaging device can be configured as an oral endoscope or an oral scanner. Generally, it has at least one camera module, a motion measurement module located inside the handle, and a computer connected to the handle. The camera module and motion measurement module are used to collect the video of the maxilla and mandible and the motion detection data corresponding to the video in the imaging method of the oral imaging device. Of course, it can also be configured in other devices that meet the requirements.

[0141] At the electronic hardware level, this embodiment adds / integrates a motion detection module to the oral imaging device and electrically connects it to the processing unit (computer, host computer, etc.). The motion detection module can be a three-axis accelerometer and a three-axis gyroscope, or optionally add three-axis magnetic field data (i.e., a nine-axis IMU sensor). In this embodiment, a nine-axis IMU sensor is preferred.

[0142] At the structural level, there are several options for integrating the motion measurement module: it can be located at the front of the handle, in the middle of the handle, at the rear of the handle, or inside the probe (it will rotate with the probe). It can also be directly integrated into the camera module. Multiple motion measurement modules can be present and placed in different locations simultaneously.

[0143] In one embodiment, the oral imaging device has a camera module, each camera module having an IMU sensor to collect kinematic information from each camera.

[0144] In another embodiment, the IMU sensor is placed at the front end of the handle, above the rotating probe. Preferably, the IMU sensor is integrated into the camera module; when space is limited in the lens module, placing the IMU sensor at the front end of the endoscope handle is also a preferred option.

[0145] At the transmission layer, the motion measurement module collects motion data at a specific sampling frequency (preferably 50Hz, but the exact frequency can be adjusted according to actual needs). The collected data can be transmitted in various ways, including wireless communication modules (such as Wi-Fi, Bluetooth, Zigbee, etc.) or wired communication interfaces (such as USB, Ethernet, etc.). In wireless transmission, Wi-Fi is preferred for synchronous transmission of motion data and video data to achieve stable real-time data transmission. In wired transmission, data can be transmitted via a high-speed interface to ensure data stability and integrity. The transmitted data is then sent to a host computer, PC, or other data processing equipment for further processing and analysis.

[0146] A preferred implementation uses a mobile app as the host computer. The app sends a TCP connection request to establish a communication connection with the device, and then the device sends the data collected by the motion measurement module to the app. The app not only processes the data collected by the motion measurement module in real time, but also integrates and stores this data with video data and other relevant information, and packages it into a data package. The processed data is then uploaded to a cloud server for remote storage, centralized management, and data analysis.

[0147] Preferably, the motion state of the probe is monitored in real time through a motion measurement module. When the probe is stationary, it enters a low-power state and is quickly woken up by detecting sudden increases in angular velocity. This ensures that the device enters working state the moment the user picks it up, avoiding operation delays.

[0148] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An imaging method for an oral imaging device, characterized in that, Includes the following steps: S1. Acquire an image sequence of at least one viewpoint collected by the oral imaging device when imaging the maxilla and / or mandible, and simultaneously acquire motion detection data generated by the motion measurement module that is time-aligned with the multi-view image sequence. S2. Extract useful video clips of the maxilla and mandible from the image sequence and organize them into at least two video segments corresponding to the maxilla and mandible respectively; S3. Identify the shooting direction of video segments by combining motion detection data with a deep learning model, and unify the video order; S4. Dynamically adjust the frame extraction method of video segments based on motion detection data, retain key motion node images, and reconstruct the calibrated video segments into a standard sequence output in a unified direction, thereby performing oral cavity 3D scene reconstruction and oral disease analysis.

2. The imaging method of an oral imaging device according to claim 1, characterized in that, In step S2, before segmentation, the original motion detection data in each video segment is denoised and standardized.

3. The imaging method of an oral imaging device according to claim 1, characterized in that, Step S2 includes the following steps: S201. Use a fixed-length sliding window on the time axis of each video segment, and count the point-by-point labels inside each sliding window. Based on the proportion of positive samples in the point-by-point labels, mark the window as a dental arch scanning window or a non-dental arch scanning window. S202, Training a deep learning model; S203. Using a deep learning model, a sliding window inference is performed on the motion detection data to obtain the probability that each window belongs to "dental arch scanning behavior". The window-level probability is mapped back to the original sampling point to form a point-by-point probability curve. Then, multiple dental arch scanning candidate segments are extracted. Combined with the minimum duration constraint, short-term interference is filtered out, and the effective scanning interval that conforms to the clinical operation rules is retained. Based on the effective scanning interval, the corresponding video segments are obtained through image reconstruction algorithm.

4. The imaging method of an oral imaging device according to claim 1, characterized in that, The deep learning model is a time-series model.

5. The imaging method of an oral imaging device according to claim 3, characterized in that, Step S3 includes the following steps: S301. Obtain motion detection data corresponding to each video segment and form an independent motion detection data file; S302. These motion detection data files are unified to fixed sampling values ​​by resampling. S303. Input the resampled motion detection data into the trained temporal model, extract temporal features and determine the scanning direction of each video segment; S304. Based on the direction labels output by the model, rearrange all video segments into a standard order from left to right or right to left. S305. Align the transition frames at the boundaries of each segment to ensure that the image at the stitching point is continuous without any jumps.

6. The imaging method of an oral imaging device according to claim 1, characterized in that, The frame extraction method in step S4 specifically involves obtaining the acceleration of the device movement corresponding to the video segment through the motion measurement module. When the acceleration of some frames in the video segment is less than a threshold, it is determined that the device is stationary or moving slowly. Any frame of that segment is taken as the image of that segment, and the remaining frames are deleted. If the acceleration is greater than the threshold, it is determined to be a fast-moving frame segment. For that frame segment, a sufficient number of frames are retained according to the ratio of acceleration for three-dimensional reconstruction.

7. The imaging method of an oral imaging device according to claim 6, characterized in that, Step S4 also includes visual detection-assisted similarity detection of similar frames.

8. The imaging method of an oral imaging device according to claim 6, characterized in that, When the acceleration is maintained for at least two frames for a duration less than a threshold, the device is determined to be stationary or in slow motion.

9. The imaging method of an oral imaging device according to claim 1, characterized in that, In step S1, when the device is detected to be in a stable and static state, it automatically switches to high-resolution imaging mode and enhances the image bitstream. Simultaneously, it triggers the computer to perform automated analysis of local areas and comparison with oral diseases, thereby realizing the identification of oral diseases.

10. An oral imaging device, characterized in that, The device includes a handle and a probe, at least one set of camera modules mounted on the probe, a motion measurement module located inside the handle, and a computer connected to the handle. The camera modules and the motion measurement module are used to collect video of the maxilla and mandible and the motion detection data corresponding to the video in the imaging method of the oral imaging device of claims 1-9.

Images