Preoperative doctor-patient communication system for dental implant based on ar glasses multi-person real-time interaction

Abstract

The application relates to a preoperative doctor-patient communication system for oral implantation based on multi-person real-time interaction of AR glasses, characterized in that the system comprises N AR glass sub-ends and a server end, and N is greater than or equal to 2; one AR glass sub-end in the N AR glass sub-ends is connected with a main service box, and the other AR glass sub-ends are connected with secondary service boxes. The AR glasses based on the spatial perception interaction slam technology can present a three-dimensional image of a patient, and the three-dimensional image can be anchored at a specific position in space. After doctors and patients wear the AR glasses at the same time, spatial three-dimensional image sharing can be realized, zooming, rotating, splitting and other functions of the image can be realized through voice and gesture interaction, and through the shared three-dimensional image, doctors and patients can intuitively understand each other's ideas, so that the communication efficiency between doctors and patients can be effectively improved, and space can be fully utilized.

Description

Technical Field

[0001] This invention relates to a system that enables doctors and patients to share and communicate in real time before surgery using SLAM maps (instantaneous localization and mapping) and location data. This invention applies the SLAM map technology of AR glasses to doctor-patient communication in dental implant clinical practice. Background Technology

[0002] Currently, there isn't a particularly effective form of communication between doctors and patients before surgery. When communicating with patients, doctors tend to use technical jargon, but this jargon is often difficult for patients to understand. Without alternative methods, patients often cannot comprehend what the doctor is trying to convey. Therefore, in clinical practice, to ensure patients understand the doctor's ideas, doctors often try to express their thoughts fully through gestures or other means, but even then, patients cannot intuitively grasp the doctor's meaning.

[0003] Currently, published patents and patent applications related to doctor-patient communication focus on enabling online communication between doctors and patients to address the difficulty patients face in accessing offline medical care. For example, patent application CN105894418A, published on August 24, 2016, discloses a doctor-patient communication emergency nursing platform that allows doctors and patients to communicate before face-to-face treatment, making full use of emergency time. Patent application CN112365357A, published on February 12, 2021, discloses a doctor-patient communication management system. This system enables patients and doctors to communicate online in real time, facilitating doctors' answering of patient questions, long-term patient management, and regular follow-ups, allowing doctors to monitor patients' conditions in real time. Simultaneously, the aforementioned doctor-patient communication management system also saves patients' medical costs, protects patient privacy, reduces doctor-patient conflicts, improves medical quality, and prevents serious medical incidents. Summary of the Invention

[0004] The technical problem that this invention aims to solve is that current patent solutions regarding preoperative communication between doctors and patients only focus on providing communication methods other than offline communication channels, without addressing how to better enable patients to understand the doctor's meaning. There is no particularly good form of communication between patients and doctors, and patients cannot intuitively understand the doctor's meaning.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is to provide a multi-person real-time interactive pre-operative doctor-patient communication system for dental implantation based on AR glasses, characterized in that it includes N AR glasses sub-endpoints and one server, where N≥2; one of the N AR glasses sub-endpoints is connected to the main server box, and the other AR glasses sub-endpoints are connected to the secondary server box.

[0006] Each AR glasses sub-endpoint includes a display unit and a camera unit, wherein: the display unit is used to process the 3D image data received from the autonomous service box or the sub-service box and then display it to the user;

[0007] The camera unit is used to scan the surrounding environment and collect environmental texture data;

[0008] The main service box includes a main VSLAM localization algorithm module, a map building module, a main communication module, a main near-eye display driver module, and a main patient CT 3D image rendering module, among which:

[0009] The main VSLAM localization algorithm module is used for tracking and localizing the corresponding AR glasses sub-endpoints;

[0010] The main patient CT 3D image rendering module is used to obtain the digital twin CT 3D image data rendered by the server through the main communication module, and then anchor the digital twin CT 3D image data at a specific position in the virtual space based on the tracking and positioning information of the main VSLAM positioning algorithm module, and then render the 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data.

[0011] The main near-eye display driver module is used to drive the display unit of the AR glasses sub-endpoint based on the three-dimensional image data;

[0012] The map building module is used to create virtual space information based on the tracking and positioning information of the corresponding AR glasses sub-endpoints obtained by the main VSLAM positioning algorithm module, and upload the virtual space information to the server through the main communication module;

[0013] The secondary service box includes a secondary VSLAM positioning algorithm module, a secondary communication module, a secondary near-eye display driver module, and a secondary patient CT 3D image rendering module, wherein:

[0014] The secondary VSLAM positioning algorithm module is used for tracking and positioning of the corresponding AR glasses sub-endpoints;

[0015] The secondary patient CT 3D image rendering module is used to obtain digital twin CT 3D image data rendered by the server and virtual space map data uploaded by the main service box from the server through the secondary communication module. It merges the virtual space map data uploaded by the main service box with the space where the current secondary service box is located to obtain a virtual space where all AR glasses sub-endpoints are in the same coordinate. Then, based on the tracking and positioning information of the secondary VSLAM positioning algorithm module, it anchors the digital twin CT 3D image data to a specific position in the virtual space and renders a 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data.

[0016] The near-eye display driver module is used to drive the display unit at the sub-end of the AR glasses based on 3D image data.

[0017] Preferably, the display unit includes a display module and a VPU module. The VPU module processes the three-dimensional image data received from the main service box or the secondary service box and then displays it to the user through the display module.

[0018] Preferably, the camera unit is a binocular camera module.

[0019] Preferably, the AR glasses sub-endpoint further includes a headband structure for fixing the display unit and the camera unit to the user's head.

[0020] Preferably, the user can also use the main patient CT 3D image rendering module or the secondary patient CT 3D image rendering module to adjust the posture of the digital twin CT 3D image data in real time, and feed it back to the server in real time through the main communication module.

[0021] Preferably, creating virtual space information includes the following steps:

[0022] Step 1: The camera unit scans the surrounding environment and collects environmental texture data. Then, a feature point extraction algorithm is used to collect the location of image feature points. At the same time, a template matching algorithm is used to analyze the environmental texture data to obtain feature description values, which include key points and descriptors.

[0023] Step 2: Use the Intensity centroid method to determine the orientation of feature points;

[0024] Step 3: Extract random feature point pairs that conform to a normal distribution using a greedy algorithm. Take a region around each feature point pair and divide the region into 4×4 sub-regions. Calculate the Haar wavelet response for each sub-region. Describe a sub-region using four values: the sum of the Haar wavelet responses in the X-axis direction, the sum of the absolute values ​​of the Haar wavelet responses in the X-axis direction, the sum of the Haar wavelet responses in the Y-axis direction, and the sum of the absolute values ​​of the Haar wavelet responses in the Y-axis direction.

[0025] Step 4: Using key points and descriptors, combined with feature point matching algorithm and affine transformation, obtain three-dimensional feature point cloud data based on the data obtained in Step 3, and use spatial position estimation to calculate virtual space map data with the initial position of the corresponding AR glasses endpoint as the origin.

[0026] Step 5: Upload the virtual space map data to the server via the main communication module for other secondary service boxes to download and join.

[0027] Preferably, in step 1, the FAST algorithm is used to extract feature points, and the Harris Corner algorithm is used to detect corner points among the feature points.

[0028] Preferably, when fusing virtual space map data with the space where the current sub-service box is located, the camera unit of the corresponding AR glasses sub-endpoint scans the surrounding environment, collects environmental texture data, uses a feature point extraction algorithm to collect the image feature point positions, and uses a template matching algorithm to analyze the environmental texture data to obtain feature description values ​​including key points and descriptors. Then, virtual space map data on the server is obtained, and the virtual space map data is fused with the space where the current sub-service box is located using the Umeyama algorithm, so that all AR glasses sub-endpoints are in the same coordinate system.

[0029] This invention relates to AR glasses based on spatial perception interactive SLAM technology, which can present a three-dimensional image of the patient, anchored at a specific location in space. When both the doctor and patient wear the AR glasses, they can share the three-dimensional image. Through voice and gesture interaction, functions such as zooming, rotating, and splitting the image can be achieved. Through the shared three-dimensional image, both the doctor and patient can intuitively understand each other's thoughts, thereby effectively improving communication efficiency and making full use of space. Attached Figure Description

[0030] Figure 1 This illustrates the coordinate system for locating and tracking doctor-patient communication.

[0031] Figure 2 It illustrates the process of creating a virtual space;

[0032] Figure 3 It illustrates the process of joining a virtual space;

[0033] Figure 4 This illustrates the process of synchronizing virtual space information. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0035] This embodiment discloses a multi-user real-time interactive pre-operative doctor-patient communication system for dental implant surgery based on AR glasses, comprising N AR glasses sub-endpoints and one server, where N ≥ 2. One of the N AR glasses sub-endpoints is connected to the main server box, and the other AR glasses sub-endpoints are connected to secondary server boxes.

[0036] Each AR glasses sub-endpoint includes a display module, a VPU module, a binocular camera module, and a headband structure. The headband structure is used to fix the display module, VPU module, and binocular camera module to the user's head. The binocular camera module is used to scan the surrounding environment and collect environmental texture data. The VPU module is used to process the 3D image data received from the autonomous service box or secondary service box and then display it to the user through the display module.

[0037] The main service box mainly consists of the main VSLAM localization algorithm module, map building module, main communication module, main near-eye display driver module, and main patient CT 3D image rendering module.

[0038] The main VSLAM localization algorithm module is used for tracking and localizing the corresponding AR glasses sub-endpoints.

[0039] The main patient CT 3D image rendering module is used to obtain the digital twin CT 3D image data rendered by the server through the main communication module. Based on the tracking and positioning information of the main VSLAM positioning algorithm module, the module anchors the digital twin CT 3D image data at a specific position in the virtual space and then renders a 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data. Users can also use the main patient CT 3D image rendering module to adjust the posture of the digital twin CT 3D image data in real time and feed it back to the server in real time through the main communication module.

[0040] The primary and near-eye display driver module is used to drive the VPU module of the AR glasses sub-endpoint based on 3D image data.

[0041] The map building module is used to create virtual space information based on the tracking and positioning information of the corresponding AR glasses sub-endpoints obtained by the main VSLAM positioning algorithm module, and uploads the virtual space information to the server through the main communication module. The creation of virtual space information includes the following steps:

[0042] Step 1: The binocular camera module scans the surrounding environment and collects environmental texture data. Then, a feature point extraction algorithm is used to collect the locations of image feature points. In this embodiment, the FAST algorithm is used to extract feature points, and the Harris Corner algorithm is used to detect corner points. Simultaneously, a template matching algorithm is used to analyze the environmental texture data to obtain feature description values, which include key points and descriptors.

[0043] Step 2: Use the Intensity centroid method to determine the orientation of feature points.

[0044] Step 3: Extract random feature point pairs that conform to a normal distribution using a greedy algorithm. Take a region around each feature point pair and divide the region into 4×4 sub-regions. Calculate the Haar wavelet response for each sub-region. Describe a sub-region using four values: the sum of the Haar wavelet responses in the X-axis direction, the sum of the absolute values ​​of the Haar wavelet responses in the X-axis direction, the sum of the Haar wavelet responses in the Y-axis direction, and the sum of the absolute values ​​of the Haar wavelet responses in the Y-axis direction.

[0045] Step 4: Using key points and descriptors, combined with feature point matching algorithms and affine transformations, obtain 3D feature point cloud data based on the data obtained in Step 3, and use spatial position estimation to calculate virtual space map data with the initial position of the corresponding AR glasses endpoint as the origin.

[0046] Step 5: Upload the virtual space map data to the server via the main communication module for other secondary service boxes to download and join.

[0047] The secondary service box mainly consists of a secondary VSLAM positioning algorithm module, a secondary communication module, a secondary near-eye display driver module, and a secondary patient CT 3D image rendering module.

[0048] The VSLAM positioning algorithm module is used for tracking and positioning of the corresponding AR glasses sub-endpoints.

[0049] The secondary patient CT 3D image rendering module is used to obtain digital twin CT 3D image data rendered by the server and virtual space map data uploaded by the main service box from the server via the secondary communication module. It merges the virtual space map data uploaded by the main service box with the space where the current secondary service box is located to obtain a virtual space where all AR glasses endpoints are at the same coordinates. Then, based on the tracking and positioning information of the secondary VSLAM positioning algorithm module, it anchors the digital twin CT 3D image data to a specific position in the virtual space and renders a 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data. Users can also use the secondary patient CT 3D image rendering module to adjust the posture of the digital twin CT 3D image data in real time and feed it back to the server in real time through the secondary communication module.

[0050] When fusing virtual space map data with the space where the current sub-service box is located, the binocular camera module of the corresponding AR glasses sub-endpoint scans the surrounding environment, collects environmental texture data, and then uses a feature point extraction algorithm to collect the image feature point positions. At the same time, a template matching algorithm is used to analyze the environmental texture data to obtain feature description values ​​including key points and descriptors. Then, virtual space map data is obtained from the server, and the umeyama algorithm is used to fuse the virtual space map data with the space where the current sub-service box is located, so that all AR glasses sub-endpoints are in the same coordinate system.

[0051] The sub-near-eye display driver module is used to drive the VPU module of the AR glasses sub-endpoint based on 3D image data.

[0052] Through the above system, the action information of the operation of the digital twin 3D image during the doctor-patient communication process can be transmitted in real time through the network, and the posture of the digital twin 3D image can be adjusted in real time through the rendering module, so as to realize the synchronous operation and interaction of multiple people.

Claims

1. A multi-person real-time interactive pre-operative doctor-patient communication system for dental implant surgery based on AR glasses, characterized in that, It includes N AR glasses sub-endpoints and one server, where N≥2; one of the N AR glasses sub-endpoints is connected to the main server box, and the other AR glasses sub-endpoints are connected to the secondary server box; Each AR glasses sub-endpoint includes a display unit and a camera unit, wherein: the display unit is used to process the 3D image data received from the autonomous service box or the sub-service box and then display it to the user; The camera unit is used to scan the surrounding environment and collect environmental texture data; The main service box includes a main VSLAM localization algorithm module, a map building module, a main communication module, a main near-eye display driver module, and a main patient CT 3D image rendering module, among which: The main VSLAM localization algorithm module is used for tracking and localizing the corresponding AR glasses sub-endpoints; The main patient CT 3D image rendering module is used to obtain the digital twin CT 3D image data rendered by the server through the main communication module, and then anchor the digital twin CT 3D image data at a specific position in the virtual space based on the tracking and positioning information of the main VSLAM positioning algorithm module, and then render the 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data. The main near-eye display driver module is used to drive the display unit of the AR glasses sub-endpoint based on the three-dimensional image data; The map building module is used to create virtual space information based on the tracking and positioning information of the corresponding AR glasses sub-endpoints obtained by the main VSLAM positioning algorithm module, and upload the virtual space information to the server through the main communication module; The secondary service box includes a secondary VSLAM positioning algorithm module, a secondary communication module, a secondary near-eye display driver module, and a secondary patient CT 3D image rendering module, wherein: The secondary VSLAM positioning algorithm module is used for tracking and positioning of the corresponding AR glasses sub-endpoints; The secondary patient CT 3D image rendering module is used to obtain digital twin CT 3D image data rendered by the server and virtual space map data uploaded by the main service box from the server through the secondary communication module. It merges the virtual space map data uploaded by the main service box with the space where the current secondary service box is located to obtain a virtual space where all AR glasses sub-endpoints are in the same coordinate. Then, based on the tracking and positioning information of the secondary VSLAM positioning algorithm module, it anchors the digital twin CT 3D image data to a specific position in the virtual space and renders a 3D image containing the digital twin CT 3D image data and the virtual space to generate 3D image data. The near-eye display driver module is used to drive the display unit at the sub-end of the AR glasses based on 3D image data.

2. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, The display unit includes a display module and a VPU module. The VPU module processes the three-dimensional image data received from the main service box or the secondary service box and then displays it to the user through the display module.

3. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, The camera unit uses a binocular camera module.

4. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, The AR glasses sub-end also includes a headband structure for fixing the display unit and the camera unit to the user's head.

5. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, Users can also use the main patient CT 3D image rendering module or the secondary patient CT 3D image rendering module to adjust the orientation of the digital twin CT 3D image data in real time, and feed it back to the server in real time through the main communication module.

6. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, Creating virtual space information includes the following steps: Step 1: The camera unit scans the surrounding environment and collects environmental texture data. Then, a feature point extraction algorithm is used to collect the location of image feature points. At the same time, a template matching algorithm is used to analyze the environmental texture data to obtain feature description values, which include key points and descriptors. Step 2: Use the Intensity centroid method to determine the orientation of feature points; Step 3: Extract random feature point pairs that conform to a normal distribution using a greedy algorithm. Take a region around each feature point pair and divide the region into 4×4 sub-regions. Calculate the Haar wavelet response for each sub-region. Describe a sub-region by taking four values: the sum of the Haar wavelet responses in the X-axis direction, the sum of the absolute values ​​of the Haar wavelet responses in the X-axis direction, the sum of the Haar wavelet responses in the Y-axis direction, and the sum of the absolute values ​​of the Haar wavelet responses in the Y-axis direction. Step 4: Using key points and descriptors, combined with feature point matching algorithm and affine transformation, obtain three-dimensional feature point cloud data based on the data obtained in Step 3, and use spatial position estimation to calculate virtual space map data with the initial position of the corresponding AR glasses endpoint as the origin. Step 5: Upload the virtual space map data to the server via the main communication module for other secondary service boxes to download and join.

7. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 6, characterized in that, In step 1, the FAST algorithm is used to extract feature points, and the Harris Corner algorithm is used to detect corner points among the feature points.

8. The pre-operative doctor-patient communication system for dental implant surgery based on AR glasses with real-time multi-person interaction as described in claim 1, characterized in that, When fusing virtual space map data with the space where the current sub-service box is located, the camera unit of the corresponding AR glasses sub-endpoint scans the surrounding environment, collects environmental texture data, and then uses a feature point extraction algorithm to collect the location of image feature points. At the same time, a template matching algorithm is used to analyze the environmental texture data to obtain feature description values ​​including key points and descriptors. Then, virtual space map data on the server is obtained, and the virtual space map data is fused with the space where the current sub-service box is located through the Umeyama algorithm, so that all AR glasses sub-endpoints are in the same coordinate system.

Images