Oral treatment table training system and method based on spatial computing technology

Abstract

The present application provides an oral treatment table training system and method based on spatial computing technology, which relates to augmented reality technology and is used to solve the problems that existing simulators only support single-person use and only perform training based on oral models, lack full-process doctor-patient communication, and have low simulation authenticity. With the help of spatial computing technology, the virtual-reality integrated head-mounted viewer obtains the positioning data of entities and the dental chair of the treatment table in the real space, and the situation observation server arranges virtual entities and generates a scene file. The simulation computer generates a virtual-reality integrated treatment scene, virtual patients, and treatment tools according to the scene file, and can simulate the whole process of the patient from entry to treatment. And generate force sense information to control the force sense tool in the force sense information and match the pose of the virtual treatment tool with the force feedback tool. The user wears the virtual-reality integrated head-mounted viewer to perform treatment operations. With the help of network synchronization technology, it supports multiple people to watch the operation process of the same treatment table. The present application improves the comprehensive simulation authenticity.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular, to an oral treatment table training system and method based on spatial computing technology. Background Art

[0002] With the rapid development of augmented reality technology, it has been widely applied in multiple fields. By establishing an oral treatment table training system through augmented reality technology, it can effectively help doctors conduct oral treatment training.

[0003] In the prior art, there is an oral surgery skill training simulator, which includes a simulation training platform, an oral operation training system based on a force feedback device, and an observation system based on an augmented reality helmet. Among them, the simulation training platform is used to generate a simulated head model, the oral operation training system based on the force feedback device is used to generate specific forces to simulate the force sensation when touching the oral cavity, and the observation system based on the augmented reality helmet is used to generate corresponding virtual training images according to the simulated head model and the force sensation.

[0004] However, there is a gap between the prior art and real clinical operations, and the simulation fidelity is low, which affects the user experience. Summary of the Invention

[0005] Embodiments of this application provide an oral treatment table training system and method based on spatial computing technology to achieve the technical effect of improving simulation fidelity.

[0006] In a first aspect, this application proposes an oral treatment table training system based on spatial computing technology, including: a plurality of treatment tables and a situation observation server. The plurality of treatment tables communicate with the situation observation server through the TCP / IP protocol. Each treatment table includes: a simulation computer, a virtual-real fusion head-mounted viewer, and a two-handed force feedback tool;

[0007] The virtual-real fusion head-mounted viewer: is used to obtain the positioning data of entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology. The entity objects do not include the dental chair of the treatment table;

[0008] The situation observation server: is used to generate a scene file for oral treatment according to the positioning data;

[0009] The simulation computer: is used to load the scene file and generate a virtual-real fusion space according to the scene file. In the virtual-real fusion space, a virtual patient to be treated and virtual treatment tools corresponding to real treatment tools created in advance are loaded, and the simulation entity data of the virtual patient and the virtual treatment tools are recorded;

[0010] The simulation computer is further configured to generate a virtual diagnosis and treatment scenario based on the scenario file and the simulation entity data, and generate force sense information according to the diagnosis and treatment operations performed by a real user through operating the two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Based on a matching algorithm, the force sense tool and the virtual diagnosis and treatment tool are controlled to match the pose of the two-handed force feedback tool, so as to obtain the matched simulation entity data;

[0011] The virtual-real fusion head-mounted viewer is configured to generate a first virtual-real fusion diagnosis and treatment image based on the matched simulation entity data, and the first virtual-real fusion diagnosis and treatment image is used for a real user to perform diagnosis and treatment operations;

[0012] The situation observation server is further configured to obtain the simulation entity data corresponding to any diagnosis and treatment table;

[0013] The virtual-real fusion head-mounted viewer is further configured to generate a second virtual-real fusion diagnosis and treatment image according to the simulation entity data of any diagnosis and treatment table forwarded by the situation observation server;

[0014] Wherein, in the first virtual-real fusion diagnosis and treatment image and the second virtual-real fusion diagnosis and treatment image, the process of a virtual doctor closely fitting on a dental chair of a diagnosis and treatment table in the real space to receive diagnosis and treatment operations is displayed.

[0015] Optionally, when the virtual-real fusion head-mounted viewer is configured to obtain the positioning data of the entity object and the dental chair of the diagnosis and treatment table existing in the real space based on the spatial computing technology, it is specifically configured to:

[0016] Scan a positioning picture with significant features at a specified position in the real space to determine a reference coordinate system and a reference point;

[0017] Obtain the pose information of the reference point in the current device coordinate system;

[0018] Establish a mapping relationship between the reference coordinate system and the device coordinate system according to the pose information of the reference point in the current device coordinate system;

[0019] Measure the positioning data of the entity object existing in the real space in the reference coordinate system;

[0020] Scan a positioning picture with significant features at the position of the positioning reference point of the dental chair of the diagnosis and treatment table in the real space to obtain the pose information of the dental chair of the diagnosis and treatment table in the device coordinate system;

[0021] Obtain the positioning data of the dental chair of the diagnosis and treatment table in the reference coordinate system according to the pose information of the dental chair of the diagnosis and treatment table in the device coordinate system and the mapping relationship.

[0022] Optionally, when generating a scenario file for oral diagnosis and treatment based on the positioning data, the situation observation server is specifically configured to:

[0023] Generate a virtual space based on the positioning data of the entity object in the reference coordinate system;

[0024] Generate a virtual dental chair of the examination table at the corresponding position in the virtual space according to the positioning data of the dental chair of the examination table in the reference coordinate system;

[0025] In response to the operation of the user, generate preset virtual facilities in the virtual space;

[0026] Obtain the coordinate and pose data of the virtual space, the virtual dental chair of the examination table, and the virtual facilities, and generate a scenario file for oral diagnosis and treatment according to the coordinate and pose data.

[0027] Optionally, when the simulation computer is used to generate force sense information according to the diagnosis and treatment operations of a real user through operating the two-handed force feedback tool, and the force sense information includes the pose information of the force sense tool, and based on a matching algorithm, control the force sense tool and the virtual diagnosis and treatment tool to match the pose of the two-handed force feedback tool to obtain the matched virtual-real entity data, it is specifically configured to:

[0028] Determine the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair of the examination table;

[0029] Obtain the pose information of the head model of the virtual examiner in the force sense coordinate system;

[0030] Generate force sense information according to the diagnosis and treatment operations of the two-handed force feedback tool based on the pose information of the head model in the force sense coordinate system, and the force sense information includes the pose information of the force sense tool;

[0031] Obtain the pose information of the virtual diagnosis and treatment tool in the force sense coordinate system;

[0032] In the force sense coordinate system, match the pose information of the force sense tool and the virtual diagnosis and treatment tool in the force sense coordinate system with the pose information of the two-handed force feedback tool, and convert it to the reference coordinate system to obtain the matched virtual-real entity data in the reference coordinate system.

[0033] Optionally, when the virtual-real fusion head-mounted viewer is used to generate the first virtual-real fusion diagnosis and treatment image according to the matched virtual-real entity data, it is specifically configured to:

[0034] Obtain the mapping relationship between the reference coordinate system and the device coordinate system;

[0035] According to the mapping relationship, transfer the simulated entity data in the matched reference coordinate system to the device coordinate system to obtain the simulated entity data in the device coordinate system;

[0036] Obtain the scene file, and based on the scene file and the simulated entity data in the device coordinate system, obtain the virtual image in the virtual-real fusion space;

[0037] Obtain the real-time image in the captured real space;

[0038] Overlay the virtual image in the virtual-real fusion space and the real-time image to obtain the first virtual-real fusion diagnosis and treatment image.

[0039] Optionally, the situation observation server is further configured to:

[0040] Display the virtual diagnosis and treatment scenes corresponding to different diagnosis and treatment tables.

[0041] Optionally, the situation observation server is further configured to:

[0042] Set the virtual patient calling mode, where the calling mode includes the ordinary calling mode and the enhanced calling mode. The virtual patient corresponding to the ordinary calling mode has a random disease cause, and the virtual patient corresponding to the enhanced calling mode has a preset fixed disease cause;

[0043] Edit the virtual patient queue corresponding to the calling mode.

[0044] Optionally, the diagnosis and treatment table further includes: a speaker and a microphone;

[0045] When the simulation computer loads the pre-created virtual patient to be diagnosed and treated in the virtual-real fusion space, the real user sends control instructions to the virtual patient through the microphone and receives the interactive voice output by the virtual patient through the speaker.

[0046] Optionally, the diagnosis and treatment table further includes: a height sensor and a pitch sensor;

[0047] When the simulation computer detects that the lifting height of the dental chair measured by the height sensor or the rotation angle of the dental chair measured by the pitch sensor changes, the simulation computer, based on the matching algorithm, controls the force feedback tool and the virtual diagnosis and treatment tool to re-match the pose of the two-handed force feedback tool to obtain new matched simulated entity data.

[0048] Optionally, the oral diagnosis and treatment table training system based on spatial computing technology is connected to at least one external handheld terminal, so that the at least one external handheld terminal displays the virtual-real fusion diagnosis and treatment image, and the at least one external terminal has an augmented reality function.

[0049] In a second aspect, the present application proposes an oral diagnosis and treatment training method based on spatial computing technology, which is applied to an oral diagnosis and treatment training system based on spatial computing technology, and includes:

[0050] Obtaining positioning data of entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology, where the entity objects do not include the dental chair of the treatment table;

[0051] Generating a scene file for oral diagnosis and treatment according to the positioning data;

[0052] Loading the scene file, and generating a virtual-real fusion space according to the scene file. In the virtual-real fusion space, a virtual patient to be diagnosed and treated created in advance and a virtual diagnosis and treatment tool corresponding to the real diagnosis and treatment tool are loaded, and the simulation entity data of the virtual patient and the virtual diagnosis and treatment tool are recorded;

[0053] Generating a virtual diagnosis and treatment scene according to the scene file and the simulation entity data, and generating force sense information according to the diagnosis and treatment operations performed by the real user through operating a two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Based on a matching algorithm, controlling the force sense tool and the virtual diagnosis and treatment tool to match the pose of the two-handed force feedback tool to obtain the matched simulation entity data;

[0054] Generating a first virtual-real fusion diagnosis and treatment image according to the matched simulation entity data, where the first virtual-real fusion diagnosis and treatment image is used for the real user to perform diagnosis and treatment operations;

[0055] Obtaining the simulation entity data corresponding to any treatment table;

[0056] Generating a second virtual-real fusion diagnosis and treatment image according to the forwarded simulation entity data of any treatment table;

[0057] Wherein, in the first virtual-real fusion diagnosis and treatment image and the second virtual-real fusion diagnosis and treatment image, the process of a virtual patient closely adhering to the dental chair of the treatment table in the real space to receive diagnosis and treatment operations is displayed.

[0058] Optionally, the obtaining the positioning data of entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology includes:

[0059] Scanning a positioning picture with significant features at a specified position in the real space to determine a reference coordinate system and a reference point;

[0060] Obtaining the pose information of the reference point in the current device coordinate system;

[0061] Establish the mapping relationship between the reference coordinate system and the device coordinate system according to the pose information of the reference point in the current device coordinate system;

[0062] Measure the positioning data of the entity object existing in the real space in the reference coordinate system;

[0063] Scan the positioning picture with significant features at the positioning reference point of the dental chair on the treatment table in the real space to obtain the pose information of the dental chair on the treatment table in the device coordinate system;

[0064] According to the pose information of the dental chair on the treatment table in the device coordinate system and the mapping relationship, obtain the positioning data of the dental chair on the treatment table in the reference coordinate system.

[0065] Optionally, generating a scene file for oral diagnosis and treatment according to the positioning data includes:

[0066] Generate a virtual space according to the positioning data of the entity object in the reference coordinate system;

[0067] Generate a virtual dental chair on the treatment table at the corresponding position in the virtual space according to the positioning data of the dental chair on the treatment table in the reference coordinate system;

[0068] In response to the user's operation, generate preset virtual facilities in the virtual space;

[0069] Obtain the coordinate and pose data of the virtual space, the virtual dental chair on the treatment table, and the virtual facilities, and generate a scene file for oral diagnosis and treatment according to the coordinate and pose data.

[0070] Optionally, generating force sense information according to the real user's diagnosis and treatment operation by operating the two-handed force feedback tool, where the force sense information includes the pose information of the force sense tool. Based on the matching algorithm, control the force sense tool and the virtual diagnosis tool to match the pose of the two-handed force feedback tool, and obtain the matched virtual entity data, including:

[0071] Determine the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair on the treatment table;

[0072] Obtain the pose information of the head model of the virtual examiner in the force sense coordinate system;

[0073] Generate force sense information according to the diagnosis and treatment operation of the two-handed force feedback tool based on the pose information of the head model in the force sense coordinate system, where the force sense information includes the pose information of the force sense tool;

[0074] Obtain the pose information of the virtual diagnosis tool in the force sense coordinate system;

[0075] Under the force sense coordinate system, match the pose information of the force sense tool and the virtual diagnosis and treatment tool in the force sense coordinate system with the pose information of the two-handed force feedback tool, and transform it to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching.

[0076] Optionally, generating the first virtual-real fusion diagnosis and treatment image according to the matched simulated entity data includes:

[0077] Obtain the mapping relationship between the reference coordinate system and the device coordinate system;

[0078] According to the mapping relationship, transfer the simulated entity data in the reference coordinate system after matching to the device coordinate system to obtain the simulated entity data in the device coordinate system;

[0079] Obtain the scene file, and according to the scene file and the simulated entity data in the device coordinate system, obtain the virtual image in the virtual-real fusion space;

[0080] Obtain the real-time image in the captured real space;

[0081] Overlay the virtual image in the virtual-real fusion space and the real-time image to obtain the first virtual-real fusion diagnosis and treatment image.

[0082] An embodiment of the present application provides an oral treatment table training system and method based on spatial computing technology. The system includes: multiple treatment tables and a situation observation server. The multiple treatment tables communicate with a situation observation server through the TCP / IP protocol. Each treatment table includes: a simulation computer, a virtual-real fusion head-mounted viewer, and a two-handed force feedback tool. Among them, the virtual-real fusion head-mounted viewer obtains the positioning data of the entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology. The situation observation server generates a scene file for oral treatment according to the positioning data. The simulation computer loads the scene file and generates a virtual-real fusion space according to it. In the virtual-real fusion space, a pre-created virtual patient to be treated and virtual treatment tools corresponding to real treatment tools are loaded, and the simulation entity data of the virtual patient and the virtual treatment tools are recorded. The simulation computer also generates a virtual treatment scene according to the scene file and the simulation entity data, and obtains force sense information according to the treatment operation of the real user through operating the two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Furthermore, based on a matching algorithm, the force sense tool and the virtual treatment tool are controlled to match the pose of the two-handed force feedback tool, and the matched simulation entity data is obtained. The virtual-real fusion head-mounted viewer generates a first virtual-real fusion treatment image according to the matched simulation entity data. The situation observation server can also obtain the simulation entity data corresponding to any treatment table, so that the virtual-real fusion head-mounted viewer can generate a corresponding second virtual-real fusion treatment image according to the simulation entity data of any treatment table forwarded by the situation observation server. In this application, by generating a virtual-real fusion treatment scene, the simulation authenticity is improved, and the system can support multiple people to watch the treatment process of the same treatment table, which is convenient for users to communicate and improves the user experience. Description of the Drawings

[0083] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0084] Figure 1 It is a schematic structural diagram of an oral treatment table training system based on spatial computing technology provided by an embodiment of the present application;

[0085] Figure 2 It is a schematic flow diagram of a method for obtaining the positioning data of entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology provided by an embodiment of the present application;

[0086] Figure 3 It is a schematic flow diagram of a method for obtaining the matched simulation entity data provided by an embodiment of the present application;

[0087] Figure 4 It is a schematic diagram after virtual-real matching provided by an embodiment of the present application;

[0088] Figure 5 Schematic diagram of an oral treatment table training system based on spatial computing technology provided by an embodiment of the present application;

[0089] Figure 6 Application schematic diagram of an oral treatment table training system based on spatial computing technology provided by an embodiment of the present application;

[0090] Figure 7 Scene schematic diagram provided by an embodiment of the present application;

[0091] Figure 8 Flow schematic diagram of an oral treatment training method based on spatial computing technology provided by an embodiment of the present application.

[0092] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0093] Here, exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0094] In the description of the embodiments of the present application, terms such as "inner" and "outer" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present application.

[0095] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, the terms "connected" and "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those skilled in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific situations.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0097] The mastery of oral clinical skills requires medical students or doctors to conduct a large number of practical operation trainings. However, in traditional teaching, medical students have limited opportunities to obtain practical operations. Junior doctors often also face the dilemma of lacking patients to practice on in the early stage of their careers, which poses challenges to their skill development and confidence building.

[0098] In the prior art, in order to improve the diagnosis and treatment skills of medical students or doctors, it is usually through using a surgical skill training simulator to simulate the diagnosis and treatment scenario for training. Such simulators generally include a support platform, a simulation computer, a display screen, a force feedback device, etc. Among them, the support platform has a specific structure to integrate each device into an overall simulator. The simulation computer is built into the support platform and is used to provide simulation control of the diagnosis and treatment operation process and output visual and haptic information. The output virtual oral 3D model is viewed by the user through the display screen, and the output haptic information is transmitted through the force feedback device and a series of connecting rods, and finally is sensed by the connecting rod end handpiece (dental diagnosis and treatment tool) in the user's hand. The connecting rod end handpiece is usually located below the display screen and is in the same space as the virtual oral 3D model presented on the display screen in terms of perception, so as to simulate the real diagnosis and treatment process.

[0099] Exemplarily, there is currently an oral surgical skill training simulator, which includes a simulation training platform, an oral operation training system based on a force feedback device, and an observation system based on an augmented reality helmet. Among them, the simulation training platform is used to generate a simulation head model, the oral operation training system based on the force feedback device is used to generate specific forces to simulate the haptic feeling when touching the oral cavity, and the observation system based on the augmented reality helmet is used to generate corresponding virtual training images according to the simulation head model and the haptic feeling.

[0100] However, the current training simulators only support single-person use, cannot realize multiple people watching the diagnosis and treatment operations on the same diagnosis and treatment table at the same time, are not convenient for users to communicate in a timely manner, and only train based on the oral cavity model, without taking into account the patient reception and communication process, lacking practice in skills of communicating with patients for diagnosis, and do not construct the entire virtual space of the diagnosis and treatment room, with relatively low simulation authenticity and lacking a sense of on-site immersion, which affects the user experience.

[0101] Therefore, in view of the above technical problems in the prior art, the inventors found during the research process that based on spatial computing technology, by superimposing virtual viewers in the real space and integrating the real space with the virtual space, the resulting virtual-real fusion diagnosis and treatment images can be made closer to reality to improve the simulation realism. By enabling multiple people to simultaneously view the virtual-real fusion diagnosis and treatment images of the same examination table, it is convenient for users to communicate in a timely manner to enhance the user experience. Based on this, the present application proposes an oral examination table training system and method based on spatial computing technology.

[0102] The application scenario of the oral examination table training system based on spatial computing technology provided by the present application can be applied in schools for teaching, in hospitals for training, or in other scenarios where there is a need, and the present application does not make any limitations.

[0103] The following will specifically describe the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0104] Figure 1 FIG. is a schematic structural diagram of an oral examination table training system based on spatial computing technology provided by an embodiment of the present application. As Figure 1 shown, the system includes: a plurality of examination tables 01 and a situation observation server 02. Each examination table 01 communicates with the situation observation server 02 through the TCP / IP protocol.

[0105] Among them, each examination table includes: a simulation computer 011, a virtual-real fusion head-mounted viewer 012, and a two-handed force feedback tool 013.

[0106] The virtual-real fusion head-mounted viewer 012: is used to obtain the positioning data of the entity objects existing in the real space and the dental chair of the examination table 01 based on spatial computing technology. The entity objects do not include the dental chair of the examination table 01.

[0107] In this embodiment, the virtual-real fusion head-mounted viewer 012 can be a mixed reality device with spatial computing capabilities, capable of object positioning, generating virtual images and real images for superimposed display, and having a wireless transmission function, such as a mixed display helmet, an augmented reality glasses, etc.

[0108] In the virtual-real fusion head-mounted viewer 012, there is installed a scene setting tool based on spatial computing technology. The scene setting tool is an application component of the spatial computing technology scene system, which is used to measure the positions of the entity objects existing in the real space, that is, the real examination room, and is also used to locate the pose of the dental chair of the examination table 01 in the created virtual space.

[0109] Optionally, the entity object can be each wall, floor, ceiling, door, etc. It can be understood that the type of the entity object is not limited in this embodiment, and it can also be other things existing in the real space.

[0110] After the virtual-real fusion head-mounted viewer 012 obtains the positioning data of the entity object existing in the real space and the examination table 01 dental chair, it communicates the same to the situation observation server 02 through the TCP / IP protocol.

[0111] Situation observation server 02: It is used to generate a scene file for oral diagnosis and treatment according to the positioning data.

[0112] In the scene layout stage, the situation observation server 02 can receive the positioning data of the entity object and the examination table 01 dental chair sent by the virtual-real fusion head-mounted viewer 012, generate a virtual space according to the positioning data, and can also arrange virtual facilities related to the consulting room in the virtual space, such as medical cabinets, call screens, etc.

[0113] The situation observation server 02 can also generate a scene file according to the coordinate attitude data of the generated virtual space, virtual examination table dental chair and virtual facilities, and save it in a preset storage module.

[0114] In this embodiment, the situation observation server 02 can also set a virtual patient calling mode and edit a virtual patient queue corresponding to the calling mode. The calling mode can include a normal calling mode and an enhanced calling mode. Among them, the virtual patients corresponding to the normal calling mode have random causes, and the virtual patients corresponding to the enhanced calling mode have preset fixed causes.

[0115] In the normal calling mode, all examination tables 01 share the same virtual patient queue with random causes. When an examination table 01 calls a number, a patient can be dequeued from the shared queue in turn for treatment. In the enhanced calling mode, each examination table 01 corresponds to a dedicated queue with a specified cause sequence for intensive practice.

[0116] The situation observation server 02 is also equipped with a large display screen. When the situation observation server 02 receives the calling information sent by the examination table 01, according to the calling mode set by the situation observation server 02 and the identification information of the examination table 01 included in the calling information, the target virtual patient can be determined from the virtual patient queue corresponding to the calling mode and sent to the corresponding examination table 01, and the large display screen will also display the target virtual patient.

[0117] Simulation computer 011: It is used to load a scenario file and generate a virtual-real fusion space according to the scenario file. In the virtual-real fusion space, a pre-created virtual patient to be diagnosed and virtual diagnostic tools corresponding to real diagnostic tools are loaded, and the simulation entity data of the virtual patient and the virtual diagnostic tools are recorded.

[0118] In this embodiment, the simulation computer 011 can be connected to the situation observation server 02 in a wired manner for communication connection, so that the scenario file distributed by the situation observation server 02 can be loaded.

[0119] An oral diagnosis and treatment simulation training system runs in the simulation computer 011.

[0120] The simulation computer 011 loads the scenario file distributed by the situation observation server 02 and generates a virtual-real fusion space according to the scenario file to simulate a diagnosis and treatment room for diagnosis and treatment training.

[0121] The simulation computer 011 loads a pre-created virtual patient to be diagnosed through a queuing operation and loads the virtual oral model corresponding to the virtual patient. Among them, the virtual oral model has been pre-constructed based on oral data. The oral data can be, for example, cone beam CT (CBCT) scan data and true color scan data, etc.

[0122] Optionally, the oral data of the virtual patient can be obtained by pre-collecting the oral data of a real patient. For example, use a CBCT device and a true color scanning device to scan the oral cavity of the patient to obtain DICOM-format CBCT oral data and stl-format oral scan data. Through a preset three-dimensional modeling model, the above data is superimposed and reconstructed to obtain a complete three-dimensional surface and physical model of the lower half of the skull, thereby pre-constructing a virtual oral model.

[0123] Optionally, a virtual oral operation environment can be pre-created according to the virtual oral model and virtual diagnostic tools, and the virtual diagnostic tools can be pre-created according to the measurement data such as the size and shape of the obtained diagnostic tools.

[0124] Optionally, the virtual patient can be pre-created. Specifically, based on a pre-constructed instruction-based behavior-driven model and knowledge graph model, a diagnosis and treatment information set of the virtual patient is established, and the diagnosis and treatment information set is fused with the virtual oral operation environment, and finally a virtual patient is generated. The virtual patient has force feedback interaction ability.

[0125] The simulation computer 011 records the simulation entity data of the virtual patient and the virtual diagnostic tools and shares the simulation entity data to the situation observation server 02.

[0126] In this embodiment, each simulation computer 011 will be assigned a unique Id Sim , when the simulation computer 011 loads the virtual patient and virtual diagnosis and treatment tools, local unique Ids will be assigned to the loaded virtual patient and virtual diagnosis and treatment tools Local , and based on the preset Id Sim <<16|Id Local , a method is used to generate a globally unique Id for the loaded virtual patient and virtual diagnosis and treatment tools Global , which serves as the synchronization identifier for the unified virtual-real fusion space

[0127] Simulation computer 011: It is used to generate a virtual diagnosis and treatment scenario according to the scenario file and simulation entity data, and obtain force sense information based on the diagnosis and treatment operations of the real user through operating the two-handed force feedback tool 013. The force sense information includes the pose information of the force sense tool, and also includes the magnitude and direction of the force, etc

[0128] The simulation computer 011 generates a virtual diagnosis and treatment scenario according to the loaded scenario file and the created simulation entities. Among them, each simulation entity has a different Id Global identifier

[0129] The simulation computer 011 is also used to control the force sense tool and the virtual diagnosis and treatment tool to match the pose of the two-handed force feedback tool based on the matching algorithm, and obtain the matched simulation entity data

[0130] After the simulation computer 011 loads the virtual patient and virtual diagnosis and treatment tools, it performs diagnosis and treatment operations on the virtual patient through the two-handed force feedback tool 013 to obtain force sense information. Furthermore, based on the preset triple matching algorithm, it controls the force sense tool and the virtual diagnosis and treatment tool to match the pose of the two-handed force feedback tool, so that the presented virtual-real diagnosis and treatment images are closer to reality

[0131] Furthermore, the virtual-real fusion head-mounted viewer 012 generates the first virtual-real fusion diagnosis and treatment image according to the matched simulation entity data

[0132] The virtual-real fusion head-mounted viewer 012 first detects whether there is an object corresponding to the identification Id Global , if it exists, it updates the existing object according to the obtained simulation entity data, if it does not exist, it creates a corresponding object according to the obtained simulation entity data, and finally obtains the virtual-real fusion diagnosis and treatment image

[0133] In this application, the situation observation server 02 can also obtain the simulation entity data in any diagnosis and treatment table simulation computer, so that the virtual-real fusion head-mounted viewer 012 generates the corresponding virtual-real fusion diagnosis and treatment image according to the simulation entity data of any diagnosis and treatment table forwarded by it

[0134] In the generated virtual-real fusion diagnosis and treatment images, the process of a virtual patient closely adhering to the dental chair on the treatment table in the real space and receiving treatment operations will be displayed.

[0135] Through communication based on the TCP / IP protocol, that is, the multi-person network synchronization technology, multiple treatment tables are networked for use, enabling any real user wearing the virtual-real fusion head-mounted viewer 012 to observe the virtual-real fusion diagnosis and treatment images of other treatment tables at any time, facilitating timely communication among users and thus enhancing the user experience.

[0136] In this application, the large display screen equipped on the situation observation server 02 can also display the virtual diagnosis and treatment scenarios corresponding to any treatment table.

[0137] In the above-mentioned embodiment of this application, the system includes: multiple treatment tables and one situation observation server. The multiple treatment tables communicate with one situation observation server through the TCP / IP protocol. Each treatment table includes: a simulation computer, a virtual-real fusion head-mounted viewer, and a two-handed force feedback tool. Among them, the virtual-real fusion head-mounted viewer obtains the positioning data of the entity objects and the dental chair on the treatment table existing in the real space based on spatial computing technology. The situation observation server generates a scenario file for oral diagnosis and treatment according to this positioning data. The simulation computer loads this scenario file and generates a virtual-real fusion space according to it. In the virtual-real fusion space, a pre-created virtual patient to be treated and virtual treatment tools corresponding to real treatment tools are loaded, and the simulation entity data of the virtual patient and the virtual treatment tools are recorded. The simulation computer also generates a virtual diagnosis and treatment scenario according to the scenario file and the simulation entity data, and obtains force sense information according to the treatment operations performed by the real user through the two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Furthermore, based on the matching algorithm, the force sense tool and the virtual treatment tools are controlled to match the pose of the two-handed force feedback tool, and the matched simulation entity data is obtained. The virtual-real fusion head-mounted viewer generates the first virtual-real fusion diagnosis and treatment image according to the matched simulation entity data. The situation observation server can also obtain the simulation entity data corresponding to any treatment table, so that the virtual-real fusion head-mounted viewer can generate the corresponding second virtual-real fusion diagnosis and treatment image according to the simulation entity data of any treatment table forwarded by the situation observation server. The system of this embodiment improves the simulation fidelity by generating a virtual-real fusion diagnosis and treatment scenario, and this system can support multiple people to watch the treatment process of the same treatment table, facilitating communication among users and enhancing the user experience.

[0138] Further, on the basis of the above-mentioned embodiment, the following embodiment illustrates the process of the virtual-real fusion head-mounted viewer obtaining the positioning data of the entity objects and the dental chair on the treatment table existing in the real space based on spatial computing technology.

[0139] Figure 2Schematic flowchart of a method for obtaining positioning data of entity objects and treatment tables / dental chairs existing in a real space based on spatial computing technology provided by an embodiment of this application. As Figure 2 shown, the method may include:

[0140] S201. Scan a positioning picture with significant features at a specified position in the real space to determine a reference coordinate system and a reference point.

[0141] S202. Obtain the pose information of the reference point in the current device coordinate system.

[0142] Previously select a position as the reference point O real in the real empty room R ref , and stipulate the orientations of each coordinate axis to form a reference coordinate system Coord ref . Place a Marker picture at the reference point O ref . Among them, the Marker picture can be a QR code picture or other pictures with obvious features.

[0143] After the virtual-reality fusion head-mounted viewer is started, determine the device coordinate system Coord dev and the origin O dev of the current device.

[0144] The virtual-reality fusion head-mounted viewer scans the Marker picture to obtain the pose information O ref of the reference point O of the reference coordinate system in the current device coordinate system ref '=(x,y,z,α,β,γ), where x, y, and z are the position information of the reference point O ref ', and α, β, and γ are the orientation information of the reference point O ref '.

[0145] S203. Establish a mapping relationship between the reference coordinate system and the device coordinate system according to the pose information of the reference point in the current device coordinate system.

[0146] Combined with the pose information of O ref ', use coordinate basis transformation to construct the translation matrix T from O dev to O ref , and the rotation matrix R. Multiply the two matrices, that is, T·R, to obtain the transformation matrix M from the reference coordinate system Coord ref to the device coordinate system Coord dev .

[0147] Invert the transformation matrix M to obtain M -1 . This matrix M -1 is the transformation matrix from the device coordinate system Coord dev to the reference coordinate system Coord reftransformation matrix.

[0148] transformation matrix M and M -1 represents the mapping relationship between the reference coordinate system and the device coordinate system.

[0149] S204. Measure the positioning data of the entity objects existing in the real space in the reference coordinate system.

[0150] Measure the entity objects existing in the real space, such as each wall, floor, ceiling, door, etc., relative to the reference point O ref The positioning data is used by the situation observation server to construct the virtual space R virtual such that each wall, floor, ceiling, door, etc. in the virtual space R virtual coincide with the positions of each wall, floor, ceiling, door, etc. in the real space R real exactly.

[0151] S205. Scan the positioning pictures with significant features at the positioning reference point of the dental chair on the treatment table in the real space, and obtain the pose information of the dental chair on the treatment table in the device coordinate system.

[0152] Pre-arrange the treatment tables (C1, C2,... C real ) including dental chairs for simulation training in the real space R n ), and place different Marker maps M i at the origin of each treatment table C i , where M i contains pictures of the dental chair on the treatment table in the real space.

[0153] The virtual-reality fusion head-mounted viewer scans M i to obtain the Marker identification and the pose information of the treatment table Ci in the device coordinate system Coord dev below.

[0154] S206. According to the pose information of the dental chair on the treatment table in the device coordinate system and the mapping relationship, obtain the positioning data of the dental chair on the treatment table in the reference coordinate system.

[0155] The virtual-reality fusion head-mounted viewer uses the transformation matrix M -1 to transform the pose information of the dental chair on the treatment table in the device coordinate system to the reference coordinate system Coord ref below, and combines it with the Marker identification for storage to obtain the positioning data of the dental chair on the treatment table in the reference coordinate system.

[0156] The virtual-real fusion head-mounted viewer communicates the positioning data of the entity objects existing in the real space and the dental chair of the treatment table to the situation observation server based on the TCP / IP protocol. The situation observation server generates a virtual space according to the positioning data of the entity objects in the reference coordinate system, and generates a virtual dental chair of the treatment table at the corresponding position in the virtual space according to the positioning data of the dental chair of the treatment table in the reference coordinate system. And in response to the operation of the user, preset virtual facilities are generated in the virtual space, and then the coordinate and pose data of the virtual space, the virtual dental chair of the treatment table, and the virtual facilities are obtained, and a scene file for oral treatment is generated according to the coordinate and pose data.

[0157] In the above embodiment of the present application, the virtual-real fusion head-mounted viewer scans the positioning pictures with significant features at the specified positions in the real space, determines the reference coordinate system and the reference point, and obtains the pose information of the reference point in the current device coordinate system. Then, according to the pose information of the reference point in the current device coordinate system, the mapping relationship between the reference coordinate system and the device coordinate system is established. The virtual-real fusion head-mounted viewer measures the positioning data of the entity objects existing in the real space in the reference coordinate system, and scans the positioning pictures with significant features at the position of the positioning reference point of the dental chair of the treatment table in the real space to obtain the pose information of the dental chair of the treatment table in the device coordinate system, and then according to the pose information of the dental chair of the treatment table in the device coordinate system and the mapping relationship, the positioning data of the dental chair of the treatment table in the reference coordinate system is obtained. In this embodiment, by establishing the mapping relationship between the reference coordinate system Coord ref and the device coordinate system Coord dev it is convenient for the virtual-real fusion head-mounted viewer to perform visual display, and the spatial coordinate virtual-real fusion is realized by using the mapping relationship between the two.

[0158] Further, on the basis of the above embodiment, the process of the simulation computer obtaining the matched virtual entity data is described by the following embodiment.

[0159] Figure 3 It is a schematic flowchart of a method for obtaining matched virtual entity data provided by an embodiment of the present application. As Figure 3 shown, the method may include the following steps:

[0160] S301. Determine the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair of the treatment table.

[0161] A position is pre-selected as the reference point O real in the real empty room R ref , and the orientations of the respective coordinate axes are specified to form the reference coordinate system Coord ref ,

[0162] According toFigure 2 In the method of the illustrated embodiment, a mapping relationship between the reference coordinate system and the device coordinate system is established, that is, a transformation matrix M is constructed.

[0163] For any dental chair C in the real space i , its world transformation matrix M is obtained Ci , and C is included in this matrix i In the reference coordinate system Coord ref The pose information.

[0164] Select the center point of the connection line of the two-handed force feedback ink cartridge fixed on the backrest of the dental chair C i On the panel as the reference point of the force tactile workspace, denoted as O Fi , O Fi The pose in the local coordinate system of the dental chair is T L-Ofi , and the pose in the reference coordinate system Coord ref Is T W-Ofi =M Ci ·T L-Ofi , denoted as M W-Ofi . Taking O Fi As the origin, a force sense coordinate system Coord Ofi Is established, and the inverse of M W-Ofi Is obtained to get Coord ref To Coord Ofi The transformation matrix M Ofi -1 .

[0165] S302 Obtain the pose information of the head model of the virtual patient in the force sense coordinate system.

[0166] When the simulation computer constructs a virtual patient with force feedback interaction ability, it records the local pose information T L-HB Of the coordinate origin of its force tactile head model relative to the animation bone Head Bone.

[0167] When the real user sends a control command through the microphone and the simulation computer controls the virtual patient to lie on the dental chair C i , the simulation computer obtains the pose information T ref Of Head Bone in the reference coordinate system Coord W-HB , calculates the pose information T ref Of the force sense head model in Coord H-Ref =T W-HB ·T L-HB , transforms it to the force sense coordinate system Coord Ofi , and obtains the pose information T H-Ofi =MOfi -1 ·T H-Ref 。

[0168] S303. Perform a diagnosis and treatment operation based on the pose information of the two-handed force feedback tool in the force perception coordinate system based on the head model, and obtain force perception information.

[0169] The simulation computer transmits the pose information T of the force perception head model H-Ofi to the force feedback module. The real user uses the two-handed force feedback tool to perform a diagnosis and treatment operation, driving the force feedback module to perform force perception calculation to obtain force perception information. The force perception information includes the pose information of the force perception tool, as well as the magnitude and direction of the force, and feeds the calculated force perception information back to the real user via the two-handed force feedback tool.

[0170] S304. Obtain the pose information of the virtual diagnosis and treatment tool in the force perception coordinate system.

[0171] Translate the handles of the two-handed force feedback tool (the tools of the left and right force feedback devices) from the ink cartridge to O respectively Fi to obtain the pose information of their respective handles in their device coordinate systems, which are respectively denoted as T L 、T R , invert T L 、T R respectively to obtain the transformation matrix T L -1 、T R -1 , which is the transformation matrix from their respective force feedback device coordinate systems to the force perception coordinate system Coord Ofi below.

[0172] S305. In the force perception coordinate system, match the pose information of the force perception tool and the virtual diagnosis and treatment tool in the force perception coordinate system with the pose information of the two-handed force feedback tool, and transform it to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching. The simulated entity data after matching is used for the virtual-real fusion head-mounted viewer to generate a virtual-real fusion diagnosis and treatment image with a better matching effect.

[0173] Use the transformation matrix T L -1 、T R -1 to transform the diagnosis and treatment tools of the two-handed force feedback to the force perception coordinate system Coord Ofi respectively to obtain the pose information P Ti-Ofi in the force perception coordinate system, where i can be L or R.

[0174] Transform the pose information P Ti-Ofi to the reference coordinate system Coord ref below to obtain the pose information P in the reference coordinate systemTi-Ref = T W-Ofi ·P Ti-Ofi 。

[0175] Use the transformation matrix M to transform P Ti-Ref to the device coordinate system of the virtual-real fusion head-mounted viewer device, obtaining the virtual entity data in the device coordinate system. The virtual-real fusion head-mounted viewer obtains the scene file, and based on the scene file and the virtual entity data, obtains the virtual image in the virtual-real fusion space, and obtains the real-time image in the captured real space. The virtual image in the virtual-real fusion space is superimposed on the real-time image, and finally the virtual-real fusion diagnosis and treatment image is obtained, thereby realizing the triple matching of the force sense of the diagnosis and treatment tool and the virtual diagnosis and treatment tool and the two-handed force feedback tool in the fusion space.

[0176] As Figure 4 shown, Figure 4 is a schematic diagram after virtual-real matching provided by an embodiment of the present application. In Figure 4 , where 401 is a virtual doctor, 402 is a physical dental chair, 403 is the handle of the right-handed force feedback device, 404 is the right-handed virtual diagnosis and treatment tool, 405 is the handle of the left-handed force feedback device, and 406 is the left-handed virtual diagnosis and treatment tool. It can be seen from Figure 4 that the force feedback physical handle and the virtual diagnosis and treatment tool are coincidentally matched in the virtual-real fusion space. In terms of actual force touch, the virtual diagnosis and treatment tool and the force sense tool model are also matched. Among them, the real user perceives the force through the two-handed force feedback device based on the force sense tool model.

[0177] In the above embodiment of the present application, by determining the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair on the diagnosis and treatment table, obtaining the pose information of the head model of the virtual doctor in the force sense coordinate system, and performing diagnosis and treatment operations according to the two-handed force feedback tool based on the pose information of the head model in the force sense coordinate system, the force sense information is obtained. And obtaining the pose information of the virtual diagnosis and treatment tool in the force sense coordinate system. In the force sense coordinate system, the pose information of the force sense tool and the virtual diagnosis and treatment tool in the force sense coordinate system is matched with the pose information of the two-handed force feedback tool, and is converted to the reference coordinate system to obtain the virtual entity data in the reference coordinate system after matching. This embodiment realizes the matching of the force sense space, the virtual space, and the real space by matching the force sense tool and the virtual diagnosis and treatment tool with the two-handed force feedback tool, making the virtual-real fusion diagnosis and treatment image generated in the virtual-real fusion head-mounted viewer more accurate.

[0178] Next, through the following embodiments, the more detailed structural composition of the oral diagnosis and treatment table training system based on the spatial computing technology of the present application will be described.

[0179] In this embodiment, in addition to the simulation computer 011, the virtual-real fusion head-mounted viewer 012, and the two-handed force feedback tool 013, the examination table 01 further includes: a touch display 014, a dental chair height sensor 015, a pitch sensor 016, a dental chair height / pitch controller panel 017, a speaker 018, a microphone 019, and a retractable headrest assembly 0110.

[0180] As Figure 5 shown, Figure 5 FIG. is a schematic diagram of an oral examination table 01 training system provided by an embodiment of the present application based on spatial computing technology:

[0181] Among them, the dental chair of the examination table 01 includes functions of lifting and backrest pitching. The side is designed as a housing that can internally accommodate the main unit of the examination table 01. The mechanical structure of the dental chair can stably support the entire examination table 01. Two connecting drive rods are used to control the lifting of the dental chair and the pitching of the backrest respectively. Each drive rod is connected by a motor and is controlled by the lift / pitch controller panel 017.

[0182] The two-handed force feedback tool 013 is composed of two force feedback devices, which are fixed on the backrest of the dental chair and are symmetrically arranged along the longitudinal central axis of the dental chair, leaving enough space to prevent physical interference between the devices, and can be lifted and pitched synchronously with the dental chair.

[0183] The touch display 014 connects the display to the main chassis housing in the form of a 6-axis connector, meeting the requirements of any height, left-right angle, and pitch angle of the display for users.

[0184] The dental chair height sensor 015 and the pitch sensor 016 use inclination sensors to measure the real-time angle between the support rod of the dental chair and the horizontal plane, and the height sensor 015 is fixed on the support rod of the main bracket, and the pitch sensor 016 is fixed on the backrest.

[0185] The dental chair height / pitch controller panel is connected to the dental chair main unit in the form of a 2-axis connector, which can provide the left-right overall deflection of the controller panel 017 and the connecting rod, as well as the rotation of the panel itself, facilitating user operation.

[0186] The speaker 018 and the microphone 019 select mature speaker and microphone industrial devices, design corresponding installation positions, install the speaker 018 inside the backrest of the dental chair, and install the microphone 019 in the middle of the two-handed force feedback and open a hole in the panel so that it can receive sound.

[0187] The retractable headrest assembly 0110 can be adjusted according to virtual patients of different heights, so that the head of the virtual patient is just in the appropriate position of the headrest, providing support during actual user operation.

[0188] To facilitate the understanding of the above oral treatment table training system based on spatial computing technology in this application, the following takes a specific example to illustrate its application process. Figure 6 The following is a schematic diagram of the application of an oral treatment table training system based on spatial computing technology provided by an embodiment of this application, as Figure 6 shown: It includes a preparation stage and a training stage.

[0189] In the preparation stage ( Figure 6 shown by the dashed line):

[0190] Construct a virtual-real fusion space, that is, a virtual-real fusion treatment room:

[0191] In the constructed virtual-real fusion treatment room, it includes virtual objects corresponding to the entity objects existing in the real space, a virtual treatment table dental chair, virtual medical cabinets, virtual queuing screens and other virtual objects.

[0192] Construct a virtual oral operation environment:

[0193] Based on oral data such as CBCT scans and true-color scan data, establish a virtual oral model, based on the measurement data of treatment tools, establish virtual treatment tools, and according to the virtual oral model and virtual treatment tools, establish a virtual oral operation environment.

[0194] Construct a virtual patient:

[0195] Using a digital holographic real-time three-dimensional model with voice recognition interaction, define a behavior-driven model based on instructions. Through this behavior-driven model and a knowledge graph model, establish a diagnosis and treatment information set of the virtual patient, and fuse the diagnosis and treatment information set with the virtual oral operation environment to generate a virtual patient. This virtual patient can receive control instructions from the simulation computer to complete behaviors such as walking, standing, getting on / off the dental chair, and expression control.

[0196] Establish a communication network:

[0197] Enable communication among multiple treatment tables, that is, multiple simulation computers, virtual-real fusion head-mounted viewers and a situation observation server. Among them, the simulation computers can be connected to the situation observation server in a wired manner, and the virtual-real fusion head-mounted viewers can be connected to the situation observation server in a wireless manner, respectively establishing TCP / IP communication links.

[0198] In the simulation training stage ( Figure 6 shown by the solid line):

[0199] Real users conduct simulation training based on dual-hand force feedback tools:

[0200] The simulation computer loads the scenario file distributed by the situation observation server to generate a virtual-real fusion space. The real user performs a queuing operation through the operation interface UI of the touch display to generate a virtual patient, and conducts a medical interview with the generated virtual patient in the form of voice through a microphone, and receives the interactive voice output by the virtual patient through a speaker.

[0201] The real user uses the two-handed force feedback tool as an operation medium to perform medical treatment operations in the virtual-real fusion space. The simulation computer collects the position data of the two-handed force feedback tool in real time to calculate the force sense information, and feeds the calculated force sense information back to the operator, that is, the real user, through the two-handed force feedback tool, so that the operator can feel the force sense of the virtual scene. Among them, the force sense information includes the pose information, the magnitude and direction of the force of the force sense tool.

[0202] The simulation computer performs triple matching of the pose of the force sense tool and the virtual medical treatment tool with the two-handed force feedback tool:

[0203] The simulation computer performs pose matching of the force sense tool and the virtual medical treatment tool with the two-handed force feedback tool based on a preset matching algorithm to ensure that the poses of the virtual medical treatment tools in terms of force sense and vision always coincide with the two-handed force feedback tool.

[0204] After adjusting the lifting and pitching states of the dental chair, the virtual patient rises and falls and pitches synchronously with the dental chair. When the simulation computer obtains a change in the lifting height of the dental chair measured by the height sensor or the rotation angle of the dental chair measured by the pitching sensor, based on the above matching algorithm, it controls the force sense tool and the virtual medical treatment tool to be re-matched with the pose of the real medical treatment tool. At this time, a new pose T Fi in the local coordinate system of the dental chair is obtained, and M L-Ofi is updated, and M W-Ofi and M Ofi -1 are updated, as well as the pose information of the virtual medical treatment tool in the virtual-real fusion head-mounted viewer, and triple matching is realized again to obtain new simulation entity data.

[0205] Network synchronization of simulation entity data:

[0206] After the simulation server obtains the simulation entity data matched by the simulation computer, it can process or forward the simulation entity data according to corresponding strategies. Each simulation entity data will carry a globally unique identifier Id Global and other associated information, such as entity type, entity pose, simulation grid deformation data, and animation data, etc.

[0207] The real user can observe the medical treatment process through the virtual-real fusion head-mounted viewer:

[0208] The virtual-reality fusion head-mounted viewer scans the Marker map to determine the reference coordinate system and reference points, and establishes the mapping relationship between the reference coordinate system and the device coordinate system, that is, the mapping relationship between the virtual space and the real space. It loads the scene file and receives the matched virtual entity data sent by the simulation computer to obtain the virtual image in the virtual-reality fusion space. Finally, after superimposing the real-time image of the real space, the virtual-reality fusion diagnosis and treatment image is obtained.

[0209] The real user can observe the diagnosis and treatment process by holding a tablet computer:

[0210] In this application, the oral diagnosis and treatment table system based on spatial computing technology can also be connected to at least one external handheld terminal so that the external handheld terminal can display the virtual-reality fusion diagnosis and treatment image. Among them, the external handheld terminal has the augmented reality function and is connected to the simulation computer wirelessly. The external handheld terminal takes the handheld tablet computer as an example. The handheld tablet computer scans the Marker map to determine the reference coordinate system and reference points, and establishes the mapping relationship between the reference coordinate system and the device coordinate system, that is, the mapping relationship between the virtual space and the real space. It loads the scene file and receives the matched virtual entity data sent by the simulation computer to obtain the virtual image in the virtual-reality fusion space. Finally, after superimposing the real-time image of the real space, the virtual-reality fusion diagnosis and treatment image is obtained.

[0211] In this application, there is no limit to the number of external handheld terminals that can be connected, and there can be one or more.

[0212] The diagnosis and treatment process ends:

[0213] After the diagnosis and treatment process ends, the simulation computer can also control the virtual doctor to leave the diagnosis and treatment table and give a systematic evaluation of the real user's diagnosis and treatment operations in the virtual-reality fusion space.

[0214] In this application, multiple diagnosis and treatment tables are connected to the situation observation server, as Figure 7 shown, Figure 7 which is a schematic diagram of a scene provided by an embodiment of this application. In Figure 7 :

[0215] Suppose there are 4 examination tables in the real diagnosis and treatment space. The four examination tables are connected to the situation observation server 02 through a local wired network 03. Among them, the situation observation server 02 is equipped with a large display screen. 101 is the No. 1 examination table, 102 is the virtual viewer of the No. 1 examination table, and 103 is the real user of the No. 1 examination table. Similarly, 201 is the No. 2 examination table, 202 is the virtual viewer of the No. 2 examination table, and 203 is the real user of the No. 2 examination table. 301 is the No. 3 examination table, 302 is the virtual viewer of the No. 3 examination table, 303 is the real user of the No. 3 examination table, and 304 is another real user of this examination table. In the No. 4 examination table, in addition to 401, 402, and 403 being the No. 4 examination table, virtual viewer, and real user, it also includes 404. 404 is a teacher who can observe the diagnosis and treatment process of the No. 4 examination table through a handheld tablet computer 405.

[0216] In this application, through the situation observation server, real users can observe the diagnosis and treatment process of any examination table. By controlling the "camera" function in the large display screen of the situation observation server, the "camera" is controlled to move to the preset position of the corresponding examination table for viewing.

[0217] The oral examination table training system based on spatial computing technology of this application improves the simulation authenticity by generating a virtual-real fusion diagnosis and treatment scene. And this system can support multiple people to watch the diagnosis and treatment process of the same examination table, which is convenient for users to communicate and improves the user experience.

[0218] This application also provides an oral diagnosis and treatment training method based on spatial computing technology, which is applied to an oral diagnosis and treatment training system based on spatial computing technology, as Figure 8 shown Figure 8 is a schematic flowchart of an oral diagnosis and treatment training method based on spatial computing technology provided by an embodiment of this application. The method includes:

[0219] S801. Obtain the positioning data of the entity objects existing in the real space and the dental chair of the examination table based on spatial computing technology. The entity objects do not include the dental chair of the examination table.

[0220] A possible implementation is:

[0221] Scan the positioning picture with significant features at a specified position in the real space, determine the reference coordinate system and the reference point, and obtain the pose information of the reference point in the current device coordinate system. According to the pose information of the reference point in the current device coordinate system, establish the mapping relationship between the reference coordinate system and the device coordinate system. And measure the positioning data of the entity object existing in the real space in the reference coordinate system, scan the positioning picture with significant features at the position of the positioning reference point of the dental chair on the treatment table in the real space, so as to obtain the pose information of the dental chair on the treatment table in the device coordinate system, and then according to the pose information of the dental chair on the treatment table in the device coordinate system and the mapping relationship, obtain the positioning data of the dental chair on the treatment table in the reference coordinate system.

[0222] S802. Generate a scene file for oral diagnosis and treatment according to the positioning data.

[0223] A possible implementation method is:

[0224] Generate a virtual space according to the positioning data of the entity object in the reference coordinate system, generate a virtual dental chair on the treatment table at the corresponding position in the virtual space according to the positioning data of the dental chair on the treatment table in the reference coordinate system, and generate preset virtual facilities in the virtual space in response to the user's operation. Obtain the coordinate and pose data of the virtual space, the virtual dental chair on the treatment table and the virtual facilities, and generate a scene file for oral diagnosis and treatment according to the coordinate and pose data.

[0225] S803. Load the scene file, and generate a virtual-real fusion space according to the scene file. In the virtual-real fusion space, load the virtual patient to be diagnosed and treated and the virtual treatment tools corresponding to the real treatment tools that are created in advance, and record the simulation entity data of the virtual patient and the virtual treatment tools.

[0226] S804. Generate a virtual diagnosis and treatment scene according to the scene file and the simulation entity data, and generate force sense information according to the real user's diagnosis and treatment operation by operating the two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Based on the matching algorithm, control the force sense tool and the virtual treatment tools to match the pose of the two-handed force feedback tool, and obtain the matched simulation entity data.

[0227] A possible implementation method is:

[0228] According to the preset position of the dental chair on the examination table, determine the reference point of the force sense coordinate system and the force sense coordinate system, and obtain the pose information of the head model of the virtual examiner in the force sense coordinate system. Based on the pose information of the head model in the force sense coordinate system, perform a diagnosis and treatment operation using the dual-hand force feedback tool to generate force sense information, where the force sense information includes the pose information of the force sense tool. Obtain the pose information of the virtual diagnosis and treatment tool in the force sense coordinate system. In the force sense coordinate system, match the pose information of the force sense tool and the virtual diagnosis and treatment tool in the force sense coordinate system with the pose information of the dual-hand force feedback tool, and convert it to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching.

[0229] S805. Generate a first virtual-real fusion diagnosis and treatment image based on the simulated entity data after matching. The first virtual-real fusion diagnosis and treatment image is used for the real user to perform a diagnosis and treatment operation.

[0230] A possible implementation method is:

[0231] Obtain the mapping relationship between the reference coordinate system and the device coordinate system. According to this mapping relationship, transfer the simulated entity data in the reference coordinate system after matching to the device coordinate system to obtain the simulated entity data in the device coordinate system. Obtain the scene file. According to the scene file and the simulated entity data in the device coordinate system, obtain the virtual image in the virtual-real fusion space, and obtain the real-time image in the captured real space. Superimpose the virtual image in the virtual-real fusion space and the real-time image to finally obtain the first virtual-real fusion diagnosis and treatment image.

[0232] S806. Obtain the simulated entity data corresponding to any examination table.

[0233] S807. Generate a second virtual-real fusion diagnosis and treatment image based on the forwarded simulated entity data of any examination table.

[0234] Among them, in both the first virtual-real fusion diagnosis and treatment image and the second virtual-real fusion diagnosis and treatment image, the process of the virtual examiner closely adhering to the dental chair on the examination table in the real space to receive a diagnosis and treatment operation is displayed.

[0235] In the above method of the present application, for the specific implementation process and technical effects, please refer to the above embodiments. To avoid redundancy, no further description will be repeated.

[0236] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0237] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above method is implemented.

[0238] The unit described as a separating component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, in each embodiment of the present invention, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0240] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs and other various media that can store program codes.

[0241] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When this program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: ROMs, RAMs, magnetic disks, or optical discs and other various media that can store program codes.

[0242] Finally, it should be noted that: after considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other implementation schemes of the present invention. The present invention is intended to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An oral treatment table training system based on spatial computing technology, characterized in that, Including: A plurality of diagnosis and treatment tables and a situation observation server. The plurality of diagnosis and treatment tables communicate with the one situation observation server through the TCP / IP protocol. Each diagnosis and treatment table includes: a simulation computer, a virtual-real fusion head-mounted viewer, and a two-handed force feedback tool; The virtual-real fusion head-mounted viewer: is used to obtain the positioning data of the entity objects existing in the real space and the dental chair of the diagnosis and treatment table based on spatial computing technology. The entity objects do not include the dental chair of the diagnosis and treatment table; The situation observation server: is used to generate a scene file for oral diagnosis and treatment according to the positioning data; The simulation computer: is used to load the scene file and generate a virtual-real fusion space according to the scene file. In the virtual-real fusion space, a pre-created virtual patient to be diagnosed and a virtual diagnosis and treatment tool corresponding to the real diagnosis and treatment tool are loaded, and the simulation entity data of the virtual patient and the virtual diagnosis and treatment tool are recorded; The simulation computer: is also used to generate a virtual diagnosis and treatment scene according to the scene file and the simulation entity data, and generate force sense information according to the diagnosis and treatment operations performed by the real user by operating the two-handed force feedback tool. The force sense information includes the pose information of the force sense tool. Based on a matching algorithm, the force sense tool and the virtual diagnosis and treatment tool are controlled to match the pose of the two-handed force feedback tool to obtain the matched simulation entity data; The virtual-real fusion head-mounted viewer: is used to generate a first virtual-real fusion diagnosis and treatment image according to the matched simulation entity data. The first virtual-real fusion diagnosis and treatment image is used for the real user to perform diagnosis and treatment operations; The situation observation server: is also used to obtain the simulation entity data corresponding to any diagnosis and treatment table; The virtual-real fusion head-mounted viewer: is also used to generate a second virtual-real fusion diagnosis and treatment image according to the simulation entity data of any diagnosis and treatment table forwarded by the situation observation server; Wherein, in the first virtual-real fusion diagnosis and treatment image and the second virtual-real fusion diagnosis and treatment image, the process of the virtual patient closely adhering to the dental chair of the diagnosis and treatment table in the real space to receive diagnosis and treatment operations is displayed.

2. The system according to claim 1, wherein When the virtual-real fusion head-mounted viewer is used to obtain the positioning data of the entity objects existing in the real space and the dental chair of the diagnosis and treatment table based on spatial computing technology, it is specifically used for: Scanning the positioning pictures with significant features at the specified positions in the real space to determine the reference coordinate system and the reference point; Obtaining the pose information of the reference point in the current device coordinate system; Establishing the mapping relationship between the reference coordinate system and the device coordinate system according to the pose information of the reference point in the current device coordinate system; Measuring the positioning data of the entity objects existing in the real space in the reference coordinate system; Scanning the positioning pictures with significant features at the position of the positioning reference point of the dental chair of the diagnosis and treatment table in the real space to obtain the pose information of the dental chair of the diagnosis and treatment table in the device coordinate system; Obtaining the positioning data of the dental chair of the diagnosis and treatment table in the reference coordinate system according to the pose information of the dental chair of the diagnosis and treatment table in the device coordinate system and the mapping relationship.

3. The system according to claim 2, wherein When the situation observation server is used to generate a scene file for oral diagnosis and treatment according to the positioning data, it is specifically used for: Generating a virtual space according to the positioning data of the entity object in the reference coordinate system; Generating a virtual dental chair of the examination table at the corresponding position in the virtual space according to the positioning data of the dental chair of the examination table in the reference coordinate system; Generating preset virtual facilities in the virtual space in response to the operation of the user; Obtaining the coordinate and pose data of the virtual space, the virtual dental chair of the examination table, and the virtual facilities, and generating a scene file for oral diagnosis and treatment according to the coordinate and pose data.

4. The system according to claim 3, wherein When the simulation computer is used to generate force sense information according to the diagnosis and treatment operations of a real user by operating the two-handed force feedback tool, and the force sense information includes the pose information of the force sense tool, and based on a matching algorithm, controlling the force sense tool and the virtual diagnosis and treatment tool to match the pose of the two-handed force feedback tool to obtain the simulated entity data after matching, it is specifically used for: Determining the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair of the examination table; Obtaining the pose information of the head model of the virtual patient in the force sense coordinate system; Generating force sense information according to the diagnosis and treatment operations of the two-handed force feedback tool based on the pose information of the head model in the force sense coordinate system, and the force sense information includes the pose information of the force sense tool; Obtaining the pose information of the virtual diagnosis and treatment tool in the force sense coordinate system; In the force sense coordinate system, matching the pose information of the force sense tool and the virtual diagnosis and treatment tool in the force sense coordinate system with the pose information of the two-handed force feedback tool, and converting it to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching.

5. The system according to claim 4, characterized in that, When the virtual-real fusion head-mounted viewer is used to generate the first virtual-real fusion diagnosis and treatment image according to the simulated entity data after matching, it is specifically used for: Obtaining the mapping relationship between the reference coordinate system and the device coordinate system; According to the mapping relationship, transferring the simulated entity data in the reference coordinate system after matching to the device coordinate system to obtain the simulated entity data in the device coordinate system; Obtaining the scene file, and obtaining the virtual image in the virtual-real fusion space according to the scene file and the simulated entity data in the device coordinate system; Obtaining the real-time image in the captured real space; Performing superposition processing on the virtual image in the virtual-real fusion space and the real-time image to obtain the first virtual-real fusion diagnosis and treatment image.

6. The system according to claim 1, characterized in that, The situation observation server is also used for: Displaying the virtual diagnosis and treatment scenes corresponding to different examination tables.

7. The system according to claim 1, wherein The situation observation server is also used for: Setting the call mode for the virtual patient, the call mode includes the ordinary call mode and the enhanced call mode, the virtual patient corresponding to the ordinary call mode has random causes, and the virtual patient corresponding to the enhanced call mode has preset fixed causes; Editing the virtual patient queue corresponding to the call mode.

8. The system according to claim 1, characterized in that The examination table further includes: a speaker and a microphone; After the simulation computer loads a pre-created virtual patient to be diagnosed and treated in the virtual-real fusion space, the real user sends control instructions to the virtual patient through the microphone and receives interactive voices output by the virtual patient through the speaker.

9. The system according to claim 1, wherein The dental treatment table further includes: a height sensor and a pitch sensor; When the simulation computer detects a change in the lifting height of the dental chair measured by the height sensor or the rotation angle of the dental chair measured by the pitch sensor, the simulation computer, based on the matching algorithm, controls the haptic tool and the virtual treatment tool to be re-matched with the pose of the two-handed force feedback tool, and obtains new matched virtual entity data.

10. The system according to any one of claims 1-9, characterized in that, The oral treatment table training system based on spatial computing technology is connected to at least one external handheld terminal, so that the at least one external handheld terminal displays virtual-real fusion diagnosis and treatment images, and the at least one external terminal has an augmented reality function.

11. An oral diagnosis and treatment training method based on spatial computing technology, characterized in that, Applied to an oral treatment training system based on spatial computing technology, it includes: Obtaining the positioning data of the entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology, where the entity objects do not include the dental chair of the treatment table; Generating a scene file for oral treatment according to the positioning data; Loading the scene file and generating a virtual-real fusion space according to the scene file. In the virtual-real fusion space, load a pre-created virtual patient to be diagnosed and treated and virtual treatment tools corresponding to real treatment tools, and record the virtual entity data of the virtual patient and the virtual treatment tools; Generating a virtual treatment scene according to the scene file and the virtual entity data, and generating haptic information according to the treatment operations performed by the real user by operating the two-handed force feedback tool. The haptic information includes the pose information of the haptic tool. Based on the matching algorithm, control the haptic tool and the virtual treatment tool to be matched with the pose of the two-handed force feedback tool, and obtain the matched virtual entity data; Generating a first virtual-real fusion diagnosis and treatment image according to the matched virtual entity data, and the first virtual-real fusion diagnosis and treatment image is used for the real user to perform treatment operations; Obtaining the virtual entity data corresponding to any treatment table; Generating a second virtual-real fusion diagnosis and treatment image according to the forwarded virtual entity data of any treatment table; Wherein, in the first virtual-real fusion diagnosis and treatment image and the second virtual-real fusion diagnosis and treatment image, the process of a virtual patient closely adhering to the dental chair of the treatment table in the real space to receive treatment operations is displayed.

12. The method according to claim 11, wherein The obtaining the positioning data of the entity objects existing in the real space and the dental chair of the treatment table based on spatial computing technology includes: Scanning the positioning pictures with significant features at specified positions in the real space to determine the reference coordinate system and the reference point; Obtaining the pose information of the reference point in the current device coordinate system; Establishing the mapping relationship between the reference coordinate system and the device coordinate system according to the pose information of the reference point in the current device coordinate system; Measuring the positioning data of the entity objects existing in the real space in the reference coordinate system; Scan the positioning picture with significant features at the positioning reference point of the dental chair on the treatment table in the real space to obtain the pose information of the dental chair on the treatment table in the device coordinate system; According to the pose information of the dental chair on the treatment table in the device coordinate system and the mapping relationship, obtain the positioning data of the dental chair on the treatment table in the reference coordinate system.

13. The method according to claim 12, wherein Generating a scene file for oral diagnosis and treatment according to the positioning data includes: Generate a virtual space according to the positioning data of the entity object in the reference coordinate system; Generate a virtual dental chair on the treatment table at the corresponding position in the virtual space according to the positioning data of the dental chair on the treatment table in the reference coordinate system; In response to the user's operation, generate a preset virtual facility in the virtual space; Obtain the coordinate and pose data of the virtual space, the virtual dental chair on the treatment table, and the virtual facility, and generate a scene file for oral diagnosis and treatment according to the coordinate and pose data.

14. The method according to claim 13, wherein Generating force sense information according to the real user's diagnosis and treatment operation by operating the two-handed force feedback tool, where the force sense information includes the pose information of the force sense tool. Based on the matching algorithm, control the force sense tool and the virtual diagnosis tool to match the pose of the two-handed force feedback tool, and obtain the simulated entity data after matching, including: Determine the reference point and the force sense coordinate system of the force sense coordinate system according to the preset position of the dental chair on the treatment table; Obtain the pose information of the head model of the virtual viewer in the force sense coordinate system; Generate force sense information according to the two-handed force feedback tool's diagnosis and treatment operation based on the pose information of the head model in the force sense coordinate system, where the force sense information includes the pose information of the force sense tool; Obtain the pose information of the virtual diagnosis tool in the force sense coordinate system; In the force sense coordinate system, match the pose information of the force sense tool and the virtual diagnosis tool in the force sense coordinate system with the pose information of the two-handed force feedback tool, and convert it to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching.

15. The method according to claim 14, wherein Generating the first virtual-real fusion diagnosis and treatment image according to the simulated entity data after matching includes: Obtain the mapping relationship between the reference coordinate system and the device coordinate system; According to the mapping relationship, transfer the simulated entity data in the reference coordinate system after matching to the device coordinate system to obtain the simulated entity data in the device coordinate system; Obtain the scene file, and obtain the virtual image in the virtual-real fusion space according to the scene file and the simulated entity data in the device coordinate system; Obtain the real-time image in the captured real space; Overlay the virtual image in the virtual-real fusion space with the real-time image to obtain the first virtual-real fusion diagnosis and treatment image.

Images