Information sharing method, system, device, equipment, storage medium and program product

Abstract

The invention relates to an information sharing method, system, device and equipment, a storage medium and a program product, and the method is applied to a first mixed reality device, and comprises the steps: obtaining a to-be-displayed operation planning model; in response to a first operation on the surgical planning model, transmitting a first data stream of the operation process to a second mixed reality device; the second mixed reality device and the first mixed reality device are mutually registered, and the second mixed reality device is used for displaying an operation process corresponding to the first data stream; and transmitting the first data stream to a third-party display device, wherein the third-party display device is used for displaying an operation process corresponding to the first data stream. By using the information sharing method provided by the invention, the family members of the patient can accurately know the operation process.

Description

Technical Field

[0001] The present application relates to the field of information transmission technology, and in particular to an information sharing method, system, device, equipment, storage medium and program product. Background Art

[0002] During the medical treatment, the patient's family members do not have a clear understanding of the condition, surgical process and surgical results, that is, the information asymmetry between doctors and patients' families, which can easily lead to medical disputes. In some surgical procedures, before the operation, the patient's family members need to be explained the surgical process and matters that need to be paid attention to after the operation.

[0003] In traditional technology, doctors usually explain the operation to the patient's family through text and pictures of the patient. However, in this way, the patient's family cannot accurately understand the operation process. Summary of the invention

[0004] Based on this, it is necessary to provide an information sharing method, system, device, equipment, storage medium and program product that can enable the patient's family to accurately understand the surgical process in response to the above technical problems.

[0005] In a first aspect, the present application provides an information sharing method, which is applied to a first mixed reality device, and the method includes:

[0006] obtaining a surgical planning model to be presented;

[0007] In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream;

[0008] The first data stream is transmitted to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0009] In one embodiment, the method further comprises:

[0010] A registration notification is sent to the second mixed reality device, and a preset registration algorithm is used to perform a registration operation; the second mixed reality device is used to perform a registration operation based on the registration notification using the preset registration algorithm; the registration notification includes registration parameters.

[0011] In one embodiment, obtaining a surgical planning model to be displayed includes:

[0012] determining whether the current storage space includes a surgical planning model;

[0013] If the current storage space does not include the surgery planning model, the surgery planning model is searched for in the remaining storage space until the surgery planning model is obtained.

[0014] In one embodiment, the first mixed reality device includes a control unit and a mixed reality unit; in response to a first operation on a surgical planning model, transmitting a first data stream of the operation process to a second mixed reality device, comprising:

[0015] In response to a first operation on the surgical planning model, the control unit transmits a first data stream of the operation process to the mixed reality unit for display, and transmits it to the second mixed reality device.

[0016] In a second aspect, an embodiment of the present application provides an information sharing method, which is applied to a second mixed reality device, and the method includes:

[0017] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the first operation on the surgical planning model; the second mixed reality device is mutually registered with the first mixed reality device.

[0018] In one embodiment, the method further comprises:

[0019] Receive a registration notification sent by the first mixed reality device, and use a preset registration algorithm to perform a registration operation; the registration notification includes registration parameters.

[0020] In one embodiment, the method further comprises:

[0021] In response to a second operation on the surgical planning model, a second data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the second data stream.

[0022] In a third aspect, an embodiment of the present application provides an information sharing method, which is applied to a third-party display device, and the method includes:

[0023] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0024] In one embodiment, the method further comprises:

[0025] Obtain video data corresponding to the operation process;

[0026] Correspondingly, the first data stream showing the operation process includes:

[0027] The video data and the first data stream are merged to obtain and display a merged video.

[0028] In one embodiment, the third-party display device includes a third-party control unit; obtaining video data corresponding to the operation process includes:

[0029] Displaying position information of the surgical planning model corresponding to the first data stream on a display interface of the third-party control unit;

[0030] The third-party control unit adjusts the shooting parameters according to the position information, and based on the adjusted shooting parameters, shoots the background of the operation process of the surgical planning model in the reality augmented mode to obtain video data corresponding to the operation process.

[0031] In one embodiment, the third-party display device further includes a panoramic camera to obtain video data corresponding to the operation process, including:

[0032] The panoramic camera is started to shoot the background of the operation process of the surgical planning model to obtain video data corresponding to the operation process.

[0033] In one embodiment, the method further comprises:

[0034] In response to a third operation on the surgical planning model, a third data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the third data stream.

[0035] In a fourth aspect, an embodiment of the present application provides an information sharing system, the information sharing system comprising: a first mixed reality device, a second mixed reality device, and a third-party display device, the first mixed reality device, the second mixed reality device, and the third-party display device being communicatively connected;

[0036] A first mixed reality device, used to execute the steps of the method provided in the first aspect above;

[0037] A second mixed reality device, used to execute the steps of the method provided in the second aspect above;

[0038] A third-party display device is used to execute the steps of the method provided in the third aspect above.

[0039] In a fifth aspect, an embodiment of the present application provides an information sharing device, the device comprising:

[0040] An acquisition module, used for acquiring a surgical planning model to be displayed;

[0041] a response module, configured to respond to a first operation on the surgical planning model and transmit a first data stream of the operation process to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is configured to display the operation process corresponding to the first data stream;

[0042] The transmission module is used to transmit the first data stream to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0043] In a sixth aspect, an embodiment of the present application provides an information sharing device, the device comprising:

[0044] A receiving module is used to receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which the first mixed reality device responds to a user performing a first operation on a surgical planning model; and the second mixed reality device is registered with the first mixed reality device.

[0045] In a seventh aspect, an embodiment of the present application provides an information sharing device, the device comprising:

[0046] The display module is used to receive and display a first data stream of the operation process; the first data stream is the data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0047] In an eighth aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the methods provided in the first, second and third aspects are implemented.

[0048] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the methods provided in the first, second and third aspects above are implemented.

[0049] In the tenth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the methods provided in the first aspect, the second aspect and the third aspect are implemented.

[0050] The above-mentioned information sharing method, system, device, equipment, storage medium and program product, the method obtains the surgical planning model to be displayed; responds to the first operation on the surgical planning model, and transmits the first data stream of the operation process to the second mixed reality device; the second mixed reality device and the first mixed reality device are mutually registered, and the second mixed reality device is used to display the operation process corresponding to the first data stream; the first data stream is transmitted to a third-party display device, and the third-party display device is used to display the operation process corresponding to the first data stream. In this embodiment, the operation process of the first operation on the surgical planning model in the first mixed reality device can be synchronized in real time to the second mixed reality device and the third-party display device, so that the user carrying the second mixed reality device and the third-party display device can obtain the operation process of the first user in the first mixed reality device in real time and accurately, so that the user can accurately understand the first operation performed by the first user on the surgical planning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the structure of an information sharing system in one embodiment;

[0052] Figure 2 A schematic diagram of a step flow of an information sharing method in an embodiment;

[0053] Figure 3 A schematic diagram of an application environment of an information sharing method in an embodiment;

[0054] Figure 4 A schematic diagram of the structure of an information sharing system in another embodiment;

[0055] Figure 5 A schematic diagram of a process flow of an information sharing method in another embodiment;

[0056] Figure 6 A schematic diagram of the structure of an information sharing system in another embodiment;

[0057] Figure 7 A schematic diagram of a process flow of an information sharing method in another embodiment;

[0058] Figure 8 A schematic diagram of a process flow of an information sharing method in another embodiment;

[0059] Fig. 9 A schematic diagram of data displayed by a third-party display device in one embodiment;

[0060] Fig.10 A schematic diagram of a process flow of an information sharing method in another embodiment;

[0061] Fig.11A schematic diagram of a process flow of an information sharing method in another embodiment;

[0062] Fig.12 A schematic diagram of the software structure of each device in an information sharing system in an embodiment;

[0063] Fig.13 A schematic diagram of the structure of an information sharing device in one embodiment;

[0064] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The serial numbers assigned to the components in this application, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0067] First of all, before specifically introducing the technical solutions of the disclosed embodiments of the present application, the background technology or technical evolution context on which the embodiments of the present application are based is introduced. In the current medical environment, during the medical treatment process, the patient's family members do not have a clear understanding of the condition, surgical procedure and surgical results, that is, there is information asymmetry between doctors and patients' families, which easily leads to medical disputes. In some surgical procedures, before the operation, the patient's family members need to be explained the surgical procedure and matters that need to be paid attention to after the operation. In traditional technology, doctors usually explain to the patient's family members through text and pictures of the patient, however, in this way, the patient's family members cannot accurately understand the surgical procedure. In this regard, the present application provides an information sharing method.

[0068] The technical solution of the present application and how the technical solution of the present application solves the technical problem are described in detail below with specific embodiments.

[0069] The information sharing method provided in the embodiment of the present application is applied to an information sharing system. The structure of the information sharing system is as follows: Figure 1As shown. The information sharing system includes a first mixed reality device 11, a second mixed reality device 12 and a third-party display device 13. The first mixed reality device 11 is communicatively connected with the second mixed reality device 12 and the third-party display device 13. That is, the first mixed reality device 11 can transmit data to the second mixed reality device 12 and the third-party display device 13; the second mixed reality device 12 can transmit data to the first mixed reality device 11 and the third-party display device 13; the third-party display device 13 can transmit data to the first mixed reality device 11 and the second mixed reality device 12.

[0070] In an optional embodiment, the first mixed reality device 11, the second mixed reality device 12 and the third-party display device 13 are all provided with a communication unit, through which the first mixed reality device 11, the second mixed reality device 12, and the third-party display device 13 can communicate with each other.

[0071] Please continue to see Figure 1 In another optional embodiment, the information sharing system also includes a server 14, and the first mixed reality device 11, the second mixed reality device 12 and the third-party display device 13 are all connected to the server 14, and the first mixed reality device 11, the second mixed reality device 12, and the third-party display device 13 communicate using the WebSocket protocol through the server 14.

[0072] The first mixed reality device 11 can superimpose virtual images on real images and videos so that the user can see the superimposed images or videos through the first mixed reality device 11. The structures of the first mixed reality device 11 and the second mixed reality device 12 can be the same or different. The third-party display device 13 is used to receive and display data transmitted by the first mixed reality device 11 and the second mixed reality device 12. The first mixed reality device 11 can also receive and display data transmitted by the second mixed reality device 12 and the third-party display device 13. The second mixed reality device 12 can also receive and display data transmitted by the first mixed reality device 11 and the third-party display device 13. This embodiment does not limit the types and specific structures of the first mixed reality device 11, the second mixed reality device 12 and the third-party display device 13, as long as their functions can be achieved.

[0073] In one embodiment, Figure 2 As shown, an information sharing method is provided. This embodiment is illustrated by applying the method to the first mixed reality device in the above information sharing system. In this embodiment, the steps of the method include:

[0074] Step 200: Obtain a surgical planning model to be displayed.

[0075] The surgical planning model refers to a three-dimensional model obtained by processing the image of the tissue and organ that needs surgery using the surgical planning software. The first mixed reality device obtains the surgical planning model to be displayed. This embodiment does not limit the specific method of obtaining the surgical planning model to be displayed, as long as its function can be achieved.

[0076] In an optional embodiment, the surgical planning model may be pre-stored in a storage unit of the first mixed reality device, and the first mixed reality device directly obtains the surgical planning model to be displayed in the storage unit. The storage format of the surgical planning model may be PLY (Polygon File Format) / STL (stereolithograpiy), or ProtoBuf (Google Protocol Buffer).

[0077] In another optional embodiment, when the information sharing system includes a server, the surgical planning model can be stored in the server, and the first mixed reality device obtains the surgical planning model to be displayed from the server. The server can be a PACS (Picture Archiving and Communication Systems) server.

[0078] Step 210: In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream. 。

[0079] The first mixed reality device and the second mixed reality device are pre-registered. After the first mixed reality device and the second mixed reality device are registered with each other, the operation process of the first operation performed by the first user carrying the first mixed reality device in the first mixed reality device can be displayed in real time and synchronously in the second mixed reality device. In other words, the operation process viewed by the second user carrying the second mixed reality device through the second mixed reality device is the same as the operation process of the first operation performed by the first user in the first mixed reality device. This embodiment does not limit the registration process between the first mixed reality device and the second mixed reality device, as long as its function can be achieved.

[0080] After the first mixed reality device acquires the surgical planning model, the first user can view the image after the surgical planning model and the background image are superimposed through the first mixed reality device. On this basis, the first user performs a first operation on the surgical planning model displayed by the first mixed reality device, and the first mixed reality device responds to the first operation and transmits the first data stream corresponding to the operation process to the second mixed reality device. In other words, the data stream of the entire operation process of the first user performing the first operation on the surgical planning model will be transmitted to the second mixed reality device, and the second mixed reality device can display the operation process corresponding to the first data stream, that is, the second user can view the operation process corresponding to the first data stream through the second mixed reality device.

[0081] The first operation may include routine operations and specific operations. The first routine operations include translation, rotation, scaling, and one-key restoration. One-key restoration means that after the first user performs routine operations and / or operations, the surgical planning model can be restored to its initial state through a one-key restoration control, that is, restored to the state before various operations. Specific operations are used to characterize the use of specific tools for operations, and specific operations include using a cropping box to crop the surgical planning model; using a free brush to perform annotation operations in the surgical planning model, such as marking the surgical planning path in the surgical planning model; reconstructing and displaying the surgical planning model in multiple planes; moving or scaling the safety boundary ball to mark the lesion area in the surgical planning model; selecting some organs in the surgical planning model to hide or display; splitting the surgical planning model into multiple areas including the lesion, so as to be able to more intuitively display the relationship between the various areas in the surgical planning model, as well as the route of the surgical planning path.

[0082] Step 220: Transmit the first data stream to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0083] While transmitting the first data stream to the second mixed reality device, the first mixed reality device also transmits the first data stream to the third-party display device. After receiving the first data stream, the third-party display device will display the operation process corresponding to the first data stream. In other words, the third user carrying the third-party display device can view the operation process corresponding to the first data stream.

[0084] In an optional embodiment, the application environment of the information sharing method is as follows: Figure 3 As shown. The first user is a doctor, the second user is a patient family representative, and the third user is other family members. The first mixed reality device and the second mixed reality device are mixed reality glasses (Hololens), and the third-party display device is a smart display. There can be multiple patient family representatives, for example, at least 4 patient family representatives. The structure of the information sharing system is as follows Figure 4As shown. The doctor and the patient's family representative both wear mixed reality glasses. The doctor performs a first operation on the surgical planning model through the first mixed reality device, and explains the surgical process during the first operation. The patient's family representative can use the second mixed reality device, and other family members can use the third-party display device to synchronously see the doctor's operation process on the surgical planning model while listening to the doctor's explanation, so as to understand the doctor's explanation of the surgical process based on the surgical planning model more accurately and realistically.

[0085] The information sharing method provided in the embodiment of the present application is applied to a first mixed reality device, and the method obtains a surgical planning model to be displayed; responds to a first operation on the surgical planning model, and transmits a first data stream of the operation process to a second mixed reality device; the second mixed reality device and the first mixed reality device are mutually aligned, and the second mixed reality device is used to display the operation process corresponding to the first data stream; the first data stream is transmitted to a third-party display device, and the third-party display device is used to display the operation process corresponding to the first data stream. In this embodiment, the operation process of the first operation on the surgical planning model in the first mixed reality device can be synchronized in real time to the second mixed reality device and the third-party display device, so that users carrying the second mixed reality device and the third-party display device can obtain the operation process of the first user in the first mixed reality device in real time and accurately, so that the user can accurately understand the first operation performed by the first user on the surgical planning model.

[0086] In an optional embodiment, the first mixed reality device also receives a first data stream transmission control instruction of the first user for the operation process, and the transmission control instruction includes controlling the first data stream to be transmitted to at least one of the second mixed reality device and the third-party display device. In other words, the first user can control whether to transmit the first data stream to the second mixed reality device and / or the third-party display device through the transmission control instruction, that is, the first user can control whether to synchronize the operation process corresponding to the first operation to the second mixed reality device and / or the third-party display device.

[0087] In one embodiment, when registering the first mixed reality device and the second mixed reality device, the information sharing method further includes:

[0088] A registration notification is sent to the second mixed reality device, and a preset registration algorithm is used to perform a registration operation; the second mixed reality device is used to perform a registration operation based on the registration notification using the preset registration algorithm; the registration notification includes registration parameters.

[0089] When registration is required, the first mixed reality device will send a registration notification to the second mixed reality device, and the first mixed reality device will use a preset registration algorithm to perform the registration operation. The preset registration algorithm can be pre-stored in the first mixed reality device, and the first mixed reality device can directly use it when performing the registration operation.

[0090] The registration parameters in the registration notification are used to inform the second mixed reality device of the markers that need to be registered. After receiving the registration notification sent by the first mixed reality device, the second mixed reality device will use a preset registration algorithm to perform a registration operation based on the registration notification. The registration algorithm in the second mixed reality device is the same as the registration algorithm used by the first mixed reality device. The registration algorithm can be pre-stored in the second mixed reality device, or included in the registration parameters sent from the first mixed reality device to the second mixed reality device.

[0091] In an optional embodiment, the specific registration process includes: the first mixed reality device starts the registration algorithm and detects whether the registration algorithm is accurately ready, that is, determines whether there is a preset registration algorithm in the first mixed reality device. If it exists, it means that the registration algorithm is accurately ready; if it does not exist, it is necessary to re-acquire the registration algorithm. When the registration algorithm is accurately ready, the registration algorithm is used for image recognition. The image includes the markers required for registration, and the position information of the marker is determined by identifying the image. After the image is successfully recognized, an empty node is determined according to the position information of the marker, and the local coordinate system of the empty node is used as the coordinate system when the surgical planning model is displayed. An empty node can be determined at a preset position of the marker (for example, 10 cm directly above the marker).

[0092] Similarly, when the second mixed reality device receives the registration notification, the registration notification may include the registration algorithm, the markers required for registration, and the positional relationship between the empty node and the marker. Based on the registration notification, the second mixed reality device uses the registration algorithm to perform image recognition, determines the position information of the marker by recognizing the image, and determines the coordinate system for displaying the surgical planning model according to the positional relationship between the empty node and the marker. In this way, the surgical planning models in the first mixed reality device and the second mixed reality device are displayed in the same coordinate system, achieving registration.

[0093] In this embodiment, the first mixed reality device directly sends a registration notification to the second mixed reality device to instruct the second mixed reality device to register with the first mixed reality device. In this way, the registration process of the first mixed reality device and the second mixed reality device is fast and easy to implement.

[0094] In one embodiment, when storing the surgical planning model, the storage unit or server of the first mixed reality device divides the storage unit into different storage spaces, and stores surgical planning models of different patients or different types of surgical planning models in different storage spaces. Figure 5 As shown, an implementation method of obtaining a surgical planning model to be displayed is involved, and the steps of the implementation method include:

[0095] Step 500: Determine whether the current storage space includes a surgery planning model.

[0096] The current storage space may refer to the storage space corresponding to the first mixed reality device, that is, the first mixed reality device preferentially obtains the storage space of the surgical planning model. When obtaining the surgical planning model, the first mixed reality device first determines whether the current storage space includes the surgical planning model to be displayed that it needs to obtain.

[0097] If the first mixed reality device determines that the current storage space includes the surgery planning model to be displayed, the surgery planning model is directly acquired.

[0098] Step 510: If the current storage space does not include the surgery planning model, search for the surgery planning model in the remaining storage space until the surgery planning model is obtained.

[0099] If the first mixed reality device determines that the current storage space does not include the surgery planning model to be displayed, it will search for the surgery planning model to be displayed from the remaining storage space until the surgery planning model to be displayed is found and the surgery planning model is acquired.

[0100] In an optional embodiment, the remaining storage space includes a list of surgical planning models, and the first mixed reality device searches the list of surgical planning models for the surgical planning model to be displayed.

[0101] In this embodiment, when the first mixed reality device obtains the surgical planning model to be displayed, it will first determine whether the surgical planning model exists in the current storage space corresponding to it; if so, it will be directly obtained; if not, it will be searched and obtained from the remaining storage space. In this way, it is ensured that the surgical planning model to be displayed can be obtained, making the information sharing method more reliable and practical.

[0102] In one embodiment, the first mixed reality device includes a control unit and a mixed reality unit. The control unit and the mixed reality unit are connected in communication, and the control unit is also connected in communication with the second mixed reality device and the third-party display device. This embodiment does not limit the type and structure of the control unit and the mixed reality unit, as long as their functions can be achieved. The structure of the information sharing system is as follows Figure 6 shown.

[0103] In this case, it is related to an implementation method of responding to a first operation on a surgical planning model and transmitting a first data stream of the operation process to a second mixed reality device, and the implementation method includes:

[0104] In response to a first operation on the surgical planning model, the control unit transmits a first data stream of the operation process to the mixed reality unit for display, and transmits it to the second mixed reality device.

[0105] A surgical planning model is displayed in the control unit, and a first user performs a first operation on the surgical planning model on the control unit. In response to the first operation on the surgical planning model, the control unit transmits a first data stream of an operation process corresponding to the first operation to the mixed reality unit for display. At the same time, the control unit transmits the first data stream of the operation process to a second mixed reality device, so that the second mixed reality device displays the operation process corresponding to the first data stream.

[0106] In an optional embodiment, the control unit may be a tablet computer, a smart phone, a laptop computer, etc., and the mixed reality unit may be mixed reality glasses.

[0107] In this embodiment, the control unit responds to the first operation, that is, the first user can directly perform the first operation on the surgical planning model on the control unit, so that the first user has a physical sense of the operation, which can reduce the difficulty of performing the first operation on the surgical planning model, thereby making the information sharing method more practical.

[0108] In one embodiment, an information sharing method is provided. This embodiment is illustrated by applying the method to a second mixed reality device in the above-mentioned information sharing system. In this embodiment, the method includes:

[0109] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model; the second mixed reality device is aligned with the first mixed reality device.

[0110] After the first user performs a first operation on the surgical planning model based on the first mixed reality device, the first mixed reality device transmits a data stream of the operation process corresponding to the first operation to the second mixed reality device.

[0111] The second mixed reality device receives the first data stream of the operation process corresponding to the first operation transmitted by the first mixed reality device, and displays the operation process corresponding to the first data stream, so that a second user carrying the second mixed reality device can view the operation process corresponding to the first data stream.

[0112] For the description of the mutual registration of the first mixed reality device and the second mixed reality device, and the description of the first operation, reference may be made to the specific description in the above embodiments, which will not be repeated here.

[0113] In this embodiment, the second mixed reality device can receive and display the first data stream of the first mixed reality device in response to the first user performing the first operation on the surgical planning model. In this way, the second user (i.e., the patient's family representative) carrying the second mixed reality device can obtain the operation process of the first user (doctor) in the first mixed reality device in real time and accurately.

[0114] In one embodiment, when the second mixed reality device and the first mixed reality device are registered with each other, the information sharing method further includes:

[0115] Receive a registration notification sent by the first mixed reality device, and use a preset registration algorithm to perform a registration operation; the registration notification includes registration parameters.

[0116] The second mixed reality device receives the registration notification sent by the first mixed reality device, and the registration notification includes registration parameters, and the registration parameters may be the positional relationship between the markers and the empty nodes required for the registration, etc. The second mixed reality device performs a registration operation based on the registration notification using a preset registration algorithm to achieve mutual registration between the second mixed reality device and the first mixed reality device.

[0117] For the description of mutual registration between the first mixed reality device and the second mixed reality device, reference may be made to the specific description in the above embodiments, which will not be repeated here.

[0118] In this embodiment, the second mixed reality device directly performs a registration operation according to the registration notification sent by the first mixed reality device, so that the registration process is quick and easy to implement.

[0119] In one embodiment, the information sharing method further includes:

[0120] In response to a second operation on the surgical planning model, a second data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the second data stream.

[0121] A surgical planning model is displayed in the second mixed reality device. The second user can perform a second operation on the surgical planning model based on the first mixed reality device, and transmit a second data stream of the operation process corresponding to the second operation to the first mixed reality device and the third-party display device.

[0122] After receiving the second data stream, the first mixed reality device will display the operation process corresponding to the second data stream. Similarly, after receiving the second data stream, the third-party display device will display the operation process corresponding to the second data stream.

[0123] The second operation may include a conventional operation and / or a specific operation. The description of the conventional operation and the specific operation can refer to the specific description in the above embodiment, which will not be repeated here.

[0124] In an optional embodiment, the first mixed reality device receives the permission issuing operation input by the first user, and in response to the permission issuing operation, sends a permission opening instruction to the second mixed reality device, and the second mixed reality device responds to the second operation of the second user based on the permission opening instruction. The permission issuing operation includes issuing the permission for regular operations, issuing the permission for specific operations, and issuing the permission for regular operations and specific operations, and the corresponding permission opening instructions include opening instructions for regular operations, opening instructions for specific operations, and opening instructions for regular operations and specific operations at the same time. If the second mixed reality device does not receive the permission opening instruction, it means that the first user has not performed the permission issuing operation, and the second user cannot perform the second operation.

[0125] If the permission operation includes issuing the permission for routine operations, the corresponding permission opening instruction includes the opening instruction for routine operations, and the second mixed reality device can receive the routine operations of the second user on the surgical planning model. In this case, if the second user performs a specific operation on the surgical planning model, the second mixed reality device does not respond, or displays a prompt message to the second user to prompt that the second user has no permission to operate. If the permission operation includes issuing the permission for specific operations, the corresponding permission opening instruction includes the opening instruction for specific operations, and the second mixed reality device can receive the specific operations of the second user on the surgical planning model. In this case, if the second user performs routine operations on the surgical planning model, the second mixed reality device does not respond, or displays a prompt message to the second user to prompt that the second user has no permission to operate. If the permission operation includes issuing the permission for routine operations and specific operations, the corresponding permission opening instruction includes the opening instruction for both routine operations and specific operations, and the second mixed reality device can receive the routine operations and specific operations of the second user on the surgical planning model.

[0126] In an optional embodiment, the second mixed reality device also receives a second data stream transmission control instruction of the second user for the operation process, and the transmission control instruction includes controlling the second data stream to be transmitted to at least one of the first mixed reality device and the third-party display device. In other words, the second user can control whether to transmit the second data stream to the first mixed reality device and / or the third-party display device through the transmission control instruction, that is, the second user can control whether to synchronize the operation process corresponding to the second operation to the first mixed reality device and / or the third-party display device.

[0127] In an optional embodiment, the first user is a doctor, the second user is a patient's family representative, and the third user is other family members. The first mixed reality device and the second mixed reality device are both mixed reality glasses (Hololens), and the third-party display device is a smart display. The doctor and the patient's family representative both wear mixed reality glasses. When the patient's family representative has questions to ask, the second operation is performed on the surgical planning model through the second mixed reality device, and the questions are asked during the second operation. The doctor can use the first mixed reality device, and other family members can use the third-party display device to synchronously see the operation process of the patient's family representative on the surgical planning model while listening to the patient's family representative's questions, so as to more accurately and truly understand the patient's family representative's inquiries about the surgical process with respect to the surgical planning model, thereby facilitating the doctor to make targeted responses to the inquiries of the patient's family representative, thereby improving the communication efficiency and communication information symmetry between the doctor and the patient's family representative.

[0128] In an optional embodiment, the second mixed reality device may also include a control component and a mixed reality component. In response to the second operation on the surgical planning model, the control component transmits a second data stream of the operation process corresponding to the second operation to the mixed reality component and the third-party display device; the mixed reality component is used to display the operation process corresponding to the second data stream; and the third-party display device is used to display the operation process corresponding to the second data stream.

[0129] In this embodiment, the second user carrying the second mixed reality device can perform a second operation on the surgical planning model based on the second mixed reality device. In this way, the second mixed reality device can not only receive the data stream for display, but also be operated by the second user, which makes the information sharing method more practical.

[0130] In one embodiment, an information sharing method is provided. This embodiment is illustrated by applying the method to a third-party display device in the above-mentioned information sharing system. In this embodiment, the method includes:

[0131] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0132] After the first user performs a first operation on the surgical planning model based on the first mixed reality device, the first mixed reality device will transmit a data stream of the operation process corresponding to the first operation to a third-party display device.

[0133] The third-party display device receives the first data stream of the operation process corresponding to the first operation transmitted by the first mixed reality device, and displays the operation process corresponding to the first data stream, so that a third user carrying the third-party display device can view the operation process corresponding to the first data stream.

[0134] In this embodiment, the third-party display device can receive and display the first data stream of the first mixed reality device in response to the first user performing the first operation on the surgical planning model. In this way, the third user (other family members) carrying the third-party display device can obtain the operation process of the first user (doctor) in the first mixed reality device in real time and accurately.

[0135] In one embodiment, when a third-party display device displays the received first data stream, such as Figure 7 As shown, the steps of the information sharing method also include:

[0136] Step 700: Obtain video data corresponding to the operation process.

[0137] The video data corresponding to the operation process may include video data composed of a background image corresponding to the surgical planning model during the process in which the first user performs the first operation on the surgical planning model; or may include video data of the entire process in which the first user and the second user communicate and operate based on the surgical planning model. The specific method of obtaining the video data is different depending on the content of the video data.

[0138] The third-party display device obtains video data corresponding to the operation process.

[0139] In the case where a third-party display device can obtain video data corresponding to the operation process, an implementation method of a first data stream for displaying the operation process includes:

[0140] Step 710: merge the video data and the first data stream to obtain and display a merged video.

[0141] If the first data stream received by the third-party display device corresponds to the operation process, the third-party display device will fuse the first data stream and the video data after acquiring the video data of the operation process to obtain a fused video, and then display the fused video. In other words, the operation process of the first user performing the first operation on the surgical planning model included in the first data stream is superimposed on the acquired video data, so that the fused video includes the video data and the operation process corresponding to the first data stream.

[0142] In an optional embodiment, the third-party display device can determine the position information corresponding to the surgical planning model based on the acquired first data stream, that is, the position information of the surgical planning model and the marker. When the first data stream is fused with the video data, the surgical planning model corresponding to the first data stream can be placed at the same position in the video data as the position information corresponding to the surgical planning model, thereby realizing the fusion of the video data and the first data stream. The third-party display device includes the image information of the marker; the third-party display device performs image recognition on the shooting area to determine whether there is an image that matches the image information of the marker; if so, the corresponding position of the marker in the video data is placed according to the position information corresponding to the surgical planning model.

[0143] In this embodiment, the content displayed by the third-party display device to the third user not only includes the operation process of the first user's first operation on the surgical planning model, but also includes the background corresponding to the operation process (i.e., video data), so that the third user can understand the operation process of the first user on the surgical planning model more realistically and accurately.

[0144] In one embodiment, the third-party display device includes a third-party control unit and a third-party display unit. The third-party control unit is communicatively connected to the third-party display unit, and the third-party control unit is also communicatively connected to the second mixed reality device and the first mixed reality device. The third-party control unit is used to receive data streams transmitted by the first mixed reality device and the second mixed reality device, and transmit the received data streams (first data stream and second data stream) to the third-party display unit. The third-party display unit is used to display the operation process corresponding to the received data stream. This embodiment does not limit the type and structure of the third-party control unit and the third-party display unit, as long as their functions can be achieved.

[0145] In an optional embodiment, the third-party control unit may be a tablet computer, a smart phone, a laptop computer, etc., and the third-party display unit may be a smart display screen.

[0146] In this case, if Figure 8 As shown, an implementation method of obtaining video data corresponding to an operation process is involved, and the steps of the implementation method include:

[0147] Step 800: Display the position information of the surgical planning model corresponding to the first data stream on the display interface of the third-party control unit.

[0148] The third-party control unit includes a shooting component, through which video data can be shot. The third-party control unit also includes a display screen, on which the position information of the surgical planning model corresponding to the first data stream is displayed, that is, the relative position of the surgical planning model corresponding to the first data stream in the shooting component.

[0149] In an optional embodiment, the third-party control unit can determine the coordinate position of the surgical planning model based on the acquired first data stream, and can also obtain the coordinate position of its shooting component. The third-party control unit can determine the relative position of the surgical planning model corresponding to the first data stream in the shooting component based on the coordinate position of the surgical planning model and the coordinate position of the shooting component.

[0150] Specifically, the third-party control unit calculates the position difference, that is, the direction of the surgical planning model corresponding to the shooting component, through the formula Vector3 dir=target.position-transform.position; wherein target.position represents the coordinate position of the shooting component, transform.position represents the coordinate position of the surgical planning model, and Vector3 di represents the position difference. The dot multiplication method is used to determine whether the surgical planning model is in front of or behind the shooting component, float dot1=Vector3.Dot(transform.forward,dir.normalized). If dot1 is greater than zero, it means that the surgical planning model is in front of the shooting component. If dot1 is less than zero, it means that the surgical planning model is behind the shooting component. The dot multiplication method is used to determine whether the surgical planning model is on the right or left side of the shooting component, float dot2=Vector3.Dot(transform.right,dir.normalized). If dot2 is greater than zero, it means that the surgical planning model is on the right side of the shooting component. If dot2 is less than zero, it means that the surgical planning model is on the left side of the shooting component. Determine whether the surgical planning model is above or below the shooting component by dot product method, float dot3 = Vector3.Dot (transform.up, dir.normalized). If dot3 is greater than zero, it means that the surgical planning model is above the shooting component. If dot3 is less than zero, it means that the surgical planning model is below the shooting component. Determine the angle between the surgical planning model and the shooting component by dot product method, float angle = Mathf.Acos (Vector3.Dot (transform.forward.normalized, dir.normalized)) * Math f.Rad2Deg. Calculate the distance between the surgical planning model and the shooting component by float distance = (target.position-transform.position).magnitude.

[0151] Assuming that the surgical planning model is above the shooting component (dot3>0), an upward arrow and numerical prompts (dot3, angle angle, distance distance) are displayed on the display interface; if the surgical planning model is below the shooting component (dot3<0), a downward arrow and numerical prompts (dot3, angle angle, distance distance) are displayed on the display interface; if the surgical planning model is on the left side of the shooting component (dot2<0), a left arrow and numerical prompts (dot2, angle angle, distance distance) are displayed on the display interface; if the surgical planning model is on the right side of the shooting component (dot2>0), a right arrow and numerical prompts (dot2, angle angle, distance distance) are displayed on the display interface.

[0152] Step 810: The third-party control unit adjusts the shooting parameters according to the position information, and based on the adjusted shooting parameters, shoots the background of the operation process of the surgical planning model in the reality augmented mode to obtain video data corresponding to the operation process.

[0153] The third user adjusts the position of the third-party control unit based on the position information displayed on the display interface of the third-party control unit so that the third-party control unit can capture the background of the surgical planning model corresponding to the first data stream. The third-party control unit adjusts the shooting parameters of the shooting component in the third-party control unit in response to the adjustment operation performed by the third user according to the position information. After the shooting parameters of the shooting component in the third-party control unit are adjusted, the background of the operation process of the surgical planning model is shot in the reality enhancement mode to obtain video data corresponding to the operation process. The reality enhancement mode is a working mode of the shooting component of the third-party control unit. The background of the operation process of the surgical planning model may include a video of the space in which the first user (doctor) and the second user (patient's family representative) communicate based on the surgical planning model.

[0154] In this embodiment, the position information of the surgical planning model corresponding to the first data stream can be displayed on the real interface of the third-party control unit, and the third-party control unit adjusts the shooting parameters based on the position information. This enables the third-party control unit to accurately capture the video data corresponding to the operation process. In addition, the third-party control unit is filmed in enhanced mode, which facilitates better fusion of subsequent video data with the first data stream, and the displayed fused video is more realistic and immersive.

[0155] In one embodiment, the third-party display device further includes a panoramic camera, which can be communicatively connected with the third control unit in the third-party display device.

[0156] In this case, another implementation method of obtaining video data corresponding to the operation process is involved, and the implementation method includes:

[0157] The panoramic camera is started to shoot the background of the operation process of the surgical planning model to obtain video data corresponding to the operation process.

[0158] The third-party display device controls the panoramic camera to start, and the panoramic camera shoots the background of the operation process of the surgical planning model to obtain video data corresponding to the operation process.

[0159] When the third-party display device includes a third-party control unit, the third-party control unit sends a start instruction to the panoramic camera to drive the panoramic camera to start. After capturing video data corresponding to the operation process, the panoramic camera transmits the video data to the third-party control unit.

[0160] In this embodiment, a method is proposed to use the panoramic camera in the third-party display device to obtain the video data corresponding to the operation process, so that the information sharing method has higher practicality. The fused video displayed by the third-party display device can be as follows Fig. 9 shown.

[0161] In one embodiment, the information sharing method further includes:

[0162] In response to a third operation on the surgical planning model, a third data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the third data stream.

[0163] A surgical planning model is displayed in the third-party display device. A third user can perform a third operation on the surgical planning model based on the third-party display device, and transmit a third data stream of the operation process corresponding to the third operation to the first mixed reality device and the second mixed reality device.

[0164] After receiving the third data stream, the first mixed reality device will display the operation process corresponding to the third data stream. Similarly, after receiving the third data stream, the second mixed reality device will also display the operation process corresponding to the third data stream.

[0165] The third operation may include a normal operation and / or a specific operation. The description of the normal operation and the specific operation can refer to the specific description in the above embodiment, which will not be repeated here.

[0166] In an optional embodiment, the first mixed reality device receives the permission issuing operation input by the first user, and in response to the permission issuing operation, sends a permission opening instruction to the third-party display device, and the third-party display device responds to the third operation of the third user based on the permission opening instruction. The permission issuing operation includes issuing the permission of regular operation, issuing the permission of specific operation, and issuing the permission of regular operation and specific operation, and the corresponding permission opening instruction includes the opening instruction of regular operation, the opening instruction of specific operation, and the opening instruction of regular operation and specific operation at the same time. If the third-party display device does not receive the permission opening instruction, it means that the first user has not performed the permission issuing operation, and the third user cannot perform the third operation.

[0167] If the permission operation includes issuing the permission for routine operations, the corresponding permission opening instruction includes the opening instruction for routine operations, and the third-party display device can receive routine operations of the third user on the surgical planning model. In this case, if the third user performs a specific operation on the surgical planning model, the third-party display device does not respond, or displays a prompt message to the third user to prompt that the third user has no operation permission. If the permission operation includes issuing the permission for specific operations, the corresponding permission opening instruction includes the opening instruction for specific operations, and the third-party display device can receive specific operations of the third user on the surgical planning model. In this case, if the third user performs routine operations on the surgical planning model, the third-party display device does not respond, or displays a prompt message to the third user to prompt that the third user has no operation permission. If the permission operation includes issuing the permission for routine operations and specific operations, the corresponding permission opening instruction includes the opening instruction for both routine operations and specific operations, and the third-party display device can receive routine operations and specific operations of the third user on the surgical planning model.

[0168] In an optional embodiment, the third-party display device also receives a third data stream transmission control instruction of the third user for the operation process, and the transmission control instruction includes controlling the third data stream to be transmitted to at least one of the second mixed reality device and the second mixed reality device. In other words, the third user can control whether to transmit the third data stream to the second mixed reality device and / or the first mixed reality device through the transmission control instruction, that is, the third user can control whether to synchronize the operation process corresponding to the third operation to the second mixed reality device and / or the first mixed reality device.

[0169] In an optional embodiment, the first user is a doctor, the second user is a patient's family representative, and the third user is other family members. The first mixed reality device and the second mixed reality device are both mixed reality glasses (Hololens), and the third-party display device is a smart display. The doctor and the patient's family representative both wear mixed reality glasses. When other family members have questions to ask, they perform a third operation on the surgical planning model through a third-party display device and ask questions during the third operation. The doctor can use the first mixed reality device, and the patient's family representative can use the second mixed reality device to synchronously see the operation process of other family members on the surgical planning model while listening to other family members' questions, so that they can more accurately and truly understand other family members' inquiries about the surgical process with respect to the surgical planning model, thereby facilitating the doctor to make targeted responses to other family members' inquiries, thereby improving the communication efficiency and communication information symmetry between the doctor and other family representatives.

[0170] When the third-party display device includes a third-party control unit and a third-party display unit, a third user can perform a third operation on the surgical planning model on the third-party control unit, and the third-party control unit transmits a third data stream of the operation process corresponding to the third operation to the third-party display unit, the first mixed reality device, and the second mixed reality device in response to the third operation.

[0171] In an optional embodiment, the first mixed reality device includes a control unit (master tablet computer) and a mixed reality unit (mixed reality glasses), and the third-party display device includes a third-party control unit (slave tablet computer) and a third-party display unit. When the control unit to be operated is described as the execution subject, the workflow of the information sharing method is as follows: Fig.10 As shown, including:

[0172] Step 101: The control unit to be operated receives the user's login information and displays a mode selection interface if the login information is correct; the mode selection interface includes a linkage mode and a three-party viewing mode;

[0173] Step 102: If the control unit to be operated responds to the user's selection operation of the linkage mode on the mode selection interface, the control unit to be operated is determined as a control unit in the first mixed reality device;

[0174] Step 103: The control unit determines whether the current storage space includes a surgical planning model;

[0175] Step 104: If the current storage space does not include the surgery planning model, search for the surgery planning model in the remaining storage space until the surgery planning model is obtained;

[0176] Step 105: If the current storage space includes the surgical planning model, the control unit actively enters a three-party perspective film reading mode, and in response to a first operation of the user on the surgical planning model, transmits a first data stream of the operation process corresponding to the first operation to the third-party control unit and the second mixed reality device;

[0177] Step 106: The third-party control unit transmits the first data stream to the third-party display unit, and the third-party display unit and the second mixed reality device are both used to display the operation process corresponding to the first data stream;

[0178] Step 107: if the control unit to be operated responds to the user's selection operation of the three-party viewing mode on the mode selection interface, the control unit to be operated is determined as a third-party control unit;

[0179] Step 108: The third-party control unit receives the first data stream sent by the control unit.

[0180] In an optional embodiment, the first mixed reality device includes a master mixed reality glasses, and the second mixed reality device includes a slave mixed reality glasses. When describing the mixed reality glasses to be operated as the execution subject, the workflow of the information sharing method is as follows: Fig.11 As shown, including:

[0181] Step 111, the mixed reality glasses to be operated receive the user's login information, and if the login information is correct, start the linkage module;

[0182] Step 112: The server to which the mixed reality glasses to be operated are connected determines whether the mixed reality glasses to be operated are the main mixed reality glasses;

[0183] The server determines the mixed reality device to be operated that logs in first as the main mixed reality glasses.

[0184] Step 113: If the server determines that the mixed reality glasses to be operated are the main mixed reality glasses, the server determines whether there is a control unit corresponding to the main mixed reality glasses;

[0185] Step 114: If it is determined that there is a control unit corresponding to the main mixed reality glasses, the main mixed reality glasses enter a three-party viewing mode;

[0186] Step 115: If it is determined that there is no control unit corresponding to the master mixed reality glasses, the master mixed reality glasses obtain the operation authority;

[0187] Step 116: The main mixed reality glasses determine whether the current storage space includes the surgical planning model;

[0188] Step 117: If the current storage space does not include the surgery planning model, search for the surgery planning model in the remaining storage space until the surgery planning model is obtained;

[0189] Step 118: If the current storage space includes the surgical planning model, the master mixed reality glasses actively enter the three-party viewing mode, and in response to the user's first operation on the surgical planning model, transmit a first data stream of the operation process corresponding to the first operation to the slave mixed reality device and the third-party display device;

[0190] Step 119: The mixed reality device and the third-party display device are used to display the operation process corresponding to the first data stream;

[0191] Step 120: If the server determines that the mixed reality device to be operated is a slave mixed reality pair of glasses, the slave mixed reality pair of glasses enters a three-party viewing mode and receives a first data stream sent by the master mixed reality pair of glasses.

[0192] In an optional embodiment, the first mixed reality device includes a control unit and mixed reality glasses, the second mixed reality device includes mixed reality glasses, and the third-party display device includes a third-party control unit and a third-party display device. The software of the control unit, mixed reality glasses and third-party control unit are all developed based on the Unity (user interface) platform for releasing rendered three-dimensional models, and the surgical planning model can be operated through gestures. The software of the control unit and the third-party control unit use UGUI as the user (UI) framework. The mixed reality glasses use MRTK (Mixed Reality Toolkit) as the mixed reality (MR) framework, and AR Foundation (augmented reality function) provides the framework capability of AR mode for the third-party control unit. The software structure of the control unit, mixed reality glasses and third-party control unit is as follows: Fig.12 shown.

[0193] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0194] Please continue to see Figure 1 An embodiment of the present application provides an information sharing system, which includes a first mixed reality device 11, a second mixed reality device 12 and a third-party display device 13. The first mixed reality device 11, the second mixed reality device 12, and the third-party display device 13 are communicatively connected.

[0195] A first mixed reality device 11, used to execute the steps described in the embodiment of the above-mentioned information sharing method applied to the first mixed reality device 11;

[0196] A second mixed reality device 12, used to perform the steps described in the embodiment of the above-mentioned information sharing method applied to the second mixed reality device 12;

[0197] The third-party display device 13 is used to execute the steps described in the embodiment of the above-mentioned information sharing method applied to the third-party display device 13.

[0198] The information sharing system provided in the embodiment of the present application is used to execute the steps of the above-mentioned information sharing method. Therefore, the information sharing system has all the beneficial effects of the above-mentioned information sharing method, which will not be repeated here.

[0199] Based on the same inventive concept, the embodiment of the present application also provides an information sharing device for implementing the information sharing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more information sharing device embodiments provided below can refer to the limitations on the information sharing method above, and will not be repeated here.

[0200] In one embodiment, Fig.13 As shown, an information sharing device 20 is provided, comprising: an acquisition module 21, a response module 22 and a transmission module 23, wherein:

[0201] The acquisition module 21 is used to acquire the surgical planning model to be displayed.

[0202] The response module 22 is used to respond to the first operation on the surgical planning model and transmit the first data stream of the operation process to the second mixed reality device; wherein the second mixed reality device and the first mixed reality device are aligned with each other, and the second mixed reality device is used to display the operation process corresponding to the first data stream.

[0203] The transmission module 23 is used to transmit the first data stream to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0204] In one embodiment, the information sharing device 20 further includes a sending module. The sending module is used to send a registration notification to the second mixed reality device and perform a registration operation using a preset registration algorithm; the second mixed reality device is used to perform a registration operation using a preset registration algorithm based on the registration notification; the registration notification includes registration parameters.

[0205] In one embodiment, the acquisition module 21 includes a determination unit and a search unit. The determination unit is used to determine whether the current storage space includes the surgery planning model; if the current storage space does not include the surgery planning model, the search unit is used to search for the surgery planning model in the remaining storage space until the surgery planning model is obtained.

[0206] In one embodiment, the response module 22 is specifically configured to transmit a first data stream of the operation process to the mixed reality unit for display, and to the second mixed reality device in response to a first operation on the surgical planning model.

[0207] In one embodiment, an information sharing device is provided, comprising a receiving module. The receiving module is used to receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process of a first mixed reality device in response to a user performing a first operation on a surgical planning model; and a second mixed reality device is registered with the first mixed reality device.

[0208] In one embodiment, the receiving module is further used to receive a registration notification sent by the first mixed reality device, and perform a registration operation using a preset registration algorithm; the registration notification includes registration parameters.

[0209] In one embodiment, the information sharing device also includes an operation response module for responding to a second operation on the surgical planning model and transmitting a second data stream of the operation process to the first mixed reality device and a third-party display device; both the first mixed reality device and the third-party display device are used to display the operation process corresponding to the second data stream.

[0210] In one embodiment, an information sharing device is provided, comprising a display module for receiving and displaying a first data stream of an operation process; the first data stream is a data stream of an operation process of a first mixed reality device in response to a user performing a first operation on a surgical planning model.

[0211] In one embodiment, the information sharing device further includes a data acquisition module. The data acquisition module is used to acquire video data corresponding to the operation process; and the display module is specifically used to merge the video data and the first data stream to obtain and display the merged video.

[0212] In one embodiment, the data acquisition module includes a display unit and a shooting unit. The display unit is used to display the position information of the surgical planning model corresponding to the first data stream on the display interface of the third-party control unit. The shooting unit is used to adjust the shooting parameters according to the position information, and based on the adjusted shooting parameters, shoot the background of the operation process of the surgical planning model in the reality enhancement mode to obtain video data corresponding to the operation process.

[0213] In one embodiment, the shooting unit is further used to start the panoramic camera to shoot the background of the operation process of the surgical planning model to obtain video data corresponding to the operation process.

[0214] In one embodiment, the information sharing device further includes a data stream transmission module. The data stream transmission module is used to respond to the third operation on the surgical planning model and transmit the third data stream of the operation process to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the third data stream.

[0215] Each module in the above information sharing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0216] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an information sharing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0217] Those skilled in the art will understand that Fig.14The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0218] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0219] obtaining a surgical planning model to be presented;

[0220] In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream;

[0221] The first data stream is transmitted to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0222] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0223] Receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which a first mixed reality device responds to a first operation on a surgical planning model; and a second mixed reality device is registered with the first mixed reality device.

[0224] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0225] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0226] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0227] obtaining a surgical planning model to be presented;

[0228] In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream;

[0229] The first data stream is transmitted to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0230] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0231] Receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which a first mixed reality device responds to a first operation on a surgical planning model; and a second mixed reality device is registered with the first mixed reality device.

[0232] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0233] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0234] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0235] obtaining a surgical planning model to be presented;

[0236] In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream;

[0237] The first data stream is transmitted to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

[0238] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0239] Receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which a first mixed reality device responds to a first operation on a surgical planning model; and a second mixed reality device is registered with the first mixed reality device.

[0240] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0241] Receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

[0242] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0243] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0244] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An information sharing method, characterized in that: Applied to a first mixed reality device, the method comprises: obtaining a surgical planning model to be presented; In response to a first operation on the surgical planning model, a first data stream of the operation process is transmitted to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is used to display the operation process corresponding to the first data stream; The first data stream is transmitted to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

2. The method according to claim 1, characterized in that The method further comprises: A registration notification is sent to the second mixed reality device, and a preset registration algorithm is used to perform a registration operation; the second mixed reality device is used to perform a registration operation based on the registration notification using the preset registration algorithm; the registration notification includes registration parameters.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining the surgical planning model to be displayed includes: determining whether the current storage space includes the surgical planning model; If the current storage space does not include the surgery planning model, the surgery planning model is searched for in the remaining storage space until the surgery planning model is obtained.

4. The method according to claim 1 or 2, characterized in that: The first mixed reality device includes a control unit and a mixed reality unit; the first data stream of the operation process is transmitted to the second mixed reality device in response to the first operation on the surgical planning model, including: In response to the first operation on the surgical planning model, the control unit transmits a first data stream of the operation process to the mixed reality unit for display, and transmits it to the second mixed reality device.

5. An information sharing method, characterized in that: Applied to a second mixed reality device, the method comprises: Receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which the first mixed reality device responds to a first operation on a surgical planning model; and the second mixed reality device is registered with the first mixed reality device.

6. The method according to claim 5, characterized in that The method further comprises: Receive a registration notification sent by the first mixed reality device, and use a preset registration algorithm to perform a registration operation; the registration notification includes registration parameters.

7. The method according to claim 5, characterized in that The method further comprises: In response to a second operation on the surgical planning model, a second data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the second data stream.

8. An information sharing method, characterized in that: Applied to a third-party display device, the method includes: Receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process of the first mixed reality device in response to a user performing a first operation on a surgical planning model.

9. The method according to claim 8, characterized in that The method further comprises: Acquire video data corresponding to the operation process; Correspondingly, the first data flow showing the operation process includes: The video data and the first data stream are merged to obtain and display a merged video.

10. The method according to claim 9, characterized in that The third-party display device includes a third-party control unit; the obtaining of video data corresponding to the operation process includes: Displaying the position information of the surgical planning model corresponding to the first data stream on the display interface of the third-party control unit; The third-party control unit adjusts the shooting parameters according to the position information, and based on the adjusted shooting parameters, shoots the background of the operation process of the surgical planning model in the reality enhancement mode to obtain video data corresponding to the operation process.

11. The method according to claim 9, characterized in that The third-party display device also includes a panoramic camera, and the obtaining of video data corresponding to the operation process includes: The panoramic camera is started to shoot the background of the operation process of the surgical planning model to obtain video data corresponding to the operation process.

12. The method according to any one of claims 8 to 10, characterized in that: The method further comprises: In response to a third operation on the surgical planning model, a third data stream of the operation process is transmitted to the first mixed reality device and the third-party display device; the first mixed reality device and the third-party display device are both used to display the operation process corresponding to the third data stream.

13. An information sharing system, characterized in that: The information sharing system includes: a first mixed reality device, a second mixed reality device, and a third-party display device, wherein the first mixed reality device, the second mixed reality device, and the third-party display device are communicatively connected; The first mixed reality device is used to perform the steps of the method according to any one of claims 1 to 4; The second mixed reality device is used to perform the steps of the method according to any one of claims 5 to 7; The third-party display device is used to perform the steps of the method described in any one of claims 8-12.

14. An information sharing device, characterized in that: The device comprises: An acquisition module, used for acquiring a surgical planning model to be displayed; a response module, configured to respond to a first operation on the surgical planning model and transmit a first data stream of the operation process to a second mixed reality device; wherein the second mixed reality device is registered with the first mixed reality device, and the second mixed reality device is configured to display the operation process corresponding to the first data stream; The transmission module is used to transmit the first data stream to a third-party display device; the third-party display device is used to display the operation process corresponding to the first data stream.

15. An information sharing device, characterized in that: The device comprises: A receiving module is used to receive and display a first data stream of an operation process; the first data stream is a data stream of an operation process in which the first mixed reality device responds to a user performing a first operation on a surgical planning model; and the second mixed reality device is registered with the first mixed reality device.

16. An information sharing device, characterized in that: The device comprises: The display module is used to receive and display a first data stream of the operation process; the first data stream is a data stream of the operation process of the first mixed reality device in response to the user performing a first operation on the surgical planning model.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

19. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images