Display method, device and system of wearable equipment and wearable equipment

Abstract

The invention discloses a wearable device display method, device and system and a wearable device.The display method applied to a first wearable device comprises the steps that a marker is detected, and a first position and a first orientation are obtained; displaying the first model in the target space according to the first position and the first orientation; in response to the operation for the first model, determining the offset of the first model in the target space before and after the operation; sending the offset to the second wearable device; and the second wearable device is used for displaying a second model in the target space according to the offset and a second position and a second orientation obtained by detecting the marker. According to the invention, users of different wearable devices share the model, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device is triggered to be the same operation, and the models seen by an operator and an observer are always in the same position and orientation in the real world, so that the user experience is improved. And the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a display method, device, system and wearable device for wearable devices. Background Art

[0002] Wearable devices, also known as wearable devices, are portable devices that are worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also realize powerful functions through software support, data interaction, and cloud interaction.

[0003] Currently, in scenarios where multiple wearable devices share a model, the model is displayed based on the position offset relative to a certain wearable device. Although this display method can achieve synchronous translation, rotation, scaling and other operations on the model, the model viewed by different users through different wearable devices is not moved to the position the operator wants to move to, resulting in a poor user experience. Summary of the invention

[0004] In order to overcome the defect in the prior art that when multiple wearable devices share a model, the models viewed by different users through different wearable devices are not moved to the position the operator wants them to move to, resulting in a poor user experience, the present invention provides a display method, device, system and wearable device for a wearable device.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] A first aspect of the present invention provides a display method for a wearable device, which is applied to a first wearable device and includes:

[0007] Detecting the marker to obtain a first position and a first orientation of the marker;

[0008] displaying a first model in a target space according to the first position and the first orientation;

[0009] In response to an operation on the first model, determining an offset of the first model in the target space before and after the operation;

[0010] The offset is sent to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the offset and a second position and a second orientation obtained by detecting the marker.

[0011] Optionally, detecting the marker to obtain the first position and the first orientation of the marker includes:

[0012] Detecting the marker to obtain the position and orientation of the marker in a first coordinate system; wherein the first coordinate system is the coordinate system of the first wearable device;

[0013] The position of the marker in the first coordinate system is converted into a first position in the first world coordinate system, and the orientation of the marker in the first coordinate system is converted into a first orientation in the first world coordinate system.

[0014] Optionally, sending the offset to the second wearable device includes: if the offset satisfies a preset condition, sending the offset to the second wearable device.

[0015] A second aspect of the present invention provides a display method for a wearable device, which is applied to a second wearable device, comprising:

[0016] Detecting the marker to obtain a second position and a second orientation of the marker;

[0017] displaying a second model in a target space according to the second position and the second orientation;

[0018] Receiving an offset sent by a first wearable device; wherein the first wearable device is used to display a first model in the target space according to a first position and a first orientation obtained by detecting the marker, and in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation;

[0019] The second model is adjusted according to the offset.

[0020] Optionally, detecting the marker to obtain the second position and the second orientation of the marker includes:

[0021] Detecting the marker to obtain the position and orientation of the marker in a second coordinate system; wherein the second coordinate system is the coordinate system of the second wearable device;

[0022] The position of the marker in the second coordinate system is converted into a second position in the second world coordinate system, and the orientation of the marker in the second coordinate system is converted into a second orientation in the second world coordinate system.

[0023] Optionally, adjusting the second model according to the offset includes: if the offset satisfies a preset condition, adjusting the second model according to the offset.

[0024] A third aspect of the present invention provides a display method for a wearable device, which is applied to a first wearable device and includes:

[0025] Displaying the first model in the target space according to the preset position and the preset orientation;

[0026] Determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting the marker;

[0027] Sending the target transformation matrix to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker;

[0028] In response to an operation on the first model, determining an offset of the first model in the target space before and after the operation;

[0029] The offset is sent to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset.

[0030] Optionally, determining the target transformation matrix according to the preset position and the preset orientation and a first position and a first orientation obtained by detecting a marker includes:

[0031] Acquire the position and orientation of the marker in a first coordinate system obtained by detecting the marker; wherein the first coordinate system is the coordinate system of the first wearable device;

[0032] Converting the position of the marker in the first coordinate system to a first position in the first world coordinate system, and converting the orientation of the marker in the first coordinate system to a first orientation in the first world coordinate system;

[0033] A target transformation matrix is ​​determined according to the first position and the preset position and the first orientation and the preset orientation.

[0034] Optionally, sending the offset to the second wearable device includes:

[0035] If the offset meets the preset condition, the offset is sent to the second wearable device.

[0036] A fourth aspect of the present invention provides a display method for a wearable device, which is applied to a second wearable device, comprising:

[0037] Receiving a target transformation matrix sent by a first wearable device; wherein the first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, and determining the target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker;

[0038] Displaying a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker;

[0039] Receiving the offset sent by the first wearable device; wherein the first wearable device is used to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model;

[0040] The second model is adjusted according to the offset.

[0041] Optionally, displaying a second model in the target space according to the target transformation matrix and the second position and the second orientation obtained by detecting the marker includes:

[0042] Acquire the position and orientation of the marker in a second coordinate system obtained by detecting the marker; wherein the second coordinate system is the coordinate system of the second wearable device;

[0043] Converting the position of the marker in the second coordinate system to a second position in the second world coordinate system, and converting the orientation of the marker in the second coordinate system to a second orientation in the second world coordinate system;

[0044] A second model is displayed in the target space according to the second position, the second orientation, and the target transformation matrix.

[0045] Optionally, adjusting the second model according to the offset includes:

[0046] If the offset meets a preset condition, the second model is adjusted according to the offset.

[0047] The fifth aspect of the present invention provides a display device for a wearable device, which is applied to a first wearable device, and includes: a first detection module, used to detect a marker to obtain a first position and a first orientation of the marker; a first display module, used to display a first model in a target space according to the first position and the first orientation; an offset determination module, used to determine, in response to an operation on the first model, the offset of the first model in the target space before and after the operation; and an offset sending module, used to send the offset to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the offset and the second position and the second orientation obtained by detecting the marker.

[0048] Optionally, the first detection module is specifically used to detect the marker to obtain the position and orientation of the marker in a first coordinate system; wherein the first coordinate system is the coordinate system of the first wearable device; and convert the position of the marker in the first coordinate system into a first position in a first world coordinate system, and convert the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system.

[0049] Optionally, the offset sending module is specifically configured to send the offset to a second wearable device when the offset meets a preset condition.

[0050] The sixth aspect of the present invention provides a display device of a wearable device, which is applied to a second wearable device, including: a second detection module, used to detect a marker to obtain a second position and a second orientation of the marker; a second display module, used to display a second model in a target space according to the second position and the second orientation; an offset receiving module, used to receive an offset sent by a first wearable device; wherein the first wearable device is used to display a first model in the target space according to the first position and the first orientation obtained by detecting the marker, and in response to an operation on the first model, determines the offset of the first model in the target space before and after the operation; and a model adjustment module, used to adjust the second model according to the offset.

[0051] Optionally, the second detection module is specifically used to detect the marker to obtain the position and orientation of the marker in a second coordinate system; wherein the second coordinate system is the coordinate system of the second wearable device; and convert the position of the marker in the second coordinate system into a second position in a second world coordinate system, and convert the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system.

[0052] Optionally, the model adjustment module is specifically configured to adjust the second model according to the offset when the offset meets a preset condition.

[0053] The seventh aspect of the present invention provides a display system for a wearable device, comprising a first wearable device and a second wearable device; the first wearable device is used to detect a marker, obtain a first position and a first orientation of the marker, and display a first model in a target space according to the first position and the first orientation, and in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation, and send the offset to the second wearable device; the second wearable device is used to detect the marker, obtain a second position and a second orientation of the marker, and display a second model in the target space according to the second position and the second orientation, and receive the offset sent by the first wearable device, and adjust the second model according to the offset.

[0054] An eighth aspect of the present invention provides a display device for a wearable device, applied to a first wearable device, comprising:

[0055] A first display module, used for displaying a first model in a target space according to a preset position and a preset orientation;

[0056] A matrix determination module, configured to determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting the marker;

[0057] A matrix sending module, used for sending the target transformation matrix to a second wearable device; wherein the second wearable device is used for displaying a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker;

[0058] an offset determination module, configured to determine, in response to an operation on the first model, an offset of the first model in the target space before and after the operation;

[0059] An offset sending module is used to send the offset to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset.

[0060] Optionally, the matrix determination module is specifically used to obtain the position and orientation of the marker obtained by detecting the marker in a first coordinate system, wherein the first coordinate system is the coordinate system of the first wearable device, and convert the position of the marker in the first coordinate system into a first position in a first world coordinate system, convert the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system, and determine the target transformation matrix based on the first position and the preset position and the first orientation and the preset orientation.

[0061] Optionally, the offset sending module is specifically configured to send the offset to the second wearable device when the offset meets a preset condition.

[0062] A ninth aspect of the present invention provides a display device of a wearable device, applied to a second wearable device, comprising:

[0063] A matrix receiving module, configured to receive a target transformation matrix sent by a first wearable device; wherein the first wearable device is configured to display a first model in a target space according to a preset position and a preset orientation, and to determine the target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker;

[0064] A second display module, configured to display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker;

[0065] an offset receiving module, configured to receive the offset sent by the first wearable device; wherein the first wearable device is configured to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model;

[0066] A model adjustment module is used to adjust the second model according to the offset.

[0067] Optionally, the second display module is specifically used to obtain the position and orientation of the marker obtained by detecting the marker in a second coordinate system, wherein the second coordinate system is the coordinate system of the second wearable device, and convert the position of the marker in the second coordinate system into a second position in a second world coordinate system, convert the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system, and display the second model in the target space according to the second position, the second orientation and the target transformation matrix.

[0068] Optionally, the model adjustment module is specifically configured to adjust the second model according to the offset when the offset meets a preset condition.

[0069] A tenth aspect of the present invention provides a display system for a wearable device, including a first wearable device and a second wearable device;

[0070] The first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker, and send the target transformation matrix to the second wearable device;

[0071] The second wearable device is used to receive the target transformation matrix sent by the first wearable device, and display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker;

[0072] The first wearable device is further configured to, in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation, and send the offset to the second wearable device;

[0073] The second wearable device is further used to receive the offset sent by the first wearable device and adjust the second model according to the offset.

[0074] An eleventh aspect of the present invention provides a wearable device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the display method as described above when executing the computer program.

[0075] A sixth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the display method as described above when executed by a processor.

[0076] Based on the common sense in the art, the above optional conditions can be combined arbitrarily to obtain the preferred embodiments of the present invention.

[0077] The positive and progressive effect of the present invention is that the first wearable device and the second wearable device respectively detect the same marker, the first wearable device displays a first model in the real space where the first wearable device and the second wearable device are located, that is, the target space, according to the detected first position and first orientation, and the second wearable device displays a second model in the target space according to the detected second position and second orientation. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, and the second wearable device receives the offset and adjusts the second model, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also trigger the same operation, providing the user with a visual experience similar to operating the entity corresponding to the first model in the real world, and the models seen by the operator and the observer are always in the same position and orientation in the real world, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flowchart of a display method for a wearable device provided in Example 1 of the present invention.

[0079] Figure 2A schematic diagram of an application scenario provided by an embodiment of the present invention.

[0080] Figure 3 A schematic diagram of another application scenario provided for an embodiment of the present invention.

[0081] Figure 4 A schematic flow chart of step S11 provided in embodiment 1 of the present invention.

[0082] Figure 5 This is a structural block diagram of a display device of a wearable device provided in Example 1 of the present invention.

[0083] Figure 6 This is a flowchart of a display method for a wearable device provided in Example 2 of the present invention.

[0084] Figure 7 A flowchart of step S21 provided in Example 2 of the present invention.

[0085] Figure 8 This is a structural block diagram of a display device for a wearable device provided in Example 2 of the present invention.

[0086] Fig. 9 This is a structural block diagram of a display system for a wearable device provided in Example 3 of the present invention.

[0087] Fig.10 This is a flowchart of a display method for a wearable device provided in Example 4 of the present invention.

[0088] Fig.11 A flowchart of step S32 provided in embodiment 4 of the present invention.

[0089] Fig.12 This is a structural block diagram of a display device for a wearable device provided in Example 4 of the present invention.

[0090] Fig.13 This is a flowchart of a display method for a wearable device provided in Example 5 of the present invention.

[0091] Fig.14 A flowchart of step S42 provided in Example 5 of the present invention.

[0092] Fig.15 This is a structural block diagram of a display device for a wearable device provided in Example 5 of the present invention.

[0093] Fig.16 A structural schematic diagram of a wearable device provided in Example 7 of the present invention. DETAILED DESCRIPTION

[0094] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0095] The wearable device involved in this embodiment (including the first wearable device and the second wearable device) can specifically be smart glasses, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, etc., and can also be a smart helmet, a smart watch, etc.

[0096] Example 1

[0097] Figure 1 This is a flow chart of a display method for a wearable device provided in this embodiment. The display method for a wearable device can be executed by a display device of the wearable device. The display device can be implemented by software and / or hardware. The display device can be part or all of the wearable device. The display method for a wearable device provided in this embodiment is introduced below with the first wearable device as the execution subject.

[0098] like Figure 1 As shown, the display method of the wearable device provided in this embodiment may include the following steps S11 to S14:

[0099] Step S11: Detect the marker to obtain a first position and a first orientation of the marker.

[0100] Specifically, the marker can be detected using a sensor device built into the first wearable device. The sensor device may include an image acquisition device such as a camera, and may also include a gyroscope, etc. The first position obtained by detecting the marker is usually represented by a three-dimensional coordinate, and the first orientation obtained by detecting the marker refers to the direction in which the marker points, which is usually represented by a direction vector.

[0101] Step S12: Displaying a first model in the target space according to the first position and the first orientation. In this embodiment, the first position of the marker is directly used as the position of the first model, and the first orientation of the marker is used as the orientation of the first model.

[0102] Step S13: In response to the operation on the first model, determine the offset of the first model in the target space before and after the operation.

[0103] The operation on the first model may include at least one of movement, rotation and scaling, and correspondingly, the offset may include at least one of position offset, orientation offset and size offset. In a specific implementation, the position offset Δ may be calculated using the following formula: p, Direction offset Δ r , size offset Δ s :

[0104] Δ p =P new -P old ;

[0105] Δ r =R new -R old ;

[0106] Δ s =S new -S old ;

[0107] Among them, P new is the position of the first model after the operation, P old is the position of the first model before operation, R new is the orientation of the first model after the operation, R old is the orientation of the first model before the operation, S new is the size of the first model after the operation, S old is the size of the first model before the operation.

[0108] Step S14: sending the offset to the second wearable device. The target space is a real space where the first wearable device and the second wearable device are located, and the second wearable device is used to display a second model in the target space according to the offset and a second position and a second orientation obtained by detecting the marker.

[0109] The second model and the first model are the same model, usually a three-dimensional model. In a specific implementation, the first model and the second model can be three-dimensional models of human body parts, and the user of the first wearable device and the user of the second wearable device can be different doctors.

[0110] In a specific implementation, the second wearable device can first display the second model in the target space according to the second position and the second orientation obtained by detecting the marker, that is, directly use the second position of the marker as the position of the second model, and use the second orientation of the marker as the orientation of the second model. In this way, the connection between the first wearable device and the second wearable device is established through the marker, so that the first model and the second model have the same posture in the target space. After receiving the offset sent by the first wearable device, the second model is adjusted according to the offset. Specifically, if the offset includes a position offset and an orientation offset, the position of the second model in the target space can be adjusted according to the position offset, and the orientation of the second model in the target space can be adjusted according to the orientation offset.

[0111] Among them, the number of second wearable devices can be one or more. It should be noted that the first wearable device and the second wearable device are at different positions in the target space, for example, the first wearable device and the second wearable device can be at different positions in the same room. The marker can be any object in the target space, and its position is usually fixed. For example, the marker can be a painting on the wall. In a specific implementation, the marker can also have an orientation feature, that is, the marker itself contains information that can indicate the orientation. The marker can also be a fixed-position marking point. In a specific implementation, the target space can be an operating room or a consultation room, etc., and different doctors wear the first wearable device and the second wearable device to realize the sharing of the three-dimensional model of a certain part of the patient, so as to better assist medical diagnosis.

[0112] In this embodiment, the user of the first wearable device can see the first model displayed in the target space but cannot see the second model, and the user of the second wearable device can see the second model displayed in the target space but cannot see the first model. It should be noted that users other than the first wearable device cannot see the first model displayed in the target space, and users other than the second wearable device cannot see the second model displayed in the target space.

[0113] Figure 2 A usage scenario diagram for illustrating a display method of a wearable device. Figure 2 In the example shown, the first wearable device 100 and the second wearable device 200 are both HoloLens, and the marker 300 is a picture containing an arrow with an orientation feature. Among them, HoloLens is an MR head display developed by Microsoft, that is, a mixed reality head-mounted display. The first wearable device 100 detects the marker 300 to obtain the first position and the first orientation of the marker, and displays the first model in the target space according to the first position and the first orientation. The user can view the first model through the first wearable device 100, and can also perform different operations on the first model. The first wearable device 100 responds to the operation, determines the offset of the first model in the target space before and after the operation, and sends the offset to the second wearable device 200. The second wearable device 200 detects the marker 300 to obtain the second position and the second orientation of the marker, and displays the second model in the target space according to the offset sent by the first wearable device 100 and the second position and the second orientation. The user can view the second model through the second wearable device 200.

[0114] In this embodiment, the first wearable device and the second wearable device respectively detect the same marker, the first wearable device displays the first model in the target space according to the detected first position and first orientation, and the second wearable device displays the second model in the target space according to the detected second position and second orientation. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, so that the second wearable device can display the second model in the target space according to the offset and the second position and second orientation obtained by detecting the marker, thereby enabling users of different wearable devices to share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also trigger the same operation, providing the user with a visual experience similar to operating the entity corresponding to the first model in the real world, and the models seen by the operator and the observer are always in the same position and orientation in the real world, thereby improving the user experience.

[0115] In a specific example, the first model and the second model are the target model M. The user of the first wearable device can see the front of the target model A, and the user of the second wearable device can see the right side of the target model M. After the user of the first wearable device rotates the target model M to the right, the user of the first wearable device can see the front and left sides of the target model M, and the target model M seen by the user of the second wearable device will also be triggered to rotate right, and the user of the second wearable device can see the right side and front of the target model M.

[0116] In the specific implementation of step S14, the offset may be directly sent to the second wearable device. In this implementation, the first wearable device may be communicatively connected with the second wearable device, and the first wearable device directly sends the offset to the second wearable device.

[0117] In the specific implementation of step S14, the offset may also be sent to the second wearable device through the server. In this embodiment, the first wearable device is connected to the server for communication, the first wearable device sends the offset to the server, the server is connected to the second wearable device for communication, and the server forwards the offset to the second wearable device. Figure 3In the example shown, the server is connected to a first wearable device and two second wearable devices respectively. The first wearable device and the two second wearable devices detect the same marker respectively. The first wearable device displays the first model according to the detected marker position and orientation, and the two second wearable devices display the second model according to the position and orientation of the markers detected by each. Among them, the first wearable device and the two second wearable devices display the first model and the second model according to the original size, and the first model and the second model are the same model. Afterwards, if the user of the first wearable device operates the first model, the first wearable device receives the operation and determines the offset of the first model in the target space before and after the operation. The server receives the offset sent by the first wearable device and forwards the offset to the two second wearable devices. The two second wearable devices calculate the new position, new orientation and / or new size according to the received offset respectively, and finally update the current position of the second model to the new position, the current orientation of the second model to the new orientation, and the current size of the second model to the new size, so that the user of the first wearable device and the users of the two second wearable devices share the same model.

[0118] In an optional embodiment, Figure 4 As shown, the above step S11 specifically includes the following steps S111-S112:

[0119] Step S111: Detect the marker to obtain the position and orientation of the marker in a first coordinate system, wherein the first coordinate system is the coordinate system of the first wearable device.

[0120] Step S112: converting the position of the marker in the first coordinate system into a first position in the first world coordinate system, and converting the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system.

[0121] It should be noted that the first coordinate system is the object coordinate system of the first wearable device. When the first wearable device moves or rotates, the first coordinate system also undergoes the same translation or rotation. The first world coordinate system is a coordinate system established with the initial position of the first wearable device as the origin, and the first coordinate system is a coordinate system established with the current position of the first wearable device as the origin.

[0122] In a specific implementation, the conversion from the first coordinate system to the first world coordinate system may be performed according to the following formula:

[0123]

[0124]

[0125]

[0126] P world1 =R z R y R x P obj1 +T Cam2World1 ;

[0127]

[0128] Among them, P obj1 is the position of the marker in the first coordinate system, P world1 is the first position of the marker in the first world coordinate system, T Cam2World1 is the relative offset between the first wearable device and the first world coordinate system, that is, the position of the first wearable device in the first world coordinate system, is the orientation of the marker in the first coordinate system, is the first orientation of the marker in the first world coordinate system, R x is the rotation matrix of the x-axis in the first coordinate system, R y is the rotation matrix of the y axis in the first coordinate system, R z is the rotation matrix of the z-axis in the first coordinate system, α is the angle of rotation around the x-axis in the first coordinate system, β is the angle of rotation around the y-axis in the first coordinate system, and γ is the angle of rotation around the z-axis in the first coordinate system.

[0129] In actual applications, if the first wearable device sends the offset to the second wearable device as soon as it detects it, the second wearable device will cause the second wearable device to frequently adjust the second model. However, the amount of data of the second model is usually relatively large. If the second model is adjusted frequently, it will affect the performance of the second wearable device. In an optional implementation, the above step S14 specifically includes: if the offset meets the preset condition, the offset is sent to the second wearable device. Among them, the preset condition can be that the position offset reaches the corresponding position threshold, and the orientation offset reaches the corresponding orientation threshold, and the size offset reaches the corresponding size threshold. The preset condition can also be that one of the offsets reaches the corresponding threshold. The preset condition can also be that the weighted sum of the three offsets reaches a certain threshold, wherein the weight corresponding to each offset can be set according to the actual situation.

[0130] In this embodiment, the offset is sent to the second wearable device when the offset meets the preset conditions. When the offset does not meet the preset conditions, it is considered that the offset is small and will not affect the users of the first wearable device and the second wearable device sharing the same model. Therefore, the offset is not sent to the second wearable device at this time, thereby avoiding the situation where frequent sending of the offset to the second wearable device causes the performance of the second wearable device to deteriorate.

[0131] This embodiment also provides a display device for a wearable device, which is applied to a first wearable device, such as Figure 5 As shown, the display device of the wearable device includes a first detection module 61, a first display module 62, an offset determination module 63 and an offset sending module 64. The first detection module 61 is used to detect a marker to obtain a first position and a first orientation of the marker. The first display module 62 is used to display a first model in a target space according to the first position and the first orientation. The offset determination module 63 is used to determine the offset of the first model in the target space before and after the operation in response to an operation on the first model. The offset sending module 64 is used to send the offset to a second wearable device; wherein the target space is a real space where the first wearable device and the second wearable device are located, and the second wearable device is used to display a second model in the target space according to the offset and the second position and the second orientation obtained by detecting the marker, and the second model and the first model are the same model.

[0132] In an optional embodiment, the first detection module is specifically used to detect the marker to obtain the position and orientation of the marker in a first coordinate system; wherein the first coordinate system is the coordinate system of the first wearable device; and convert the position of the marker in the first coordinate system into a first position in a first world coordinate system, and convert the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system.

[0133] In an optional implementation manner, the offset sending module is specifically configured to send the offset to the second wearable device when the offset meets a preset condition.

[0134] It should be noted that the display device of the wearable device in this embodiment may specifically be a separate chip, a chip module or a wearable device, or may be a chip or a chip module integrated in the wearable device.

[0135] Regarding the various modules / units included in the display device of the wearable device described in this embodiment, they can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units.

[0136] Example 2

[0137] Figure 6This is a flow chart of a display method for a wearable device provided in this embodiment. The display method for the wearable device can be executed by a display device of the wearable device. The display device can be implemented by software and / or hardware. The display device can be part or all of the wearable device. The display method for the wearable device provided in this embodiment is introduced below with the second wearable device as the execution subject.

[0138] like Figure 6 As shown, the display method of the wearable device provided in this embodiment may include the following steps S21 to S24:

[0139] Step S21: Detect the marker to obtain a second position and a second orientation of the marker.

[0140] Specifically, the marker can be detected using a sensor device built into the second wearable device. The sensor device may include an image acquisition device such as a camera, and may also include a gyroscope, etc. The second position obtained by detecting the marker is usually represented by a three-dimensional coordinate, and the second orientation obtained by detecting the marker refers to the direction in which the marker faces or turns, and is usually represented by a direction vector.

[0141] Step S22: display the second model in the target space according to the second position and the second orientation. In this embodiment, the second position of the marker is directly used as the position of the second model, and the second orientation of the marker is used as the orientation of the second model. In this way, the connection between the first wearable device and the second wearable device is established through the marker, so that the first model and the second model have the same position and posture in the target space.

[0142] Step S23, receiving the offset sent by the first wearable device. The target space is the real space where the first wearable device and the second wearable device are located, and the first wearable device is used to display the first model in the target space according to the first position and the first orientation obtained by detecting the marker, and in response to an operation on the first model, determine the offset of the first model in the target space before and after the operation.

[0143] The first model and the second model are the same model, usually a three-dimensional model. In a specific implementation, the first model and the second model can be three-dimensional models of human body parts, and the user of the first wearable device and the user of the second wearable device can be different doctors.

[0144] The operations on the first model may include movement, rotation, scaling, etc. Correspondingly, the offset may include position offset, orientation offset, size offset, etc.

[0145] It should be noted that the first wearable device and the second wearable device are at different positions in the target space, for example, the first wearable device and the second wearable device can be at different positions in the same room. The marker can be any object in the target space, and its position is usually fixed. For example, the marker can be a painting on the wall. In a specific implementation, the marker can also have an orientation feature, that is, the marker itself contains information that can indicate the orientation. The marker can also be a fixed-position marking point. In a specific implementation, the target space can be an operating room or a consultation room, etc., and different doctors wear the first wearable device and the second wearable device to realize the sharing of the three-dimensional model of a certain part of the patient, so as to better assist medical diagnosis.

[0146] Step S24: adjusting the second model according to the offset. Specifically, calculating a new position, a new orientation, and / or a new size according to the offset, updating the current position of the second model to the new position, updating the current orientation of the second model to the new orientation, and updating the current size of the second model to the new size.

[0147] In one specific implementation example, the offset includes a position offset and an orientation offset, and the position of the second model in the target space can be adjusted according to the position offset, and the orientation of the second model in the target space can be adjusted according to the orientation offset. In another specific implementation example, the offset includes a position offset and a size offset, and the position of the second model in the target space can be adjusted according to the position offset, and the size of the second model in the target space can be adjusted according to the size offset.

[0148] In another example of a specific implementation, the offset includes a position offset Δ p , Direction offset Δ r and size offset Δ s , the new position P can be calculated using the following formula new 、New Direction R new , new size S new :

[0149] P new =P cur +Δ p ;

[0150] R new =R cur +Δ r ;

[0151] S new =S cur +Δ s ;

[0152] Among them, P curis the current position of the second model, R cur is the current orientation of the second model, S cur is the current size of the second model.

[0153] It should be noted that the first wearable device involved in this embodiment is the same as the first wearable device involved in Example 1. In this embodiment, the user of the first wearable device can see the first model displayed in the target space but cannot see the second model, and the user of the second wearable device can see the second model displayed in the target space but cannot see the first model. It should be noted that users other than the first wearable device cannot see the first model displayed in the target space, and users other than the second wearable device cannot see the second model displayed in the target space.

[0154] In this embodiment, the first wearable device and the second wearable device respectively detect the same marker, the first wearable device displays the first model in the target space according to the detected first position and first orientation, and the second wearable device displays the second model in the target space according to the detected second position and second orientation. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, and the second wearable device receives the offset and adjusts the second model, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also trigger the same operation, providing the user with a visual experience similar to operating the entity corresponding to the first model in the real world, and the models seen by the operator and the observer are always in the same position and orientation in the real world, thereby improving the user experience.

[0155] In an optional embodiment, Figure 7 As shown, the above step S21 specifically includes the following steps S211-S212:

[0156] Step S211: Detect the marker to obtain the position and orientation of the marker in a second coordinate system, wherein the second coordinate system is the coordinate system of the second wearable device.

[0157] Step S212: converting the position of the marker in the second coordinate system into a second position in the second world coordinate system, and converting the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system.

[0158] It should be noted that the second coordinate system is the object coordinate system of the second wearable device. When the second wearable device moves or rotates, the second coordinate system also undergoes the same translation or rotation. The second world coordinate system is a coordinate system established with the initial position of the second wearable device as the origin, and the second coordinate system is a coordinate system established with the current position of the second wearable device as the origin.

[0159] In a specific implementation, the second coordinate system to the second world coordinate system can be converted according to the following formula:

[0160]

[0161]

[0162]

[0163] P world2 =R z R y R x P obj2 +T Cam2World2 ;

[0164]

[0165] Among them, P obj2 is the position of the marker in the second coordinate system, P world2 is the second position of the marker in the second world coordinate system, T Cam2World2 is the relative offset between the second wearable device and the second world coordinate system, that is, the position of the second wearable device in the second world coordinate system, is the orientation of the marker in the second coordinate system, is the first orientation of the marker in the second world coordinate system, R x is the rotation matrix of the x-axis in the second coordinate system, R y is the rotation matrix of the y axis in the second coordinate system, R z is the rotation matrix of the z-axis in the second coordinate system, α is the angle of rotation around the x-axis in the second coordinate system, β is the angle of rotation around the y-axis in the second coordinate system, and γ is the angle of rotation around the z-axis in the second coordinate system.

[0166] In practical applications, the amount of data of the second model is usually relatively large. If the second wearable device adjusts the second model as soon as it receives the offset, the performance of the second wearable device will be reduced. In an optional implementation, the above step S24 specifically includes: if the offset meets the preset condition, adjusting the second model according to the offset. Among them, the preset condition can be that the position offset reaches the corresponding position threshold, and the orientation offset reaches the corresponding orientation threshold, and the size offset reaches the corresponding size threshold. The preset condition can also be that one of the offsets reaches the corresponding threshold. The preset condition can also be that the weighted sum of the three offsets reaches a certain threshold, wherein the weight corresponding to each offset can be set according to the actual situation.

[0167] In this embodiment, when the offset meets the preset conditions, the second wearable device adjusts the second model according to the offset. When the offset does not meet the preset conditions, it is considered that the offset is small and will not affect the users of the second wearable device and the first wearable device sharing the same model. Therefore, the second wearable device does not adjust the second model at this time, avoiding the situation where frequent adjustment of the second model leads to a decrease in the performance of the second wearable device.

[0168] This embodiment also provides a display device for a wearable device, which is applied to a second wearable device, such as Figure 8 As shown, the display device of the wearable device includes a second detection module 71, a second display module 72, an offset receiving module 73 and a model adjustment module 74. The second detection module 71 is used to detect the marker to obtain the second position and the second orientation of the marker; the second display module 72 is used to display the second model in the target space according to the second position and the second orientation; the offset receiving module 73 is used to receive the offset sent by the first wearable device; wherein the target space is the real space where the first wearable device and the second wearable device are located, the first wearable device is used to display the first model in the target space according to the first position and the first orientation obtained by detecting the marker, and in response to the operation on the first model, determine the offset of the first model in the target space before and after the operation, the first model and the second model are the same model; the model adjustment module 74 is used to adjust the second model according to the offset.

[0169] In an optional embodiment, the second detection module is specifically used to detect the marker to obtain the position and orientation of the marker in a second coordinate system; wherein the second coordinate system is the coordinate system of the second wearable device; and convert the position of the marker in the second coordinate system into a second position in a second world coordinate system, and convert the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system.

[0170] In an optional implementation manner, the model adjustment module is specifically configured to adjust the second model according to the offset when the offset satisfies a preset condition.

[0171] It should be noted that the display device of the wearable device in this embodiment may specifically be a separate chip, a chip module or a wearable device, or may be a chip or a chip module integrated in the wearable device.

[0172] Regarding the various modules / units included in the display device of the wearable device described in this embodiment, they can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units.

[0173] Example 3

[0174] This embodiment provides a display system for a wearable device, such as Fig. 9 As shown, it includes a first wearable device and a second wearable device.

[0175] The first wearable device is used to detect a marker, obtain a first position and a first orientation of the marker, and display a first model in a target space according to the first position and the first orientation, and in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation, and send the offset to the second wearable device; wherein the target space is a real space where the first wearable device and the second wearable device are located.

[0176] The second wearable device is used to detect the marker, obtain the second position and the second orientation of the marker, and display a second model in the target space according to the second position and the second orientation, and receive the offset sent by the first wearable device, and adjust the second model according to the offset, and the second model and the first model are the same model.

[0177] It should be noted that the first wearable device involved in this embodiment is the same as the first wearable device involved in Embodiments 1 and 2, and the second wearable device involved in this embodiment is the same as the second wearable device involved in Embodiments 1 and 2.

[0178] In this embodiment, the first wearable device and the second wearable device respectively detect the same marker, the first wearable device displays the first model in the target space according to the detected first position and first orientation, and the second wearable device displays the second model in the target space according to the detected second position and second orientation. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, and the second wearable device receives the offset and adjusts the second model, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also trigger the same operation, providing the user with a visual experience similar to operating the entity corresponding to the first model in the real world, and the models seen by the operator and the observer are always in the same position and orientation in the real world, thereby improving the user experience.

[0179] Example 4

[0180] Fig.10 This is a flow chart of a display method for a wearable device provided in this embodiment. The display method for a wearable device can be executed by a display device of the wearable device. The display device can be implemented by software and / or hardware. The display device can be part or all of the wearable device. The display method for a wearable device provided in this embodiment is introduced below with the first wearable device as the execution subject.

[0181] like Fig.10 As shown, the display method of the wearable device provided in this embodiment may include the following steps S31 to S35:

[0182] Step S31, displaying the first model in the target space according to the preset position and the preset orientation. In this embodiment, any position and any orientation can be used as the position and orientation of the first model. Specifically, the preset position and the preset orientation can be set according to the actual situation, and the preset position and the preset orientation are used as the position and orientation of the first model. The preset position is usually represented by three-dimensional coordinates, and the preset orientation is usually represented by a direction vector; the first wearable device can know the preset position and the preset orientation.

[0183] Step S32: determining a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting the marker.

[0184] Specifically, the marker can be detected in advance using a sensor device built into the first wearable device. The sensor device may include an image acquisition device such as a camera, and may also include a gyroscope, etc. The first position obtained by detecting the marker is usually represented by a three-dimensional coordinate, and the first orientation obtained by detecting the marker refers to the direction in which the marker points, which is usually represented by a direction vector.

[0185] The preset position and the preset orientation are combined and defined as a preset posture, the first position and the first orientation are collectively defined as a first posture, and the conversion matrix between the preset posture and the first posture is the target transformation matrix mentioned in this embodiment.

[0186] Step S33, sending the target transformation matrix to the second wearable device; wherein the second wearable device is used to display the second model in the target space according to the target transformation matrix and the second position and the second orientation obtained by detecting the marker. Specifically, the target space is the real space where the first wearable device and the second wearable device are located, and the second model and the first model are the same model, usually a three-dimensional model. In a specific implementation, the first model and the second model can be three-dimensional models of human body parts, and the user of the first wearable device and the user of the second wearable device can be different doctors.

[0187] The second wearable device detects the marker to obtain the second position and the second orientation of the marker, and defines the second position and the second orientation together as the second posture. The second wearable device can transform the second posture through the target transformation matrix to obtain the target posture, and display the second model in the target posture. In this way, the connection between the first wearable device and the second wearable device is established through the marker, so that the first model and the second model have the same posture in the target space. In addition, in this embodiment, each wearable device does not need to project the model to the marker for display, nor does it need to set the marker at the position where the model is to be displayed, which improves the flexibility of use.

[0188] Step S34: In response to the operation on the first model, determine the offset of the first model in the target space before and after the operation.

[0189] The operation on the first model may include at least one of movement, rotation and scaling, and correspondingly, the offset may include at least one of position offset, orientation offset and size offset. In a specific implementation, the position offset Δ may be calculated using the following formula: p , Direction offset Δ r , size offset Δ s :

[0190] Δ p =P new -P old ;

[0191] Δ r =R new -R old ;

[0192] Δs =S new -S old ;

[0193] Among them, P new is the position of the first model after the operation, P old is the position of the first model before operation, R new is the orientation of the first model after the operation, R old is the orientation of the first model before the operation, S new is the size of the first model after the operation, S old is the size of the first model before the operation.

[0194] Step S35: sending the offset to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset. Specifically, if the offset includes a position offset and an orientation offset, the position of the second model in the target space can be adjusted according to the position offset, and the orientation of the second model in the target space can be adjusted according to the orientation offset.

[0195] Among them, the number of second wearable devices can be one or more. It should be noted that the first wearable device and the second wearable device are at different positions in the target space, for example, the first wearable device and the second wearable device can be at different positions in the same room. The marker can be any object in the target space, and its position is usually fixed. For example, the marker can be a painting on the wall. In a specific implementation, the marker can also have an orientation feature, that is, the marker itself contains information that can indicate the orientation. The marker can also be a fixed-position marking point. In a specific implementation, the target space can be an operating room or a consultation room, etc., and different doctors wear the first wearable device and the second wearable device to realize the sharing of the three-dimensional model of a certain part of the patient, so as to better assist medical diagnosis.

[0196] In this embodiment, the user of the first wearable device can see the first model displayed in the target space but cannot see the second model, and the user of the second wearable device can see the second model displayed in the target space but cannot see the first model. It should be noted that users other than the first wearable device cannot see the first model displayed in the target space, and users other than the second wearable device cannot see the second model displayed in the target space.

[0197] In this embodiment, the first wearable device and the second wearable device can detect the same marker in advance, and the first wearable device determines the target transformation matrix according to the preset position and preset orientation of the first model and the first position and first orientation of the marker detected, and the second wearable device displays the second model in the target space according to the target transformation matrix and the second position and second orientation of the marker detected. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, so that the second wearable device can adjust the second model according to the offset, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also be triggered by the same operation, providing users with a visual experience similar to operating an entity corresponding to the first model in the real world, and the model seen by the operator and the observer is always in the same position and orientation in the real world, which improves the user experience. In addition, in this embodiment, each wearable device does not need to project the model to the marker for display, nor does it need to set the marker at the position where the model is to be displayed, which improves the flexibility of use.

[0198] In a specific example, the first model and the second model are the target model M. The user of the first wearable device can see the front of the target model A, and the user of the second wearable device can see the right side of the target model M. After the user of the first wearable device rotates the target model M to the right, the user of the first wearable device can see the front and left sides of the target model M, and the target model M seen by the user of the second wearable device will also be triggered to rotate right, and the user of the second wearable device can see the right side and front of the target model M.

[0199] In the specific implementation of step S35, the offset may be directly sent to the second wearable device. In this implementation, the first wearable device may be communicatively connected with the second wearable device, and the first wearable device directly sends the offset to the second wearable device.

[0200] In the specific implementation of step S35, the offset may also be sent to the second wearable device through the server. In this embodiment, the first wearable device is connected to the server for communication, the first wearable device sends the offset to the server, the server is connected to the second wearable device for communication, and the server forwards the offset to the second wearable device.

[0201] In an optional embodiment, Fig.11 As shown, the above step S32 specifically includes the following steps S321 to S323:

[0202] Step S321: Obtain the position and orientation of the marker in a first coordinate system obtained by detecting the marker; wherein the first coordinate system is the coordinate system of the first wearable device.

[0203] Step S322: convert the position of the marker in the first coordinate system into a first position in the first world coordinate system, and convert the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system.

[0204] It should be noted that the first coordinate system is the object coordinate system of the first wearable device. When the first wearable device moves or rotates, the first coordinate system also undergoes the same translation or rotation. The first world coordinate system is a coordinate system established with the initial position of the first wearable device as the origin, and the first coordinate system is a coordinate system established with the current position of the first wearable device as the origin.

[0205] Step S323: determine a target transformation matrix according to the first position and the preset position and the first orientation and the preset orientation.

[0206] In actual applications, if the first wearable device sends the offset to the second wearable device as soon as it detects it, the second wearable device will frequently adjust the second model. However, the amount of data of the second model is usually relatively large. If the second model is adjusted frequently, it will affect the performance of the second wearable device. In an optional implementation, the above step S35 specifically includes: if the offset meets the preset condition, the offset is sent to the second wearable device. Among them, the preset condition can be that the position offset reaches the corresponding position threshold, and the orientation offset reaches the corresponding orientation threshold, and the size offset reaches the corresponding size threshold. The preset condition can also be that one of the offsets reaches the corresponding threshold. The preset condition can also be that the weighted sum of the three offsets reaches a certain threshold, wherein the weight corresponding to each offset can be set according to the actual situation.

[0207] In this embodiment, the offset is sent to the second wearable device when the offset meets the preset conditions. When the offset does not meet the preset conditions, it is considered that the offset is small and will not affect the users of the first wearable device and the second wearable device sharing the same model. Therefore, the offset is not sent to the second wearable device at this time, thereby avoiding the situation where frequent sending of the offset to the second wearable device causes the performance of the second wearable device to deteriorate.

[0208] This embodiment also provides a display device for a wearable device, which is applied to a first wearable device, such as Fig.12As shown, the display device of the wearable device includes a first display module 81, a matrix determination module 82, a matrix sending module 83, an offset determination module 84 and an offset sending module 85. The first display module 81 is used to display a first model in a target space according to a preset position and a preset orientation. The matrix determination module 82 is used to determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker. The matrix sending module 83 is used to send the target transformation matrix to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker. The offset determination module 84 is used to determine the offset of the first model in the target space before and after the operation in response to an operation on the first model. The offset sending module 85 is used to send the offset to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset.

[0209] In an optional embodiment, the matrix determination module is specifically used to obtain the position and orientation of the marker obtained by detecting the marker in a first coordinate system, wherein the first coordinate system is the coordinate system of the first wearable device, and convert the position of the marker in the first coordinate system into a first position in a first world coordinate system, convert the orientation of the marker in the first coordinate system into a first orientation in the first world coordinate system, and determine the target transformation matrix based on the first position and the preset position and the first orientation and the preset orientation.

[0210] In an optional implementation, the offset sending module is specifically configured to send the offset to the second wearable device when the offset meets a preset condition.

[0211] It should be noted that the display device of the wearable device in this embodiment may specifically be a separate chip, a chip module or a wearable device, or may be a chip or a chip module integrated in the wearable device.

[0212] Regarding the various modules / units included in the display device of the wearable device described in this embodiment, they can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units.

[0213] Example 5

[0214] Fig.13This is a flow chart of a display method for a wearable device provided in this embodiment. The display method for the wearable device can be executed by a display device of the wearable device. The display device can be implemented by software and / or hardware. The display device can be part or all of the wearable device. The display method for the wearable device provided in this embodiment is introduced below with the second wearable device as the execution subject.

[0215] like Fig.13 As shown, the display method of the wearable device provided in this embodiment may include the following steps S41 to S44:

[0216] Step S41, receiving a target transformation matrix sent by a first wearable device; wherein the first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, and determine the target transformation matrix according to the preset position, the preset orientation, and the first position and the first orientation obtained by detecting a marker.

[0217] Among them, any position and any orientation can be used as the position and orientation of the first model. Specifically, a preset position and a preset orientation can be set according to actual conditions, and the preset position and the preset orientation are used as the position and orientation of the first model. The preset position is usually represented by three-dimensional coordinates, and the preset orientation is usually represented by a direction vector; the first wearable device can know the preset position and the preset orientation. Specifically, the preset position and the preset orientation are combined and defined as a preset posture, and the first position and the first orientation are jointly defined as the first posture. The conversion matrix between the preset posture and the first posture is the target transformation matrix mentioned in this embodiment.

[0218] Step S42: display a second model in the target space according to the target transformation matrix and the second position and the second orientation obtained by detecting the marker.

[0219] Specifically, the marker can be detected in advance using a built-in sensor device in the second wearable device. The sensor device may include an image acquisition device such as a camera, and may also include a gyroscope, etc. The second position obtained by detecting the marker is usually represented by a three-dimensional coordinate, and the second orientation obtained by detecting the marker refers to the direction in which the marker faces or turns, and is usually represented by a direction vector.

[0220] In this embodiment, the position of the marker is not directly used as the position of the second model, nor is the orientation of the marker directly used as the orientation of the second model. Instead, the position and orientation of the marker need to be transformed using the target transformation matrix, and the transformed position and orientation are used as the position and orientation of the second model. Specifically, the second wearable device obtains the second position and second orientation of the marker by detecting the marker, and defines the second position and second orientation together as the second posture. The second wearable device can transform the second posture through the target transformation matrix to obtain the target posture, and display the second model in the target posture. In this way, the connection between the first wearable device and the second wearable device is established through the marker, so that the first model and the second model have the same posture in the target space. In addition, in this embodiment, each wearable device does not need to project the model to the marker for display, nor does it need to set the marker at the position where the model is to be displayed, which improves the flexibility of use.

[0221] Step S43: receiving the offset sent by the first wearable device; wherein the first wearable device is used to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model.

[0222] The first model and the second model are the same model, usually a three-dimensional model. In a specific implementation, the first model and the second model can be three-dimensional models of human body parts, and the user of the first wearable device and the user of the second wearable device can be different doctors.

[0223] The operations on the first model may include movement, rotation, scaling, etc. Correspondingly, the offset may include position offset, orientation offset, size offset, etc.

[0224] It should be noted that the first wearable device and the second wearable device are at different positions in the target space, for example, the first wearable device and the second wearable device can be at different positions in the same room. The marker can be any object in the target space, and its position is usually fixed. For example, the marker can be a painting on the wall. In a specific implementation, the marker can also have an orientation feature, that is, the marker itself contains information that can indicate the orientation. The marker can also be a fixed-position marking point. In a specific implementation, the target space can be an operating room or a consultation room, etc., and different doctors wear the first wearable device and the second wearable device to realize the sharing of the three-dimensional model of a certain part of the patient, so as to better assist medical diagnosis.

[0225] Step S44: adjusting the second model according to the offset. Specifically, calculating a new position, a new orientation, and / or a new size according to the offset, updating the current position of the second model to the new position, updating the current orientation of the second model to the new orientation, and updating the current size of the second model to the new size.

[0226] In one specific implementation example, the offset includes a position offset and an orientation offset, and the position of the second model in the target space can be adjusted according to the position offset, and the orientation of the second model in the target space can be adjusted according to the orientation offset. In another specific implementation example, the offset includes a position offset and a size offset, and the position of the second model in the target space can be adjusted according to the position offset, and the size of the second model in the target space can be adjusted according to the size offset.

[0227] In another example of a specific implementation, the offset includes a position offset Δ p , Direction offset Δ r and size offset Δ s , the new position P can be calculated using the following formula new 、New Direction R new , new size S new :

[0228] P new =P cur +Δ p ;

[0229] R new =R cur +Δ r ;

[0230] S new =S cur +Δ s ;

[0231] Among them, P cur is the current position of the second model, R cur is the current orientation of the second model, S cur is the current size of the second model.

[0232] It should be noted that the first wearable device involved in this embodiment is the same as the first wearable device involved in Example 4. In this embodiment, the user of the first wearable device can see the first model displayed in the target space but cannot see the second model, and the user of the second wearable device can see the second model displayed in the target space but cannot see the first model. It should be noted that users other than the first wearable device cannot see the first model displayed in the target space, and users other than the second wearable device cannot see the second model displayed in the target space.

[0233] In this embodiment, the first wearable device and the second wearable device can detect the same marker in advance, and the first wearable device determines the target transformation matrix according to the preset position and preset orientation of the first model and the first position and first orientation of the marker detected, and the second wearable device displays the second model in the target space according to the target transformation matrix and the second position and second orientation of the marker detected. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, so that the second wearable device can adjust the second model according to the offset, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also be triggered by the same operation, providing users with a visual experience similar to operating an entity corresponding to the first model in the real world, and the model seen by the operator and the observer is always in the same position and orientation in the real world, which improves the user experience. In addition, in this embodiment, each wearable device does not need to project the model to the marker for display, nor does it need to set the marker at the position where the model is to be displayed, which improves the flexibility of use.

[0234] In an optional embodiment, Fig.14 As shown, the above step S42 may specifically include the following steps S421 to S423:

[0235] Step S421: Obtain the position and orientation of the marker in a second coordinate system obtained by detecting the marker; wherein the second coordinate system is the coordinate system of the second wearable device.

[0236] Step S422: convert the position of the marker in the second coordinate system into a second position in the second world coordinate system, and convert the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system.

[0237] It should be noted that the second coordinate system is the object coordinate system of the second wearable device. When the second wearable device moves or rotates, the second coordinate system also undergoes the same translation or rotation. The second world coordinate system is a coordinate system established with the initial position of the second wearable device as the origin, and the second coordinate system is a coordinate system established with the current position of the second wearable device as the origin.

[0238] In a specific implementation, the second coordinate system to the second world coordinate system can be converted according to the following formula:

[0239]

[0240]

[0241]

[0242] P world2 =R z R y R x P obj2 +T Cam2World2 ;

[0243]

[0244] Among them, P obj2 is the position of the marker in the second coordinate system, P world2 is the second position of the marker in the second world coordinate system, T Cam2World2 is the relative offset between the second wearable device and the second world coordinate system, that is, the position of the second wearable device in the second world coordinate system, is the orientation of the marker in the second coordinate system, is the first orientation of the marker in the second world coordinate system, R x is the rotation matrix of the x-axis in the second coordinate system, R y is the rotation matrix of the y axis in the second coordinate system, R z is the rotation matrix of the z-axis in the second coordinate system, α is the angle of rotation around the x-axis in the second coordinate system, β is the angle of rotation around the y-axis in the second coordinate system, and γ is the angle of rotation around the z-axis in the second coordinate system.

[0245] Step S423: display a second model in the target space according to the second position, the second orientation, and the target transformation matrix.

[0246] In practical applications, the amount of data of the second model is usually relatively large. If the second wearable device adjusts the second model as soon as it receives the offset, the performance of the second wearable device will be reduced. In an optional implementation, the above step S44 specifically includes: if the offset meets the preset condition, adjusting the second model according to the offset. Among them, the preset condition can be that the position offset reaches the corresponding position threshold, and the orientation offset reaches the corresponding orientation threshold, and the size offset reaches the corresponding size threshold. The preset condition can also be that one of the offsets reaches the corresponding threshold. The preset condition can also be that the weighted sum of the three offsets reaches a certain threshold, wherein the weight corresponding to each offset can be set according to the actual situation.

[0247] In this embodiment, when the offset meets the preset conditions, the second wearable device adjusts the second model according to the offset. When the offset does not meet the preset conditions, it is considered that the offset is small and will not affect the users of the second wearable device and the first wearable device sharing the same model. Therefore, the second wearable device does not adjust the second model at this time, avoiding the situation where frequent adjustment of the second model leads to a decrease in the performance of the second wearable device.

[0248] This embodiment also provides a display device for a wearable device, which is applied to a second wearable device, such as Fig.15 As shown, the display device of the wearable device includes a matrix receiving module 91, a second display module 92, an offset receiving module 93 and a model adjustment module 94. The matrix receiving module 91 is used to receive the target transformation matrix sent by the first wearable device; wherein the first wearable device is used to display the first model in the target space according to the preset position and the preset orientation, and determine the target transformation matrix according to the preset position, the preset orientation and the first position and the first orientation obtained by detecting the marker. The second display module 92 is used to display the second model in the target space according to the target transformation matrix and the second position and the second orientation obtained by detecting the marker. The offset receiving module 93 is used to receive the offset sent by the first wearable device; wherein the first wearable device is used to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model. The model adjustment module 94 is used to adjust the second model according to the offset.

[0249] In an optional embodiment, the second display module is specifically used to obtain the position and orientation of the marker in a second coordinate system obtained by detecting the marker, wherein the second coordinate system is the coordinate system of the second wearable device, and convert the position of the marker in the second coordinate system into a second position in a second world coordinate system, convert the orientation of the marker in the second coordinate system into a second orientation in the second world coordinate system, and display the second model in the target space according to the second position, the second orientation and the target transformation matrix.

[0250] In an optional implementation manner, the model adjustment module is specifically configured to adjust the second model according to the offset when the offset satisfies a preset condition.

[0251] It should be noted that the display device of the wearable device in this embodiment may specifically be a separate chip, a chip module or a wearable device, or may be a chip or a chip module integrated in the wearable device.

[0252] Regarding the various modules / units included in the display device of the wearable device described in this embodiment, they can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units.

[0253] Example 6

[0254] This embodiment provides a display system for a wearable device, including a first wearable device and a second wearable device.

[0255] The first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker, and send the target transformation matrix to the second wearable device.

[0256] The second wearable device is used to receive the target transformation matrix sent by the first wearable device, and display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker.

[0257] The first wearable device is further used to determine, in response to an operation on the first model, an offset of the first model in the target space before and after the operation, and send the offset to the second wearable device.

[0258] The second wearable device is further used to receive the offset sent by the first wearable device and adjust the second model according to the offset.

[0259] It should be noted that the first wearable device involved in this embodiment is the same as the first wearable device involved in Embodiments 4 and 5, and the second wearable device involved in this embodiment is the same as the second wearable device involved in Embodiments 4 and 5.

[0260] In this embodiment, the first wearable device and the second wearable device can detect the same marker in advance, and the first wearable device determines the target transformation matrix according to the preset position and preset orientation of the first model and the first position and first orientation of the marker detected, and the second wearable device displays the second model in the target space according to the target transformation matrix and the second position and second orientation of the marker detected. When the user of the first wearable device operates the first model, the offset of the first model in the target space before and after the operation is sent to the second wearable device, so that the second wearable device can adjust the second model according to the offset, so that users of different wearable devices share the same model, that is, no matter how the user of the first wearable device operates the first model, the second model seen by the user of the second wearable device will also be triggered by the same operation, providing users with a visual experience similar to operating an entity corresponding to the first model in the real world, and the model seen by the operator and the observer is always in the same position and orientation in the real world, which improves the user experience. In addition, in this embodiment, each wearable device does not need to project the model to the marker for display, nor does it need to set the marker at the position where the model is to be displayed, which improves the flexibility of use.

[0261] Example 7

[0262] Fig.16 A schematic diagram of the structure of a wearable device provided in this embodiment. The wearable device includes at least one processor and a memory connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the display method of the wearable device of Embodiment 1, 2, 4 or 5. Fig.16 The wearable device 3 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0263] The components of the wearable device 3 may include, but are not limited to: the at least one processor 4 mentioned above, the at least one memory 5 mentioned above, and a bus 6 connecting different system components (including the memory 5 and the processor 4).

[0264] The bus 6 includes a data bus, an address bus and a control bus.

[0265] The memory 5 may include a volatile memory, such as a random access memory (RAM) 51 and / or a cache memory 52 , and may further include a read-only memory (ROM) 53 .

[0266] The memory 5 may also include a program / utility 55 having a set (at least one) of program modules 54, such program modules 54 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0267] The processor 4 executes various functional applications and data processing by running the computer program stored in the memory 5, such as the display method of the wearable device of the above-mentioned embodiment 1, 2, 4 or 5.

[0268] The wearable device 3 may also communicate with one or more external devices 7 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 8. Furthermore, the wearable device 3 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 9. Fig.16 As shown, the network adapter 9 communicates with other modules of the wearable device 3 via the bus 6. It should be understood that although Fig.16 Not shown, other hardware and / or software modules may be used in conjunction with the wearable device 3, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0269] It should be noted that although several units / modules or sub-units / modules of the wearable device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0270] Example 8

[0271] This embodiment provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the display method of the wearable device of Embodiment 1, 2, 4 or 5 is implemented.

[0272] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0273] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a wearable device, the program code is used to enable the wearable device to execute the display method of the wearable device implementing Example 1, 2, 4 or 5.

[0274] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the wearable device, partially on the wearable device, as an independent software package, partially on the wearable device and partially on a remote device, or completely on the remote device.

[0275] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A display method for a wearable device, applied to a first wearable device, characterized in that: include: Detecting the marker to obtain a first position and a first orientation of the marker; displaying a first model in a target space according to the first position and the first orientation; In response to an operation on the first model, determining an offset of the first model in the target space before and after the operation; The offset is sent to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the offset and a second position and a second orientation obtained by detecting the marker.

2. The display method according to claim 1, characterized in that: The detecting of the marker to obtain the first position and the first orientation of the marker includes: Detecting the marker to obtain the position and orientation of the marker in a first coordinate system; wherein the first coordinate system is the coordinate system of the first wearable device; The position of the marker in the first coordinate system is converted into a first position in the first world coordinate system, and the orientation of the marker in the first coordinate system is converted into a first orientation in the first world coordinate system.

3. The display method according to claim 1 or 2, characterized in that: The sending the offset to the second wearable device includes: If the offset meets the preset condition, the offset is sent to the second wearable device.

4. A display method for a wearable device, applied to a second wearable device, characterized in that: include: Detecting the marker to obtain a second position and a second orientation of the marker; displaying a second model in a target space according to the second position and the second orientation; Receiving an offset sent by a first wearable device; wherein the first wearable device is used to display a first model in the target space according to a first position and a first orientation obtained by detecting the marker, and in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation; The second model is adjusted according to the offset.

5. The display method according to claim 4, characterized in that: The detecting of the marker to obtain the second position and the second orientation of the marker includes: Detecting the marker to obtain the position and orientation of the marker in a second coordinate system; wherein the second coordinate system is the coordinate system of the second wearable device; The position of the marker in the second coordinate system is converted into a second position in the second world coordinate system, and the orientation of the marker in the second coordinate system is converted into a second orientation in the second world coordinate system.

6. The display method according to claim 4 or 5, characterized in that: The adjusting the second model according to the offset includes: If the offset meets a preset condition, the second model is adjusted according to the offset.

7. A display method for a wearable device, applied to a first wearable device, characterized in that: include: Displaying the first model in the target space according to the preset position and the preset orientation; Determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting the marker; Sending the target transformation matrix to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker; In response to an operation on the first model, determining an offset of the first model in the target space before and after the operation; The offset is sent to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset.

8. The display method according to claim 7, characterized in that: The determining of the target transformation matrix according to the preset position and the preset orientation and the first position and the first orientation obtained by detecting the marker includes: Acquire the position and orientation of the marker in a first coordinate system obtained by detecting the marker; wherein the first coordinate system is the coordinate system of the first wearable device; Converting the position of the marker in the first coordinate system to a first position in the first world coordinate system, and converting the orientation of the marker in the first coordinate system to a first orientation in the first world coordinate system; A target transformation matrix is ​​determined according to the first position and the preset position and the first orientation and the preset orientation.

9. The display method according to claim 7 or 8, characterized in that: The sending the offset to the second wearable device includes: If the offset meets the preset condition, the offset is sent to the second wearable device.

10. A display method for a wearable device, applied to a second wearable device, characterized in that: include: Receiving a target transformation matrix sent by a first wearable device; wherein the first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, and determining the target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker; Displaying a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker; Receiving the offset sent by the first wearable device; wherein the first wearable device is used to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model; The second model is adjusted according to the offset.

11. The display method according to claim 10, characterized in that: The displaying of the second model in the target space according to the target transformation matrix and the second position and the second orientation obtained by detecting the marker comprises: Acquire the position and orientation of the marker in a second coordinate system obtained by detecting the marker; wherein the second coordinate system is the coordinate system of the second wearable device; Converting the position of the marker in the second coordinate system to a second position in the second world coordinate system, and converting the orientation of the marker in the second coordinate system to a second orientation in the second world coordinate system; A second model is displayed in the target space according to the second position, the second orientation, and the target transformation matrix.

12. The display method according to claim 10 or 11, characterized in that: The adjusting the second model according to the offset includes: If the offset meets a preset condition, the second model is adjusted according to the offset.

13. A display device for a wearable device, applied to a first wearable device, characterized in that: include: A first detection module, used to detect the marker to obtain a first position and a first orientation of the marker; A first display module, configured to display a first model in a target space according to the first position and the first orientation; an offset determination module, configured to determine, in response to an operation on the first model, an offset of the first model in the target space before and after the operation; An offset sending module is used to send the offset to a second wearable device; wherein the second wearable device is used to display a second model in the target space according to the offset and a second position and a second orientation obtained by detecting the marker.

14. A display device for a wearable device, applied to a second wearable device, characterized in that: include: A second detection module is used to detect the marker to obtain a second position and a second orientation of the marker; A second display module, configured to display a second model in a target space according to the second position and the second orientation; an offset receiving module, configured to receive an offset sent by a first wearable device; wherein the first wearable device is configured to display a first model in the target space according to a first position and a first orientation obtained by detecting the marker, and to determine an offset of the first model in the target space before and after the operation in response to an operation on the first model; A model adjustment module is used to adjust the second model according to the offset.

15. A display system for a wearable device, characterized in that: including a first wearable device and a second wearable device; The first wearable device is used to detect the marker, obtain a first position and a first orientation of the marker, display a first model in a target space according to the first position and the first orientation, and determine an offset of the first model in the target space before and after the operation in response to an operation on the first model, and send the offset to the second wearable device; The second wearable device is used to detect the marker, obtain the second position and the second orientation of the marker, display the second model in the target space according to the second position and the second orientation, receive the offset sent by the first wearable device, and adjust the second model according to the offset.

16. A display device for a wearable device, applied to a first wearable device, characterized in that: include: A first display module, used for displaying a first model in a target space according to a preset position and a preset orientation; A matrix determination module, configured to determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting the marker; A matrix sending module, used for sending the target transformation matrix to a second wearable device; wherein the second wearable device is used for displaying a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker; an offset determination module, configured to determine, in response to an operation on the first model, an offset of the first model in the target space before and after the operation; An offset sending module is used to send the offset to the second wearable device; wherein the second wearable device is used to adjust the second model according to the offset.

17. A display device for a wearable device, applied to a second wearable device, characterized in that: include: A matrix receiving module, configured to receive a target transformation matrix sent by a first wearable device; wherein the first wearable device is configured to display a first model in a target space according to a preset position and a preset orientation, and to determine the target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker; A second display module, configured to display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker; an offset receiving module, configured to receive the offset sent by the first wearable device; wherein the first wearable device is configured to determine the offset of the first model in the target space before and after the operation in response to the operation on the first model; A model adjustment module is used to adjust the second model according to the offset.

18. A display system for a wearable device, characterized in that: including a first wearable device and a second wearable device; The first wearable device is used to display a first model in a target space according to a preset position and a preset orientation, determine a target transformation matrix according to the preset position, the preset orientation, and a first position and a first orientation obtained by detecting a marker, and send the target transformation matrix to the second wearable device; The second wearable device is used to receive the target transformation matrix sent by the first wearable device, and display a second model in the target space according to the target transformation matrix and a second position and a second orientation obtained by detecting the marker; The first wearable device is further configured to, in response to an operation on the first model, determine an offset of the first model in the target space before and after the operation, and send the offset to the second wearable device; The second wearable device is further used to receive the offset sent by the first wearable device and adjust the second model according to the offset.

19. A wearable device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the display method according to any one of claims 1 to 12 is implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the display method according to any one of claims 1 to 12 is implemented.

Images