Virtual object display method and device and computer equipment

Abstract

The invention relates to a virtual object display method and device and computer equipment. The method comprises the following steps: displaying a real object and a virtual object corresponding to the real object in a display interface; and periodically changing the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface. By adopting the method, the illusion of position judgment when the virtual object is displayed can be reduced, and the accuracy of position judgment of the real object and the virtual object in the display interface is improved.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular, to a method, device, and computer device for displaying virtual objects. Background Art

[0002] In open surgery, considering factors such as surgical risks and postoperative recovery, the surgical incision is usually controlled within a certain range, so there is a problem of limited vision. Augmented Reality (AR) technology can superimpose virtual objects on the real scene in the field of vision to achieve the effect of enhancing the field of vision. During surgery, doctors can see the real scene integrated with three-dimensional virtual surgical organs through augmented reality display devices, and establish a complete understanding of the three-dimensional structure and spatial position of the organs, thus overcoming the limited vision.

[0003] However, the representation form of virtual objects in augmented reality is usually triangular meshes. When fusing and rendering virtual organs with the real scene, if the meshes are directly rendered, due to their occlusion effect on the real scene, it is easy to create an illusion in the eyes of the observer that the virtual object floats above or in front of the real object, affecting the observer's judgment of the real positions of the virtual object and the real object in three-dimensional space.

[0004] Therefore, there is a problem of inaccurate position judgment in current virtual object display technologies. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method, device, computer device, computer-readable storage medium, and computer program product for displaying virtual objects that can accurately determine the positions of virtual objects and real objects.

[0006] In a first aspect, this application provides a method for displaying virtual objects, including:

[0007] Displaying a real object and a virtual object corresponding to the real object in a display interface;

[0008] Periodically changing the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

[0009] In a second aspect, this application further provides a device for displaying virtual objects, including:

[0010] A display module for displaying a real object and a virtual object corresponding to the real object in a display interface;

[0011] A flashing module for periodically changing the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

[0012] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0013] Display a real object and a virtual object corresponding to the real object on a display interface;

[0014] Periodically change the transparency of the virtual object to determine the positions of the real object and the virtual object on the display interface.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0016] Display a real object and a virtual object corresponding to the real object on a display interface;

[0017] Periodically change the transparency of the virtual object to determine the positions of the real object and the virtual object on the display interface.

[0018] In a fifth aspect, the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0019] Display a real object and a virtual object corresponding to the real object on a display interface;

[0020] Periodically change the transparency of the virtual object to determine the positions of the real object and the virtual object on the display interface.

[0021] For the above virtual object display method, device, computer device, computer-readable storage medium, and computer program product, by displaying a real object and a virtual object corresponding to the real object on a display interface and periodically changing the transparency of the virtual object, when the transparency of the virtual object is relatively high and the occlusion of the real object is reduced, the position of the real object on the display interface can be accurately determined, and when the transparency of the virtual object is relatively low, the position of the virtual object on the display interface can be accurately determined, thereby reducing the illusion of position judgment during the display of the virtual object and improving the accuracy of position judgment for the real object and the virtual object on the display interface. Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0023] Figure 1 It is a schematic flowchart of a method for displaying a virtual object in an embodiment;

[0024] Figure 2 It is a schematic diagram of a surgical guide plate, a fiducial marker bracket, and fiducial markers installed at a vertebral bone site in an embodiment;

[0025] Figure 3 It is a schematic diagram of an operation interface for aligning a virtual object and a real object in an embodiment;

[0026] Figure 4 It is a schematic diagram of an operation interface for adjusting the display effect of a virtual object in an embodiment;

[0027] Figure 5 It is a schematic diagram of a virtual vertebra in an embodiment;

[0028] Figure 6 It is a schematic diagram of performing a stroke display on a virtual vertebra in an embodiment;

[0029] Figure 7 It is a schematic diagram of displaying the skeleton of a virtual guide plate and the bounding box of a fiducial marker bracket in an embodiment;

[0030] Figure 8 It is a schematic diagram of displaying the skeleton of a virtual guide plate, the bounding box of a fiducial marker bracket, and the centerline of a virtual nail track in an embodiment;

[0031] Figure 9 It is a schematic flowchart of a method for displaying a virtual object in another embodiment;

[0032] Figure 10 It is a structural block diagram of a display device for a virtual object in an embodiment. Detailed implementation manners

[0033] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further details the present application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of the present disclosure are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] Before introducing the specific embodiments of the present application, the professional terms involved in the present application are first explained:

[0036] AR sensing device: A sensing device used to scan the real scene during surgery, on which sensors such as optical cameras, depth cameras, lidar, etc. are provided, and can be, but is not limited to, various head-mounted devices, or other non-head-mounted devices with optical cameras, depth cameras, lidar, etc.;

[0037] AR display device: A device used to superimpose and display virtual objects and real scenes, on which a display is provided, and can be, but is not limited to, various head-mounted display devices, or displays, mobile phones, tablets, etc.;

[0038] Preoperative image: A three-dimensional image scan performed on the surgical object before surgery, including but not limited to CT, Magnetic Resonance (MR), etc., which records the anatomical features of the target part of the surgical object;

[0039] Surgical guide: A guide used to guide the doctor to perform surgery during the operation, which is a customized guide designed according to the anatomical features of the relevant parts in the preoperative image of the surgical object, and is manufactured and formed by 3D printing;

[0040] Reference marker and its installation and fixation: The reference marker is a bracket with a characteristic pattern that can be fixed on the surgical site and is used for the initial registration of the virtual model and the real scene during the operation. The reference marker bracket can be designed according to the anatomical features of the surgical site, formed by 3D printing, installed and fixed on the corresponding surgical site, and its geometric features are all known, such as the position and deflection angle of the reference marker plane, etc.;

[0041] Reference marker tracking algorithm: Used to detect and track the pose of the reference marker in the real scene in real time;

[0042] Manual Fine-tuning Alignment Operation Interface: A human-computer interaction interface that can control the position and posture of virtual objects and / or real objects, used to align virtual objects with the real scene. Exemplarily, this operation interface can be, but is not limited to, an application independently developed on a head-mounted display device, or desktop, mobile operating systems, etc. The operation methods include, but are not limited to, using external devices such as gestures, joysticks, keyboards, mice, microphones, touchscreens, etc. to control the components in the interface. The form of the interface can be in window form or windowless form;

[0043] Display Effect Adjustment Operation Interface: A human-computer interaction interface for adjusting the display of virtual objects in ways such as color, transparency, or blinking, including implementation methods on different operating system platforms. For example, a head-mounted AR operating system, or desktop, mobile operating systems, etc. The operation methods include, but are not limited to, using external devices such as gestures, joysticks, keyboards, mice, microphones, touchscreens, etc. The form of the interface can be in window form or windowless form.

[0044] The following uses specific embodiments to elaborate in detail on the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems. These several specific embodiments can be combined with each other. Concepts or processes that are the same or similar may not be repeated in some embodiments. The following will describe the embodiments of the present application in conjunction with the accompanying drawings.

[0045] In an exemplary embodiment, as Figure 1 shown, a method for displaying virtual objects is provided. This embodiment takes the application of this method to an AR device as an example for illustration. Among them, the AR device includes an AR sensing device and an AR display device. It can be understood that this method can also be applied to other terminals, which can be, but are not limited to, various personal computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and other wearable devices, etc. It can also be applied to a server or a system including a terminal and a server. In this embodiment, the method includes the following steps:

[0046] Step S102, display a real object and a virtual object corresponding to the real object in the display interface.

[0047] Among them, the display interface can be an interface for displaying virtual objects and real objects, including but not limited to the interfaces displayed by various displays, display screens, or the interfaces formed by projecting through optical devices into three-dimensional space. The real object can be a human body part, surgical instrument, etc. in the real scene. The virtual object can be a virtual human body part, surgical instrument, etc.

[0048] In specific implementation, the AR device can display the real object in the display interface, and can also obtain the information of the virtual object and display the virtual object in the display interface according to this information.

[0049] In practical applications, taking lumbar spine surgery as an example, during the operation, only the spine is exposed to the root of the articular process (the position where the screw is inserted), and deeper structures such as the transverse process of the spine are not further exposed, so as to reduce the harm to the patient during the operation. In order to determine whether the screw insertion point and angle are adapted to the complete three-dimensional structure of the spine, currently, multiple X-ray fluoroscopies are usually performed during the operation to confirm and adjust the screw insertion point and angle. In order to reduce the number of X-ray fluoroscopies during the operation and reduce the radiation risk, through an AR display device, a real scene integrated with a three-dimensional virtual surgical organ can be displayed, so as to establish a complete understanding of the three-dimensional structure and spatial position of the organ, and overcome the difficulty of limited intraoperative vision. However, this requires aligning the virtual surgical organ with the real organ in the surgical field and displaying the virtual object in the doctor's field of vision. Due to the occlusion effect of the virtual object on the real scene, it is easy to create an illusion in the observer's eyes that the virtual object floats above or in front of the real object, affecting the observer's judgment of the real positions of the virtual object and the real object in three-dimensional space, and further affecting the alignment of the virtual object and the real object.

[0050] In this embodiment, the AR device can project the human body part and / or surgical instrument in the real scene onto the display interface, obtain preoperative images, project virtual human body parts according to the information provided by the preoperative images, and can also perform preoperative planning based on the preoperative images to obtain the relative position relationship between the intraoperative surgical instrument and the human body part. According to this relative position relationship, taking the virtual human body part as a reference, virtual surgical instruments are projected.

[0051] Step S104, periodically change the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

[0052] In specific implementation, the AR device can periodically adjust the transparency of the virtual object in the display interface. When the transparency of the virtual object is relatively high, the occlusion of the virtual object on the real object can be reduced, and the position of the real object in the display interface can be accurately determined. When the transparency of the virtual object is relatively low, the clarity of the virtual object is relatively high, and the position of the virtual object in the display interface can be accurately determined.

[0053] It can be understood that by periodically changing the transparency of the virtual object, a flickering effect of the virtual object in the display interface can be formed, which is convenient for observing the actual positions of the virtual object and the real object at the same time, and then aligning the virtual object with the real object. In practical applications, the virtual human body part can be displayed in a flickering manner to facilitate aligning the virtual human body part with the real human body part, and the virtual surgical instrument can also be displayed in a flickering manner to facilitate aligning the real surgical instrument with the virtual surgical instrument. In addition, all virtual human body parts, surgical instruments, etc. in the display interface can be made to flicker synchronously to avoid visual interference caused by asynchronous flickering and improve the alignment efficiency of the virtual object and the real object.

[0054] The above method for displaying virtual objects can accurately determine the position of the real object in the display interface when the transparency of the virtual object is high and the occlusion of the real object is reduced, and can accurately determine the position of the virtual object in the display interface when the transparency of the virtual object is low, by displaying the real object and the virtual object corresponding to the real object in the display interface and periodically changing the transparency of the virtual object. This can reduce the illusion of position judgment when the virtual object is displayed and improve the accuracy of position judgment of the real object and the virtual object in the display interface.

[0055] In an exemplary embodiment, the above step S104 may specifically include: obtaining the timestamp of the virtual object; and determining the transparency of the virtual object according to the timestamp and a predetermined periodic function.

[0056] Among them, the timestamp may be a time tag added to the virtual object. The periodic function may be a function with periodic fluctuations, including but not limited to trigonometric functions, etc.

[0057] In specific implementation, a timestamp may be added to the virtual object and a periodic function may be set. The AR device can read the timestamp of the virtual object, substitute the timestamp into the periodic function to obtain the value of the transparency of the virtual object, and adjust the transparency of the virtual object according to this value.

[0058] In practical applications, the internal time of the AR device can be used as the timestamp of the virtual object. When it is necessary to make multiple virtual objects in the display interface synchronously blink and display, multiple virtual objects can use the same timestamp and the same periodic function. For example, multiple virtual objects in the display interface can use the same trigonometric function and uniformly use the current internal time of the device as the timestamp; when it is necessary to make multiple virtual objects in the display interface blink and display in turn, multiple virtual objects can use different timestamps and the same periodic function. For example, multiple virtual objects in the display interface can use the same trigonometric function, and for different virtual objects, different time delays are added to the current internal time of the device as the timestamps of each virtual object. Further, the time delays corresponding to different virtual objects can also have a certain regularity, such as an arithmetic progression, for use in automatically aligning virtual objects to real objects or enhancing the orderliness of the alternating display.

[0059] In this embodiment, by obtaining the timestamp of the virtual object and determining the transparency of the virtual object according to the timestamp and a predetermined periodic function, the frequency and phase of the blinking display of the virtual object can be adjusted using the timestamp, so as to achieve synchronous blinking display or alternating blinking display of multiple virtual objects, etc., and enhance the display effect of the virtual object.

[0060] In an exemplary embodiment, there are at least two virtual objects, and each virtual object corresponds to the same timestamp and the same periodic function to synchronously flash and display each virtual object in the display interface. Alternatively, each virtual object corresponds to different timestamps and the same periodic function to alternately flash and display each virtual object in the display interface.

[0061] In a specific implementation, the AR device can make all virtual objects in the display interface use the same timestamp and the same periodic function to synchronously flash and display all virtual objects in the display interface. It can also make all virtual objects in the display interface use different timestamps and the same periodic function to alternately flash and display all virtual objects in the display interface. Among them, different timestamps can also have a certain regularity. For example, the time delay between adjacent timestamps can form an arithmetic progression, so as to be applied to automatically align virtual objects to real objects or enhance the orderliness of alternate display.

[0062] In this embodiment, by making each virtual object correspond to the same timestamp and the same periodic function, all virtual objects in the display interface can be synchronously flashed and displayed, avoiding the influence of asynchronous flashing of virtual objects on the alignment effect. By making each virtual object correspond to different timestamps and the same periodic function, all virtual objects in the display interface can be alternately flashed and displayed, enhancing the orderliness of the display of virtual objects.

[0063] Figure 2 A schematic diagram of placing a surgical instrument at a human body part during a vertebra surgery is provided. Among them, the human body part can specifically be a vertebra part. The surgical instrument can include a surgical guide plate 202 placed on the vertebra, a reference marker bracket 204 connected to the surgical guide plate 202, and a reference marker 206 fixed on the reference marker bracket 204. The surgical guide plate is used for positioning and navigation during the surgery and is usually customized based on the patient's preoperative images. However, during the surgery, there may be soft tissue residues at the human body part where the guide plate is placed, resulting in a deviation between the actual placement position of the guide plate and the preoperative plan. To improve the accuracy of the surgery, this deviation needs to be corrected.

[0064] In an exemplary embodiment, the real part and the real instrument acting on the real part can be displayed in the display interface, and the initial virtual part can be displayed. The initial virtual part is adjusted to be aligned with the real part to obtain the adjusted virtual part. Based on the relative position relationship between the pre-determined real part and the real instrument, the adjusted virtual part is used as a reference to display the virtual instrument, and the actual placement position of the real instrument is adjusted according to the virtual instrument.

[0065] Among them, the real part can be a human body part in a real scenario, for example, the vertebra part. The real instrument can be a surgical instrument in a real scenario, including but not limited to a surgical guide plate, a fiducial marker bracket, and fiducial markers. The virtual part can be a virtual human body part projected by an AR device. The virtual instrument can be a virtual surgical instrument projected by an AR device.

[0066] In specific implementation, a preoperative image of the real part can be obtained, the real instrument can be designed based on the preoperative image, and the relative position relationship between the real instrument and the real part can be planned when the real instrument is placed on the real part. During the operation, the real instrument can be placed on the real part, and the AR device can display the real part and the real instrument on the display interface, and display the initial virtual part according to the preoperative plan. The initial virtual part can be adjusted to align with the real part to obtain the adjusted virtual part. According to the relative position relationship between the real instrument and the real part planned preoperatively, the AR device can use the adjusted virtual part as a reference object and determine the position and pose of the virtual instrument to be displayed according to the relative position relationship, and display the virtual instrument according to the position and pose. Since the virtual part and the real part are aligned at this time, and the virtual instrument is generated based on the virtual part according to the preoperative plan and has the correct position and pose, adjusting the real instrument according to the virtual instrument can correct the deviation of the placement position of the real instrument and improve the accuracy of the operation.

[0067] Taking the vertebra surgery as an example for illustration, a preoperative image of the vertebra part can be obtained, designed and 3D printed according to the preoperative image to obtain a surgical guide plate and a fiducial marker bracket, and the position and pose of the surgical guide plate placed on the vertebra part can be planned to obtain the relative position relationship R1 between the surgical guide plate and the vertebra part. Since the fiducial marker is connected to the surgical guide plate through the fiducial marker bracket, the relative position relationship R2 between the fiducial marker and the vertebra part can also be obtained according to the position and pose of the surgical guide plate placed on the vertebra part. During the operation, the surgical guide plate, the fiducial marker bracket, and the fiducial markers are placed on the vertebra part. Affected by soft tissue residues, etc., the placement position may have deviations. To correct the deviations, the AR device can observe the vertebra part, the surgical guide plate, the fiducial marker bracket, and the fiducial markers in the real scenario. The AR device can also use the fiducial marker as a reference object on the display interface and display the virtual vertebra according to R2 to obtain the initial virtual vertebra. The initial virtual vertebra may not be aligned with the real vertebra. Adjust the initial virtual vertebra to align it with the real vertebra to obtain the adjusted virtual vertebra. Then, the virtual guide plate can be displayed according to R1 with the adjusted virtual vertebra as a reference object. Since the virtual guide plate is generated according to the preoperative plan and has the correct position and pose, the real guide plate can be aligned with the virtual guide plate to correct the deviation of the placement position of the real guide plate. Among them, the virtual vertebra refers to the virtual vertebra part. The real vertebra refers to the vertebra part in the real scenario. The virtual guide plate refers to the virtual surgical guide plate. The real guide plate refers to the surgical guide plate in the real scenario.

[0068] In view of the fact that the above correction process for the position deviation of the surgical guide plate requires aligning the virtual vertebra to the real vertebra and aligning the real guide plate to the virtual guide plate, in order to avoid the virtual objects (virtual vertebra and virtual guide plate) obscuring the real objects, resulting in inaccurate position judgment and further affecting the alignment effect, the present application also proposes a method for flashing display of virtual objects. In an exemplary embodiment, the virtual object includes a virtual part and a virtual instrument; the above step S104 may specifically include: periodically changing the transparency of the virtual part; displaying the virtual instrument on the display interface according to the virtual part; and periodically changing the transparency of the virtual instrument.

[0069] In specific implementation, the AR device can display the virtual part and periodically change the transparency of the virtual part to achieve the flashing display of the virtual part, facilitating the adjustment of the virtual part to be aligned with the real part. After the virtual part is adjusted to be aligned with the real part, the AR device can also use the virtual part as a reference to display the virtual instrument on the display interface according to the relative position relationship planned before the operation, and periodically change the transparency of the virtual instrument to achieve the flashing display of the virtual instrument, facilitating the adjustment of the real instrument to be aligned with the virtual instrument.

[0070] For example, during the operation, place the surgical guide plate, the fiducial marker bracket and the fiducial marker at the vertebra site. Observe the vertebra site, the surgical guide plate, the fiducial marker bracket and the fiducial marker in the real scene through the AR device. The AR device can use the fiducial marker as a reference object on the display interface and display the virtual vertebra according to the relative position relationship between the fiducial marker and the vertebra site planned before the operation to obtain the initial virtual vertebra. The initial virtual vertebra may not be aligned with the real vertebra. Flash-display the initial virtual vertebra and adjust it to be aligned with the real vertebra to obtain the adjusted virtual vertebra. Then, use the adjusted virtual vertebra as a reference object and display the virtual guide plate on the display interface according to the relative position relationship between the surgical guide plate and the vertebra site planned before the operation. Flash-display the virtual guide plate and adjust the real guide plate according to the virtual guide plate to align the real guide plate with the virtual guide plate.

[0071] It should be noted that the above process of aligning the virtual vertebra can be automatically aligned by the AR device, or the user can send an adjustment instruction to the AR device, and the AR device aligns according to the received adjustment instruction. The present application does not limit this; the above process of aligning the real guide plate can be aligned by the AR device controlling devices such as a robotic arm, or manually by the user. The present application does not limit this.

[0072] In this embodiment, by periodically changing the transparency of the virtual part and displaying a virtual instrument in the display interface according to the virtual part, and periodically changing the transparency of the virtual instrument, it is possible to avoid the virtual part and the virtual instrument from blocking the real part and the real instrument when aligning the virtual part to the real part and aligning the real instrument to the virtual instrument, thereby improving the accuracy of alignment.

[0073] In an exemplary embodiment, the real object includes a real part and a real instrument acting on the real part; after periodically changing the transparency of the virtual part, it may specifically further include: in response to a first operation instruction for the virtual part, aligning the virtual part to the real part; after periodically changing the transparency of the virtual instrument, it may specifically further include: in response to a second operation instruction for the real instrument, aligning the real instrument to the virtual instrument.

[0074] Among them, the first operation instruction may be information indicating adjustment of the position and attitude of the virtual part, including but not limited to instructions automatically generated by the AR device, or instructions sent by the user to the AR device through external devices such as gestures, handles, keyboards, mice, microphones, touchscreens, etc.

[0075] Among them, the second operation instruction may be information indicating adjustment of the position and attitude of the real instrument, including but not limited to instructions automatically generated by the AR device and sent to devices such as robotic arms, or instructions sent by the user to devices such as robotic arms through external devices such as gestures, handles, keyboards, mice, microphones, touchscreens, etc.

[0076] In specific implementation, the AR device can display the virtual part and periodically change the transparency of the virtual part to achieve a flashing display of the virtual part. When receiving the first operation instruction, the AR device can adjust the position and attitude of the virtual part according to the first operation instruction until the virtual part is aligned to the real part. The AR device can use the virtual part as a reference and display the virtual instrument in the display interface according to the relative position relationship planned before the operation, and periodically change the transparency of the virtual instrument to achieve a flashing display of the virtual instrument. When receiving the second operation instruction, the AR device can adjust the position and attitude of the real instrument according to the second operation instruction until the real instrument is aligned to the virtual instrument.

[0077] For example, Figure 3 a schematic diagram of an operation interface for aligning a virtual object and a real object is provided, and it is possible to Figure 3The operation interface shown is projected into a three-dimensional space. When it is necessary to adjust the head-tail direction of a virtual part or a real instrument, the head-tail direction of the virtual part or the real instrument can be rotated and / or translated through gesture operations. Similarly, rotation and / or translation can also be performed in the vertical or horizontal direction until the virtual part is aligned with the real part, or the real instrument is aligned with the virtual instrument.

[0078] In this embodiment, by responding to the first operation instruction for the virtual part to align the virtual part with the real part, and responding to the second operation instruction for the real instrument to align the real instrument with the virtual instrument, efficient and accurate alignment between the virtual part and the real part, and between the real instrument and the virtual instrument can be achieved.

[0079] In an exemplary embodiment, the above step S104 may specifically include: responding to a third operation instruction for a virtual object to adjust the display effect of the virtual object; the display effect includes periodically changing the transparency of the virtual object.

[0080] Among them, the third operation instruction may be information indicating an adjustment to the display effect of the virtual part, including but not limited to instructions automatically generated by the AR device, or instructions sent by the user to the AR device through external devices such as gestures, handles, keyboards, mice, microphones, touchscreens, etc. The display effect includes but not limited to transparency, color, blinking effect, etc., where the blinking effect includes but not limited to no blinking, synchronous blinking, alternating blinking, etc.

[0081] In specific implementation, when receiving the third operation instruction, the AR device can adjust the display effect of the virtual object according to the third operation instruction, including but not limited to adjusting the transparency, color, blinking effect, etc. of the virtual object.

[0082] In practical applications, the AR device can project the operation interface for changing the display effect of the virtual object into a three-dimensional space. When receiving a moving operation of the user on the operation interface, the AR device can adjust the position of the operation interface in the three-dimensional space according to the indication of the moving operation. When receiving a triggering operation of the user on the operation interface, the AR device can change the current display effect of the virtual object according to the virtual object indicated by the triggering operation and the display effect of the virtual object.

[0083] Figure 4 A schematic diagram of an operation interface for adjusting the display effect of a virtual object is provided. It is possible to Figure 4 project the graphical operation interface shown into a three-dimensional space. The user can drag the graphical operation interface to a suitable position in the three-dimensional space with their hand, and can also change the transparency, color, blinking effect, etc. of the virtual object by dragging a slider, clicking a button, etc. in the graphical operation interface.

[0084] In this embodiment, by responding to a third operation instruction for a virtual object and adjusting the display effect of the virtual object, the display effect of the virtual object can be changed through human-computer interaction, making the virtual object easier to observe.

[0085] In an exemplary embodiment, the virtual object includes a virtual part and a virtual instrument; the above step S102 may include at least one of the following: performing a stroke display on the virtual part in the display interface; displaying the skeleton and / or bounding box of the virtual instrument in the display interface; identifying a target display area of the virtual part and displaying the target display area in the display interface.

[0086] Among them, the target display area may be an area on the virtual part that needs to be displayed. For example, the top of the vertebral spinous process.

[0087] In specific implementation, the AR device can perform a stroke display on the virtual part in the display interface to only display the outline of the virtual part. The AR device can also display the skeleton, bounding box, etc. of the virtual instrument in the display interface. For example, display the skeleton of the surgical guide plate and the bounding box of the fiducial marker bracket in the display interface. In addition, the AR device can also identify the virtual part to obtain the target display area and only display the target display area in the display interface. For example, the top of the vertebral spinous process in the virtual vertebra can be identified through methods such as machine learning and neural networks, and the top of the vertebral spinous process is displayed in the display interface. By aligning the virtual top of the vertebral spinous process with the real top of the vertebral spinous process, rapid alignment of the virtual vertebra and the real vertebra can be achieved.

[0088] It should be noted that when performing a stroke display, a first virtual model and a second virtual model corresponding to the virtual vertebra can be generated. The volume of the first virtual model is larger than that of the second virtual model. When the virtual vertebra needs to be displayed, the second virtual model can be used to block the first virtual model, and the outline of the first virtual model can be exposed. By adjusting the transparency of the second virtual model and using the face culling operation, the stroke display of the virtual vertebra can be achieved.

[0089] In this embodiment, by performing a stroke display on the virtual part in the display interface, displaying the skeleton and / or bounding box of the virtual instrument in the display interface, identifying the target display area of the virtual part, and displaying the target display area in the display interface, the occlusion of the virtual object to the real object can be further reduced, facilitating the inspection of whether the virtual object is aligned with the real object.

[0090] To facilitate the in-depth understanding of the embodiments of the present application by those skilled in the art, a specific example will be described below.

[0091] The present application proposes a display method in augmented reality. When fusing virtual objects into a real surgical scene, it reduces the illusion that virtual objects float above or in front of real objects caused by the occlusion effect of virtual objects, thereby enabling a more accurate judgment of the position of virtual objects in the real scene. At the same time, the present application also provides an operation interface for the observer to adjust the display effect of virtual objects in real time, such as color, blinking effect, and opacity, etc., to adjust the fusion display effect of virtual objects in the real scene during surgery, thereby reducing the illusion of the observer when observing virtual objects, enhancing the judgment of the position of virtual objects, and ultimately reducing the number of X-ray fluoroscopies during surgery and reducing the risk of ionizing radiation.

[0092] In the following embodiments, four different display schemes applied to spinal surgery-assisted pedicle screw placement are proposed, enabling the observer to see both virtual objects and real objects at the same time. In addition, a graphical operation interface based on an AR display device is proposed, enabling the observer to adjust the display mode of virtual objects as needed.

[0093] In an exemplary embodiment, a stroke display method is proposed. In this display method, only the contour edges of virtual objects are displayed, and other parts of the object surface are not displayed. The displayed contour edges are three-dimensional object contours, closely attached to the surface of real objects, and can be updated in real time as the observer's viewing angle changes. In this display method, the observer can see both real objects and their virtual contours through the AR device.

[0094] Specifically, the stroke display method can be implemented by means of two virtual object models of different sizes (the first virtual model and the second virtual model). Utilize the occlusion effect between virtual objects (the virtual object in front can occlude the virtual object behind it), and by reasonably adjusting the transparency of the virtual object to be close to zero and using the operation of face culling, the observer can see the real surgical scene behind the approximately transparent three-dimensional model while seeing the outer contour of the virtual object. It can be understood that the present application does not limit the specific implementation manner of the stroke display. Any display method that visually appears to display the edge contour of two-dimensional or three-dimensional objects should be included.

[0095] Figure 5 A schematic diagram of a virtual vertebra displayed on a display interface is provided. The opacity of the virtual vertebra 301 is 1.0. At this time, due to the occlusion of the virtual vertebra on the real scene, the observer is prone to the illusion that the virtual vertebra is located above the real vertebra. Figure 6A schematic diagram of outlining and displaying a virtual vertebra in a display interface is provided, which shows the contour edge 302 of the virtual vertebra, the virtual vertebra 301 (opacity adjusted to be close to 0), and the top 303 of the vertebral spinous process. At this time, the observer can see both the virtual vertebra and the real scene simultaneously, thus greatly reducing the illusion of the position of the virtual vertebra.

[0096] In an exemplary embodiment, a flashing display method is proposed. The flashing display is to let the AR display system automatically adjust the opacity of the virtual object periodically, so that the observer visually sees the virtual object flickering. The specific implementation method can be to represent the opacity of the virtual object as a periodic function of the current timestamp, for example, using trigonometric functions, so as to obtain an opacity value that changes periodically with time. This display method can be used to check whether the virtual object coincides with the real object in the AR scene. It can be understood that the present application does not limit the specific implementation method of the flashing display, and any method that visually appears to periodically change the opacity of the virtual object should be included.

[0097] In an exemplary embodiment, a skeleton and bounding box display method is proposed. This display method is to present only the virtual skeleton or bounding box of the real object to the observer, rather than all the external features of the real object. In this display method, since the virtual object only partially occludes the real object, the observer can see both the real object and the virtual object at the same time, and can judge whether the virtual object is aligned with the key features of the real object. To implement this display method, a three-dimensional model of the virtual skeleton and bounding box needs to be made in advance and then displayed on the AR device.

[0098] Reference Figure 7 and Figure 8 , Figure 7 shows the contour edge 302 of the virtual vertebra, the skeleton 304 of the virtual guide plate, and the bounding box 305 of the reference marker bracket above it. Figure 8 shows the contour edge 302 of the virtual vertebra, the bounding box 305 of the reference marker bracket above the virtual guide plate, and the center line 306 of the virtual nail track. Here, it can be seen that the bounding box of the virtual guide plate is aligned with the reference marker bracket above the real guide plate, and the center line of the virtual nail track is aligned with the real Kirschner wire. It can be understood that the present application does not limit the surgical instruments mentioned in the skeleton and bounding box display, and can include any surgical instruments displayed in this way.

[0099] In an exemplary embodiment, a partial display method is proposed. This display method means to only display a part of the real object as a virtual object, so that the observer can see both the real object and the virtual object at the same time, and can judge whether the virtual object is aligned with the corresponding part of the real object. For example,Figure 6 The top 303 of the vertebral spinous process shown in

[0100] In an exemplary embodiment, a graphical operation interface is proposed, enabling the observer to change the transparency, color, and blinking effect of the virtual object in real time, Figure 4 Taking the real-time change of the transparency of the virtual object as an example, a schematic diagram of a graphical operation interface for adjusting the display effect of the virtual object is provided. This operation interface can be dragged by hand to a suitable position in the three-dimensional space according to the observer's needs. It can be understood that the adjustment method of the transparency described in the embodiments of the present application is not limited to the observer manually adjusting the overall transparency of the target object in the interface. The adjustment of the transparency can consider other forms. For example, the area of the target object whose transparency needs to be adjusted can be customized according to the user's needs to ensure the visibility of the key structure.

[0101] In addition, other ways to reduce the observer's misperception of the position of the virtual object can be considered. For example, by adding guiding lines or marking lines around the virtual object to help the observer better understand the spatial relationship between the virtual object and the real object and reduce the possibility of misjudgment.

[0102] It can be understood that the format of the three-dimensional model of the virtual object in the embodiments of the present application includes but is not limited to usdz. In addition, it can also be any other format that can be used to store and represent three-dimensional objects, such as.stl,.obj, etc. The color of the virtual object can be any color, and the present application does not limit this.

[0103] The present application proposes a fusion display method of a virtual object in a real scene to reduce the observer's misperception of the position of the virtual object in the real scene, thereby presenting an image that more conforms to the normal visual law. Specifically, the present application realizes the fusion of the virtual object and the real scene by displaying part of the features of the target object in the real scene. When simultaneously displaying the real scene and the virtual object to the human eye, the occlusion of the real object by part of the features enables the human brain to automatically process the visual relationship between the part of the features and the whole, thereby establishing the position cognition of the virtual object in the real scene in the human brain, and thus reducing the misperception of the position of the virtual object generated by the human brain. This method uses the principle of the human brain's cognitive process to realize the fusion display of the virtual object and the real scene, so there is no need for additional equipment or computational processes to estimate the occlusion relationship between the virtual object and the real scene, which is a simple and efficient display method. The display method of the virtual object proposed in the present application can be used to check whether the virtual object is aligned with the real scene.

[0104] In one embodiment, as Figure 9 shown, a method for displaying a virtual object is provided, including the following steps:

[0105] Step S401, display a real part and a real instrument acting on the real part in a display interface, and display a virtual part;

[0106] Step S402, periodically change the transparency of the virtual part according to a time stamp and a pre-determined periodic function, and adjust the virtual part to be aligned with the real part in response to a first operation instruction for the virtual part;

[0107] Step S403, display a virtual instrument in the display interface according to the virtual part;

[0108] Step S404, periodically change the transparency of the virtual instrument according to a time stamp and a pre-determined periodic function, and adjust the real instrument to be aligned with the virtual instrument in response to a second operation instruction for the real instrument.

[0109] In a specific implementation, a preoperative image of the real part can be obtained, the real instrument can be designed according to the preoperative image, and the relative position relationship between the real instrument and the real part when the real instrument is placed on the real part can be planned. During the operation, the real instrument can be placed on the real part, and the AR device can display the real part and the real instrument in the display interface, and display an initial virtual part according to the preoperative plan. According to the time stamp and the pre-determined periodic function, the transparency of the initial virtual part is periodically changed to make the initial virtual part flash. When a first operation instruction is received, the initial virtual part is adjusted to be aligned with the real part to obtain an adjusted virtual part. According to the relative position relationship between the real instrument and the real part planned preoperatively, the AR device can use the adjusted virtual part as a reference object to determine the position and attitude of the virtual instrument to be displayed according to the relative position relationship, and display the virtual instrument according to the position and attitude. According to the time stamp and the pre-determined periodic function, the transparency of the virtual instrument is periodically changed to make the virtual instrument flash. When a second operation instruction is received, the real instrument is adjusted to be aligned with the virtual instrument.

[0110] The above method for displaying a virtual object displays a real part and a real instrument acting on the real part in a display interface, and also displays a virtual part. According to a time stamp and a predetermined periodic function, the transparency of the virtual part is periodically changed. In response to a first operation instruction for the virtual part, the virtual part is adjusted to align with the real part. According to the virtual part, a virtual instrument is displayed in the display interface. According to the time stamp and the predetermined periodic function, the transparency of the virtual instrument is periodically changed. In response to a second operation instruction for the real instrument, the real instrument is adjusted to align with the virtual instrument. When the transparency of the virtual object is relatively high, the occlusion of the real object can be reduced, and the position of the real object in the display interface can be accurately determined. When the transparency of the virtual object is relatively low, the position of the virtual object in the display interface can be accurately determined. Therefore, the illusion of position judgment during the display of the virtual object can be reduced, and the accuracy of position judgment for the real object and the virtual object in the display interface can be improved.

[0111] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this document, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0112] Based on the same inventive concept, an embodiment of the present application also provides a virtual object display device for implementing the above-described virtual object display method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the virtual object display device provided below can refer to the limitations on the virtual object display method in the above text, and will not be repeated here.

[0113] In an exemplary embodiment, as Figure 10 shown, a virtual object display device is provided, including: a display module 502 and a flashing module 504, where:

[0114] The display module 502 is configured to display a real object and a virtual object corresponding to the real object in a display interface;

[0115] The flashing module 504 is configured to periodically change the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

[0116] In an exemplary embodiment, the above-mentioned blinking module 504 is further configured to obtain a timestamp of the virtual object; and determine the transparency of the virtual object according to the timestamp and a pre-determined periodic function.

[0117] In an exemplary embodiment, there are at least two of the above-mentioned virtual objects, and each of the virtual objects corresponds to the same timestamp and the same periodic function to synchronously blink and display each of the virtual objects in the display interface, or each of the virtual objects corresponds to different timestamps and the same periodic function to alternately blink and display each of the virtual objects in the display interface.

[0118] In an exemplary embodiment, the above-mentioned virtual object includes a virtual part and a virtual instrument; the above-mentioned blinking module 504 is further configured to periodically change the transparency of the virtual part; display the virtual instrument in the display interface according to the virtual part; and periodically change the transparency of the virtual instrument.

[0119] In an exemplary embodiment, the above-mentioned real object includes a real part and a real instrument acting on the real part; the display device of the above-mentioned virtual object further includes an alignment module, which is configured to align the virtual part to the real part in response to a first operation instruction for the virtual part; and align the real instrument to the virtual instrument in response to a second operation instruction for the real instrument.

[0120] In an exemplary embodiment, the above-mentioned blinking module 504 is further configured to adjust the display effect of the virtual object in response to a third operation instruction for the virtual object; the display effect includes periodically changing the transparency of the virtual object.

[0121] In an exemplary embodiment, the above-mentioned virtual object includes a virtual part and a virtual instrument; the above-mentioned display module 502 is further configured to perform a stroking display on the virtual part in the display interface; display the skeleton and / or bounding box of the virtual instrument in the display interface; identify a target display area of the virtual part, and display the target display area in the display interface.

[0122] Each module in the display device of the above-mentioned virtual object can be implemented in whole or in part by software, hardware, and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to each of the above-mentioned modules.

[0123] In an exemplary embodiment, a computer device is provided, which may be an augmented reality device. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for displaying a virtual object. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0124] Those skilled in the art can understand that the above structure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0125] In an embodiment, a computer device is further provided, which may be, but is not limited to, an augmented reality device, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0126] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0127] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0129] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0130] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0131] The above-described embodiments merely represent several implementation manners of this application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several deformations and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A method for displaying a virtual object, characterized in that, The method includes: displaying a real object and a virtual object corresponding to the real object in a display interface; periodically changing the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

2. The method according to claim 1, wherein The periodically changing the transparency of the virtual object includes: obtaining a timestamp of the virtual object; determining the transparency of the virtual object according to the timestamp and a pre-determined periodic function.

3. The method according to claim 2, wherein There are at least two virtual objects, and each virtual object corresponds to the same timestamp and the same periodic function to synchronously flash and display each virtual object in the display interface, or each virtual object corresponds to different timestamps and the same periodic function to alternately flash and display each virtual object in the display interface.

4. The method according to claim 1, characterized in that The virtual object includes a virtual part and a virtual instrument; the periodically changing the transparency of the virtual object includes: periodically changing the transparency of the virtual part; displaying a virtual instrument in the display interface according to the virtual part; periodically changing the transparency of the virtual instrument.

5. The method according to claim 4, wherein The real object includes a real part and a real instrument acting on the real part; after periodically changing the transparency of the virtual part, it further includes: in response to a first operation instruction for the virtual part, aligning the virtual part to the real part; after periodically changing the transparency of the virtual instrument, it further includes: in response to a second operation instruction for the real instrument, aligning the real instrument to the virtual instrument.

6. The method according to claim 1, characterized in that The periodically changing the transparency of the virtual object includes: in response to a third operation instruction for the virtual object, adjusting the display effect of the virtual object; the display effect includes periodically changing the transparency of the virtual object.

7. The method according to claim 1, characterized in that, The virtual object includes a virtual part and a virtual instrument; the displaying a real object and a virtual object corresponding to the real object in the display interface includes at least one of the following: performing a stroked display on the virtual part in the display interface; displaying the skeleton and / or bounding box of the virtual instrument in the display interface; identifying a target display area of the virtual part and displaying the target display area in the display interface.

8. A display device for virtual objects, characterized in that, The device includes: a display module for displaying a real object and a virtual object corresponding to the real object in a display interface; a flashing module for periodically changing the transparency of the virtual object to determine the positions of the real object and the virtual object in the display interface.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images