Three-dimensional visual field adjustment method, adjustment device and electronic equipment

Abstract

The invention discloses a three-dimensional visual field adjusting method and device and electronic equipment. The three-dimensional visual field adjusting method comprises the steps that three-dimensional models of multiple to-be-displayed target objects are acquired; acquiring pose information of the plurality of target objects in a coordinate system of the optical navigation system in a physical space; according to the three-dimensional model and the pose information, constructing and displaying a real-time three-dimensional visual field of view; in response to an adjustment instruction, updating and displaying the three-dimensional visual field; the three-dimensional visual view field comprises a target view field. According to the three-dimensional visual view field adjusting method and device and the electronic equipment, the target object and the target view field can be visually displayed, three-dimensional visual adjustment of the target view field is achieved, the target view field can be conveniently adjusted so that the target view field can completely cover the target object, and operation is convenient and visual.

Description

Technical Field

[0001] The present invention relates to a method, an adjusting device and an electronic device for adjusting a field of view, and in particular to a method, a device and an electronic device for adjusting a three-dimensional visualization field of view. Background Art

[0002] When the surgical robot registers and navigates based on NDI, it must first adjust the NDI field of view to meet the scene requirements. The current NDI field of view adjustment guidance method is based on the 2D three-view method. During the adjustment process, the reflective balls seen by NDI can fall into the three-view, i.e., the top view, front view, and side view recommended field of view frames. This type of adjustment strongly relies on the user's sense of spatial perception and transformation, and usually requires multiple attempts. The adjustment guidance is not intuitive and time-consuming. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the defects in the prior art and provide a three-dimensional visualization field of view adjustment method, adjustment device and electronic device that are convenient and intuitive while improving the field of view adjustment efficiency.

[0004] The present invention solves the above technical problems through the following technical solutions: a three-dimensional visualization field of view adjustment method, characterized in that:

[0005] Acquire three-dimensional models of multiple target objects to be displayed;

[0006] Obtaining the position and posture information of multiple target objects in the physical space in the coordinate system of the optical navigation system;

[0007] Constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information;

[0008] In response to the adjustment instruction, updating and displaying the three-dimensional visualization field of view;

[0009] The three-dimensional visualization field of view includes a target field of view.

[0010] Preferably, after the step of constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information, the method further includes:

[0011] Determining whether the multiple target objects are within the target field of view, and determining a target field of view recommendation result;

[0012] If all target objects are within the target field of view, determining the target field of view recommendation result as a recommendation;

[0013] If there is a situation where the target object is not within the target field of view, the target field of view recommendation result is determined to be to be adjusted. Preferably, if the field of view recommendation result is to be adjusted, a direction vector for indicating field of view adjustment is determined and displayed.

[0014] Preferably, determining a direction vector indicating field of view adjustment is specifically determining the direction vector for indicating field of view adjustment based on the posture information of the target object that is not within the target field of view and its relative posture relationship with the optical navigation system.

[0015] Preferably, the target object includes a surgical cart array, a robotic arm end array and an end workspace.

[0016] Preferably, the adjustment instruction comprises translation, scaling, or rotation of at least one of the plurality of target objects, the target field of view, or the three-dimensional visualization field of view.

[0017] Preferably, when displaying the three-dimensional visualization field of view, the three-dimensional visualization field of view is displayed at a first viewing angle of the optical navigation system.

[0018] Preferably, the three-dimensional model of the target object to be displayed is obtained by a depth camera, or by a three-dimensional scanner, or by a pre-stored model file, or by three-dimensionally reconstructing multiple two-dimensional images.

[0019] Another aspect of the present invention provides a three-dimensional visualization field of view adjustment device, characterized in that it includes

[0020] A first acquisition module, used to acquire three-dimensional models of multiple target objects to be displayed;

[0021] A second acquisition module is used to acquire the position and posture information of the multiple target objects in the physical space in the optical navigation system coordinate system;

[0022] Constructing a display module, used to construct and display a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information;

[0023] The adjustment module is used to update the three-dimensional visualization field of view of the simulated adjustment in response to the adjustment instruction.

[0024] Another aspect of the present invention provides an electronic device, comprising:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions for execution by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any one of the above methods.

[0028] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described above.

[0029] The positive and progressive effect of the present invention is that the three-dimensional visualization field of view adjustment method, adjustment device and electronic equipment can intuitively display the target object and the target field of view, realize the three-dimensional visualization adjustment of the target field of view, facilitate the adjustment of the target field of view so that the target field of view completely covers the target object, and the operation is convenient and intuitive. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a flow chart of a three-dimensional visualization field of view adjustment method provided in Example 1 of the present invention;

[0031] Figure 2 A schematic diagram of a target field of view provided in Example 1 of the present invention;

[0032] Figure 3 A schematic diagram of another direction of the target field of view provided by Embodiment 1 of the present invention;

[0033] Figure 4 A schematic diagram of a three-dimensional visualization field of view provided in Example 1 of the present invention;

[0034] Figure 5 A schematic diagram of a flow chart of a three-dimensional visualization field of view adjustment method provided in Embodiment 2 of the present invention;

[0035] Figure 6 A schematic diagram of a three-dimensional visualization field of view provided by Embodiment 2 of the present invention;

[0036] Figure 7 Another schematic diagram of the three-dimensional visualization field of view provided by Embodiment 2 of the present invention;

[0037] Figure 8 A schematic diagram of the framework of a three-dimensional visualization field of view adjustment device provided in Example 3 of the present invention;

[0038] Fig. 9 A schematic block diagram of an exemplary electronic device 500 provided in accordance with Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0040] Example 1

[0041] This embodiment provides a three-dimensional visualization field of view adjustment method, such as Figure 1 As shown, the following steps are included:

[0042] S101, obtaining three-dimensional models of multiple target objects to be displayed;

[0043] S102, obtaining position information of multiple target objects in the physical space in the optical navigation system coordinate system;

[0044] S103, constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information;

[0045] S104, in response to the adjustment instruction, updating and displaying the three-dimensional visualization field of view of the simulated adjustment.

[0046] In the three-dimensional visualization field of view, Figure 4 As shown, it includes a target field of view 70, which is the field of view for subsequent surgical operations, specifically the target operation field of view generated by the detector of the optical navigation system, preferably, the field of view obtained by the NDI binocular camera, such as Figure 2 , 4 As shown, it is a stereoscopic field of view determined by at least 8 points. The NDI binocular camera is arranged on the navigation cart.

[0047] In the step S101, three-dimensional models of multiple target objects to be displayed are obtained, which can be obtained through a depth camera, which can be a structured light camera, a binocular camera or a TOF camera, or obtained through a three-dimensional scanner, such as a laser scanner, or obtained through a pre-stored model file, or obtained by three-dimensional reconstruction of multiple two-dimensional images.

[0048] In step S102, the position and posture information of multiple target objects in the physical space in the optical navigation system coordinate system is obtained; the physical space is a real space, and the position and posture information of the multiple target objects in the optical navigation system coordinate system in the physical space is usually obtained by a measurement detector of the optical navigation system. The optical navigation system may be, for example, an NDI optical navigation system.

[0049] In step S103, a real-time three-dimensional visualization field of view is constructed and displayed based on the three-dimensional model and the posture information. A real-time three-dimensional visualization field of view is jointly constructed based on the three-dimensional model and the posture information of the three-dimensional model in the coordinates of the optical navigation system, that is, the posture information relative to the detector of the optical navigation system, and displayed in the display space of the display device.

[0050] Step S104, in response to the adjustment instruction, updating and displaying the three-dimensional visualization field of view. The adjustment instruction includes translation, scaling, or rotation of at least one of the plurality of target objects, the target field of view, and the three-dimensional visualization field of view.

[0051] According to the instructions for translation, scaling, or rotation of at least one of the multiple target objects or the three-dimensional visualization field of view, the three-dimensional visualization field of view is updated and adjusted in real time; or the adjustment instructions may also include adjustment instructions for the target field of view, specifically, the adjustment of the orientation of the detector in the optical navigation system, that is, the adjustment of the up, down, left, and right orientations of the target field of view, and the adjustment of scaling or translation of the target field of view. With the adjustment of the orientation of the target field of view, the coordinate system of the optical navigation system also changes, and then the position information of the multiple target objects in the coordinate system of the optical navigation system also changes. At this time, according to the adjustment of the optical adjustment system, the three-dimensional visualization field of view is updated and displayed in real time. The adjustment of the detector in the optical navigation system is mainly to adjust the orientation of the target field of view so that all target objects can be located in the target field of view, while meeting the user's personal habits and needs, and facilitating subsequent operations.

[0052] like Figure 4 As shown, the target object includes a bed 10, a patient 20 on the bed, a surgical trolley 30, a surgical execution arm 40, a surgical execution arm end 50, and an end working space 60. At least the surgical trolley, the surgical execution arm end, and the end working space can be increased as needed, and at least these three can ensure the normal progress of subsequent operations.

[0053] The bed is associated with the patient on the bed, and a surgical execution arm is provided on the surgical cart. The surgical execution arm includes a plurality of bendable joints, and the end of the surgical execution arm is the execution end operated by the robot, which is located at the end of the surgical execution arm and can move with the bending movement of the surgical execution arm. The end workspace, i.e., the surgical area, is a three-dimensional area including a part of the patient area, which can be obtained after the patient's position is determined.

[0054] The method for adjusting the three-dimensional visualization field of view can display the optical navigation system and the target object in three dimensions, and then can adjust the optical navigation system and the entire three-dimensional visualization field of view, so as to make the adjustment process more intuitive and improve the efficiency of the adjustment, and facilitate subsequent operations.

[0055] Example 2

[0056] The adjustment method disclosed in this embodiment 2 is as follows: Figure 5 As shown, specifically including,

[0057] S201, obtaining three-dimensional models of multiple target objects to be displayed;

[0058] S202, obtaining position information of multiple target objects in the physical space in the optical navigation system coordinate system;

[0059] S203, constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information;

[0060] S204 , in response to the adjustment instruction, updating and displaying the simulated adjusted three-dimensional visualization field of view.

[0061] After step S203, the following steps are also included:

[0062] S2031, determining whether the multiple target objects are within the target field of view, and determining a field of view recommendation result;

[0063] Specifically, if all target objects are within the target field of view, the field of view recommendation result is determined to be recommended; Figure 6 As shown, the frame in the figure shows the target field of view and the terminal workspace. If all target objects are within the target field of view, the field of view recommendation result is determined to be recommended, and all target objects can be highlighted in the display area to be within the target field of view.

[0064] If there is a situation where the target object is not within the target field of view, the field of view recommendation result is determined to be to be adjusted. Figure 7 As shown, if the surgical area, that is, the terminal working space, is not within the target field of view, the output field of view recommendation result is to be adjusted, specifically, it is highlighted in the display area that the surgical area is not within the target field of view.

[0065] Determine whether the target object is within the target field of view, such as Figure 2 , 3 As shown in the figure, it is a schematic diagram of the target field of view, that is, the NDI field of view. p ,y p ,z p ) as an example, taking the point Z=z p The coordinates intercept the xy section. If the point is within the field of view, the corresponding (x p ,y p ) will fall within the extended rectangle, and its mathematical expression is as follows:

[0066] |x p |<=224+x_board and |y p |<=240+y_board

[0067] Where x_board=(|z p |)-950)×k x ,y_board=(|z p |-950)×k y ;k x and k y are the near and far section ratios of the X and Y axes, respectively.

[0068] As another optional embodiment, when part of the target object is within the target field of view and part of the target object is outside the target field of view, information of the target object may be displayed at a part of the display area, for example Figure 6 As shown, it is convenient for the user to make subsequent adjustments.

[0069] As another optional implementation, the step S2031 further includes a step S2032, if the field of view recommendation result is to be adjusted, determining and displaying a direction vector for indicating field of view adjustment. The field of view is the target field of view. Specifically, the direction vector for indicating field of view adjustment is determined, specifically, based on the posture information of the target object that is not in the target field of view and the relative posture relationship with the optical navigation system, to determine the direction vector for indicating field of view adjustment.

[0070] like Figure 7 As shown, if the surgical area, i.e., the terminal working space, is not within the target field of view, then the direction vector for indicating the field of view adjustment is determined based on the posture information of the target object that is not within the target field of view, and its relative posture relationship with the optical navigation system. Specifically, the posture information of the target object that is not within the target field of view is the posture information in the optical navigation system coordinate system. Therefore, the relative posture relationship with the optical navigation system can be obtained based on the posture information of the target object, and then, the direction vector for the target field of view adjustment is determined based on the posture information of the target object and its relative posture relationship with the optical navigation system. For example, Figure 7 As shown, the surgical area is not within the target field of view, and its posture information can be determined, thereby determining its relative posture relationship with the optical navigation system, and then determining the direction in which the field of view should be adjusted so that all target objects are included in the target field of view.

[0071] As a preferred implementation, when displaying the three-dimensional visualization field of view, the three-dimensional visualization field of view is displayed at a first viewing angle of the optical navigation system to facilitate subsequent adjustment. The first viewing angle, such as Figure 7 It is the direction that the detector of the optical navigation system is facing.

[0072] The three-dimensional visualization field of view adjustment method can intuitively display the target object and the target field of view, realize the three-dimensional visualization adjustment of the target field of view, facilitate the adjustment of the target field of view so that the target field of view completely covers the target object, and the operation is convenient and intuitive.

[0073] Example 3

[0074] This embodiment provides a three-dimensional visualization field of view adjustment device, specifically including: Figure 8 As shown, including,

[0075] The first acquisition module 1 is used to acquire three-dimensional models of multiple target objects to be displayed; the three-dimensional models of multiple target objects to be displayed can be acquired through a depth camera, which can be a structured light camera, a binocular camera or a TOF camera, or through a three-dimensional scanner, such as a laser scanner, or through a pre-stored model file, or by performing three-dimensional reconstruction on multiple two-dimensional images.

[0076] The second acquisition module 2 is used to acquire the position and posture information of the multiple target objects in the physical space in the optical navigation system coordinate system; the physical space is the real space, and acquiring the position and posture information of the multiple target objects in the optical navigation system coordinate system in the physical space usually also requires a measurement detector of the optical navigation system. The optical navigation system can be, for example, an NDI optical navigation system.

[0077] A display module 3 is constructed to construct and display a real-time three-dimensional visualization field of view based on the three-dimensional model and the posture information; a real-time three-dimensional visualization field of view is jointly constructed based on the three-dimensional model and the posture information of the three-dimensional model in the coordinates of the optical navigation system, that is, the posture information relative to the detector of the optical navigation system, and displayed in the display space of the display device.

[0078] The adjustment module 4 is used to update and display the simulated adjusted three-dimensional visualization field of view in response to the adjustment instruction. The three-dimensional visualization field of view is updated and displayed. The adjustment instruction includes translation, scaling, or rotation of at least one of the multiple target objects, the target field of view, and the three-dimensional visualization field of view.

[0079] Preferably, the adjusting module 4 may further include a judging module 41 , a determining module 42 and an indicating module 43 .

[0080] The judging module 41 is used to judge whether the multiple target objects are within the target field of view;

[0081] The determination module 42 is used to determine the field of view recommendation result according to the judgment result of the judgment module. Specifically, if all target objects are within the target field of view, the field of view recommendation result is determined to be recommended; if there is a situation where the target object is not within the target field of view, the field of view recommendation result is determined to be to be adjusted.

[0082] The indication module 43 is used to determine and display a direction vector for indicating field of view adjustment when the determination module determines that the field of view recommendation result is to be adjusted. Specifically, the direction vector for indicating field of view adjustment is determined based on the posture information of the target object that is not within the target field of view, and its relative posture relationship with the optical navigation system.

[0083] The three-dimensional visualization field of view adjustment device can intuitively display the target object and the target field of view, realize three-dimensional visualization adjustment of the target field of view, facilitate the adjustment of the target field of view so that the target field of view completely covers the target object, and is convenient and intuitive to operate.

[0084] Example 4

[0085] According to Embodiment 4 of the present invention, the present invention further provides an electronic device, a readable storage medium and a computer program product. Fig. 9 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present invention is shown. The device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 into a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0086] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a three-dimensional visualization field of view adjustment method. For example, in some embodiments, the three-dimensional visualization field of view adjustment method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned three-dimensional visualization field of view adjustment method in any other appropriate manner (for example, by means of firmware).

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0091] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A three-dimensional visualization field of view adjustment method, characterized in that: Acquire three-dimensional models of multiple target objects to be displayed; Obtaining the position and posture information of multiple target objects in the physical space in the coordinate system of the optical navigation system; Constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information; In response to the adjustment instruction, updating and displaying the three-dimensional visualization field of view; The three-dimensional visualization field of view includes a target field of view.

2. The three-dimensional visualization field of view adjustment method according to claim 1, characterized in that: After the step of constructing and displaying a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information, the method further includes: Determining whether the multiple target objects are within the target field of view, and determining a target field of view recommendation result; If all target objects are within the target field of view, determining the target field of view recommendation result as a recommendation; If there is a situation where the target object is not within the target field of view, it is determined that the target field of view recommendation result is to be adjusted.

3. The three-dimensional visualization field of view adjustment method according to claim 2, characterized in that: If the field of view recommendation result is to be adjusted, a direction vector for indicating field of view adjustment is determined and displayed.

4. The three-dimensional visualization field of view adjustment method according to claim 3, characterized in that: Determining a direction vector indicating field of view adjustment is specifically determining the direction vector for indicating field of view adjustment according to the posture information of the target object that is not in the target field of view and its relative posture relationship with the optical navigation system.

5. The three-dimensional visualization field of view adjustment method according to claim 3, characterized in that: The target objects include a surgical cart array, a robotic arm end array, and an end workspace.

6. The three-dimensional visualization field of view adjustment method according to claim 1, characterized in that: The adjustment instruction includes translation, scaling, or rotation of at least one of the plurality of target objects, the target field of view, or the three-dimensional visualization field of view.

7. The three-dimensional visualization field of view adjustment method according to claim 1, characterized in that: When displaying the three-dimensional visualization field of view, the three-dimensional visualization field of view is displayed at a first viewing angle of the optical navigation system.

8. The three-dimensional visualization field of view adjustment method according to claim 1, characterized in that: The three-dimensional model of the target object to be displayed is obtained by a depth camera, or by a three-dimensional scanner, or by a pre-stored model file, or by three-dimensionally reconstructing multiple two-dimensional images.

9. A three-dimensional visualization field of view adjustment device, characterized in that: include A first acquisition module, used to acquire three-dimensional models of multiple target objects to be displayed; A second acquisition module is used to acquire the position and posture information of the multiple target objects in the physical space in the optical navigation system coordinate system; Constructing a display module, used to construct and display a real-time three-dimensional visualization field of view according to the three-dimensional model and the posture information; The adjustment module is used to update and display the three-dimensional visualization field of view in response to the adjustment instruction.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions for execution by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images