3D-VR binocular stereoscopic vision image construction method and endoscope operation teaching device

Abstract

The present invention belongs to the field of medical equipment technology, and specifically discloses a 3D-VR binocular stereoscopic vision image construction method and an endoscope operation teaching device, which obtains the three-dimensional original image information of the endoscope inspection by setting two endoscope imaging probes; determines the displayed three-dimensional original image information; renders and coordinates the displayed three-dimensional original image information to construct an image model of parallel binocular vision; based on the image model of parallel binocular vision, displays the visual image on a 3D-VR display device; uses a 3D-VR endoscope operation platform to operate the endoscope and display it on the display device. The use of this technical solution to construct an image model and use a 3D-VR display device for display can not only enhance the visual reception ability of the trainees, but also improve the corresponding medical diagnosis and treatment teaching efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of medical equipment and relates to a 3D-VR binocular stereoscopic vision image construction method and an endoscope operation teaching device. Background Art

[0002] With the development of modern medical technology and medical teaching technology, endoscopy, as an extension of the doctor's eyes and hands, has reached the level of "penetrating all holes". Endoscopic examination can diagnose the following diseases: ① Differentiation and diagnosis of inflammation, ulcers, benign and malignant tumors in the digestive tract (including esophagus, stomach, duodenum, small intestine and large intestine); ② Diagnosis of benign and malignant lesions of abdominal organs. ③ Surgical treatment of the digestive tract and abdominal organs (such as endoscopic ESD for patients with early esophageal cancer). Therefore, it has become an important teaching subject of internal medicine.

[0003] The teaching of endoscopic operation and treatment technology currently relies mainly on the instructor to perform the corresponding endoscopic diagnosis and treatment operations, and the students observe and accumulate experience to achieve a certain teaching effect. However, since the human eye uses the principle of binocular imaging to achieve stereoscopic imaging, the Information Department has found through long-term scientific research and daily work that from 2002 to 2018, the development of endoscopic diagnosis and treatment related equipment has lagged behind and cannot keep up with the development of the relevant medical industry. The teaching field of endoscopic diagnosis and treatment has developed slowly and the methods are backward.

[0004] In the process of endoscopic medical treatment and teaching at home and abroad, ordinary flat imaging equipment is completely relied on. Since flat displays can only display so-called two-dimensional spatial images, the eyes cannot perceive information such as the depth, distance, and angle of objects. This also affects the accuracy of students' grasp of endoscopic teaching knowledge, which is contrary to the current national call for "precision medicine". The students themselves lack experience, and much of their knowledge remains only at the theoretical level. Therefore, when the teaching teacher makes an error in operation, it will have a great degree of adverse impact on the students. Summary of the invention

[0005] The purpose of the present invention is to provide a 3D-VR binocular stereoscopic vision image construction method and an endoscope operation teaching device, which can not only enhance the visual reception ability of trainees, but also improve the corresponding medical diagnosis and treatment teaching efficiency.

[0006] In order to achieve the above object, the basic scheme of the present invention is: a 3D-VR binocular stereoscopic vision image construction method, comprising the following steps:

[0007] Two endoscopic imaging probes are set to obtain three-dimensional original image information of endoscopic examination;

[0008] Determining the three-dimensional original image information to be displayed;

[0009] Render and transform the coordinates of the displayed 3D original image information to build an image model of parallel binocular vision;

[0010] Based on the image model of parallel binocular vision, the visual image is displayed on the 3D-VR display device;

[0011] The endoscope is operated using the 3D-VR endoscope operating platform and displayed on the display device.

[0012] The working principle and beneficial effect of this basic solution are: using two endoscopes to collect corresponding image information, obtain more three-dimensional and comprehensive image information, which is convenient for viewing. Through the 3D-VR endoscope operation platform and 3D-VR display equipment, the image information is displayed in front of the user, the viewing distance is closer, the image is clearer, the visual reception ability of the trainees is enhanced, it is conducive to medical teaching, and the corresponding medical diagnosis and treatment teaching efficiency is improved.

[0013] Further, the method for determining the displayed three-dimensional original image information is:

[0014] A three-dimensional original image is acquired, and an image range to be displayed is determined, wherein the image range includes the endoscopic imaging probe and the endoscopic detection area, and the endoscopic imaging probe does not exceed 50% of the area within the image range.

[0015] Acquire a suitable image range for easy use.

[0016] Further, the method for rendering the three-dimensional original image information is as follows:

[0017] Rendering the three-dimensional raw image information based on points, using points as primitives for video production;

[0018] Use three mutually perpendicular light sources to illuminate the object, and record the normal vector and color information at the intersection of each light ray with the object model;

[0019] Adopting a layered deep cube structure, when rendering, the concept of facets is designed. By calculating the number of facets in each pixel, the visibility of the levels in the octree of the layered deep cube structure is determined, and then the occlusion judgment is completed. The occluded data is discarded and not rendered;

[0020] The illumination is calculated by normal vectors to complete stereoscopic video rendering.

[0021] Simple operation and easy to use.

[0022] Furthermore, the method for calculating illumination to complete stereoscopic video rendering is:

[0023] For the light source carried by the mirror, establish the illumination model function:

[0024]

[0025] Where I is the RGB value of the image pixel, I a is the ambient light color, I P is the brightness of the ambient light, k a , k d and k s are ambient light, diffuse reflection and specular reflection coefficients respectively; S P ∈[0,1] is shadow; H P is the specular coefficient, is the light source direction, is the normal vector;

[0026] According to the illumination direction of the light source, obtain the highlight and shadow parts in the image;

[0027] The pixels corresponding to the highlight and shadow parts are fused with the image and rendered to output the final rendered image.

[0028] Based on the difference in pixels between the highlight and shadow parts, they are distinguished to facilitate fusion rendering.

[0029] Furthermore, the 3D-VR endoscope operation platform is used to operate the endoscope for anti-collision detection. The specific method is as follows:

[0030] Obtain the motion range of all endoscopes;

[0031] The endoscope with the largest range of motion is selected as the reference endoscope;

[0032] Determine the real-time relative position information of other endoscope bodies and the reference endoscope body;

[0033] Determine the safety area corresponding to other endoscope bodies according to the relative position information, wherein the safety area is an area in the relative position information where the distance from the reference body is greater than a safety threshold;

[0034] The other endoscope mirror bodies are suspended when the mirror body with collision risk is determined according to the safety area and the collision threshold is reached.

[0035] In this way, anti-collision is achieved, the operation is simple and the use is convenient.

[0036] The present invention also provides an endoscopic operation teaching device, comprising an image acquisition unit, a processing unit and a helmet display, wherein the image acquisition unit is used to acquire original images, the output end of the image acquisition unit is connected to the input end of the processing unit, the processing unit executes the method of the present invention to construct a stereoscopic visual image, and the output end of the processing unit is connected to the input end of the helmet display.

[0037] The device of this solution is used to display image information in the helmet display. When the user wears the helmet display, the viewing distance is closer and the image is clearer, which enhances the visual reception ability of the trainees and is beneficial to medical teaching.

[0038] Furthermore, a convex lens is arranged between the display screen of the helmet display and the human eye.

[0039] By using a convex lens, the image is processed to allow the eye to obtain a wide viewing angle, thereby increasing the immersiveness of the device.

[0040] Furthermore, the helmet display is provided with a head tracking unit, and the head tracking unit sets the head as a coordinate, and determines the origin position of the tracked viewpoint through the coordinate setting;

[0041] The default origin is the initial position of the head, and the trapezoidal cone is the tracking range. When the head moves within this range, the head position shift will be obtained on the display screen of the helmet display through the head tracking unit to obtain the new position coordinates of the tracking point;

[0042] After obtaining the head position, the offset position of the head is obtained through offset calculation based on the origin, and then the offset displacement data is transmitted to the display system. The system recalculates and sets the new viewpoint of the internal viewfinder model to achieve synchronization with the head position.

[0043] Enhance system immersion through head tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural block diagram of the 3D-VR binocular stereoscopic vision image construction device of the present invention;

[0045] Figure 2 It is a schematic diagram of the structure of the convex lens of the 3D-VR binocular stereoscopic vision image construction device of the present invention. DETAILED DESCRIPTION

[0046] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0047] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0048] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0049] The present invention discloses a 3D-VR binocular stereoscopic vision image construction method, comprising the following steps:

[0050] Two endoscopic imaging probes are set to obtain three-dimensional original image information of endoscopic examination;

[0051] Determining the three-dimensional original image information to be displayed;

[0052] The displayed three-dimensional original image information is rendered and the coordinates are transformed to construct an image model of parallel binocular vision; the camera coordinate system is a very critical coordinate system in binocular vision, which describes the coordinates of the image in three-dimensional space from the perspective of the camera. The camera coordinate system takes the center of the camera as the origin, and the X-axis and Y-axis are parallel to the x-axis and y-axis of the image coordinate system respectively, and the directions are consistent. The Z direction is obtained according to the right-hand coordinate system rule. Because the y-axis directions in the image coordinate system are different, the Z-axis directions of photogrammetry and computer vision are exactly opposite. The world coordinate system is an absolute coordinate system in the objective three-dimensional world. The camera coordinate system can be converted to and from the world coordinate system, and the conversion between pixels and the world coordinate system can be completed.

[0053] Based on the image model of parallel binocular vision, the visual image is displayed on the 3D-VR display device;

[0054] The endoscope is operated using a 3D-VR (virtual reality) endoscope operating platform and displayed on a display device.

[0055] In a preferred embodiment of the present invention, the method for determining the displayed three-dimensional original image information is:

[0056] A three-dimensional original image is acquired, and an image range to be displayed is determined, wherein the image range includes the endoscopic imaging probe and the endoscopic detection area, and the endoscopic imaging probe does not exceed 50% of the area within the image range.

[0057] In a preferred embodiment of the present invention, a method for rendering three-dimensional original image information is as follows:

[0058] Rendering the three-dimensional raw image information based on points, using points as primitives for video production;

[0059] Use three mutually perpendicular light sources to illuminate the object, and record the normal vector and color information at the intersection of each light ray with the object model;

[0060] Adopting a layered deep cube structure, when rendering, the concept of facets is designed. By calculating the number of facets in each pixel, the visibility of the levels in the octree of the layered deep cube structure is determined, and then the occlusion judgment is completed. The occluded data is discarded and not rendered;

[0061] In a preferred embodiment of the present invention, the method for calculating illumination to complete stereoscopic video rendering is:

[0062] For the light source carried by the mirror, establish the illumination model function:

[0063]

[0064] Where I is the RGB value of the image pixel, I a is the ambient light color, I P is the brightness of the ambient light, k a , k d and k s are ambient light, diffuse reflection and specular reflection coefficients respectively; S P ∈[0,1] is shadow; H P is the specular coefficient, is the light source direction, is the normal vector;

[0065] According to the illumination direction of the light source, the highlight part and the shadow part in the image are obtained; according to the difference in the pixel points of the highlight part and the shadow part, they are distinguished;

[0066] The pixels corresponding to the highlight and shadow parts are fused with the image and rendered to output the final rendered image.

[0067] In a preferred embodiment of the present invention, a 3D-VR endoscope operation platform is used to operate an endoscope for anti-collision detection, and the specific method is as follows:

[0068] Obtain the motion range of all endoscopes;

[0069] The endoscope with the largest range of motion is selected as the reference endoscope;

[0070] Determine the real-time relative position information of other endoscope bodies and the reference endoscope body;

[0071] Determine the safety area corresponding to other endoscope bodies according to the relative position information, wherein the safety area is an area in the relative position information where the distance from the reference body is greater than a safety threshold;

[0072] The other endoscope mirror bodies are suspended when the mirror body with collision risk is determined according to the safety area and the collision threshold is reached.

[0073] The present invention also provides an endoscope operation teaching device, such as Figure 1 As shown, it includes an image acquisition unit, a processing unit and a helmet display (such as the Oculus Rift CV1 head-mounted VR display device), the image acquisition unit is used to acquire original images, the image acquisition unit includes an endoscope (such as the Pentax EPM-3500 endoscope host) and a 3D camera system, the 3D camera system is installed on the endoscope, and the 3D camera system includes two miniature 3D cameras (such as HERO3 Black Edition), the two miniature 3D cameras are fixedly set (such as welding, bonding, etc.) on both sides of the endoscope to collect image information in a wider range and obtain more information. The output end of the image acquisition unit is electrically connected to the input end of the processing unit, the processing unit executes the method described in the present invention to construct a stereoscopic visual image, and the output end of the processing unit is electrically connected to the input end of the helmet display.

[0074] Preferably, the processing unit includes a memory, a GPU (Graphics Processing Unit), a CPU (Central Processing Unit) and a buffer memory. The output end of the 3D camera system is electrically connected to the input end of the memory (such as 24CXX series, 93CXX series, etc.), the GPU and the CPU receive the image information in the memory and process it, and the output end of the GPU (such as DUAL-GTX1060-6G, etc.) and the CPU (such as E3300, E3400 (775 interface), G440, G530, etc.) are electrically connected to the input end of the buffer memory. The specific processing method can adopt the existing technology, such as a mine multi-camera video fusion technology based on GPU and CPU collaborative calculation mentioned in the existing patent CN201510338689.3, which constructs a GPU and CPU collaborative calculation model to perform multi-camera video fusion processing. The output end of the buffer memory is electrically connected to the helmet display, and the operating platform uses the GPU and CPU to fuse the image signal and optimize the image information.

[0075] like Figure 2 As shown, a convex lens is arranged between the display screen of the helmet display and the human eye. The display screen is used to display the processed image, and the light of the display screen reaches the retina through the lens group. Through the image processing, the retina obtains a wide viewing angle, which can completely cover the visual scene of the human eye, thereby increasing the immersion of the device.

[0076] The helmet display is equipped with a head tracking unit, which sets the head as the coordinate and determines the origin of the viewpoint to be tracked through the coordinate setting. The default origin is the initial position of the head, and the trapezoidal cone is the tracking range. When the movement of the head falls within the range, the transfer of the head position will be obtained through the head tracking unit on the display screen of the helmet display to obtain the new position coordinates of the tracking point.

[0077] After obtaining the head position, the offset position of the head is obtained through offset calculation based on the origin, and then the offset displacement data is transmitted to the display system. The system recalculates and sets the new viewpoint of the internal viewfinder model to achieve synchronization with the head position.

[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0079] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A 3D-VR binocular stereoscopic vision image construction method, characterized in that: The steps include: Two endoscopic imaging probes are set to obtain three-dimensional original image information of endoscopic examination; Determining the three-dimensional original image information to be displayed; Render and transform the coordinates of the displayed 3D original image information to build an image model of parallel binocular vision; Based on the image model of parallel binocular vision, the visual image is displayed on the 3D-VR display device; Use the 3D-VR endoscope operation platform to operate the endoscope and display it on the display device; The method for determining the displayed three-dimensional original image information is: Acquire a three-dimensional original image and determine the image range to be displayed, wherein the image range includes the endoscopic imaging probe and the endoscopic detection area, and the endoscopic imaging probe does not exceed 50% of the area within the image range; Method for rendering three-dimensional original image information: Rendering the three-dimensional raw image information based on points, using points as primitives for video production; Use three mutually perpendicular light sources to illuminate the object, and record the normal vector and color information at the intersection of each light ray with the object model; Adopting a layered deep cube structure, when rendering, the concept of facets is designed. By calculating the number of facets in each pixel, the visibility of the levels in the octree of the layered deep cube structure is determined, and then the occlusion judgment is completed. The occluded data is discarded and not rendered; The illumination is calculated by normal vectors to complete stereoscopic video rendering.

2. The 3D-VR binocular stereoscopic vision image construction method according to claim 1, characterized in that: The method for calculating lighting to complete stereo video rendering is: For the light source carried by the mirror, establish the illumination model function: Where I is the RGB value of the image pixel, I a is the ambient light color, I P is the brightness of the ambient light, k a , k d and k s are ambient light, diffuse reflection and specular reflection coefficients respectively; S P ∈[0,1] is shaded; H P is the specular coefficient, is the light source direction, is the normal vector; According to the illumination direction of the light source, obtain the highlight and shadow parts in the image; The pixels corresponding to the highlight and shadow parts are fused with the image and rendered to output the final rendered image.

3. The 3D-VR binocular stereoscopic vision image construction method according to claim 1, characterized in that: Use the 3D-VR endoscope operating platform to operate the endoscope for anti-collision detection. The specific method is as follows: Obtain the motion range of all endoscopes; The endoscope with the largest range of motion is selected as the reference endoscope; Determine the real-time relative position information of other endoscope bodies and the reference endoscope body; Determine the safety area corresponding to other endoscope bodies according to the relative position information, wherein the safety area is an area in the relative position information where the distance from the reference body is greater than a safety threshold; The other endoscope mirror bodies are suspended when the mirror body with collision risk is determined according to the safety area and the collision threshold is reached.

4. An endoscope operation teaching device, characterized in that: It includes an image acquisition unit, a processing unit and a helmet display, wherein the image acquisition unit is used to acquire original images, the output end of the image acquisition unit is connected to the input end of the processing unit, the processing unit executes the method described in one of claims 1-3 to construct a stereoscopic visual image, and the output end of the processing unit is connected to the input end of the helmet display.

5. The endoscopic operation teaching device according to claim 4, characterized in that: A convex lens is arranged between the display screen of the helmet display and the human eye.

6. The endoscopic operation teaching device according to claim 4, characterized in that: The helmet display is provided with a head tracking unit, and the head tracking unit sets the head as a coordinate, and determines the origin position of the tracked viewpoint through the coordinate setting; The default origin is the initial position of the head, and the trapezoidal cone is the tracking range. When the head moves within this range, the head position shift will be obtained on the display screen of the helmet display through the head tracking unit to obtain the new position coordinates of the tracking point; After obtaining the head position, the offset position of the head is obtained through offset calculation based on the origin, and then the offset displacement data is transmitted to the display system. The system recalculates and sets the new viewpoint of the internal viewfinder model to achieve synchronization with the head position.

Images