3D optical equipment 3D calibration method and operating microscope

Abstract

The invention provides a 3D calibration method of 3D optical equipment and an operating microscope. The method comprises the following steps: acquiring a left eye image and a right eye image respectively shot by a binocular camera in the 3D optical equipment; based on a preset window, according to the obtained left eye image and right eye image, window adjustment is carried out, and the window adjustment comprises displacement of at least one of a first segmentation window in the left eye image and a second segmentation window in the right eye image; respectively segmenting the left eye image and the right eye image according to the first segmentation window and the second segmentation window after window adjustment so as to obtain a segmented left eye image and a segmented right eye image; and amplifying the segmented left eye image and right eye image to a target image size, and outputting the amplified left eye image and right eye image to a 3D imaging display device according to a left-right arrangement format. Through the technical scheme of the invention, the 3D calibration efficiency and precision can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of 3D display technology, and in particular to a 3D calibration method for a 3D optical device and a surgical microscope. Background Art

[0002] When performing three-dimensional display based on a 3D optical device, it is necessary to calibrate the installation position of the binocular camera in the 3D optical device to meet the binocular camera shooting parallax and shooting angle required for presenting the best three-dimensional display effect.

[0003] At present, the installation position of the binocular camera in the 3D optical device is usually adjusted by additional mechanical devices or manual methods. However, the 3D calibration method using mechanical devices will greatly increase the cost and the adjustment range is limited. The manual calibration method cannot accurately control the adjustment distance, and it is easy to adjust too much or too little, which requires repeated adjustments, reducing the efficiency of 3D calibration. Summary of the invention

[0004] The present disclosure provides a 3D calibration method for a 3D optical device and a surgical microscope, which can perform 3D calibration by adjusting the relative positions of left and right eye images without adjusting the installation position of a binocular camera, thereby improving the 3D calibration efficiency and accuracy.

[0005] In a first aspect, an embodiment of the present disclosure provides a 3D calibration method for a 3D optical device, comprising:

[0006] Obtaining a left eye image and a right eye image respectively captured by a binocular camera in a 3D optical device;

[0007] Based on the preset window, according to the acquired left-eye image and right-eye image, the window adjustment includes shifting at least one of the first segmentation window in the left-eye image and the second segmentation window in the right-eye image;

[0008] According to the first segmentation window and the second segmentation window after the window adjustment, the left-eye image and the right-eye image are segmented respectively to obtain the segmented left-eye image and the right-eye image;

[0009] The segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

[0010] In a second aspect, the disclosed embodiment further provides a surgical microscope, including a binocular camera, a processing module and a 3D imaging display device; wherein:

[0011] The binocular camera is used to capture left-eye images and right-eye images;

[0012] The processing module is used to implement the method described in any embodiment of the present disclosure according to the captured left-eye image and right-eye image;

[0013] The 3D imaging display device is used for performing 3D imaging display on input left-eye images and right-eye images.

[0014] In a third aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the method as described in any embodiment of the present disclosure.

[0015] The technical solution provided by the embodiment of the present disclosure adopts a method of window adjustment based on the left eye image and the right eye image respectively captured by the binocular camera in the 3D optical device: at least one of the first segmentation window in the left eye image and the second segmentation window in the right eye image is displaced, and the first segmentation window and the second segmentation window after the window adjustment are used to segment the left eye image and the right eye image respectively, and the segmented left eye image and the right eye image are both enlarged to the target image size, thereby realizing the adjustment of the relative positions of the left and right eye images, and outputting the enlarged left eye image and the right eye image in a left-right arrangement format to the 3D imaging display device for presenting the 3D display effect, so as to achieve the best 3D display effect by adjusting the relative positions of the left and right eye images, avoiding the tediousness and uncertainty of manual operation, and not needing to add additional mechanical devices, and having the advantages of easy use, high correction accuracy, time saving and no interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1A A flowchart of a 3D calibration method for a 3D optical device according to an embodiment of the present disclosure is exemplarily shown;

[0018] Figure 1B An example diagram showing a captured image and a segmented window according to an embodiment of the present disclosure is exemplarily shown;

[0019] Figure 1C An exemplary diagram showing an image magnification process according to an embodiment of the present disclosure is shown;

[0020] Figure 1D An example diagram showing a 3D display according to an embodiment of the present disclosure is exemplified;

[0021] Figure 1EAn example diagram showing left-eye and right-eye image correction according to an embodiment of the present disclosure is exemplarily shown;

[0022] Figure 1F An example diagram of pixel size calibration according to an embodiment of the present disclosure is exemplarily shown;

[0023] Figure 2A A flowchart of a 3D calibration method for a 3D optical device according to another embodiment of the present disclosure is exemplarily shown;

[0024] Figure 2B An exemplary diagram showing a window moving to the left according to an embodiment of the present disclosure is shown;

[0025] Figure 3A A flowchart of a 3D calibration method for a 3D optical device according to another embodiment of the present disclosure is exemplarily shown;

[0026] Figure 3B An example diagram of the optical path of a binocular camera according to an embodiment of the present disclosure is exemplarily shown;

[0027] Figure 4A A flowchart of a 3D calibration method for a 3D optical device according to another embodiment of the present disclosure is exemplarily shown;

[0028] Figure 4B An exemplary diagram showing a downward movement of a window according to an embodiment of the present disclosure is shown;

[0029] Figure 5 A schematic diagram of a surgical microscope according to an embodiment of the present disclosure is exemplarily shown. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] Figure 1A The flowchart of the 3D calibration method of the 3D optical device according to the embodiment of the present disclosure is exemplarily shown. The embodiment of the present disclosure is suitable for calibrating the 3D display effect by adjusting the relative position between the left and right eye images taken by the 3D optical device. The method can be executed by a processing module in the 3D optical device, and the processing module can be implemented in the form of software and / or hardware. Optionally, the 3D optical device can be any optical device with a 3D display effect. For example, the 3D optical device can be but is not limited to a 3D surgical microscope.

[0036] like Figure 1A As shown, the 3D optical device 3D calibration method may include steps S110-S140.

[0037] In step S110, a left-eye image and a right-eye image respectively captured by a binocular camera in a 3D optical device are acquired.

[0038] In the embodiment of the present disclosure, the binocular camera in step S110 includes a left-eye camera and a right-eye camera, and the 3D optical device uses the images captured by the two cameras to record the left-eye image and the right-eye image respectively. By using the captured left-eye image and right-eye image, the binocular camera in the 3D optical device is 3D calibrated.

[0039] In step S120, based on the preset window and according to the acquired left-eye image and right-eye image, the window adjustment is performed, and the window adjustment includes shifting at least one of the first segmentation window in the left-eye image and the second segmentation window in the right-eye image.

[0040] It should be understood that the preset window is a pre-set window for splitting the left and right eye images. The window size of the preset window is smaller than the image size of the left and right eye images. The preset window includes a first split window and a second split window. The first split window is a window for cutting out a portion of the image in the left eye image. The second split window is a window for cutting out a portion of the image in the right eye image. The window size of the first split window is smaller than the image size of the left eye image. The window size of the second split window is smaller than the image size of the right eye image. The window size of the first split window can be the same as the window size of the second split window so that images of the same size can be cut out. The window sizes of the first split window and the second split window can be set based on business needs. The window adjustment in step S120 is to adjust the position of the window in the image. Different window positions result in different image areas being cut out. For example, see Figure 1B , the image size of the left and right eye images (i.e., the real frame area) is 3840×2160, and the window size of the segmentation window (i.e., the virtual frame area) is 3600×2026, so that a partial image of 3600×2026 can be cut out from the 3840×2160 complete image captured by the binocular camera through the segmentation window.

[0041] In step S120, according to the left eye image and the right eye image obtained in step S110, at least one of the first segmentation window in the left eye image and the second segmentation window in the right eye image is displaced, thereby changing the relative position between the images in the two windows, and realizing the movement of the image in the field of view. As an implementation method, only one of the segmentation windows can be adjusted in position, and the position of the other segmentation window is fixed, so that the relative position of the images in the two windows can be changed. As another implementation method, the positions of the two segmentation windows can be adjusted at the same time, that is, the positions of the first segmentation window and the second segmentation window are adjusted at the same time, so that the relative positions of the images in the two windows can also be changed. The specific implementation method can be set based on business needs and is not limited here.

[0042] In step S130, the left-eye image and the right-eye image are segmented respectively according to the first segmentation window and the second segmentation window after the window adjustment to obtain segmented left-eye image and right-eye image.

[0043] It should be understood that, according to the position of the first segmentation window in the left eye image after the window adjustment, the image in the first segmentation window is segmented from the left eye image, and the image in the first segmentation window is used as the segmented left eye image. Similarly, according to the position of the second segmentation window in the right eye image after the window adjustment, the image in the second segmentation window is segmented from the right eye image, and the image in the second segmentation window is used as the segmented right eye image. By using the first segmentation window and the second segmentation window to segment the left and right eye images, partial left and right eye images can be obtained.

[0044] In step S140 , the segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

[0045] The 3D imaging display device in step S140 may be, but is not limited to, a 3D television. The target image size is an image size adapted to the 3D imaging display device so that the enlarged left-eye image and right-eye image can be displayed in 3D imaging on the 3D imaging display device. For example, if the 3D imaging display device is a 4K display device, the segmented left-eye image and right-eye image need to be enlarged to an image size of 3840×2160, so that the 3D imaging display device can use the input 3840×2160 left-eye and right-eye images for 3D imaging and display them as 3D images on the 3D imaging display device.

[0046] For example, see Figure 1C , the image sizes of the left and right eye images captured by the binocular camera are both image sizes adapted to the 3D imaging display device, so that the image in the first split window can be enlarged to the same image size as the left eye image, and the image in the second split window can be enlarged to the same image size as the right eye image. It should be noted that in the image magnification process, the magnification factor required for the image is very small, and the negative impact of the interpolation algorithm can be ignored. This embodiment can adjust the phase position between the left and right eye images by splitting and enlarging the captured left and right eye images, thereby achieving the same effect as adjusting the installation position of the binocular camera, meeting the needs of 3D image display. The 3D calibration method provided by the disclosed embodiment is flexible in control, easy to use, has high correction accuracy, and does not increase additional costs, thereby greatly improving the installation efficiency.

[0047] For example, see Figure 1D By adjusting the window displacement of the left and right eye images taken by the binocular camera, the relative position of the captured images is changed, thereby optimizing the 3D display effect. It can also replace the adjustment of the physical position of the binocular camera, solving the problems of low efficiency, poor accuracy and susceptibility to vibration in the traditional binocular camera position calibration method.

[0048] It should be noted that if the 3D image displayed in the 3D imaging display device achieves the optimal 3D display effect after this window adjustment, it indicates that the 3D calibration is completed. There is no need to adjust the window again for the next 3D imaging. The left and right eye images taken next time can be directly split and enlarged according to the split window after this adjustment, so that the 3D image displayed in the 3D imaging display device next time can achieve the optimal 3D display effect.

[0049] In the disclosed embodiment, by displacing at least one of a first segmentation window in a left-eye image and a second segmentation window in a right-eye image, and using the adjusted first segmentation window and the second segmentation window to segment the left-eye image and the right-eye image respectively, and enlarging the segmented left-eye image and the right-eye image to a target image size, the relative positions of the left-eye and right-eye images are adjusted, and the enlarged left-eye image and the right-eye image are output to a 3D imaging display device in a left-right arrangement format for presenting a 3D display effect, so as to achieve the best 3D display effect by adjusting the relative positions of the left-eye and right-eye images, avoid the tediousness and uncertainty of manual operation, and do not need to add additional mechanical devices, and have the advantages of easy use, high correction accuracy, time saving and no interference.

[0050] As an optional embodiment, before performing the window adjustment, the method further includes: correcting the left-eye image and the right-eye image acquired in step S110, and / or performing pixel calibration.

[0051] The calibration process includes at least one of the following calibration operations:

[0052] Hardware installation calibration: Perform preliminary installation of the left and right eye cameras and fix them according to the requirements of the structural design and optical design. The left and right eye cameras must be installed straight, that is, the clockwise or counterclockwise rotation displacement between the images taken by the left and right eye cameras cannot exceed 5 degrees; the left and right eye cameras must be installed on the same horizontal plane, that is, ensure that the images taken by the two cameras are not larger or smaller. It should be noted that after the binocular camera is installed, it is fixed and there is no need to adjust the installation position of the binocular camera during the subsequent 3D calibration process.

[0053] Software alignment correction: that is, automatically initialize and correct the left eye image and the right eye image obtained in step S110. Under fixed parameters, such as focal length 10mm and working distance 20mm, shoot the calibration template. The software will obtain the rotation matrix and translation vector of the two cameras based on the captured checkerboard image, and then generate a mapping table to align the left and right images. After correction, according to the application scheme, further 3D adaptive adjustment is performed to optimize the 3D display effect, see Figure 1E shown.

[0054] Through the above correction operation, the left-eye image and the right-eye image acquired in step S110 are corrected, which can further improve the 3D calibration efficiency and the 3D display effect.

[0055] Pixel calibration is to determine the relationship between physical displacement and pixel displacement and perform calibration. For example: Figure 1F, set the magnification to 10x, if the maximum image size captured within the field of view is 32mm×18mm, then in 4K image (resolution is 3840×2160), the pixel size Lpixel=32 / 3840=8.3um / pixel. By using the pixel size, the conversion between the length unit and the number of pixels can be realized.

[0056] Figure 2A The flowchart of a 3D calibration method of a 3D optical device according to another embodiment of the present disclosure is exemplarily shown.

[0057] like Figure 2A As shown, the 3D optical device 3D calibration method may include the following steps S210-S240:

[0058] In step S210, a left-eye image and a right-eye image respectively captured by a binocular camera in a 3D optical device are obtained.

[0059] This step and Figure 1A The step S110 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0060] In step S220, based on the preset window, window adjustment is performed according to the acquired left-eye image and right-eye image, and the window adjustment includes shifting at least one window in the left-right direction according to the input moving distance of at least one window in the left-right direction.

[0061] The moving distance input in step S220 can be represented by the number of pixels or length units. For example, the moving distance is: moving 20 pixels, or moving 2 mm. The moving direction is the window moving left or the window moving right, so as to adjust the parallax between the left and right eye images.

[0062] In step S220, by observing the relative position between the left eye image and the right eye image captured by the binocular camera, the moving distance of at least one of the first segmented window and the second segmented window in the left-right direction can be input, and then at least one window can be displaced in the left-right direction according to the input moving distance, so as to achieve the left-right position movement of the image in the field of view. For example, see Figure 2B , move the window 20 pixels to the left on the captured image, which is equivalent to moving the image 20 pixels to the left in the field of view.

[0063] When only one of the split windows (i.e., the first split window or the second split window) is adjusted to the left and right positions, the split window to be adjusted, the moving direction and moving distance of the split window can be input, so that the split window is displaced in the specified direction and the specified distance according to the input moving direction and moving distance, so as to adjust the relative position between the left and right eye images in the left and right directions.

[0064] In the case of adjusting the left and right positions of two split windows (i.e., the first split window and the second split window) at the same time, as an implementation method, the moving direction and moving distance of the first split window and the moving direction and moving distance of the second split window can be input, so that according to the input moving direction and moving distance, the first split window and the second split window are controlled to move in a specified direction and a specified distance, so as to adjust the relative position between the left and right eye images in the left and right directions. Among them, the moving directions of the input first split window and the second split window can be the same or different. The moving distances of the input first split window and the second split window can be the same or different, which are input based on specific needs.

[0065] In the case of adjusting the two split windows (i.e., the first split window and the second split window) for left and right position adjustment at the same time, as another implementation method, only a moving distance with positive and negative values ​​can be input, so that according to the input moving distance, the first split window and the second split window are controlled to move the same specified distance in opposite directions, so as to adjust the relative position between the left and right eye images in the left and right directions. Exemplarily, when the first split window and the second split window are adjusted at the same time, the first split window and the second split window are symmetrically moved by the same distance in the left and right directions. Among them, the moving directions of the first split window and the second split window are opposite, and the moving distances are the same. When the input moving distance is a positive value, it indicates that the first split window moves to the left and the second split window moves to the right to increase the relative distance between the left and right eye images. When the input moving distance is a negative value, it indicates that the first split window moves to the right and the second split window moves to the left to reduce the relative distance between the left and right eye images.

[0066] It should be understood that the input movement distance information can allow the left and right eye images to be adjusted in the left and right directions according to the stepping of the pixel points, thereby improving the 3D calibration accuracy.

[0067] In step S230, the left-eye image and the right-eye image are segmented respectively according to the first segmentation window and the second segmentation window after the window adjustment to obtain segmented left-eye image and right-eye image.

[0068] This step and Figure 1A The step S130 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0069] In step S240, the segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

[0070] This step and Figure 1A The step S140 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0071] In the disclosed embodiment, the movement distance of at least one window in the left-right direction is input, and at least one window is displaced in the left-right direction according to the input information, thereby controlling the left-right displacement of the left and right eye images by the input movement distance. The use of the input control method can avoid the tediousness and uncertainty of manual calibration operations, greatly improving the calibration efficiency, while also avoiding the large size and additional cost of the mechanical device and the influence of the camera position offset caused by vibration.

[0072] Figure 3A The flowchart of a 3D calibration method of a 3D optical device according to another embodiment of the present disclosure is exemplarily shown.

[0073] like Figure 3A As shown, the 3D optical device 3D calibration method may include the following steps S310-S340:

[0074] In step S310, a left-eye image and a right-eye image respectively captured by a binocular camera in a 3D optical device are obtained.

[0075] This step and Figure 1A The step S110 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0076] In step S320, based on the preset window, according to the acquired left eye image and right eye image, the window adjustment is performed, the window adjustment includes adjusting the distance according to the input left and right eye parallax, determining the movement distance of at least one window in the left and right directions, and displacing at least one window in the left and right directions according to the movement distance in the left and right directions.

[0077] The left-eye disparity adjustment distance input in step S320 refers to the change in the relative distance between the left-eye image and the right-eye image. The input left-eye disparity adjustment distance can be a numerical value with positive and negative values. Among them, a positive value represents an increase in the left-eye disparity, and a negative value represents a decrease in the left-eye disparity. The left-eye disparity adjustment distance can also be represented by the number of pixels or length units. For example, the left-eye disparity adjustment distance is: increase 20 pixels.

[0078] In step S320, when only one of the split windows (i.e., the first split window or the second split window) is adjusted to the left and right positions, the split window to be adjusted and the left-eye disparity adjustment distance with positive and negative values ​​can be input, so that the moving direction and moving distance of the split window are determined according to the input left-eye disparity adjustment distance and the corresponding displacement is performed to adjust the relative position between the left and right eye images in the left and right directions. For example, if the split window to be adjusted is the first split window and the left-eye disparity adjustment distance is +20, the first split window is moved 20 pixels to the left to increase the left-eye disparity by 20 pixels. If the split window to be adjusted is the second split window and the left-eye disparity adjustment distance is +20, the second split window is moved 20 pixels to the right to increase the left-eye disparity by 20 pixels.

[0079] Step S320, when adjusting two split windows (i.e., the first split window and the second split window) for left and right position adjustment at the same time, determines the moving direction and moving distance of the first split window and the second split window according to the input left and right eye disparity adjustment distance with positive and negative values. The first pair of split windows and the second split window are displaced accordingly to adjust the relative position between the left and right eye images in the left and right directions. Exemplarily, when adjusting the first split window and the second split window at the same time, the first split window and the second split window are symmetrically moved by the same distance in the left and right directions. For example, when the input left and right eye disparity adjustment distance is a positive value, it indicates that the first split window moves to the left and the second split window moves to the right to increase the relative distance between the left and right eye images. When the input left and right eye disparity adjustment distance is a negative value, it indicates that the first split window moves to the right and the second split window moves to the left to reduce the relative distance between the left and right eye images. The input left-right eye parallax adjustment distance is divided by 2, and the obtained result is used as the moving distance of the first segmentation window and the second segmentation window, so that the first segmentation window and the second segmentation window are symmetrically moved by the same distance in the left-right direction.

[0080] It should be understood that the input left-right eye parallax adjustment distance allows the left-right eye images to be adjusted in the left-right direction according to the adjustment granularity of the pixel points, thereby improving the 3D calibration accuracy.

[0081] The left-right eye parallax adjustment distance input in step S320 may be determined by at least the following two implementation methods:

[0082] As an implementation method, the relative position between the left-eye image and the right-eye image captured by the binocular camera can be observed, and the left-eye and right-eye parallax adjustment distance can be manually input, so that at least one window is displaced in the left-right direction according to the input left-eye and right-eye parallax adjustment distance, thereby realizing the left-right position movement of the image in the field of view.

[0083] As another implementation method, the left and right eye parallax adjustment distance can be automatically determined and input according to the parameter information of the binocular camera and the left and right eye target parallax, so that the window can be displaced in the left and right directions more quickly and accurately according to the automatically input left and right eye parallax adjustment distance, thereby realizing the left and right position movement of the image in the field of view.

[0084] For example, according to the formula Determine the effective left-eye and right-eye parallax under the specific parameters of the binocular camera, and determine the input left-eye and right-eye parallax adjustment distance according to the left-eye and right-eye target parallax and the left-eye and right-eye effective parallax. is the effective parallax of the left and right eyes of the binocular camera, is the baseline length of the binocular camera, is the focal length of the binocular camera, is the working distance of the binocular camera.

[0085] See also Figure 3B , baseline length Refers to the middle distance between the two lenses, in millimeters. Working distance It refers to the distance between the lens and the object A, and the unit can be millimeters. The object A can be the patient's eye, etc. By dividing the given baseline length ,focal length and working distance Substitute into the formula The effective parallax of the left and right eyes of the binocular camera can be determined . Effective parallax between left and right eyes It can be used to characterize the left-eye and right-eye parallax of a binocular camera under specific parameters. It is set based on business needs and needs to be adjusted to the left and right eye parallax. The left and right eye target parallax is the effective left and right eye parallax that conforms to human eye observation. For example, the actual observer's binocular distance can be determined as the left and right eye target parallax. , so that the actual observer can see the best 3D display effect. Subtract the effective parallax of left and right eyes The obtained difference is determined as the input left-eye disparity adjustment distance, so that the window is adjusted according to the left-eye disparity adjustment distance, that is, the first segmentation window and the second segmentation window have a relative displacement of the difference, so that the disparity between the adjusted left-eye and right-eye images is equal to the left-eye target disparity , so that the optimal 3D display effect can be achieved more quickly through a single window adjustment. It should be noted that, according to the calibrated pixel size, the left and right eye parallax adjustment distance can be converted from length units to pixel numbers. In addition, when the focal length of the binocular camera is and working distance When changes occur, it is necessary to re-determine and re-enter the parallax adjustment distance for the left and right eyes, and then re-perform 3D calibration to ensure the optimal display effect of the 3D image.

[0086] It should be understood that when the parallax is moderate, the brain can fuse the two images to produce a sense of three-dimensionality. If the parallax is too large, it may cause diplopia (double vision) or visual fatigue; if the parallax is too small, it may not be able to effectively perceive depth. Therefore, according to the principle of 3D imaging of the human eye, the effective parallax B of the left and right eyes needs to be guaranteed to be within the target parallax standard range, that is, B min ≤B≤B max 3D effect can only be produced within.

[0087] For example, the maximum value B in the target parallax standard range is max and minimum value B min According to the formula and formula And the minimum field of view and maximum field of view To determine, among which, is the field of view angle. Among them, the minimum field of view angle and maximum field of view It is determined based on the optimal horizontal range of the human eye's viewing angle. For example, in the application scenario of a surgical microscope, the minimum viewing angle Set to 0.5 degrees, maximum field of view Set to 2 degrees.

[0088] Specifically, the given baseline length and minimum field of view Substitute into the formula The minimum working distance can be determined from , the given baseline length ,focal length and minimum working distance Substitute into the formula The maximum value B in the target parallax standard range can be determined max Similarly, the given baseline length and minimum field of view Substitute into the formula The maximum working distance can be determined from , the given baseline length ,focal length and maximum working distance Substitute into the formula The minimum value B in the target disparity representation range can be determined min It should be noted that when the baseline length of the binocular camera is and focal length When the change occurs, the minimum value B in the target parallax standard range needs to be re-determined. min and the maximum value B max .

[0089] For example, if the left and right eye target disparity is set Within target parallax standard range B min ≤B≤B max If the target parallax of the left and right eyes is set In order to present the best 3D display effect, the parallax of the left and right eyes can be directly The difference obtained by subtracting the effective parallax B of the left and right eyes is determined as the parallax adjustment distance of the left and right eyes, and the window is adjusted according to the difference, so that the left and right eye images after the window adjustment can present the best 3D display effect.

[0090] Exemplarily, there is also an adjustment coefficient k, and the left-eye and right-eye target disparity is adjusted according to the adjustment coefficient to update the left-eye and right-eye target disparity so that the left-eye and right-eye target disparity is within the target disparity standard range. Not within target parallax standard range B min ≤B≤B max Within, the left and right eye target disparity is adjusted according to the adjustment coefficient k Adjustment is performed so that the updated left and right eye target disparity is within the target disparity standard range. It can be a coefficient between 0 and 1. and adjustment coefficient The multiplication result is within the target disparity standard range, and the multiplication result is used as the adjusted left and right eye target disparity. ,Right now . According to the updated left and right eye target disparity after adjustment Determine the distance of left-eye and right-eye parallax adjustment. For example, adjust the updated left-eye and right-eye target parallax Subtract the effective parallax B of the left and right eyes, and the difference obtained is determined as the parallax adjustment distance of the left and right eyes, and the window is adjusted according to the difference, so that the left and right eye images after the window adjustment can also present the best 3D display effect. It should be noted that the adjustment coefficient can be a set fixed value, or a mapping table based on specific parameters, and the adjustment parameter does not limit k to only a fixed value setting. In addition, the setting of the adjustment coefficient is only a preferred setting of a target parallax.

[0091] For example, the adjustment factor This can be determined in at least three ways:

[0092] As the first implementation method: according to the left and right eye target parallax , minimum effective parallax B min and the maximum effective parallax B max Determine the adjustment factor For example, if the left and right eye target disparity Less than the minimum effective parallax B min , then the minimum effective disparity B min Divided by the left and right eye target disparity The result is used as the adjustment coefficient ,Right now =B min / ; If the left and right eye target disparity Greater than the maximum effective parallax B max , then the maximum effective disparity B max Divided by the left and right eye target disparity The result is used as the adjustment coefficient ,Right now =B max / .

[0093] As a second implementation method: adjust the coefficient Set to Sigmoid function, normalization function or hyperbolic tangent function, and use Sigmoid function, normalization function or hyperbolic tangent function to convert the left and right eye target disparity Mapping to target parallax standard range B min ≤B≤B max Inside.

[0094] As a third implementation method: pre-set different left and right eye target disparities , baseline length ,focal length and working distance Conduct multiple experiments under different conditions to determine the adjustment coefficient for each condition. And store it in the table, and then determine the adjustment coefficient by looking up the table

[0095] In step S330, the left-eye image and the right-eye image are segmented respectively according to the first segmentation window and the second segmentation window after the window adjustment to obtain segmented left-eye image and right-eye image.

[0096] This step and Figure 1A The step S130 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0097] In step S340, the segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

[0098] This step and Figure 1A The step S140 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0099] In the disclosed embodiments, a left-right eye parallax adjustment distance is input, and at least one window is displaced in the left-right direction according to the input left-right eye parallax adjustment distance, thereby controlling the left-right displacement of the left-right eye images by utilizing the input left-right eye parallax adjustment distance. The use of the input control method can avoid the tediousness and uncertainty of manual calibration operations, greatly improving the calibration efficiency, while also avoiding the large size and additional cost of the mechanical device and the influence of the camera position offset caused by vibration.

[0100] Figure 4A The flowchart of a 3D calibration method of a 3D optical device according to yet another embodiment of the present disclosure is exemplarily shown.

[0101] like Figure 4A As shown, the 3D optical device 3D calibration method may include the following steps S410-S440:

[0102] In step S410, a left-eye image and a right-eye image respectively captured by a binocular camera in a 3D optical device are obtained.

[0103] This step and Figure 1A The step S110 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0104] In step S420, based on the preset window, according to the acquired left-eye image and right-eye image, the window adjustment is performed, and the window adjustment includes shifting at least one of the first segmentation window in the left-eye image and the second segmentation window in the right-eye image in the left-right direction and the up-down direction.

[0105] The window adjustment in step S420 is: on the basis of shifting at least one of the first split window and the second split window in the left-right direction, shifting in the up-down direction is also performed, so as to achieve the movement of the image in the up-down position in the field of view. Figure 4B , move the window downward by 20 pixels on the captured image, which is equivalent to moving the image downward by 20 pixels in the field of view.

[0106] The up-and-down displacement process based on the left-right displacement of the window can be achieved at least in the following ways:

[0107] As a first implementation, the moving distance of the window in the up-down direction is a fixed value, which can be determined based on the parameter information of the 3D optical device. Exemplarily, the minimum value of the moving distance of at least one window in the up-down direction is 10 pixels, and the maximum value is 30 pixels. According to a conventional 4K display device, the moving distance of the window in the up-down direction can be set within the range of 10 to 30 pixels. Based on the adjustment of the left and right windows, the pixel displacement adjustment in the up-down direction is performed, which can enhance the stereoscopic vision of the image on the one hand, and minimize the misjudgment of the depth information on the other hand.

[0108] As a second implementation, the moving distance of at least one window in the up-down direction is determined according to the moving distance of at least one window in the left-right direction. After determining the moving distance of the window in the left-right direction according to the input parameters, the moving distance of the window in the up-down direction can be automatically determined according to the moving distance of the window in the left-right direction, thereby realizing the window linkage displacement. Exemplarily, the moving distance of the window in the left-right direction can be the same as the moving distance in the up-down direction, so that the image moves the same distance in both the up-down and left-right directions. Alternatively, the moving distance of at least one window in the up-down direction is a preset multiple of the displacement distance in the left-right direction, and the minimum value of the preset multiple is 1 / 10 and the maximum value is 1 / 5. Among them, the correspondence between the up-down direction and the left-right direction can be set based on business needs, such as moving to the left corresponds to moving up, and moving to the right corresponds to moving down. If the window moves n pixels to the left, the window also moves a×n pixels upward at the same time, wherein a is a preset multiple less than 1, such as a value range of 1 / 10 to 1 / 5, so that the adjustment of the image in the left-right direction can be highlighted.

[0109] As a third implementation, at least one window is displaced in the up-down direction according to the input moving distance of at least one window in the up-down direction. By observing the relative position between the left-eye image and the right-eye image captured by the binocular camera, the moving distance of at least one window in the first segmentation window and the second segmentation window in the up-down direction is input, and then at least one window is displaced in the up-down direction according to the input moving distance, thereby realizing flexible movement of the up-down position of the image in the field of view to meet the personalized needs of users.

[0110] In step S430, the left-eye image and the right-eye image are segmented respectively according to the first segmentation window and the second segmentation window after the window adjustment to obtain segmented left-eye image and right-eye image.

[0111] This step and Figure 1A The step S110 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0112] In step S440, the segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

[0113] This step and Figure 1A The step S110 shown corresponds to the same or similar step, and will not be described in detail in the embodiment of the present disclosure.

[0114] In the disclosed embodiment, by displacing at least one window in the first split window and the second split window in the left-right direction, a displacement is also performed in the up-down direction, thereby achieving the movement of the image in the field of view, and then better adjusting the relative position between the left and right eye images, and further improving the 3D calibration effect.

[0115] As an optional embodiment, the method further includes: when segmenting the image, reducing and adjusting the segmentation range of the preset window according to at least one window displacement information.

[0116] It should be noted that the adjustment of the window position may cause visual distortion of the edge of the image. For example, after the window is adjusted, no image is displayed on the edge of the image. Therefore, when the image is segmented, the distorted pixels on the edge are also segmented by reducing the window, thereby ensuring that the segmented image has valid pixels and eliminating the visual impact of the distorted image on the overall image.

[0117] It should be understood that when performing image segmentation, the window cutting range is determined according to at least one window displacement information, and the first segmentation window and the second segmentation window are reduced according to the window cutting range. Among them, the window reduction range depends on the window displacement information. For example, if the window moves 50 pixels to the left, the window length needs to be reduced by 50 pixels to ensure that all pixels in the window are valid pixels. Exemplarily, when the shape of the segmentation window is square, the window cutting range may include the window cutting length and cutting width. The window displacement information may include the window moving direction and the window moving distance. According to the window moving direction, it can be determined whether the length or width of the window is cut, and the specific window cutting value is determined according to the window moving distance. A specific implementation method may be: determining the relative movement distance between the first segmentation window and the second segmentation window in the left-right direction as the segmentation length of the first segmentation window and the second segmentation window, determining the relative movement distance between the first segmentation window and the second segmentation window in the up-down direction as the segmentation width of the first segmentation window and the second segmentation window, and reducing the segmentation length in the left-right direction of the first segmentation window and the second segmentation window according to the determined segmentation length, and reducing the segmentation width in the up-down direction of the first segmentation window and the second segmentation window according to the determined segmentation width, thereby reducing the first segmentation window and the second segmentation window in the left-right direction and the up-down direction, and using the reduced first segmentation window and the second segmentation window to perform image segmentation, so that the segmented left and right eye images do not contain distorted parts caused by displacement, thereby improving the 3D imaging quality. For example, if the first split window and the second split window are shifted by m pixels in the left-right direction and n pixels in the up-down direction, the first split window and the second split window are reduced by m pixels in the left-right direction and n pixels in the up-down direction, so that the distortion caused by the window displacement can be cropped out, thereby improving the 3D imaging quality.

[0118] It should be understood that after determining the window displacement information, the window can be displaced according to the window displacement information first, and then the window can be reduced after the window displacement; or the window can be reduced according to the window displacement information first, and then the window can be displaced after the window is reduced. These two implementation methods can achieve the same technical effect, that is, the pixels in the final segmented left and right eye images are all valid pixels, thereby eliminating the visual impact of the distorted part of the image on the overall image, which is not limited in this embodiment.

[0119] Figure 5 A schematic diagram of a surgical microscope according to yet another embodiment of the present disclosure is exemplarily shown.

[0120] like Figure 5 As shown, the surgical microscope may include: a binocular camera 510, a processing module 520 and a 3D imaging display device 530.

[0121] The binocular camera 510 is used to capture left-eye images and right-eye images; the processing module 520 is used to implement the method of any embodiment of the present disclosure according to the captured left-eye images and right-eye images; the 3D imaging display device 530 is used to perform 3D imaging display on the input left-eye images and right-eye images.

[0122] The surgical microscope in the disclosed embodiment uses a 3D imaging display device instead of an eyepiece, allowing the doctor to complete the surgical operation in front of the screen, shortening the operation time and improving the patient's survival rate. Alternatively, the surgical bystander can obtain real-time 3D images of the operation through a variety of 3D perspectives such as 3D glasses or naked-eye 3D. The surgical microscope in the disclosed embodiment uses software control instead of the traditional manual or mechanical device to adjust the installation position of the binocular camera. It is simple to operate, high in precision and efficiency, and effectively solves the problems of low efficiency, poor precision, and susceptibility to vibration in the prior art.

[0123] The surgical microscope in the embodiment of the present disclosure performs window adjustment according to the left eye image and the right eye image respectively captured by the binocular camera 510 through the processing module 520: at least one of the first segmentation window in the left eye image and the second segmentation window in the right eye image is displaced, and the first segmentation window and the second segmentation window after the window adjustment are used to segment the left eye image and the right eye image respectively, and the segmented left eye image and the right eye image are enlarged to the target image size, thereby achieving the adjustment of the relative positions of the left and right eye images, and outputting the enlarged left eye image and the right eye image in a left-right arrangement format to the 3D imaging display device 530 for presenting the 3D display effect, so as to achieve the best 3D display effect by adjusting the relative positions of the left and right eye images, avoiding the tediousness and uncertainty of manual operation, and not needing to add additional mechanical devices, and having the advantages of easy use, high correction accuracy, time saving and no interference.

[0124] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the 3D calibration method of the 3D optical device provided in the above embodiments is implemented.

[0125] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0126] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0127] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0128] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0131] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A 3D calibration method for a 3D optical device, characterized in that: include: Obtaining a left eye image and a right eye image respectively captured by a binocular camera in a 3D optical device; Based on the preset window, according to the acquired left-eye image and right-eye image, the window adjustment includes shifting at least one of the first segmentation window in the left-eye image and the second segmentation window in the right-eye image; According to the first segmentation window and the second segmentation window after the window adjustment, the left-eye image and the right-eye image are segmented respectively to obtain the segmented left-eye image and the right-eye image; The segmented left-eye image and right-eye image are both enlarged to a target image size, and the enlarged left-eye image and right-eye image are output to a 3D imaging display device in a left-right arrangement format.

2. The calibration method according to claim 1, characterized in that: The window adjustment includes: According to the input moving distance of the at least one window in the left-right direction, the at least one window is displaced in the left-right direction; or, The moving distance of the at least one window in the left-right direction is determined according to the input left-right eye parallax adjustment distance, and the at least one window is displaced in the left-right direction according to the moving distance in the left-right direction.

3. The method according to claim 2, characterized in that When the first segmentation window and the second segmentation window are adjusted simultaneously, the first segmentation window and the second segmentation window are symmetrically moved by the same distance in the left-right direction.

4. The method according to any one of claims 2-3, characterized in that: According to the formula Determine the effective left-eye and right-eye parallax under the specific parameters of the binocular camera, and determine the input left-eye and right-eye parallax adjustment distance according to the left-eye and right-eye target parallax and the left-eye and right-eye effective parallax; in, is the effective parallax of the left and right eyes of the binocular camera, is the baseline length of the binocular camera, is the focal length of the binocular camera, is the working distance of the binocular camera.

5. The method according to claim 4, characterized in that: It also has an adjustment coefficient k, and adjusts the left-eye and right-eye target disparity according to the adjustment coefficient to update the left-eye and right-eye target disparity so that the left-eye and right-eye target disparity is within the target disparity standard range; Among them, the maximum and minimum values ​​in the target parallax standard range are calculated according to the formula and formula And the minimum and maximum field of view angles are determined, where is the field of view.

6. The method according to claim 5, characterized in that: The minimum viewing angle is set to 0.5 degrees, and the maximum viewing angle is set to 2 degrees.

7. The method according to claim 2 or 3, characterized in that: The window adjustment also includes: At least one of the first segmentation window in the left-eye image and the second segmentation window in the right-eye image is shifted in an up-down direction.

8. The method according to claim 7, characterized in that: The minimum value of the moving distance of the at least one window in the up and down directions is 10 pixels, and the maximum value is 30 pixels.

9. The method according to claim 7, characterized in that: The moving distance of the at least one window in the up-down direction is determined according to the moving distance of the at least one window in the left-right direction.

10. The method according to claim 9, characterized in that: The moving distance of the at least one window in the up-down direction is a preset multiple of the displacement distance in the corresponding left-right direction, and the minimum value of the preset multiple is 1 / 10 and the maximum value is 1 / 5.

11. The method according to claim 1, characterized in that: When performing image segmentation, the segmentation range of the preset window is reduced and adjusted according to the at least one window displacement information.

12. The method according to claim 1, characterized in that: Before making window adjustments, also include: Correct the acquired left-eye image and right-eye image, and / or perform pixel calibration.

13. A surgical microscope, characterized in that: It includes binocular cameras, processing modules and 3D imaging display devices; among which, The binocular camera is used to capture left-eye images and right-eye images; The processing module is used to implement any one of the methods of claims 1-12 according to the captured left-eye image and right-eye image; The 3D imaging display device is used for performing 3D imaging display on input left-eye images and right-eye images.

Images