Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of generating good composite images. [Solution] The image processing device comprises a pair of first imaging units that each output a first image, a second imaging unit that has a wider field of view than the field of view of the first imaging units and outputs a second image with a lower resolution than the resolution of the first image, a distance measuring unit that acquires distance information based on the distance to the subject, and a synthesis unit that generates a pair of composite images by combining the pair of first images and the second image based on the distance information.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] In a head-mounted display, a technique of imaging a target area and a peripheral area with different cameras is known. The techniques described in Patent Documents 1 to 3 display a synthesized image by synthesizing an image of a target area acquired by a high-resolution camera and an image of a peripheral area acquired by a low-resolution camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the techniques described in Patent Documents 1 to 3, misregistration may occur between the image of the target area and the image of the peripheral area, and the image quality of the synthesized image may deteriorate.

[0005] An object of the present invention is to provide an image processing apparatus capable of generating a favorable synthesized image.

Means for Solving the Problems

[0006] According to one disclosure of this specification, an image processing apparatus is provided, comprising: a pair of first imaging units that each output a first image; a second imaging unit that has a wider field of view than the field of view of the first imaging units and outputs a second image with a resolution lower than the resolution of the first image; a distance measuring unit that acquires distance information based on the distance to a subject; and a synthesis unit that generates a pair of composite images by combining the pair of first images and the second image, respectively, based on the distance information.

[0007] Another disclosure of this specification provides an image processing method comprising the steps of: outputting a pair of first images; outputting a second image having a wider field of view than the pair of first images and a lower resolution than the pair of first images; acquiring distance information based on the distance to a subject; and generating a pair of composite images by combining the pair of first images and the second image, respectively, based on the distance information.

[0008] Another disclosure of this specification provides a program for causing an image processing device to execute an image processing method characterized by: outputting a pair of first images; outputting a second image having a wider field of view than the pair of first images and a lower resolution than the pair of first images; acquiring distance information based on the distance to a subject; and generating a pair of composite images by combining the pair of first images and the second image, respectively, based on the distance information. [Effects of the Invention]

[0009] According to the present invention, an image processing apparatus capable of generating good composite images can be provided. [Brief explanation of the drawing]

[0010] [Figure 1] This is an external view of an image processing device according to the first embodiment. [Figure 2] This is a block diagram of an image processing device according to the first embodiment. [Figure 3] It is a diagram showing the imaging range of the imaging unit according to the first embodiment. [Figure 4] It is a diagram showing an example of image processing by the image processing unit according to the first embodiment. [Figure 5] It is a diagram showing an example of image processing by the synthesizing unit according to the first embodiment. [Figure 6] It is an example of a synthesized image in the image processing apparatus according to the first embodiment. [Figure 7] It is an example of a synthesized image in the image processing apparatus according to the comparative example. [Figure 8] It is a flowchart showing an image processing method in the image processing apparatus according to the first embodiment. [Figure 9] It is a block diagram of the image processing apparatus according to the second embodiment. [Figure 10] It is a diagram showing the imaging range of the imaging unit according to the second embodiment. [Figure 11] It is a diagram showing an example of image processing by the image processing unit according to the second embodiment. [Figure 12] It is a flowchart showing an image processing method in the image processing apparatus according to the second embodiment. [Figure 13] It is a block diagram of the image processing apparatus according to the third embodiment. [Figure 14] It is a diagram showing the distance measurement range of the distance measurement device according to the third embodiment. [Figure 15] It is a flowchart showing an image processing method in the image processing apparatus according to the third embodiment. [Figure 16] It is a block diagram of the image processing apparatus according to the fourth embodiment. [Figure 17] It is a diagram showing an example of applying the image processing apparatus according to the fifth embodiment to a vehicle. [Figure 18] It is a block diagram of the endoscope surgery system according to the sixth embodiment.

Embodiments for Carrying Out the Invention

[0011] [First Embodiment] FIG. 1(a) and FIG. 1(b) are external views of the image processing apparatus 1 according to the present embodiment. FIG. 1(a) is a front perspective view of the image processing apparatus 1, and FIG. 1(b) is a rear perspective view of the image processing apparatus 1. Although a head-mounted display is shown as an example of the image processing apparatus 1 according to the present embodiment in FIGS. 1(a) and 1(b), the present invention is not limited thereto.

[0012] The image processing apparatus 1 includes a main body unit 10, a mounting unit 20, an imaging unit 30, and a display unit 40. The main body unit 10 has a shape that covers the left and right eyes of the user when the image processing apparatus 1 is worn on the user's head.

[0013] The mounting unit 20 is provided on the side surface of the main body unit 10. The mounting unit 20 may be made of an elastic material such as rubber and may include a mounting band that fixes the main body unit 10 to the user's head. The mounting unit 20 may be configured to be expandable and contractible by a user's operation.

[0014] The imaging unit 30 is provided on the front surface of the main body unit 10. The imaging unit 30 images the external world (real space) corresponding to the orientation of the user's face. The imaging unit 30 includes a right imaging unit 30a, a left imaging unit 30b, and a central imaging unit 30c. The right imaging unit 30a and the left imaging unit 30b (a pair of first imaging units) are provided at a predetermined distance apart. The predetermined distance may preferably be the IPD (Interpupillary Distance). The right imaging unit 30a images a region (attention region) near the fixation point in the user's right eye. The left imaging unit 30b images the attention region in the user's left eye. The central imaging unit 30c (second imaging unit) is provided between the right imaging unit 30a and the left imaging unit 30b. The central imaging unit 30c images a region (peripheral region) including the attention region of the right eye and the attention region of the left eye. The optical systems of the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c are preferably arranged on the same straight line. Also, the optical axes of the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c are preferably provided so as to face the same direction.

[0015] The display unit 40 is located on the back of the main unit 10. The display unit 40 displays images of the external world captured by the imaging unit 30. The display unit 40 may also display a composite image of the external world and a virtual object. This enables the realization of MR (Mixed Reality).

[0016] The display unit 40 includes a right display unit 40a and a left display unit 40b. The right display unit 40a is positioned to correspond to the right eye of the user wearing the image processing device 1 and displays an image for the right eye. The left display unit 40b is positioned to correspond to the left eye of the user wearing the image processing device 1 and displays an image for the left eye.

[0017] Figure 2 is a block diagram of the image processing apparatus 1 according to this embodiment. The image processing apparatus 1 includes an imaging unit 30, a display unit 40, and a control unit 50. The imaging unit 30 includes a right imaging unit 30a, a left imaging unit 30b, and a central imaging unit 30c. The display unit 40 includes a right display unit 40a and a left display unit 40b. The control unit 50 includes an image processing unit 51, a distance measuring unit 52, and a synthesis unit 53.

[0018] The right imaging unit 30a includes a lens 31a and an image sensor 32a. The lens 31a forms an optical image of the subject onto the imaging surface of the image sensor 32a. The image sensor 32a may be composed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, or the like. The image sensor 32a may also be composed of a SPAD (Single Photo Avalanche Diode) image sensor. The image sensor 32a converts the optical image of the subject formed by the lens 31a into an electrical signal by photoelectric conversion and outputs it as image data to the image processing unit 51. Each pixel in the image sensor 32a includes a filter of a predetermined wavelength. The wavelength filters may be, for example, red, blue, and green primary color filters, and may be provided at each pixel of the image sensor 32a according to a Bayer array.

[0019] The left imaging unit 30b has a lens 31b and an image sensor 32b. The left imaging unit 30b may be configured in the same way as the right imaging unit 30a. The image sensor 32b converts the optical image of the subject formed by the lens 31b into an electrical signal by photoelectric conversion and outputs it as image data to the image processing unit 51.

[0020] The central imaging unit 30c has a lens 31c and an image sensor 32c. Lens 31c may have a shorter focal length than lenses 31a and 31b. Therefore, lens 31c forms an optical image with a larger optical size than lenses 31a and 31b on the imaging surface of the image sensor 32c. The image sensor 32c converts the optical image of the subject formed by lens 31c into an electrical signal by photoelectric conversion and outputs it as image data to the image processing unit 51.

[0021] The image sensor 32c is configured to measure the distance between the central imaging unit 30c and the subject. For example, the image sensor 32c may have pixels capable of detecting image plane phase difference. This allows the image sensor 32c to have a distance measuring function.

[0022] The number of pixels per angle of view of the central imaging unit 30c is less than the number of pixels per angle of view of the right imaging unit 30a and the left imaging unit 30b. For example, if the angle of view of the right imaging unit 30a is 60 degrees and the number of pixels is 1200, then the number of pixels per angle of view of the right imaging unit 30a is 20 pixels / degree. Similarly, if the angle of view of the left imaging unit 30b is 60 degrees and the number of pixels is 1200, then the number of pixels per angle of view of the left imaging unit 30b is 20 pixels / degree. In this case, the number of pixels per angle of view of the central imaging unit 30c will be less than 20 pixels / degree. For example, if the angle of view of the central imaging unit 30c is 120 degrees and the number of pixels is 1200, then the number of pixels per angle of view of the central imaging unit 30c may be 10 pixels / degree. Here, the higher the number of pixels per angle of view, the higher the resolution of the captured image. Therefore, the resolution of the images captured by the right imaging unit 30a and the left imaging unit 30b is higher than the resolution of the images captured by the central imaging unit 30c.

[0023] Figure 3 shows the imaging range of the imaging unit 30 according to the first embodiment. Figure 3 schematically shows the imaging ranges of the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c. The right imaging range 300a is the imaging range of the right imaging unit 30a and includes the area of ​​focus of the right eye. The left imaging range 300b is the imaging range of the left imaging unit 30b and includes the area of ​​focus of the left eye. The central imaging range 300c is the imaging range of the central imaging unit 30c and includes the peripheral area.

[0024] The field of view of the central imaging unit 30c is wider than that of the right imaging unit 30a and the left imaging unit 30b. This allows the central imaging unit 30c to image the peripheral area, including the focus area of ​​the right eye and the focus area of ​​the left eye. The fields of view of the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c can be configured to be variable. In this case, the field of view of the central imaging unit 30c can be set to be wider than that of the right imaging unit 30a and the left imaging unit 30b.

[0025] The control unit 50 may be composed of hardware similar to that of a general information processing device. For example, the control unit 50 may include a CPU (Central Processing Unit), main memory, communication unit, input / output interface, etc. Each functional block included in the control unit 50 may be composed of hardware such as an LSI (Large Scale Integration) with a program embedded in it. It is also possible to realize the functions of the control unit 50 by software by loading a program into the main memory and executing it with the CPU. The configuration of the control unit 50 is not particularly limited as long as it can realize the functions described in this embodiment.

[0026] The image processing unit 51 performs development processing on the image data and generates an image from the image data. Development processing may include cropping to cut out the effective range of the image data, correction processing for lens distortion, brightness correction processing, and demosaicing. The image processing unit 51 deforms the image so that the shape and size of the subject are the same in the image captured by the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c. Figure 4 is a diagram showing an example of image processing by the image processing unit 51 according to this embodiment. Figure 4 shows image captured of a cylindrical subject S1 and a rectangular parallelepiped subject S2. Figure 4 shows the right image 501a, the left image 501b, and the central image 501c. The right image 501a represents the area of ​​focus of the user's right eye. The right image 501a is obtained by developing the image data of the right imaging unit 30a. The left image 501b represents the area of ​​focus of the user's left eye. The left image 501b is obtained by developing the image data of the left imaging unit 30b. The central image 501c represents the peripheral areas of the user's left and right eyes. The central image 501c is obtained by developing the image data from the central imaging unit 30c. The image processing unit 51 enlarges or reduces the images captured by the right imaging unit 30a, left imaging unit 30b, and central imaging unit 30c so that the shape and size of subjects S1 and S2 are the same, thereby generating the right image 501a, left image 501b, and central image 501c.

[0027] The distance measuring unit 52 calculates the distance from the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c to the subject based on the image data, and acquires this distance information. For example, the distance measuring unit 52 calculates the distance to the subject from the image plane phase difference detected by the pixels of the image sensor 32c. Based on the calculated distance, the distance measuring unit 52 may perform correction processing such as smoothing, opening, and closing on the image data.

[0028] The compositing unit 53 sets the camera coordinates Mw(Xw,Yw,Zw) in the world coordinate system based on distance information, using the central imaging unit 30c as the reference point. Here, the Z axis is the depth direction from the imaging unit 30 to the subject, and the X and Y axes are two different directions orthogonal to the Z axis. The compositing unit 53 converts the camera coordinates Mw(Xw,Yw,Zw) of the central imaging unit 30c to the camera coordinate system Mc(Xc,Yc,Zc) of the right imaging unit 30a. The coordinate transformation between camera coordinates Mw(Xw,Yw,Zw) and camera coordinate system Mc(Xc,Yc,Zc) is expressed by the following formula.

number

[0029] The composite unit 53 transforms the camera coordinate system Mc into image coordinates x(xi,yi) by perspective projection transformation. The coordinate transformation between the camera coordinate system Mc and image coordinates x is expressed by the following equation.

number

[0030] Furthermore, the following equation holds between the camera coordinate system Mc and the image coordinate x.

number

[0031] Figure 5 shows an example of image processing by the compositing unit 53 according to this embodiment. The right-transformed image 502a is generated by performing a coordinate transformation on the central image 501c to match the right image 501a. The left-transformed image 502b is generated by performing a coordinate transformation on the central image 501c to match the left image 501b. Due to the coordinate transformation, subjects S1 and S2 move. Since subject S1 is located closer to the image processing device 1 than subject S2, the amount of movement of subject S1 in the coordinate transformation of the compositing unit 53 is greater than the amount of movement of subject S2.

[0032] The synthesis unit 53 synthesizes the image of the surrounding region after coordinate transformation with the image of the region of interest to generate a composite image. The synthesis unit 53 outputs a pair of composite images to the right display unit 40a and the left display unit 40b. Figure 6 is an example of a composite image in the image processing device 1 according to this embodiment. In Figure 6, the boundary L represented by the dotted line represents the boundary L of the synthesis between the image of the region of interest and the image of the surrounding region. The right composite image 503a is a composite image of the right image 501a and the right transformed image 502a. In the right composite image 503a, the area inside the boundary L corresponds to the right image 501a, and the area outside the boundary L corresponds to the right transformed image 502a. Since image synthesis is performed using the right transformed image 502a, which has undergone coordinate transformation, positional shift at the boundary L due to parallax between the right imaging unit 30a and the central imaging unit 30c is suppressed. This makes it possible to generate a good composite image. The left composite image 503b is a composite image of the left image 501b and the left transformed image 502b. In the left composite image 503b, the area inside the boundary L corresponds to the left image 501b, and the area outside the boundary L corresponds to the left transformed image 502b. Since the image synthesis is performed using the left transformed image 502b, which has undergone coordinate transformation, positional shifts at the boundary L due to parallax between the left imaging unit 30b and the central imaging unit 30c are suppressed. As a result, a good composite image can be generated.

[0033] The blending section 53 may also undergo processes such as alpha blending, multiband blending, Poisson blending, and stitching.

[0034] Figure 7 shows an example of a composite image produced by the comparative example image processing device. In the comparative example image processing device, no coordinate transformation is performed on the image in the surrounding region. In Figure 7, boundary L represents the boundary of the composite between the image of the region of interest and the image of the surrounding region. The right composite image 504a is a composite image of the right image 501a and the center image 501c. In the right composite image 504a, the area inside boundary L corresponds to the right image 501a, and the area outside boundary L corresponds to the center image 501c. The left composite image 504b is a composite image of the left image 501b and the center image 501c. In the left composite image 504b, the area inside boundary L corresponds to the left image 501b, and the area outside boundary L corresponds to the center image 501c. The composite image produced by the comparative example gives the user an unnatural impression due to the positional shift at boundary L.

[0035] Figure 8 is a flowchart showing the image processing method in the image processing apparatus 1 according to this embodiment. The image processing unit 51 reads out the pixel values ​​corresponding to the pixel data output from the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c (step S101). The image processing unit 51 performs development processing on the read out pixel values ​​to generate the captured images from the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c (step S102).

[0036] The distance measuring unit 52 calculates the distance from the central imaging unit 30c to the subject and acquires it as distance information (step S103). For example, the distance measuring unit 52 calculates the distance from the central imaging unit 30c to the subject based on the image plane phase difference detected by the pixels of the central imaging unit 30c.

[0037] The image processing unit 51 deforms the captured images so that the size of the subject is the same in the images captured by the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30 (step S104). As a result, the image processing unit 51 generates the right image 501a, the left image 501b, and the central image 501c.

[0038] The compositing unit 53 performs a coordinate transformation on the central image 501c according to equations (1) to (3) above (step S105). The compositing unit 53 transforms the camera coordinates Mw of the central imaging unit 30c into the camera coordinate system Mc of the right imaging unit 30a. That is, the compositing unit 53 moves subjects S1 and S2 in the central image 501c to match the right image 501a and generates the right-transformed image 502a. The compositing unit 53 also transforms the camera coordinates Mw of the central imaging unit 30c into the camera coordinate system Mc of the left imaging unit 30b. That is, the compositing unit 53 moves subjects S1 and S2 in the central image 501c to match the left image 501b and generates the left-transformed image 502b.

[0039] The synthesis unit 53 synthesizes the image of the surrounding region after coordinate transformation with the image of the region of interest (step S106). The synthesis unit 53 synthesizes the right-transformed image 502a and the central image 501c to form the right-composite image 503a. The synthesis unit 53 also synthesizes the left-transformed image 502b and the central image 501c to form the left-composite image 503b. The right-composite image 503a is output to the right-display unit 40a, and the left-composite image 503b is output to the left-display unit 40b.

[0040] The right display unit 40a displays the right composite image 503a output from the composite unit 53, and the left display unit 40b displays the left composite image 503b output from the composite unit 53 (step S107).

[0041] As described above, in this embodiment, by performing image synthesis based on distance information, it is possible to suppress positional shifts in images and generate a good composite image.

[0042] [Second Embodiment] Next, an image processing apparatus according to the second embodiment will be described. The image processing apparatus according to this embodiment differs from the first embodiment in that it acquires distance information from two central imaging units. The following description will focus on the configuration that differs from the first embodiment.

[0043] Figure 9 is a block diagram of the image processing device 1 according to this embodiment. The imaging unit 30 further includes a central imaging unit 30d. In this embodiment, the images captured by the central imaging unit 30c and the images captured by the central imaging unit 30d are used to calculate the distance to the subject. The central imaging unit 30d includes a lens 31d and an image sensor 32d. The central imaging unit 30d can be configured in the same way as the central imaging unit 30c. The image sensor 32d converts the optical image of the subject formed by the lens 31d into an electrical signal by photoelectric conversion and outputs it as image data to the image processing unit 51.

[0044] The number of pixels per field of view of the central imaging unit 30d is less than the number of pixels per field of view of the right imaging unit 30a and the left imaging unit 30b. Therefore, the resolution of the image captured by the central imaging unit 30d is lower than the resolution of the image captured by the right imaging unit 30a and the left imaging unit 30b.

[0045] Figure 10 shows the imaging range of the imaging unit 30 according to the second embodiment. Figure 10 schematically shows the imaging ranges of the right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and central imaging unit 30d when the image processing device 1 is viewed from above. In Figure 10, the right imaging unit 30a and left imaging unit 30b are provided between the central imaging unit 30c and central imaging unit 30d, but this is not limited to this arrangement. For example, the central imaging unit 30c and central imaging unit 30d may be provided between the right imaging unit 30a and left imaging unit 30b. The central imaging range 300d is the range imaged by the central imaging unit 30d and includes the peripheral region. It is preferable that the optical systems of the right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and central imaging unit 30d are provided on the same straight line. Furthermore, it is preferable that the optical axes of the right imaging unit 30a, the left imaging unit 30b, the central imaging unit 30c, and the central imaging unit 30d are oriented in the same direction. This eliminates the need for coordinate transformation of the X or Y coordinates in equation (1) above, thereby reducing the processing load in coordinate transformation.

[0046] The field of view of the central imaging unit 30d is wider than that of the right imaging unit 30a and the left imaging unit 30b. This allows the central imaging unit 30d to image the peripheral area including the focus area of ​​the right eye and the focus area of ​​the left eye. The field of view of the central imaging unit 30d can be configured to be variable. In this case, the field of view of the central imaging unit 30d can be set to be wider than that of the right imaging unit 30a and the left imaging unit 30b.

[0047] The image processing unit 51 deforms the captured images so that the shape and size of the subjects contained in the captured images of the right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and central imaging unit 30d are the same. Figure 11 is a diagram showing an example of image processing by the image processing unit 51 according to this embodiment. Figure 11 shows the right image 505a, left image 505b, central image 505c, and central image 505d. The central image 505c and central image 505d represent the peripheral areas of the user's left and right eyes and are obtained by developing the image data of the central imaging unit 30c and central imaging unit 30d, respectively. The image processing unit 51 enlarges or reduces the images of the right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and central imaging unit 30d so that the shape or size of subjects S1 and S2 are the same.

[0048] The distance measuring unit 52 calculates the distance to the subject using the principle of triangulation based on the images from the central imaging unit 30c and the central imaging unit 30d.

[0049] Figure 12 is a flowchart of the image processing method in the image processing apparatus 1 according to this embodiment. Steps S101, S102, and S104-S107 are the same as in the first embodiment. In the flowchart of Figure 12, step S108 is performed instead of step S103 in the flowchart of the first embodiment.

[0050] The distance measuring unit 52 calculates the distance to the subject using the images from the central imaging unit 30c and the central imaging unit 30d, and acquires it as distance information (step S108). The distance measuring unit 52 calculates the distance to the subject using the principle of triangulation based on the images from the central imaging unit 30c and the central imaging unit 30d.

[0051] In this embodiment as well, a good composite image can be generated by performing image synthesis based on distance information. In particular, in this embodiment, distance information based on the distance to the subject can be acquired without using an imaging unit capable of detecting image plane phase difference.

[0052] [Third Embodiment] Next, an image processing apparatus according to the third embodiment will be described. The image processing apparatus according to this embodiment differs from the first embodiment in that it further includes a distance measuring device for measuring the distance to the subject. The following description will focus on the configuration that differs from the first embodiment.

[0053] Figure 13 is a block diagram of the image processing device 1 according to this embodiment. The image processing device 1 further includes a distance measuring device 60. The distance measuring device 60 is, for example, a LiDAR device. The distance measuring device 60 can measure the distance to a subject by emitting light within a predetermined range and detecting the reflected light from the subject.

[0054] The distance measuring device 60 has a light-emitting unit 61 and a light-receiving unit 62. The light-emitting unit 61 may be an LED (Light Emitting Diode), an LD (Laser Diode), or a VCSEL (Vertical Cavity Surface Emitting Laser). The light-emitting unit 61 may also be a surface light-emitting device in which multiple VCSELs are arranged in an array. The light-emitting unit 61 emits pulsed light, such as laser light, toward the object.

[0055] The light-receiving unit 62 has multiple pixels arranged in a matrix. The light-receiving unit 62 receives reflected light from the subject and measures the distance to the subject. The light-receiving unit 62 may be, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. Alternatively, the light-receiving unit 62 may be a SPAD (Single Photon Avalanche Diode) sensor. The light-receiving unit 62 converts the reflected light signal into an electrical signal and outputs it to the image processing unit 51.

[0056] Figure 14 shows the distance measuring range of the distance measuring device 60 according to the third embodiment. Figure 14 schematically shows the imaging ranges of the right imaging unit 30a, the left imaging unit 30b, and the central imaging unit 30c, as well as the distance measuring range 600 of the distance measuring device 60. In Figure 14, the distance measuring device 60 is provided between the right imaging unit 30a and the central imaging unit 30c, but is not limited to this. For example, the distance measuring device 60 may be provided between the left imaging unit 30b and the central imaging unit 30c.

[0057] The distance measuring range 600 is wider than the central imaging range 300c. This allows the distance measuring device 60 to measure the distance to the subject in the peripheral area.

[0058] Figure 15 is a flowchart showing the image processing method in the image processing apparatus 1 according to this embodiment. Steps S101, S102, and S104-S107 are the same as in the first embodiment. In the flowchart of Figure 15, step S109 is performed instead of step S103 in the flowchart of the first embodiment.

[0059] The distance measuring unit 52 calculates the distance to the subject based on the electrical signal output from the light receiving unit 62 and acquires it as distance information (step S109).

[0060] In this embodiment as well, a good composite image can be generated by performing image synthesis based on distance information. Furthermore, in this embodiment as well, distance information based on the distance to the subject can be acquired without using an imaging unit capable of detecting image plane phase difference.

[0061] [Fourth Embodiment] Next, an image processing apparatus according to the fourth embodiment will be described. The image processing apparatus according to this embodiment differs from the first embodiment in that the display unit 40 is provided separately from the image processing apparatus 1. The following description will focus on the configuration that differs from the first embodiment.

[0062] Figure 16 is a block diagram of the image processing device 1 according to this embodiment. The display unit 40, which was provided in the image processing device 1 of the first embodiment, is provided separately from the image processing device 1. The configuration and operation of the image processing unit 51, distance measuring unit 52, and composite unit 53 are the same as in the first embodiment. The right composite image 503a and the left composite image 503b are output to the display unit 40, which is provided separately from the image processing device 1. In this embodiment as well, a good composite image can be generated.

[0063] [Fifth Embodiment] An image processing device and mobile device according to the fifth embodiment of this disclosure will be described with reference to Figure 17. Figure 17 is a diagram showing the configuration of the image processing device and mobile device according to this embodiment. Figure 17 shows an example of an image processing device related to an in-vehicle camera. The image processing device 1 is installed in the vehicle 2. The image processing device 1 is the same as in any of the first to fourth embodiments described above. The image processing device 1 calculates parallax (phase difference of parallax images) from a plurality of image data and acquires distance information based on the distance to the object based on the calculated parallax.

[0064] The image processing device 1 may be provided on the upper part of the windshield of the vehicle 2. The right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and distance measuring device 60 are preferably provided in the same manner as in the third embodiment, but are not necessarily provided in that manner. The right imaging unit 30a, left imaging unit 30b, central imaging unit 30c, and distance measuring device 60 may each be configured individually. In Figure 15, both the right imaging unit 30a and the left imaging unit 30b are provided, but only one of the right imaging unit 30a and the left imaging unit 30b may be provided. If only one of the right imaging unit 30a and the left imaging unit 30b is provided, it is preferable that one of the right imaging unit 30a and the left imaging unit 30b is provided between the central imaging unit 30c and the light receiving unit 62. The light emitting unit 61 and the light receiving unit 62 may each be provided individually. In Figure 17, the distance measuring range 600 is shown as the light receiving range of the light receiving unit 62.

[0065] Although the control unit 50 is not shown in Figure 17, the control unit 50 can be installed in any location that can control the right imaging unit 30a, the left imaging unit 30b, the central imaging unit 30c, and the distance measuring device 60. The control unit 50 may determine whether or not there is a possibility of collision based on distance information and, based on the determination result, output a control signal to generate braking force on the vehicle. The control unit 50 may also issue a warning to the driver based on the collision possibility determination result. For example, if the control unit 50 determines that there is a high possibility of collision, the control unit 50 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The control unit 50 may also warn the user by sounding an alarm, displaying warning information on a screen such as a car navigation system, or vibrating the seat belt or steering wheel.

[0066] As described above, the image processing apparatus 1 according to this embodiment performs image synthesis based on distance information, making it possible to generate a good composite image.

[0067] The above example illustrates control to prevent collisions with other vehicles, but it can also be applied to control systems that automatically follow other vehicles or control systems that automatically stay within their lanes. Furthermore, the image processing device 1 can be applied not only to vehicles such as the vehicle itself, but also to mobile objects (mobile devices) such as ships, aircraft, or industrial robots. In addition, it can be applied not only to mobile objects but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS).

[0068] [Sixth Embodiment] The technology described herein can be applied to a variety of products. For example, the technology described herein may be applied to an endoscopic surgical system, which is an example of a photodetection system. The photodetection system includes an image processing device 1 and a signal processing device that processes the output signal output from the image processing device 1.

[0069] Figure 18 is a schematic diagram of the endoscopic surgical system in this embodiment. Figure 18 shows a surgeon (physician) 1131 performing surgery on a patient 1132 on a patient bed 1133 using the endoscopic surgical system 1103. As shown in the figure, the endoscopic surgical system 1103 comprises an endoscope 1100, surgical instruments 1110, an arm 1121, and a cart 1134 equipped with various devices for endoscopic surgery.

[0070] The endoscope 1100 comprises a barrel 1101, the tip of which is inserted into the body cavity of the patient 1132 for a predetermined length, and a camera head 1102 connected to the proximal end of the barrel 1101. Figure 18 shows the endoscope 1100 configured as a so-called rigid endoscope having a rigid barrel 1101, but the endoscope 1100 may also be configured as a so-called flexible endoscope having a flexible barrel.

[0071] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 1101. A light source device 1203 is connected to the endoscope 1100. The light generated by the light source device 1203 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 1101, and is irradiated through the objective lens towards the object to be observed inside the body cavity of the patient 1132. The endoscope 1100 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0072] The camera head 1102 is equipped with the image processing device described in each of the embodiments described above, and the reflected light (observation light) from the object to be observed is focused into the image processing device by the optical system. The image processing device converts the observation light into an electrical signal, and generates an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The image processing device can be any of the image processing devices described in each of the embodiments described above. The image signal is transmitted as RAW data to the camera control unit (CCU) 1135.

[0073] The CCU1135 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operation of the endoscope 1100 and the display device 1136. Furthermore, the CCU1135 receives an image signal from the camera head 1102 and performs various image processing on the image signal, such as development processing (demosaic processing), to display an image based on the image signal. The CCU1135 also realizes the functions of the distance measuring unit 52 and the synthesis unit 53 described in each of the embodiments described above. The CCU1135 acquires distance information based on the distance to the object to be observed and generates a composite image based on the distance information.

[0074] The display device 1136 displays a composite image generated by the CCU 1135 under control from the CCU 1135.

[0075] The light source device 1203 is equipped with a light source such as an LED (Light Emitting Diode) and supplies illumination light to the endoscope 1100 when photographing the surgical area, etc.

[0076] The input device 1137 is an input interface for the endoscopic surgical system 1103. The user can input various types of information and instructions to the endoscopic surgical system 1103 via the input device 1137.

[0077] The treatment instrument control device 1138 controls the driving of the energy treatment instrument 1112 for purposes such as tissue cauterization, incision, or blood vessel sealing.

[0078] The light source device 1203 is capable of supplying illumination light to the endoscope 1100 when photographing the surgical area, and may be, for example, an LED, a laser light source, or a combination thereof to form a white light source. When a white light source is formed by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision. Therefore, the white balance of the captured image can be adjusted in the light source device 1203. In this case, the laser light from each of the RGB laser light sources may be irradiated onto the observation target in a time-division manner, and the drive of the image sensor of the camera head 1102 may be controlled in synchronization with the irradiation timing. This makes it possible to capture images corresponding to each of the RGB colors in a time-division manner. With this method, a color image can be obtained without providing a color filter on the image sensor.

[0079] Furthermore, the drive of the light source device 1203 may be controlled so that the intensity of the light output from the light source device 1203 is changed at predetermined time intervals. By controlling the drive of the image sensor of the camera head 1102 in synchronization with the timing of the change in light intensity to acquire images in time division and combining these images, it is possible to generate a high dynamic range image without so-called black crushing and white clipping.

[0080] Furthermore, the light source device 1203 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue can be utilized. Specifically, by irradiating with narrowband light compared to the irradiation light used during normal observation (i.e., white light), predetermined tissues such as blood vessels on the surface of mucosa can be imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light can be irradiated onto body tissue and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated onto the body tissue to obtain a fluorescence image. The light source device 1203 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0081] By applying the image processing devices of each embodiment described above to the endoscopic surgical system, the endoscopic surgical system of this embodiment can display a good composite image.

[0082] [Modified Embodiment] This disclosure is not limited to the embodiments described above and can be modified in various ways. For example, examples in which a part of one embodiment is added to another embodiment, or examples in which the configuration of the position diagram of another embodiment is replaced, are also embodiments of this disclosure.

[0083] The image processing apparatus 1 of the above embodiment may change the brightness of part or all of the image, or adjust the apparent brightness, before or after combining the right composite image 503a and the left composite image 503b.

[0084] Furthermore, in the image processing apparatus 1 of the above embodiment, the imaging unit 30 may be composed of three or more imaging units. In that case, the number of composite images and the number of images used for synthesis may be increased, and the same processing can be performed by the method described above.

[0085] Furthermore, the images captured and synthesized by the image processing device 1 of the above embodiment may be applied to surveillance or other purposes, and object recognition or object detection may be performed. For example, object recognition may be performed on the synthesized image, which is close to the human field of view. Alternatively, in order to perform object detection with lightweight processing using low-resolution images, object detection may be performed on the central image 501c before synthesis.

[0086] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0087] Furthermore, the embodiments described above are merely examples of concrete implementations in carrying out this disclosure, and the technical scope of this disclosure should not be interpreted as being limited by them. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0088] The above-disclosed embodiments include the following configurations and methods. (Composition 1) A pair of first imaging units, each outputting a first image, A second imaging unit having a wider field of view than the first imaging unit and outputting a second image with a lower resolution than the resolution of the first image, A distance measuring unit that acquires distance information based on the distance to the subject, An image processing apparatus comprising a synthesis unit that generates a pair of composite images by combining a pair of first images and a pair of second images based on the distance information. (Configuration 2) The second imaging unit includes an image sensor capable of detecting image plane phase difference, The image processing apparatus according to configuration 1, characterized in that the distance measuring unit acquires distance information based on the image plane phase difference. (Composition 3) The system comprises a plurality of the aforementioned second imaging units, The image processing apparatus according to configuration 1 or 2, characterized in that the distance measuring unit acquires the distance information by performing triangulation using a plurality of second images output from a plurality of second imaging units. (Composition 4) The optical axes of the first imaging unit and the second imaging unit are oriented in the same direction. The image processing apparatus according to any one of configurations 1 to 3, characterized in that the optical systems of the first imaging unit and the second imaging unit are arranged on the same straight line. (Composition 5) The device further includes a distance measuring device for measuring the distance to the aforementioned subject, The image processing apparatus according to any one of configurations 1 to 4, characterized in that the distance measuring unit acquires the distance information from the distance measuring device. (Composition 6) The optical axes of the first imaging unit, the second imaging unit, and the distance measuring device are oriented in the same direction. The image processing apparatus according to configuration 5, characterized in that the first imaging unit, the second imaging unit, and the distance measuring device are arranged on the same straight line. (Composition 7) The image processing apparatus according to any one of configurations 1 to 6, characterized in that the synthesis unit converts the coordinates of the second image to the coordinate system of the first image and generates a converted image. (Composition 8) The synthesis unit converts the coordinates of the second image to the coordinate system of the first image according to the following equation (1):

number

number

number

[0089] 1 Image processing device 10 Main body 20 Mounting part 30 Imaging Unit 40 Display section 50 Control Unit 51 Image Processing Unit 52 Ranging section 53 Synthesis section

Claims

1. A pair of first imaging units, each outputting a first image, A second imaging unit having a wider field of view than the first imaging unit and outputting a second image with a lower resolution than the first image, A distance measuring unit that acquires distance information based on the distance to the subject, An image processing apparatus comprising a synthesis unit that generates a pair of composite images by combining a pair of first images and a second image, respectively, based on the distance information.

2. The second imaging unit includes an image sensor capable of detecting image plane phase difference, The image processing apparatus according to claim 1, characterized in that the distance measuring unit acquires the distance information based on the image plane phase difference.

3. The system comprises a plurality of the aforementioned second imaging units, The image processing apparatus according to claim 1, characterized in that the distance measuring unit acquires distance information by performing triangulation using a plurality of second images output from a plurality of second imaging units.

4. The optical axes of the first imaging unit and the second imaging unit are oriented in the same direction. The image processing apparatus according to claim 1, characterized in that the optical systems of the first imaging unit and the second imaging unit are arranged on the same straight line.

5. The device further includes a distance measuring device for measuring the distance to the aforementioned subject, The image processing apparatus according to claim 1, characterized in that the distance measuring unit acquires the distance information from the distance measuring device.

6. The optical axes of the first imaging unit, the second imaging unit, and the distance measuring device are oriented in the same direction. The image processing apparatus according to claim 5, characterized in that the first imaging unit, the second imaging unit, and the distance measuring device are arranged on the same straight line.

7. The image processing apparatus according to claim 1, characterized in that the synthesis unit converts the coordinates of the second image to the coordinate system of the first image and generates a converted image.

8. The synthesis unit converts the coordinates of the second image to the coordinate system of the first image according to the following formula (1): [Math 1] The image processing apparatus according to claim 7, characterized in that in formula (1), Mc is the coordinate system of the first image, Mw is the coordinates of the second image, and [R][t] are external parameters of the first imaging unit.

9. The synthesis unit generates the transformed image from the coordinate system of the first image according to the following equations (2) and (3): [Math 2] [Math 3] In equation (2) above, s is a constant, x is the coordinate of the transformed image, A is an internal parameter of the first imaging unit, and Mc is the coordinate system of the first image. The image processing apparatus according to claim 8, characterized in that in formula (3), xi and yi are the coordinates of the transformed image, f is the focal length of the first imaging unit, and Xc, Yc, and Zc are the coordinates of the coordinate system of the first image.

10. The image processing apparatus according to claim 7, characterized in that the shorter the distance to the subject, the greater the amount of movement of the subject in the conversion from the second image to the converted image.

11. The image processing apparatus according to claim 1, further comprising an image processing unit that deforms the first image or the second image in the composite image such that the shape and size of the subject in the first image correspond to the shape and size of the subject in the second image, respectively.

12. The image processing apparatus according to claim 1, further comprising a pair of display units that display a pair of the aforementioned composite images.

13. The image processing apparatus according to claim 1, characterized in that the field of view of the first imaging unit and the second imaging unit are variable.

14. The steps include outputting a pair of first images, The steps include outputting a second image having a wider field of view than the pair of first images and a lower resolution than the pair of first images, Steps include obtaining distance information based on the distance to the subject, An image processing method characterized by comprising the step of generating a pair of composite images by combining a pair of first images and a pair of second images based on the distance information.

15. On the computer, The steps include outputting a pair of first images, The steps include outputting a second image having a wider field of view than the pair of first images and a lower resolution than the pair of first images, Steps include obtaining distance information based on the distance to the subject, A program for causing an image processing device to execute an image processing method characterized by the step of generating a pair of composite images by combining a pair of first images and a pair of second images based on the distance information.

16. It is a mobile object, An image processing apparatus according to any one of claims 1 to 13, A mobile body characterized by comprising control means for controlling the mobile body based on the distance information.

17. An image processing apparatus according to any one of claims 1 to 13, A light detection system characterized by comprising a signal processing unit that processes signals output from the image processing unit.

Images