Image processing apparatus, image processing method, and program

Abstract

To provide a display device that displays a pseudo stereoscopic image by reflecting individual images with a plurality of half mirrors in different directions to allow visual recognition from different angles.SOLUTION: An image processing device comprises: an acquisition unit that acquires an image 40a from an imaging device 20a and acquires an image 40b from an imaging device 20b; a generation unit which arranges the image 40a in a partial region 150a corresponding to a first region and arranges the image 40b in a partial region 150b corresponding to a second region, and combines the images 40a and 40b to generate a composite image 150; and an output unit that outputs the composite image toward a display device so that by displaying the composite image on the display unit, a first part image of the partial region 150a of the composite image is reflected at a first half mirror to form a virtual image of the first partial image at a predefined position, and a second partial image of the second partial region of the composite image is reflected at a second half mirror to form a virtual image of the second partial image at a predefined position.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Patent Document 1 describes a video conferencing system that makes the direction in which a meeting participant on a display device faces match the actual position where the participant is present. Patent Document 2 describes a video display system that projects a video from a counterpart user onto a second HOE surface without projecting it onto a first HOE surface and displays it in front of the user. Patent Document 3 describes a hologram generation device that generates hologram data from image information and distance information. [Prior Art Documents] [Patent Documents] [Patent Document 1] Japanese Patent Application Laid-Open No. 11-103499 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-208273 [Patent Document 3] Japanese Patent No. 5831930

Summary of the Invention

Problems to be Solved by the Invention

[0003] In a display device that pseudo-stereoscopically displays an image by reflecting each image in a different direction with each of a plurality of half mirrors so that it can be viewed from different angles, real-time performance of the displayed image is required.

Means for Solving the Problems

[0004] An image processing apparatus according to an aspect of the present invention reflects a first partial image in a first region of an image displayed on a display unit by a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflects a second partial image in a second region of the image displayed on the display unit by a second half mirror to form a virtual image of the second partial image at a predetermined position. The image processing apparatus may provide an image to be displayed on the display unit for a display device. The image processing apparatus may include an acquisition unit that acquires a first image from a first imaging device and a second image from a second imaging device. The image processing apparatus may include a generation unit that synthesizes the first image and the second image by disposing the first image in a first partial region corresponding to the first region and the second image in a second partial region corresponding to the second region to generate a synthesized image. The image processing apparatus may include an output unit that outputs the synthesized image to the display device so that a first partial image in the first partial region of the synthesized image is reflected by a first half mirror to form a virtual image of the first partial image at a predetermined position, and a second partial image in the second partial region of the synthesized image is reflected by a second half mirror to form a virtual image of the second partial image at a predetermined position.

[0005] The generation unit may synthesize the first image and the second image by rotating the second image with respect to the first image by a first angle and disposing it in the second partial region.

[0006] The acquisition unit may acquire a first image from a first imaging device that images a subject from a first direction and a second image from a second imaging device that images the subject from a second direction.

[0007] The generation unit may perform mask processing by converting pixel values of regions other than a first subject region including a subject included in the first image and regions other than a second subject region including a subject included in the second image into predetermined pixel values.

[0008] The image processing apparatus may include a reception unit that receives a selection of a subject attribute. The image processing apparatus may include a specifying unit that refers to a storage unit storing imaging parameters of each of a first imaging device and a second imaging device corresponding to the subject attribute, and specifies imaging parameters of each of the first imaging device and the second imaging device corresponding to the received subject attribute. The image processing apparatus may include a setting unit that sets the imaging parameters of each of the first imaging device and the second imaging device to the specified imaging parameters.

[0009] The first image may be a first video composition image that constitutes a first video captured by the first imaging device. The second image may be a second video composition image that constitutes a second video captured by the second imaging device. The composite image may be a video composition image that constitutes a video displayed on the display unit.

[0010] The generation unit may sequentially generate a composite image in real time from a first video captured by the first imaging device and a second video captured by the second imaging device. The output unit may sequentially output the composite image generated by the generation unit toward the display device in real time.

[0011] The display device may reflect a third partial image in a third region of an image displayed on the display unit with a third half mirror to form a virtual image of the third partial image at a predetermined position. The acquisition unit may acquire a third image from a third imaging device that images a subject from a third direction. The generation unit may synthesize the first image, the second image, and the third image by arranging the first image in a first partial region, the second image in a second partial region, and further arranging the third image in a third partial region corresponding to the third region, thereby generating a synthesized image. The output unit may display the synthesized image on the display unit, reflect a first partial image in a first partial region of the synthesized image with a first half mirror to form a virtual image of the first partial image at a predetermined position, reflect a virtual image of a second partial image in a second partial region of the synthesized image with a second half mirror to form a virtual image of the second partial image at a predetermined position, and reflect a third partial image in a third partial region of the synthesized image with a third half mirror to form a virtual image of the third partial image at a predetermined position, and output the synthesized image toward the display device.

[0012] The generation unit may synthesize the first image, the second image, and the third image by rotating the second image by a first angle with respect to the first image and arranging it in the second partial region, and rotating the third image by a second angle with respect to the first image and arranging it in the third partial region.

[0013] The first angle and the second angle may be predetermined based on the positional relationship between the synthesized image displayed on the display unit and each of the first half mirror, the second half mirror, and the third half mirror.

[0014] The display device may reflect a fourth partial image in a fourth region of an image displayed on the display unit with a fourth half mirror to form a virtual image of the fourth partial image at a predetermined position. The acquisition unit may acquire a fourth image from a fourth imaging device that images a subject from a fourth direction. The generation unit may arrange the first image in a first partial region, the second image in a second partial region, the third image in a third partial region corresponding to a third region, and the fourth image in a fourth partial region corresponding to a fourth region, thereby synthesizing the first image, the second image, the third image, and the fourth image to generate a synthesized image. The output unit may cause the synthesized image to be displayed on the display unit, reflect a first partial image in the first partial region of the synthesized image with a first half mirror to form a virtual image of the first partial image at a predetermined position, reflect a virtual image of a second partial image in the second partial region of the synthesized image with a second half mirror to form a virtual image of the second partial image at a predetermined position, reflect a third partial image in the third partial region of the synthesized image with a third half mirror to form a virtual image of the third partial image at a predetermined position, and reflect a fourth partial image in the fourth partial region of the synthesized image with a fourth half mirror to form a virtual image of the fourth partial image at a predetermined position, and output the synthesized image toward the display device.

[0015] The generation unit may synthesize the first image, the second image, the third image, and the fourth image by rotating the second image by a first angle with respect to the first image and arranging it in the second partial region, rotating the third image by a second angle with respect to the first image and arranging it in the third partial region, and rotating the fourth image by a third angle with respect to the first image and arranging it in the fourth partial region.

[0016] The first angle, the second angle, and the third angle may be predetermined based on the positional relationship between the synthesized image displayed on the display unit and each of the first half mirror, the second half mirror, the third half mirror, and the fourth half mirror.

[0017] The first image may be a first video-constituting image that constitutes a first video captured by a first imaging device. The second image may be a second video-constituting image that constitutes a second video captured by a second imaging device. The third image may be a third video-constituting image that constitutes a third video captured by a third imaging device. The fourth image may be a fourth video-constituting image that constitutes a fourth video captured by a fourth imaging device. The composite image may be a video-constituting image that constitutes a video displayed on a display unit.

[0018] The generation unit may sequentially generate a composite image in real time from a first video captured by a first imaging device, a second video captured by a second imaging device, a third video captured by a third imaging device, and a fourth video captured by a fourth imaging device. The output unit may sequentially output the composite image generated by the generation unit in real time toward a display device.

[0019] An image processing method according to an aspect of the present invention may provide an image to be displayed on a display device by reflecting a first partial image in a first region of an image displayed on the display unit with a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflecting a second partial image in a second region of the image displayed on the display unit with a second half mirror to form a virtual image of the second partial image at a predetermined position. The image processing method may include steps of acquiring a first image from a first imaging device and acquiring a second image from a second imaging device. The image processing method may include a step of synthesizing the first image and the second image to generate a composite image by arranging the first image in a first partial region corresponding to the first region and arranging the second image in a second partial region corresponding to the second region. The image processing method may include a step of outputting the composite image toward the display device so as to reflect a first partial image in a first partial region of the composite image with a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflect a second partial image in a second partial region of the composite image with a second half mirror to form a virtual image of the second partial image at a predetermined position.

[0020] A program according to an aspect of the present invention causes a computer to execute a step of providing an image to be displayed on a display unit for a display device that reflects a first partial image in a first region of an image displayed on the display unit with a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflects a second partial image in a second region of the image displayed on the display unit with a second half mirror to form a virtual image of the second partial image at a predetermined position. The program may cause a computer to execute a step of acquiring a first image from a first imaging device and a second image from a second imaging device. The program may cause a computer to execute a step of generating a composite image by synthesizing the first image and the second image by arranging the first image in a first partial region corresponding to the first region and arranging the second image in a second partial region corresponding to the second region. The program may cause a computer to execute a step of outputting the composite image to the display device so that a virtual image of the first partial image in the first partial region of the composite image is formed at a predetermined position by reflecting the first partial image with the first half mirror, and a virtual image of the second partial image in the second partial region of the composite image is formed at a predetermined position by reflecting the second partial image with the second half mirror.

[0021] Note that the above summary of the invention does not enumerate all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0023] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0024] A computer-readable medium may include any tangible device that can store instructions executable by an appropriate device. As a result, a computer-readable medium having instructions stored thereon will comprise a product that includes instructions that can be executed to create means for performing the operations specified in a flowchart or block diagram. Examples of computer-readable media may include electronic memory media, magnetic memory media, optical memory media, electromagnetic memory media, semiconductor memory media, and the like. More specific examples of computer-readable media may include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray (RTM) disk, memory stick, integrated circuit card, and the like.

[0025] Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. The source code or object code may include conventional procedural programming languages. Conventional procedural programming languages may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and the "C" programming language or similar programming languages. The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The processor or programmable circuit may execute the computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors may include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0026] FIG. 1 is a schematic diagram of a display device 10. The display device 10 is a 3D hologram display that pseudo-stereoscopically displays an image. The display device 10 includes an optical member 12 including a plurality of half mirrors 12a, 12b, and 12c, and a display unit 14. The display unit 14 may be an organic EL display, a liquid crystal display, etc. The display unit 14 may be a portable terminal including a display such as a smartphone. The display unit 14 may also be a screen that displays an image projected by a projector. The display surface 15 of the display unit 14 faces the plurality of half mirrors 12a, 12b, and 12c. In FIG. 1, the image displayed on the display surface 15 is depicted as being visible from the ceiling side, but in reality, the image cannot be seen from the ceiling side.

[0027] The plurality of half mirrors 12a, 12b, and 12c are arranged to be inclined with respect to the display surface 15. The plurality of half mirrors 12a, 12b, and 12c may be arranged to be inclined at approximately 45 degrees with respect to the display surface 15. The plurality of half mirrors 12a, 12b, and 12c may be integrally configured. Each of the plurality of half mirrors 12a, 12b, and 12c reflects the image displayed on the display unit 14 to form a virtual image inside the plurality of half mirrors 12a, 12b, and 12c. The display surface 15 includes a display area 15a, a display area 15b, and a display area 15c. The display area 15b is between the display area 15a and the display area 15c.

[0028] The object 16a displayed in the display area 15a is reflected by the half mirror 12a and can be visually recognized from the front of the half mirror 12a. The object 16b displayed in the display area 15b is reflected by the half mirror 12b and can be visually recognized from the front of the half mirror 12b. The object 16c displayed in the display area 15c is reflected by the half mirror 12c and can be visually recognized from the front of the half mirror 12c. The user can recognize that the image displayed on the display unit 14 exists inside each of the plurality of half mirrors 12a, 12b, and 12c by looking at the reflection surfaces of each of the plurality of half mirrors 12a, 12b, and 12c from the front of each of the plurality of half mirrors 12a, 12b, and 12c.

[0029] An image to be displayed on the display unit 14 of the display device 10 as described above usually needs to be prepared in advance. However, it is desired to display a moving image or a still image captured by an imaging device on the display unit 14 in real time to realize a pseudo 3D hologram.

[0030] Therefore, in the embodiment, an image processing system is provided that causes the display device 10 to display a composite image obtained by compositing in real time a plurality of images captured by a plurality of imaging devices.

[0031] FIG. 2 is a diagram showing an example of the overall configuration of the image processing system. The image processing system includes a plurality of imaging devices 20a, 20b, and 20c, a display device 10, and an image processing device 100. The image processing device 100 and the display device 10 are communicably connected via a network 50 such as the Internet.

[0032] Each of the plurality of imaging devices 20a, 20b, and 20c images the subject 30 from different directions. Each of the plurality of imaging devices 20a, 20b, and 20c may capture a moving image or a still image.

[0033] The imaging device 20a may image the left side of the subject 30. The imaging device 20b may image the front of the subject 30. The imaging device 20c may image the right side of the subject 30. The imaging device 20a images an image 40a that is displayed in the display area 15a and reflected by the half mirror 12a. The imaging device 20b images an image 40b that is displayed in the display area 15b and reflected by the half mirror 12b. The imaging device 20c images an image 40c that is displayed in the display area 15c and reflected by the half mirror 12c.

[0034] As shown in FIG. 3, the image processing device 100 generates a composite image 150 based on the images 40a, 40b, and 40c captured by the imaging devices 20a, 20b, and 20c. The image processing device 100 rotates the image 40a captured by the imaging device 20a counterclockwise by 90 degrees and arranges it in a partial area 150a including the lower left part of the composite image 150. The image processing device 100 rotates the image 40b captured by the imaging device 20b by 180 degrees and arranges it in a partial area 150b including the upper central part of the composite image 150. The image processing device 100 rotates the image 40c captured by the imaging device 20c clockwise by 90 degrees and arranges it in a partial area 150c including the lower right part of the composite image 150. The image processing device 100 may adjust the resolution of each of the images 40a, 40b, and 40c according to the resolution of the image to be displayed on the display unit 14. The image processing device 100 may reduce or enlarge each of the images 40a, 40b, and 40c according to the size of the image to be displayed on the display unit 14.

[0035] Note that the rotation direction or rotation angle of the images 40a, 40b, and 40c is not limited to the above rotation direction or the above rotation angle, and may be predetermined based on the positional relationship between the image to be displayed on the display unit 14 and the half mirrors 12a, 12b, and 12c.

[0036] FIG. 4 is a diagram showing an example of the functional blocks of the image processing apparatus 100. The image processing apparatus 100 includes a reception unit 102, a specification unit 104, a storage unit 106, a setting unit 108, an acquisition unit 110, a generation unit 112, and an output unit 114.

[0037] The acquisition unit 110 acquires the image 40b from the imaging device 20b that images the subject 30 from the front direction. The acquisition unit 110 acquires the image 40a from the imaging device 20a that images the subject 30 from the left direction. The acquisition unit 110 acquires the image 40c from the imaging device 20c that images the subject 30 from the right direction. Each of the images 40a, 40b, and 40c captured by the imaging devices 20a, 20b, and 20c may be a moving image constituent image that constitutes a moving image. That is, each of the images 40a, 40b, and 40c captured by the imaging devices 20a, 20b, and 20c may be a frame that constitutes a moving image.

[0038] The generation unit 112 rotates the image 40b by 180 degrees and arranges it in the partial region 150b corresponding to the display region 15b of the display unit 14, rotates the image 40a clockwise by 90 degrees and arranges it in the partial region 150a corresponding to the display region 15a of the display unit 14, and rotates the image 40c counterclockwise by 90 degrees and arranges it in the partial region 150c corresponding to the display region 15c of the display unit 14, thereby generating the composite image 150.

[0039] If the composite image 150 is a moving image constituent image that constitutes a moving image, the generation unit 112 may compress each generated composite image at a predetermined frame rate according to a predetermined moving image compression method, for example, MPEG4.

[0040] The generation unit 112 may reduce or enlarge the images 40a, 40b, and 40c according to the respective sizes of the partial regions 150a, 150b, and 150c. The generation unit 112 may identify a subject region including the subject 30 from each of the images 40a, 40b, and 40c by edge detection, face detection, etc., and perform mask processing by converting the pixel values of the regions other than the respective subject regions into predetermined pixel values (for example, the pixel values of R, G, and B are each "0", that is, black). The generation unit 112 may also convert the pixel values of the pixels in the regions that are not filled by the images 40a, 40b, and 40c among the partial regions 150a, 150b, and 150c into predetermined pixel values.

[0041] The generation unit 112 may place the image 40b in which the front of the subject 30 is imaged in the partial region 150b, rotate the image 40a counterclockwise by 90 degrees with respect to the image 40b and place it in the partial region 150a, and rotate the image 40c clockwise by 90 degrees with respect to the image 40b and place it in the partial region 150c, so as to synthesize the images 40a, 40b, and 40c to generate a synthesized image 150.

[0042] The output unit 114 outputs the synthesized image generated by the generation unit 112 toward the display device 10. If the synthesized image 150 is a moving image constituent image constituting a moving image, the output unit 114 may sequentially output a plurality of synthesized images compressed by the generation unit 112 as a moving image toward the display device 10 in real time. The output unit 114 may transmit the synthesized image to the display device 10 via the network 50.

[0043] When the imaging devices 20a, 20b, and 20c are fixed at predetermined positions, the size of the subject in the imaging area of the imaging device changes depending on the size of the subject or the position of the subject. The generation unit 112 may adjust the size of the subject in the composite image by adjusting the reduction ratio or the enlargement ratio for reducing or enlarging the image. However, when the generation unit 112 digitally enlarges the image, the resolution decreases and the quality of the image may deteriorate. Therefore, it is desirable to optically adjust the zoom magnification of each of the imaging devices 20a, 20b, and 20c according to the size or position of the subject. Also, the exposure value of each of the imaging devices 20a, 20b, and 20c may be adjusted according to the brightness of the subject.

[0044] The imaging devices 20a, 20b, and 20c may be fixed to respective gimbals (support mechanisms) that control the postures of the imaging devices 20a, 20b, and 20c. In this case, the postures of the imaging devices 20a, 20b, and 20c may be controlled by controlling the respective gimbals. That is, the imaging directions of the imaging devices 20a, 20b, and 20c may be controlled by controlling the respective gimbals. For example, the imaging directions of the imaging devices 20a, 20b, and 20c may be adjusted by controlling the respective gimbals according to the height of the subject or the position of the subject.

[0045] The reception unit 102 receives the selection of the attribute of the subject. The attribute of the subject may be the size of the subject or the brightness of the subject. The attribute of the subject may also be the position where the subject is arranged. The position where the subject is arranged may be the relative positional relationship between the subject 30 and each of the imaging devices 20a, 20b, and 20c.

[0046] The memory unit 106 may store the respective imaging parameters of the imaging devices 20a, 20b, and 20c according to the attributes of the subject. The imaging parameters may be the zoom ratio, the exposure value, or the focus position. When the imaging devices 20a, 20b, and 20c are fixed to the gimbals that control the postures of the imaging devices 20a, 20b, and 20c, the memory unit 106 may store, as the imaging parameters, the control values (the respective rotation angles in the yaw direction, the pitch direction, and the roll direction) of the gimbals corresponding to the imaging directions that the imaging devices 20a, 20b, and 20c should face according to the size or the position of the subject.

[0047] The specifying unit 104 refers to the memory unit 106 and specifies the respective imaging parameters of the imaging devices 20a, 20b, and 20c according to the received attributes of the subject. The specifying unit 104 may specify, for example, the zoom ratio, the focus position, and the control values of the gimbals according to the size of the subject. The specifying unit 104 may specify the exposure value according to the brightness of the subject.

[0048] The setting unit 108 sets the respective imaging parameters of the imaging devices 20a, 20b, and 20c to the specified imaging parameters. The setting unit 108 may transmit a control command for setting to the specified imaging parameters to each of the imaging devices 20a, 20b, and 20c.

[0049] The setting unit 108 may derive the respective zoom ratios or the control values of the gimbals of the imaging devices 20a, 20b, and 20c so that the size and the position of the subject in the images captured by the imaging devices 20a, 20b, and 20c are arranged at the predetermined size and the position. The setting unit 108 may command each of the imaging devices 20a, 20b, and 20c to set to the derived zoom ratio. The imaging devices 20a, 20b, and 20c may adjust the zoom ratio according to the command. The setting unit 108 may command each gimbal to operate with the derived control value. Each gimbal may control the respective postures of the imaging devices 20a, 20b, and 20c according to the command.

[0050] FIG. 5A shows an example of the configuration of the optical member 12 including the half mirrors 12a, 12b, and 12c provided in the display device 10. In the example shown in FIG. 5A, the display device 10 can visually recognize the image displayed on the display unit 14 from three directions. FIG. 5B shows the respective display regions 15a, 15b, and 15c of the display surface 15 reflected by each of the half mirrors 12a, 12b, and 12c.

[0051] FIG. 6A shows an example of the configuration of the optical member 12 including the half mirrors 12a, 12b, 12c, and 12d provided in the display device 10. In the example shown in FIG. 6A, the display device 10 can visually recognize the image displayed on the display unit 14 from four directions. FIG. 6B shows the respective display regions 15a, 15b, 15c, and 15d of the display surface 15 reflected by each of the half mirrors 12a, 12b, 12c, and 12d.

[0052] When the display device 10 can be visually recognized from four directions, the image processing system may further include an imaging device 20d in addition to the plurality of imaging devices 20a, 20b, and 20c.

[0053] FIG. 7 shows an example of the composite image 150 generated by the image processing device 100 when the image displayed on the display unit 14 can be visually recognized from four directions. The composite image 150 includes sub-regions 150a, 150b, 150c, and 150d obtained by dividing the central rectangular region 151 into four equal parts. The rectangular region 151 may be a square. Each of the sub-regions 150a, 150b, 150c, and 150d may be an isosceles triangle. Among each of the sub-regions 150a, 150b, 150c, and 150d, the regions that can be well visually recognized from the respective fronts reflected by each of the half mirrors 12a, 12b, 12c, and 12d are the rectangular regions 152a, 152b, 152c, and 152d. Therefore, each of the images 40a, 40b, 40c, and 40d captured by the imaging devices 20a, 20b, 20c, and 20d may be appropriately rotated and reduced or enlarged so as to fit into the rectangular regions 152a, 152b, 152c, and 152d.

[0054] The acquisition unit 110 acquires the image 40b from the imaging device 20b that images the subject 30 from the front direction. The acquisition unit 110 acquires the image 40a from the imaging device 20a that images the subject 30 from the left direction. The acquisition unit 110 acquires the image 40c from the imaging device 20c that images the subject 30 from the right direction. The acquisition unit 110 acquires the image 40d from the imaging device 20d that images the subject 30 from the back direction. Each of the images 40a, 40b, 40c, and 40d captured by the imaging devices 20a, 20b, 20c, and 20d may be a moving image component image that constitutes a moving image. The generation unit 112 rotates the image 40b by 180 degrees and arranges it in the partial region 150b corresponding to the display region 15b of the display unit 14, rotates the image 40a clockwise by 90 degrees and arranges it in the partial region 150a corresponding to the display region 15a of the display unit 14, rotates the image 40c counterclockwise by 90 degrees and arranges it in the partial region 150c corresponding to the display region 15c of the display unit 14, and arranges the image 40d in the partial region 150d corresponding to the display region 15d of the display unit 14, thereby generating the composite image 150. The above rotation direction or the above rotation angle is not limited and may be determined in advance based on the positional relationship between the image to be displayed on the display unit 14 and the half mirrors 12a, 12b, 12c, and 12d.

[0055] FIG. 8 is a flowchart showing an example of a procedure for generating a composite image in the image processing apparatus 100.

[0056] The reception unit 102 receives the attributes of the subject to be displayed on the display device 10 (S100). The reception unit 102 may receive the size of the subject as an attribute of the subject. Alternatively, the reception unit 102 may receive an attribute selected from a plurality of predetermined attributes that can identify the size of the subject, such as a person (a child, a medium-sized person, an adult), a small product, a medium-sized product, or a large product, as an attribute of the subject.

[0057] The specifying unit 104 refers to the storage unit 106 and specifies the imaging parameters corresponding to the received subject attributes for each imaging device (S102). The specifying unit 104 may specify the zoom ratio of each of the imaging devices 20a, 20b, 20c, and 20d as the imaging parameter.

[0058] The setting unit 108 sets the imaging parameters for each imaging device to the specified imaging parameters (S104). For example, the setting unit 108 may instruct the imaging devices 20a, 20b, 20c, and 20d to set the zoom ratio of each of the imaging devices 20a, 20b, 20c, and 20d to the specified zoom ratio. The imaging devices 20a, 20b, 20c, and 20d may adjust the position of the zoom lens according to the instruction.

[0059] After the setting of the imaging parameters is completed, the acquisition unit 110 acquires each video composition image of the videos captured by each imaging device (S106). The generation unit 112 generates a composite image based on each video composition image (S108). The generation unit 112 may generate a composite image by rotating it by a predetermined angle in a predetermined direction as needed, reducing it at a predetermined reduction ratio, and arranging it in a predetermined partial area. The generation unit 112 may cut out a subject area including a desired subject from each video composition image as a partial image, rotate each partial image by a predetermined angle as needed, reduce it at a predetermined reduction ratio, or enlarge it at a predetermined enlargement ratio, and arrange it in each partial area to sequentially generate a composite image in real time. The output unit 114 sequentially outputs in real time to the display device 10 a video composed of each composite image (S110).

[0060] According to the image processing apparatus 100 configured as described above, a plurality of images captured by a plurality of imaging devices can be synthesized in real time, and the synthesized composite image can be displayed on the display device 10 in real time.

[0061] Note that in the above, an example has been described in which the images to be displayed on each of the half mirrors 12a, 12b, 12c, and 12d are images obtained by imaging the same subject from different directions. However, the images to be displayed on each of the half mirrors 12a, 12b, 12c, and 12d may also be images obtained by imaging different subjects respectively.

[0062] Further, as shown in FIG. 9, for example, the display device 10 may be installed on the rooftop of a building or the like as an advertising signboard.

[0063] FIG. 10 shows an example of a computer 1200 in which multiple aspects of the present invention may be embodied in whole or in part. Programs installed in the computer 1200 can cause the computer 1200 to perform operations associated with the devices according to the embodiments of the present invention or function as one or more "parts" of such devices. Alternatively, the program can cause the computer 1200 to execute the operations or the one or more "parts". The program can cause the computer 1200 to execute a process according to the embodiments of the present invention or a stage of the process. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0064] The computer 1200 according to the present embodiment includes a CPU 1212 and a RAM 1214, which are interconnected by a host controller 1210. The computer 1200 also includes a communication interface 1222 and an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

[0065] The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program etc. executed by the computer 1200 when activated, and / or a program dependent on the hardware of the computer 1200. Programs are provided via a computer-readable recording medium such as a CR-ROM, a USB memory or an IC card, or via a network. The programs are installed in the RAM 1214 which is also an example of a computer-readable recording medium, or in the ROM 1230, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, bringing about cooperation between the programs and the various types of hardware resources described above. The device or method may be configured by realizing the operation or processing of information according to the use of the computer 1200.

[0066] For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. The communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or in a recording medium such as a USB memory under the control of the CPU 1212, transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer area etc. provided on the recording medium.

[0067] Also, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as a USB memory etc. to be read into the RAM 1214 and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0068] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. Also, the CPU 1212 may search for information in files, databases, etc. within the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0069] The programs or software modules described above may be stored in a computer-readable storage medium on or near the computer 1200. Also, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0070] It should be noted that the execution order of each process such as operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is not explicitly indicated as "earlier" or "preceding" etc. in particular, and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flows in the claims, the specification, and the drawings, even if "first," "next," etc. are used for convenience of explanation, it does not mean that it is essential to implement in this order.

[0071] As described above, the present invention has been described using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

Explanation of Reference Numerals

[0072] 10 Display device 12 Optical member 12a, 12b, 12c, 12d Half mirror 14 Display unit 15 Display surface 15a, 15b, 15c, 15d Display area 16a, 16b, 16c Object 20a, 20b, 20c, 20d Imaging device 30 Subject 40a, 40b, 40c Image 50 Network 100 Image processing device 102 Reception unit 104 Identification unit 106 Storage unit 108 Setting unit 110 Acquisition unit 112 Generation unit 114 Output unit 150 Composite image 150a, 150b, 150c, 150d Sub-region 151 Rectangular region 1200 Computer 1210 Host controller 1212 CPU 1214 RAM 1220 Input / output controller 1222 Communication interface 1230 ROM

Claims

1. An image processing apparatus that provides an image to be displayed on a display unit, which reflects a first partial image in a first region of an image displayed on the display unit by a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflects a second partial image in a second region of the image displayed on the display unit by a second half mirror to form a virtual image of the second partial image at a predetermined position, comprising: An acquisition unit that acquires a first image from a first imaging device that images a subject from a first direction and acquires a second image from a second imaging device that images the subject from a second direction; A generation unit that synthesizes the first image and the second image by arranging the first image in a first partial region corresponding to the first region and arranging the second image in a second partial region corresponding to the second region to generate a synthesized image; An output unit that outputs the synthesized image to the display device so that a first partial image in the first partial region of the synthesized image is reflected by the first half mirror to form a virtual image of the first partial image at a predetermined position, and a second partial image in the second partial region of the synthesized image is reflected by the second half mirror to form a virtual image of the second partial image at a predetermined position; A reception unit that receives a selection of an attribute of the subject from a plurality of predetermined attributes that can specify the size of the subject; A specifying unit that specifies at least a zoom magnification for optically adjusting, with reference to a storage unit that stores imaging parameters of the first imaging device and the second imaging device corresponding to the plurality of predetermined attributes of the subject, the imaging parameters of the first imaging device and the second imaging device corresponding to the received attribute of the subject; A setting unit that sets the imaging parameters of the first imaging device and the second imaging device to the specified imaging parameters An image processing apparatus comprising the same.

2. The generating unit synthesizes the first image and the second image by rotating the second image with respect to the first image by a first angle and arranging the second image in the second partial region, according to the image processing apparatus of claim 1.

3. The generating unit performs mask processing by converting pixel values of regions other than a first subject region including the subject included in the first image and regions other than a second subject region including the subject included in the second image into predetermined pixel values, according to the image processing apparatus of claim 1 or 2.

4. The first image is a first moving image constituent image that constitutes a first moving image captured by the first imaging device, The second image is a second moving image constituent image that constitutes a second moving image captured by the second imaging device, The composite image is a moving image constituent image that constitutes a moving image displayed on the display unit, according to the image processing apparatus of any one of claims 1 to 3.

5. The generating unit sequentially generates the composite image in real time from the first moving image captured by the first imaging device and the second moving image captured by the second imaging device, The output unit sequentially outputs the composite image generated by the generating unit in real time toward the display device, according to the image processing apparatus of claim 4.

6. The display device reflects a third partial image in a third region of the image displayed on the display unit by a third half mirror and forms a virtual image of the third partial image at a predetermined position, The acquisition unit acquires a third image from a third imaging device that images the subject from a third direction, The generating unit arranges the first image in the first partial region, arranges the second image in the second partial region, and further arranges the third image in a third partial region corresponding to the third region, thereby synthesizing the first image, the second image, and the third image to generate the composite image, The output unit outputs the composite image toward the display device so that by causing the display unit to display the composite image, a first partial image of the first partial region of the composite image is reflected by the first half mirror to form a virtual image of the first partial image at a predetermined position, a virtual image of a second partial image of the second partial region of the composite image is reflected by the second half mirror to form a virtual image of the second partial image at a predetermined position, and a third partial image of the third partial region of the composite image is reflected by the third half mirror to form a virtual image of the third partial image at a predetermined position. The image processing apparatus according to any one of claims 1 to 5.

7. The generation unit synthesizes the first image, the second image, and the third image by rotating the second image with respect to the first image by a first angle and arranging the rotated second image in the second partial region, and rotating the third image with respect to the first image by a second angle and arranging the rotated third image in the third partial region. The image processing apparatus according to claim 6.

8. The first angle and the second angle are predetermined based on the positional relationship between the composite image displayed on the display unit and each of the first half mirror, the second half mirror, and the third half mirror. The image processing apparatus according to claim 7.

9. The display device reflects a fourth partial image in a fourth region of an image displayed on the display unit by a fourth half mirror to form a virtual image of the fourth partial image at a predetermined position. The acquisition unit acquires a fourth image from a fourth imaging device that images the subject from a fourth direction. The generation unit arranges the first image in the first partial region, arranges the second image in the second partial region, arranges the third image in a third partial region corresponding to the third region, and arranges the fourth image in a fourth partial region corresponding to the fourth region, thereby synthesizing the first image, the second image, the third image, and the fourth image to generate the composite image. The output unit outputs the composite image toward the display device so as to display the composite image on the display unit, reflect a first partial image of the first partial region of the composite image by the first half mirror, form an image of a virtual image of the first partial image at a predetermined position, reflect a virtual image of a second partial image of the second partial region of the composite image by the second half mirror, form an image of the virtual image of the second partial image at a predetermined position, reflect a third partial image of the third partial region of the composite image by the third half mirror, form an image of a virtual image of the third partial image at a predetermined position, and reflect a fourth partial image of the fourth partial region of the composite image by the fourth half mirror, and form an image of a virtual image of the fourth partial image at a predetermined position. The image processing apparatus according to claim 6.

10. The generation unit synthesizes the first image, the second image, the third image, and the fourth image by rotating the second image with respect to the first image by a first angle and arranging the second image in the second partial region, rotating the third image with respect to the first image by a second angle and arranging the third image in the third partial region, and rotating the fourth image with respect to the first image by a third angle and arranging the fourth image in the fourth partial region. The image processing apparatus according to claim 9.

11. The first angle, the second angle, and the third angle are predetermined based on the positional relationship between the composite image displayed on the display unit and each of the first half mirror, the second half mirror, the third half mirror, and the fourth half mirror. The image processing apparatus according to claim 10.

12. The first image is a first moving image constituent image that constitutes a first moving image captured by the first imaging device. The second image is a second moving image constituent image that constitutes a second moving image captured by the second imaging device. The third image is a third moving image constituent image that constitutes a third moving image captured by the third imaging device. The fourth image is a fourth moving image constituent image that constitutes a fourth moving image captured by the fourth imaging device. The image processing apparatus according to any one of claims 9 to 11, wherein the composite image is a video composition image constituting a video displayed on the display unit.

13. The generation unit sequentially generates the composite image in real time from the first video captured by the first imaging device, the second video captured by the second imaging device, the third video captured by the third imaging device, and the fourth video captured by the fourth imaging device. The output unit sequentially outputs the composite image generated by the generation unit in real time toward the display device. The image processing apparatus according to claim 12.

14. An image processing method for providing an image to be displayed on the display unit, which reflects a first partial image in a first region of the image displayed on the display unit with a first half mirror to form a virtual image of the first partial image at a predetermined position, and reflects a second partial image in a second region of the image displayed on the display unit with a second half mirror to form a virtual image of the second partial image at a predetermined position, comprising: Obtaining a first image from a first imaging device that images a subject from a first direction, and obtaining a second image from a second imaging device that images the subject from a second direction; Arranging the first image in a first partial region corresponding to the first region and arranging the second image in a second partial region corresponding to the second region to synthesize the first image and the second image to generate a composite image; Outputting the composite image toward the display device so that the first partial image in the first partial region of the composite image is reflected by the first half mirror to form a virtual image of the first partial image at a predetermined position, and the second partial image in the second partial region of the composite image is reflected by the second half mirror to form a virtual image of the second partial image at a predetermined position, by displaying the composite image on the display unit; Receiving a selection of an attribute of the subject from among a plurality of predetermined attributes that can identify the size of the subject; Referring to a storage unit that stores the imaging parameters of each of the first imaging device and the second imaging device according to the plurality of predetermined attributes of the subject, and at least specifying a zoom ratio that optically adjusts the imaging parameters of each of the first imaging device and the second imaging device according to the received attribute of the subject; Setting the imaging parameters of each of the first imaging device and the second imaging device to the specified imaging parameters; An image processing method comprising:

15. A program for causing a computer to execute a step of providing an image to be displayed on a display unit, wherein a first partial image in a first region of the image displayed on the display unit is reflected by a first half mirror to form a virtual image of the first partial image at a predetermined position, and a second partial image in a second region of the image displayed on the display unit is reflected by a second half mirror to form a virtual image of the second partial image at a predetermined position, Obtaining a first image from a first imaging device that images a subject from a first direction, and obtaining a second image from a second imaging device that images the subject from a second direction; Arranging the first image in a first partial region corresponding to the first region and arranging the second image in a second partial region corresponding to the second region, thereby synthesizing the first image and the second image to generate a synthesized image; By displaying the synthesized image on the display unit, a virtual image of the first partial image in the first partial region of the synthesized image is formed at a predetermined position by reflecting the first partial image with the first half mirror, and a virtual image of the second partial image in the second partial region of the synthesized image is formed at a predetermined position by reflecting the second partial image with the second half mirror. Outputting the synthesized image to the display device; Receiving a selection of an attribute of the subject from among a plurality of predetermined attributes that can specify the size of the subject; Referring to a storage unit that stores imaging parameters of each of the first imaging device and the second imaging device according to the plurality of predetermined attributes of the subject, at least specifying a zoom ratio that optically adjusts as imaging parameters of each of the first imaging device and the second imaging device according to the received attributes of the subject; Setting imaging parameters of each of the first imaging device and the second imaging device to the specified imaging parameters; A program for causing a computer to execute.

Images