VR camera and VR system

Abstract

To provide a VR camera and a VR system configured to reduce discomfort due to the influence of parallax caused by the motion of the head of a user, as much as possible.SOLUTION: A VR camera 10 includes a camera body 12 having a plurality of lenses, and a rotating mechanism 24 which rotates the camera body 12. A rotation axis of the rotating mechanism 24 is positioned in a minimum parallax point where the parallax generated between a first image and a second image is minimized, the first image being generated when the camera body 12 is in a first attitude, the second image being generated when the camera body is in a second attitude which is different from the first attitude. When the rotating mechanism 24 is driven, an image captured by the VR camera 10 is not displayed on a display device 50 or the imaging of the VR camera 10 is stopped.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to VR technology, and more particularly to a VR camera having a plurality of lenses and a VR system using the same.

Background Art

[0002] In recent years, in various technical fields, the development of products and services using cameras having a plurality of lenses has been underway. For example, Patent Document 1 discloses a stereo camera in which each of two cameras includes a rotation mechanism. In the stereo camera described in Patent Document 1, it is described that by adjusting the arrangement of the rotation axes in the rotation mechanisms of the two cameras, it is possible to increase the detection range and widen the detection angle while ensuring the distance measurement accuracy of the detected object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, it is considered that the above-exemplified stereo camera can also be applied to technical fields such as virtual reality (hereinafter referred to as "VR") and three-dimensional images. For example, in the case of shooting an image to be displayed on a head-mounted display (hereinafter referred to as "HMD"). In this case, since there may be a time difference between the movement of the head of the user wearing the HMD and the display of the image, the influence of parallax may increase and the user may feel discomfort.

[0005] In view of the above problems, an object of the present invention is to provide a VR camera and a VR system capable of reducing as much as possible the discomfort caused by the influence of parallax due to the movement of the user's head.

Means for Solving the Problem

[0006] To achieve the above object, the VR camera according to the present invention includes a camera body having a plurality of lenses, and a rotation mechanism for rotating the camera body, wherein the rotation axis of the rotation mechanism is between a first image generated when the camera body is in a first posture and a second image generated when the camera body is in a second posture different from the first posture, and is arranged at a parallax minimization point that minimizes the parallax generated therebetween.

[0007] Further, the VR system according to the present invention includes a VR camera provided with a rotation mechanism for rotating a camera body having a plurality of lenses, a display device for displaying an image captured by the VR camera, and a communication unit for transmitting and receiving data between the VR camera and the display device, and is configured such that when the rotation mechanism is driving, an image captured by the VR camera is not displayed on the display device, or imaging by the VR camera is stopped.

Advantages of the Invention

[0008] According to the present invention, it is possible to reduce as much as possible the discomfort caused by the influence of parallax due to the movement of the user's head.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the VR camera and the VR system according to the present invention will be described with reference to the drawings.

[0011] 〔VR System〕 As shown in FIG. 1, the VR system 100 according to this embodiment is a system that displays an image captured by the VR camera 10 on an HMD (head-mounted display) 50 which is a display device.

[0012] 〔HMD〕 For the HMD 50 used in this embodiment, various off-the-shelf devices that are widely and generally popular can be adopted. The HMD 50 includes basic components such as a CPU 52, a storage unit 54 such as a memory and a storage, a communication module 56 that communicates with external devices, a battery (not shown), a display 58, and an acceleration sensor 60.

[0013] In the storage unit 54 of the HMD 50, a program for displaying an image on the display 58 in cooperation with the VR camera 10 described later is stored. Then, when the CPU 52 executes this program, the image captured by the VR camera 10 is received via the communication module 56, and the received image is displayed on the display 58. Also, when the user wearing the HMD 50 moves their head, a change in the posture of the HMD 50 is detected by the acceleration sensor 60. This change in the posture of the HMD 50 is transmitted to the VR camera 10 via the communication module 56. In this way, the acceleration sensor 60 functions as a detection unit that detects a change in the posture of the HMD 50.

[0014] 〔VR Camera〕 The VR camera 10 according to this embodiment includes a camera body 12. In the camera body 12, basic components of a computer such as a CPU 14, a storage unit 16 such as a memory and a storage, a communication module 18 for communicating with an external device, and a battery (not shown) are implemented. Further, in the camera body 12, a plurality of lenses 20L, 20C, 20R and an image sensor 22 for forming an image of the transmitted light of these lenses 20L, 20C, 20R are respectively provided. In this embodiment, two lenses 20L, 20R are arranged at intervals from each other, and a wide-angle lens 20C having a wider angle of view than both of these two lenses is arranged between these two lenses 20L, 20R. And it is configured such that the light transmitted through these three lenses 20L, 20C, 20R forms an image on the image sensor 22. For convenience of explanation, the two lenses arranged on both sides of the wide-angle lens 20C may be referred to as a right lens 20R and a left lens 20L, respectively.

[0015] The VR camera 10 further includes a rotation mechanism 24. The rotation mechanism 24 includes a base (not shown) that supports the camera body 12. A known pan-tilt mechanism is incorporated in this base. The pan-tilt mechanism is a mechanism that rotates the direction of the camera body in two directions, the horizontal direction (pan) and the vertical direction (tilt), and has a rotation axis for the pan operation and a rotation axis for the tilt operation. The pan-tilt operation is executed by the CPU 14 controlling a servo motor 26.

[0016] In the rotation mechanism 24 of the VR camera 10 of this embodiment, by appropriately arranging the rotation axis during the pan operation, it is intended to reduce the influence of parallax caused by the movement of the user's head. This point will be specifically described below.

[0017] Generally, when performing panoramic shooting with a single-lens camera, it is known that parallax can be eliminated by setting the position of the rotation axis in the horizontal direction (pan) around a fulcrum called the "no-parallax point". However, in the case of a camera having a plurality of lenses as in this example, the no-parallax point is not determined in the same manner as the single-lens camera described above. Therefore, in the present embodiment, the camera body 12 provided with three lenses (right lens 20R, left lens 20L, wide-angle lens 20C) is integrally rotated, and the optimal position of the rotation axis that can minimize the influence of parallax during the pan operation is determined.

[0018] First, several reference positions are appropriately determined within the range of the field of view (angle of view) of the VR camera 10 in the basic posture. For example, the vertices of a rectangle with four sides parallel to the frames of adjacent fields of view are used as reference positions, and the rotation angle and the magnitude of parallax when performing the pan operation are confirmed. By repeating this experimentally, the position of the rotation axis during the pan operation of the rotation mechanism 24 is determined. In the present embodiment, the position of the rotation axis during the pan operation determined in this way is referred to as the "parallax minimization point". In the VR camera 10 of the present embodiment, the parallax minimization point of the wide-angle lens 20C located in the center of the three lenses is arranged on the rotation axis during the pan operation of the rotation mechanism 24.

[0019] The storage unit 16 of the VR camera 10 stores a program for displaying an image on the display 58 of the HMD 50 in cooperation with the above-described HMD 50. Then, by the CPU 14 executing this program, various processes and operations of the VR camera 10, such as imaging processing, image processing, data transmission / reception processing, and operation control of the servo motor 26, which will be described later, are configured to be comprehensively controlled.

[0020] The VR camera 10 and the HMD 50 having the above configuration communicate with each other via the network communication line N by their respective communication modules 18 and 56, and data transmission and reception are performed. Thus, in the present embodiment, the communication module 18 of the VR camera 10 and the communication module 56 of the HMD 50 function as a communication unit.

[0021] 〔Various Operations / Processes〕 Subsequently, the operation of the VR system 100 according to the present embodiment will be described with reference to the drawings. In the following description, it is assumed that the initial setting for enabling the VR camera 10 and the HMD 50 to communicate with each other via the network communication line N has already been completed.

[0022] First, the operation / process of the VR camera 10 will be described with reference to FIG. 2. When the initial setting is completed, the VR camera 10 acquires an image in the field of view (angle of view) at the time of the initial setting (S10). In this example, a composite image in which the transmitted lights of the right lens 20R, the left lens 20L, and the wide-angle lens 20C are imaged on a single image sensor 22 is acquired.

[0023] After acquiring the composite image, image processing for generating a transmission image to be transmitted to the HMD 50 is performed (S12). In this example, for each image portion acquired via the right lens 20R and the left lens 20L, image processing is performed such that the vicinity of the center of each image portion has higher image quality than its peripheral side. For such image processing, image generation techniques disclosed in Japanese Patent No. 6472864 and Japanese Patent No. 7219620 can be utilized. Then, an image acquired via the wide-angle lens 20C is arranged in a region of the light-receiving surface of the image sensor 22 that is not filled with the images acquired via the right lens 20R and the left lens 20L. Thus, an image for transmission to the HMD 50 is generated. The image generated by the above-described image processing is transmitted to the HMD 50 together with the field-of-view data of the VR camera 10 as a first image when the VR camera 10 is in the first posture (S14).

[0024] When the transmission of the image is completed, it is confirmed based on the received data from the HMD 50 whether there is a change in the posture (user's line of sight) of the HMD 50 (S16). Here, if it is determined that there is no change (S16: No), the above-described series of processes is repeated. On the other hand, if it is determined that there is a change (S16: Yes), the servo motor 26 is controlled according to the amount of change (S18), and the posture of the VR camera 10 is changed. Then, in the same manner as the above-described series of processes, an image in the field of view (angle of view) of the VR camera 10 whose posture has changed is acquired, and the image generated by image processing is transmitted to the HMD 50 as a second image when the VR camera 10 is in the second posture together with the field of view data after the posture change of the VR camera 10.

[0025] Next, the operations and processes of the HMD 50 will be described with reference to FIG. 3. When the initial settings are completed, the HMD 50 receives an image and field of view data from the VR camera 10 (S50). Then, it is confirmed whether there is a change in the received field of view data (S52). Here, if it is determined that there is no change (S52: No), the received image is displayed on the display 58 (S54). On the other hand, if it is determined that there is a change (S52: Yes), the amount of change is calculated (S56), and the timing for displaying the received image on the display 58 is determined (S58). In this example, according to the mode of variation (specifically, the amount of rotation) of the posture of the VR camera 10, the time required until the transition from the image currently displayed on the display 58 (first image) to the image received at the time of the posture change (second image) is completed is changed. This required time is determined based on the function shown in FIG. 4. In this example, the larger the rotation amount of the VR camera 10, the longer the time required until the transition is completed.

[0026] When the time required until the transition is completed is determined, the transition process from the first image to the second image is executed (S60). In this transition process, the display image from the start of the transition to the completion of the transition (the image to be displayed until the transition from the first image to the second image) is generated by alpha blending, and this generated image is displayed as a complementary image. Thereby, the display image until the first image transitions to the second image can be complemented without a sense of incongruity.

[0027] When the determined required time elapses while executing the transition process, the second image is displayed on the display 58 (S54). In this way, the image displayed on the display 58 transitions from the first image to the second image. When the image is displayed, it is checked whether there is a change in the posture (user's line of sight) of the HMD 50 (S62). If there is no change in the posture of the HMD 50 (S62: No), the above series of processes is repeated. On the other hand, if there is a change in the posture of the HMD 50 (S62: Yes), the line-of-sight data indicating the posture (user's line of sight) of the HMD 50 after the change is transmitted to the VR camera 10 (S64). When the VR camera 10 receives the line-of-sight data after the posture change from the HMD 50 (S16: Yes in FIG. 2), the posture of the VR camera 10 is changed based on the line-of-sight data (S18 in FIG. 2), and the image (second image) generated together with the field-of-view data after the posture change of the VR camera 10 is transmitted to the HMD 50 (S10 - S14 in FIG. 2).

[0028] Here, if the detection frequency of the posture change of the HMD 50 is set too high, the frequency of the image displayed on the display 58 also increases, and as a result, there is a risk of giving discomfort to the user. Therefore, in the present embodiment, as shown in FIG. 5, an area (gaze area 62) that the user is likely to gaze at according to the posture of the HMD 50 is set in advance, and when the position of the user's gaze area 62 changes and deviates from the visual field area 28 indicated by the visual field data of the VR camera 10 grasped by the HMD 50, the gaze data (gaze position data) is transmitted to the VR camera 10. (For the sake of convenience of explanation, the visual field area 28 is schematically shown as a rectangle, but its shape will vary depending on the type of camera and the like.) In this example, a certain area located around the user's gaze point 64 is set as the gaze area 62. The "certain area" mentioned here is set inside the edge of the visual field area 28 of the VR camera 10. Basically, the center point of the visual field 66 of the HMD 50 becomes the user's gaze point, but when the HMD 50 is equipped with an eye tracking function, it is also possible to use the point detected by the function as the user's gaze point. In this case, the position of the user's gaze area 62 (the position of the gaze point 64) may be transmitted to the VR camera 10 as gaze data. By adopting such a configuration, it is possible to prevent image blurring due to image bleeding caused by the movement of the VR camera 10 and image distortion caused by the rolling shutter, and image blur due to the synchronization error between the captured image and the rotation axis sensor.

[0029] The VR system 100 functions by executing the various operations and processes described above while the VR camera 10 and the HMD 50 cooperate. Here, the point to note is that in the VR system 100, the image captured by the VR camera 10 when the rotation mechanism 24 is driven is not displayed on the display 58. Specifically, even if there is a slight change in the posture of the HMD 50, the rotation mechanism 24 of the VR camera 10 does not drive as long as the position of the user's fixation area 62 does not deviate from the visual field area 28 of the VR camera 10. Also, even when the image displayed on the display 58 of the HMD 50 transitions with a change in the posture of the VR camera 10, the images before and after the transition (the first image and the second image) are images when the rotation mechanism 24 is stopped, and the image displayed during the transition process is a complementary image generated by alpha blending. In addition, since the position of the rotation axis of the rotation mechanism 24 of the VR camera 10 is arranged at the parallax minimization point, the influence of the parallax that can occur between the first image and the second image is also minimized.

[0030] As described above, according to the VR camera 10 and the VR system 100 of the present embodiment, from multiple viewpoints such as the structural viewpoint of arranging the position of the rotation axis of the rotation mechanism 24 of the VR camera 10 at the parallax minimization point, the viewpoint regarding the drive control of the VR camera 10, and the viewpoint regarding the timing of image display in the HMD 50, the influence of parallax caused by the movement of the user's head is reduced. And each viewpoint has independent technical features, and due to these synergistic effects, it is possible to suppress the user's discomfort caused by the influence of parallax as much as possible.

[0031] As described above, the VR camera and the VR system according to the present invention have been described based on the embodiments, but it goes without saying that the present invention is not limited to the above-described forms, and for example, it may be implemented in a modified form as follows.

[0032] <Modification Example> (1) In the above-described embodiment, the timing at which the image displayed on the display 58 changes is changed according to the change in the posture of the VR camera 10 (the amount of rotation during the pan operation). However, it can also be applied when a state other than the posture of the VR camera 10 changes. For example, when the position of the VR camera 10 changes in the front-rear direction (depth direction), the position information of the VR camera 10 is acquired by GPS or the like, and based on this position information, the image displayed on the display 58 of the HMD 50 is transitioned. Also in this case, similar to the above-described embodiment, the required time until the transition from the image currently displayed on the display 58 (the first image) to the image received after the movement (displacement) (the second image) is completed is changed based on a function according to the amount of movement (displacement amount) of the VR camera 10. The same applies when the position of the VR camera 10 changes in the left-right direction or the diagonal direction.

[0033] (2) In the above-described embodiment, the VR camera 10 had three lenses, but the number of lenses is not particularly limited as long as it has a plurality of lenses. For example, it may be applied to a stereo camera equipped with a pan-tilt mechanism, or it may be applied to a four-eye camera.

[0034] (3) In the above-described embodiment, a linear function was used as the function for determining the timing at which the image changes, but a logarithmic function may also be used.

[0035] (4) In the above-described embodiment, on the HMD 50 side, it was determined whether or not to drive the rotation mechanism 24 of the VR camera 10, but it may also be determined on the VR camera 10 side. That is, when the position of the user's gaze area 62 received from the HMD 50 changes and the gaze area 62 moves out of the current visual field area 28 of the VR camera 10, the VR camera 10 may be configured to drive the rotation mechanism 24 based on the position of the gaze area 62 after the change.

[0036] (5) In the above embodiment, by devising the image display specifications in the HMD 50, the image captured by the VR camera 10 when the rotation mechanism 24 is driven is not displayed on the display 58. However, a configuration may be adopted to stop the imaging by the VR camera 10 during the drive of the rotation mechanism 24.

[0037] (6) In the above embodiment, it had the technical features from three viewpoints: the structural viewpoint of the VR camera 10, the viewpoint regarding the drive control of the VR camera 10, and the viewpoint regarding the image display timing in the HMD 50. Needless to say, it may be implemented in a form including the technical features from any one of these viewpoints, or in a form including the technical features by combining a plurality of viewpoints.

[0038] The present invention can also be implemented in various improved, modified, or deformed forms based on the knowledge of those skilled in the art without departing from the gist thereof. Also, within the range where the same action or effect is produced, it may be implemented in a form in which any one of the invention specific matters is replaced with other technology.

Explanation of Reference Numerals

[0039] 10 VR camera 12 Camera body 24 Rotation mechanism 18 Communication module (communication unit) 50 HMD (display device) 56 Communication module (communication unit) 100 VR system

Claims

1. A camera body having a plurality of lenses, A rotation mechanism for rotating the camera body, Comprising: A VR camera in which the rotation axis of the rotation mechanism is arranged at a parallax minimization point that minimizes the parallax generated between a first image generated when the camera body is in a first posture and a second image generated when the camera body is in a second posture different from the first posture.

2. The plurality of lenses include a first lens and a second lens arranged at an interval, and a third lens provided between the first lens and the second lens, The VR camera according to claim 1, wherein the parallax minimization point of the third lens is arranged on the rotation axis.

3. The VR camera according to claim 2, wherein the third lens has a wider angle of view than the first lens and the second lens.

4. A VR system comprising a VR camera provided with a rotation mechanism for rotating a camera body having a plurality of lenses, A display device for displaying an image captured by the VR camera, A communication unit for transmitting and receiving data between the VR camera and the display device, Comprising: When the rotation mechanism is driving, the display device is configured not to display an image captured by the VR camera, or to stop imaging by the VR camera.

5. The VR system according to claim 4, wherein when the position of the user's fixation area changes and the fixation area deviates from the visual field area of the VR camera, the VR camera drives the rotation mechanism based on the position of the fixation area after the change.

6. The VR camera generates a first image when the camera body is in a first state, and generates a second image when the camera body is in a second state different from the first state, The VR system according to claim 4, wherein the image display timing for transitioning from the first image to the second image is changed based on the variation mode of the camera body when changing from the first state to the second state.

Images