Information processing device, system, posture adjustment method, and program

Abstract

To improve the efficiency of adjusting the camera's posture. [Solution] The information processing device is an information processing device that adjusts the posture of multiple cameras that photograph a target area from different directions. The information processing device includes a calibration means that calculates the current value of the camera posture based on the camera images, and a calculation means that acquires a target value for the camera posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera posture.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a system, a posture adjustment method, and a program.

Background Art

[0002] There is a technique of installing a plurality of cameras at different positions and performing synchronous shooting from multiple viewpoints, and generating a virtual viewpoint image composed of an arbitrary viewpoint using the plurality of viewpoint images obtained by the shooting.

[0003] In such a system using a plurality of cameras, the plurality of cameras are installed at predetermined positions. Thereafter, the system adjusts the posture of the camera so as to photograph a desired area. For example, when the number of cameras is large, the process of adjusting the posture of the camera takes time.

[0004] Patent Document 1 discloses a technique of adjusting the posture of a camera using a captured image of the camera and an estimated image that should be captured when the camera is installed at a desired position and posture. The user can visually compare these two types of images and change the posture of the camera so that the captured image matches the estimated image, thereby adjusting the camera posture for photographing a desired area.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the above-described technique, for example, when the accuracy of the estimated image is low, it is difficult to match the two images, so there are many setbacks and the working efficiency of adjusting the posture of the camera is low.

[0007] This disclosure was made in view of the above-mentioned problems and provides a technology to improve the work efficiency of adjusting the camera's posture. [Means for solving the problem]

[0008] To solve this problem, for example, the information processing apparatus of this disclosure has the following configuration. That is, An information processing device that adjusts the orientation of multiple cameras that photograph a target area from different directions, A calibration means for calculating the current value of the camera's orientation based on the camera's image, A calculation means that obtains a target value for the camera's posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera's posture. It has. [Effects of the Invention]

[0009] According to this disclosure, the efficiency of camera posture adjustment work can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the configuration of the camera attitude adjustment system of the first embodiment. [Figure 2] A schematic perspective view showing the camera and attitude control unit. [Figure 3] A plan view showing an example of the installation of multiple cameras in a shooting system. [Figure 4] A diagram illustrating an example of a calibration imaging method. [Figure 5] A flowchart illustrating the posture adjustment process of the image processing device of the first embodiment. [Figure 6] A block diagram showing the configuration of the camera attitude adjustment system of the second embodiment. [Figure 7] A diagram illustrating the relationship between the area to be photographed and the camera's shooting area. [Figure 8] A diagram illustrating the required imaging area. [Figure 9]Flowchart showing the posture adjustment process of the image processing apparatus according to the second embodiment. [Figure 10] Block diagram showing the hardware configuration of the image processing apparatus.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the present disclosure, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) In this embodiment, the current postures of a plurality of cameras installed at different positions are calculated by calibration in order to generate multi-viewpoint images. In this embodiment, based on the difference between the calculated current posture and the target posture, an adjustment value necessary to align with the target posture is calculated. In this embodiment, by adjusting the posture according to the adjustment value, the working efficiency of camera posture adjustment is improved.

[0013] FIG. 1 is a block diagram showing the configuration of the camera posture adjustment system 10 according to the first embodiment.

[0014] The camera posture adjustment system 10 includes a photographing system 101 and an image processing apparatus 102.

[0015] The photographing system 101 includes a plurality of cameras 111a to 111j and a plurality of posture control units 112a to 112j. In this embodiment, unless otherwise specified, the cameras 111a to 111j and the posture control units 112a to 112j are simply referred to as cameras 111 and posture control units 112, respectively.

[0016] The camera 111 may be, for example, a digital camera that captures a shooting area including a subject and generates image data such as still images and moving images. A plurality of cameras 111 are arranged at different positions and capture the shooting area in individually controlled poses. Thereby, the plurality of cameras 111 can generate and output a plurality of images for generating a multi-viewpoint image. The camera 111 includes a control unit such as a computer that controls the camera 111, an imaging element provided inside, and a zoom lens whose focal length can be remotely controlled in front of the imaging element. The imaging element may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a CCD (Charge Coupled Device) image sensor.

[0017] FIG. 2 is a perspective view schematically showing the camera 111 and the attitude control unit 112. The attitude control unit 112 is, for example, a motorized pan-tilt unit having drive members such as a plurality of motors, and controls and fixes the attitude of the camera 111. The attitude control unit 112 may be remotely controlled by the image processing device 102. The attitude control unit 112 is, for example, installed horizontally with respect to the ground and can control the attitude independently in three axial directions. The three axial directions may be Euler angles, for example, Euler angles indicating yaw angle, pitch angle, and roll angle. The camera 111 and the attitude control unit 112 are connected by a communication cable so that data can be transmitted and received. The image processing device 102 can send information such as focal length and F value to the camera 111 via the attitude control unit 112 and control the camera 111. Note that the camera 111 may directly acquire information such as focal length and F value from the image processing device 102.

[0018] FIG. 3 is a plan view showing an installation example of a plurality of cameras of the imaging system 101.

[0019] As shown in Figure 3, the multiple cameras 111 are installed at different positions surrounding the target area 300. For example, in a system that generates virtual viewpoint images, there may be more than 100 cameras 111, and they are installed at high places such as the top tier of the outer perimeter of a stadium to avoid occlusion. Cameras 111a to 111j may be connected, for example, in a daisy chain. The multiple cameras 111 capture the shooting area including the subject while their time is synchronized with each other.

[0020] The target area 300 is the area to be photographed by multiple cameras 111. The target area 300 is, for example, a photography studio and the field of a sports stadium where competitions are held. The multiple cameras 111 are arranged, for example, to surround the target area 300 from all directions. Each of the multiple cameras 111 photographs a portion of the target area (hereinafter referred to as the shooting area), so as to cover the target area almost completely. The number and arrangement of the cameras 111 are not limited to the example in Figure 3. Furthermore, the shooting system 101 may be a device that captures not only video but also audio and other sensor information.

[0021] Returning to Figure 1, let's describe the image processing device 102. The image processing device 102 is, for example, a computer and is responsible for the overall control of the camera attitude adjustment system 10. The image processing device 102 is an example of an information processing device. The image processing device 102 is connected to the shooting system 101 by a network such as a LAN (Local Area Network) or WAN (Wide Area Network). The image processing device 102 acquires multiple images from the shooting system 101 that are synchronously captured from multiple viewpoints by multiple cameras 111. The image processing device 102 controls the attitude and focal length of the camera 111 by controlling the attitude control unit 112. For example, the image processing device 102 calculates an adjustment value to adjust the attitude of the camera 111 based on the acquired images and adjusts the attitude of the camera 111 via the attitude control unit 112. Also, the image processing device 102 calculates an adjustment value to adjust the focal length of the camera 111 based on the acquired images and adjusts the focal length of the camera 111 via the attitude control unit 112. The image processing device 102 includes a user interface 121, a calibration unit 122, a target value storage unit 123, and an adjustment value calculation unit 124.

[0022] The user interface 121 receives inputs from the user, such as instructions to perform calibration and target values ​​for camera parameters, and outputs them to either the calibration unit 122 or the target value storage unit 123.

[0023] The calibration unit 122 performs an installation calibration, which involves pose estimation to calculate the orientation of the installed camera 111. The calibration unit 122 receives images captured synchronously by the camera 111 and performs a calibration process to calculate the camera parameters of each camera 111. Specifically, in the calibration process, the calibration unit 122 detects image feature points from the captured images and matches the image feature points between the captured images taken by each camera 111. Based on the matching, the calibration unit 122 associates world coordinates with image coordinates and calculates the camera parameters of each camera 111. The camera parameters include external parameters and internal parameters. External parameters are parameters related to the current position and orientation of the camera (hereinafter also referred to as current values). Note that there may be a time lag in the calculation of the current values, and they may be the values ​​at the time the image was captured. Internal parameters are parameters related to the camera's focal length, image center, and lens distortion. Furthermore, from the viewpoint of the accuracy of calculating camera parameters, it is desirable that the calibration can detect as many image feature points as possible. Therefore, each camera 111 may take multiple images of the target area 300 on which markers or the like that serve as image feature points are placed, thereby generating multiple images, and the calibration unit 122 may improve the accuracy of calculating image feature points by performing a calibration process based on these multiple images.

[0024] The target value storage unit 123 stores the target values ​​for the orientation and focal length of each camera 111. The target values ​​are camera parameters that have been pre-set to achieve the installation state desired by the user. Therefore, the orientation and focal length of each camera 111 are adjusted to match the target values.

[0025] The adjustment value calculation unit 124 calculates adjustment values ​​to adjust the attitude and focal length of each camera 111 to the target state. In this adjustment, the adjustment value calculation unit 124 obtains the current values ​​of the attitude and adjustment values ​​calculated by the calibration process from the calibration unit 122, and also obtains the target values ​​of the attitude and adjustment values ​​from the target value storage unit 123. Then, the adjustment value calculation unit 124 determines the adjustment value for each camera 111 from the difference between the current value and the target value. The adjustment value calculation unit 124 transmits the adjustment value of the attitude of the camera 111 to the respective attitude control unit 112. The adjustment value calculation unit 124 transmits the adjustment value of the focal length to each camera 111 via the attitude control unit 112. The attitude control unit 112 and the camera 111 change the attitude and focal length according to their respective adjustment values. This allows the attitude control unit 112 to adjust the attitude of the camera 111 to match the target installation state. Note that the method for obtaining the current value of the focal length is not limited to the method described above. For example, the adjustment value calculation unit 124 may calculate the adjustment value by directly obtaining the current focal length value from the camera 111, rather than using the current value obtained during calibration.

[0026] Next, we will explain the calibration imaging method. Figure 4 is a diagram illustrating an example of the calibration imaging method.

[0027] Figure 4(a) is a plan view illustrating the calibration process.

[0028] During the setup calibration, multiple cameras 111 synchronously capture images of the target area 300, where multiple markers 401 are installed, as shown in Figure 4(a). Note that the setup calibration method shown in Figure 4(a) is merely an example, and this invention is not limited to this method. For example, markers are not necessarily required for calibration; the calibration unit 122 may perform calibration by matching feature objects in the captured images.

[0029] The attitude control unit 112 performs a first attitude control and a second attitude control. In order for the attitude control unit 112 to determine the attitude adjustment values ​​necessary to adjust the camera 111 to the target installation state, the current attitude of the camera must be known.

[0030] The first attitude control is for acquiring the camera's current attitude and focal length through installation calibration. In the first attitude control, the camera's attitude and focal length are controlled so as to capture the target area 300a during the installation calibration described later.

[0031] The second attitude control is an attitude control that changes the camera's attitude and focal length based on calculated adjustment values. By performing the second attitude control, multiple cameras 111 are adjusted to the target installation state. In the first attitude control, it is not possible to control the attitude of the camera 111 using adjustment values ​​as in the second attitude adjustment, so the user may control the attitude of the camera 111 while checking the captured image from the camera.

[0032] Figure 4(b) is a perspective view illustrating the target area 300 using a cycling track as an example.

[0033] Because the attitude and focal length of the camera 111 are changed by the second attitude control, the target area 300a during installation calibration is the target area 300b captured by the camera 111 with the second attitude control applied, and its position and range may differ from the target area 300b desired by the user.

[0034] The target area 300a during calibration at installation may be an area that is easy to photograph. The size of the target area 300a may be such that each camera 111 can be pointed at the same point for photography. By narrowing the size of the target area 300a, the size of the overall shooting area of ​​the shooting system, which is a composite of the shooting areas of each camera 111, can be reduced. This reduces the number of markers and captured images required, thereby reducing the man-hours required for calibration at installation. Furthermore, an area that is easy to photograph may be, for example, an area with few obstructions to the shooting, and an area with little slope or unevenness in the ground, making it easy to place markers.

[0035] In the case of a velodrome, the target area 300b desired by the user is the entire circumference of the bank 400 (race track), but the target area 300a during installation calibration may be limited to the gently sloping home stretch only. Here, in Figure 4(b), the target area 300a during installation calibration is the area with light hatching. The target area 300b desired by the user is the area with dark hatching. Note that the position and shape of the target area 300a during installation calibration are not limited to these, and may be any position and shape. For example, the size of the target area 300a during installation calibration may be less than or equal to the size of the adjusted target area 300b desired by the user. In addition, in the first attitude control, the attitude of the camera 111 is adjusted to the target area 300a, but precise attitude control is not required. The attitude of the camera 111 only needs to be such that it can detect the marker 401.

[0036] The calibration unit 122 calculates the external and internal parameters of each camera 111 based on images captured synchronously from each camera 111. The external parameters representing the position and orientation of the camera 111 are calculated by detecting markers from the acquired images and solving the PnP (Perspective-n-Point) problem from the combination of the two-dimensional coordinates of the markers in image coordinates and the three-dimensional coordinates of the markers in world coordinates. The purpose of calibration during installation is to calculate the orientation of each camera 111, but in order to solve the PnP problem, the internal parameters must be known, so the calibration unit 122 also calculates the internal parameters.

[0037] The external parameters of camera 111 may be expressed in a world coordinate system, which is a three-axis orthogonal coordinate system consisting of the X, Y, and Z axes. The orientation of camera 111 may be expressed using Euler angles (yaw angle, pitch angle, roll angle). The world coordinate system may have the center of the field being photographed as the origin (0,0,0), with the X, Y, and Z axes representing the long, short, and vertical directions of the field, respectively. Note that the axial directions and origin position of the world coordinate system are not limited to these; any orthogonal coordinate system in which the XY plane is horizontal to the ground is acceptable.

[0038] Figure 10 is a block diagram showing the hardware configuration of the image processing device. The hardware configuration of the image processing device 102 will be explained using Figure 10. The hardware configuration of the attitude control unit 112 and the camera 111 is the same as that of the image processing device 102, which will be explained below. The image processing device 102 includes a CPU 211, ROM 212, RAM 213, auxiliary storage device 214, display unit 215, operation unit 216, communication I / F 217, and bus 218.

[0039] The CPU 211 controls the entire image processing device 102 using computer programs and data stored in either the ROM 212, auxiliary storage device 214, or RAM 213. This allows the CPU 211 to implement the various functions of the image processing device 102 shown in Figure 1, such as the calibration unit 122 and the adjustment value calculation unit 124. The image processing device 102 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of, or in addition to, the CPU 211. Furthermore, the image processing device 102 may have multiple processors of the same type, each implementing a different function.

[0040] Furthermore, the image processing device 102 may have one or more dedicated hardware components separate from the CPU 211, and at least a portion of the processing performed by the CPU 211 may be executed by the dedicated hardware. Examples of dedicated hardware include circuits such as ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0041] ROM212 may be a non-volatile memory that stores programs and other data that do not require modification. RAM213 may be a memory that can read and write data at high speed. RAM213 temporarily stores programs and data supplied from the auxiliary storage device 214, as well as data supplied from the outside via the communication interface 217. Auxiliary storage device 214 may be a non-volatile storage device. Non-volatile storage devices include, for example, hard disk drives and SSDs (Solid State Drives). Auxiliary storage device 214 stores various types of data, such as image data and audio data.

[0042] The display unit 215 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 102. The operation unit 216 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215, and as an operation control unit that controls the operation unit 216.

[0043] The communication interface 217 is used for communication between the image processing device 102 and external devices. These external devices include the camera 111 and the attitude control unit 112. For example, if the image processing device 102 is connected to an external device by a wire, a communication cable is connected to the communication interface 217. If the image processing device 102 has the function of wirelessly communicating with an external device, the communication interface 217 is equipped with an antenna. The bus 218 connects the various parts of the image processing device 102 and transmits information.

[0044] In this embodiment, the display unit 215 and the operation unit 216 are assumed to be located inside the image processing device 102, but at least one of the display unit 215 and the operation unit 216 may be located outside the image processing device 102 as a separate device.

[0045] Figure 5 is a flowchart showing the attitude adjustment process of the image processing device 102 in the first embodiment. The CPU 211 of the image processing device 102 starts the attitude adjustment process by, for example, reading and executing a program for attitude control processing stored in the auxiliary storage device 214.

[0046] In step S501, the calibration unit 122 performs installation calibration. Specifically, the attitude control unit 112 performs first attitude control on each camera 111. The calibration unit 122 receives images synchronously captured by each of the multiple cameras 111 on which the first attitude control has been performed. Based on these images, the calibration unit 122 calculates camera parameters, including the current attitude and focal length of each camera 111.

[0047] In step S502, the adjustment value calculation unit 124 obtains the current values ​​of the attitude and focal length of each camera 111 from the calibration unit 122.

[0048] In step S503, the adjustment value calculation unit 124 obtains target values ​​for the attitude and focal length of each camera 111 from the target value storage unit 123.

[0049] In step S504, the adjustment value calculation unit 124 calculates the difference between the target value and the current value for each component of the yaw angle, pitch angle, and roll angle that represent the attitude of the camera 111, and calculates the adjustment value necessary to match the target attitude and focal length based on that difference.

[0050] In step S505, the adjustment value calculation unit 124 sends adjustment instructions to the attitude control unit 112 and the camera 111 based on the adjustment values. For example, the adjustment value calculation unit 124 sends the calculated attitude adjustment value to each attitude control unit 112, and the attitude control unit 112 sends the focal length adjustment value to each camera 111. Then, the attitude control unit 112 and the camera 111 adjust their attitude and focal length according to the received adjustment values, thereby adjusting each camera 111 to the target installation state.

[0051] Furthermore, in this embodiment, the attitude control unit 112 controls the attitude electrically, but it is also possible for the adjustment value calculation unit 124 to simply output the calculated adjustment values ​​without electric control. For example, the adjustment values ​​for each component of the yaw angle, pitch angle, and roll angle may be presented to the user, and the user may manually control the camera's attitude while looking at the yaw, pitch, and roll angle scales indicated on the pan / tilt head.

[0052] Furthermore, in this embodiment, the camera posture adjustment system 10 controls the posture and focal length of each camera 111, but it is also possible to control only the posture. For example, if the target area 300b desired by the user is a narrow area, such as in a shooting studio, the target area 300a during calibration at the time of installation will be in a position and shape similar to the target area 300b desired by the user. In such a case, the camera may be adjusted to the target focal length during the first posture adjustment, and then only the camera posture may be adjusted during the second posture adjustment to match the target installation state.

[0053] In this embodiment, the image processing device 102 calculates the orientation of the installed camera by calibration during installation, and determines an adjustment value from the difference between the calculated current value and the target value. Furthermore, this embodiment adjusts the camera 111 to the target orientation by controlling the orientation based on the adjustment value. As a result, this embodiment can achieve highly accurate and efficient orientation adjustment.

[0054] (Second Embodiment) In the second embodiment, if the camera's installation position differs from the assumed position, the deviation between the camera's installation position and the assumed position is calculated. In the second embodiment, the target value of the camera's posture is corrected according to the positional deviation, and the difference between the current posture calculated by the installation calibration and the corrected target value is determined. In the second embodiment, the posture is changed according to the determined difference to adjust the camera to the target posture. In the description of the second embodiment, the same configurations and processes as in the first embodiment will be omitted.

[0055] Figure 6 is a block diagram showing the configuration of the camera posture adjustment system 10 of the second embodiment. The configuration and function of the second embodiment will be described with reference to Figure 6.

[0056] In addition to the configuration of the first embodiment, the image processing device 102 further includes a position determination unit 601, a shooting area calculation unit 602, a shooting area determination unit 603, and a correction target value calculation unit 604. The CPU 211 may realize the functions of each of the above-mentioned units by reading and executing a program for posture adjustment processing from the auxiliary storage device 214.

[0057] In addition to the functions of the first embodiment, the calibration unit 122 calculates the current position of each camera 111 based on images acquired from multiple cameras 111 during installation calibration.

[0058] The position determination unit 601 obtains the current position value of each camera 111 from the calibration unit 122 and the target position value from the target value storage unit 123. The position determination unit 601 then compares the position coordinates of the current position value and the target position value for each camera 111 and determines which camera 111 has different position coordinates.

[0059] The shooting area calculation unit 602 calculates the shooting area of ​​camera 111 based on at least one of the current position and orientation values ​​of camera 111, in cases where it has been determined that the current position coordinates and target position coordinates are different. For example, the shooting area calculation unit 602 calculates the shooting area when camera 111 is pointed at the orientation of the target value at the position indicated by the current value. In addition, for camera 111 where the current position coordinates and target position coordinates are different, the shooting area calculation unit 602 may calculate the area within the shooting area of ​​camera 111 that captures the target area without overlapping with the shooting areas of other cameras 111 (hereinafter referred to as the required shooting area). The shooting area calculation unit 602 may calculate the shooting area and the required shooting area based on the internal and external parameters of camera 111. The internal parameters may be values ​​calculated during the calibration at the time of installation and values ​​used in the calculation. The external parameters may be target values ​​or current values ​​depending on what is being calculated.

[0060] Figure 7 shows the relationship between the target area 300 and the camera's shooting area 701.

[0061] For simplicity, we will explain using a shooting system 101 consisting of four cameras as an example. Shooting areas 701a to 701d are the shooting areas of the four cameras 111a to 111d. Note that when there is no need to distinguish between shooting areas, they will be referred to as shooting area 701. Figures 7(a) and 7(b) show the relationship between the target area 300 and the camera's shooting area 701 when the camera is installed at the target position and orientation. Figure 7(a) is a top view. Figure 7(b) is a side view from the left. Figure 7(c) is a top view showing the missing area. Figure 7(d) is a top view showing the elimination of the missing area.

[0062] As shown in Figures 7(a) and 7(b), we assume that cameras 111a to 111d are each capturing areas 701a to 701d. The total capturing area 700, which is the entire capturing area of ​​the entire capturing system, is the combined area of ​​capturing areas 701a to 701d. The rectangular area shown by the dotted line is the area to be captured 300.

[0063] The entire shooting area 700 of the shooting system 101 encompasses the target shooting area 300. However, in systems that use multiple cameras 111, such as a virtual viewpoint image generation system, it is not always possible to position all cameras 111 at the assumed target positions. If a camera 111 positioned differently from the target value is adjusted to the target value, the inclusion relationship of the shooting area is disrupted. As a result, a missing area 702 may occur within the target shooting area 300, which is an area not captured by any of the cameras 111, as shown in Figure 7(c). In the example in Figure 7(c), the position of the camera 111 capturing shooting area 701a shifts, causing shooting area 701a to move to the position of shooting area 701a'. As a result, the entire shooting area 700 of the shooting system 101 changes to the entire shooting area 700', and a missing area 702 is created. The missing area 702 is a factor that degrades the image quality of the generated virtual viewpoint image. Therefore, if the camera 111 is not positioned according to the target value, it is necessary to correct the target value of the camera 111 and adjust its attitude in order to eliminate the missing area 702.

[0064] As shown in Figure 7(d), the attitude control unit 112 rotates the attitude of the camera 111, which is capturing the shooting area 701a', clockwise around the pitch axis. As a result, the shooting area 701a' of the camera 111 changes to the shooting area 701a'', and the entire shooting area 700' of the shooting system 101 changes to the entire shooting area 700''. As a result, the missing area 702 is eliminated.

[0065] Figure 8 illustrates the required shooting area 801a. Figure 8(a) shows the combined shooting area 802a obtained by combining the remaining shooting areas of camera 111a, excluding the shooting area 701a of camera 111a. Figure 8(b) shows the required shooting area, which is the area that the shooting area of ​​camera 111a should encompass.

[0066] The required shooting area 801a is the area that the shooting area 701a of the camera 111a being judged should encompass, and is the area within the target shooting area 300 that should be photographed only by camera 111a. In other words, the required shooting area 801a of camera 111a is the area within the target shooting area 300 that is included only by camera 111a's shooting area 701a. To put it another way, the required shooting area 801a is the area outside the composite shooting area 802a, which is formed by combining the shooting areas of the remaining cameras 111 other than camera 111a, within the target shooting area 300. If camera 111's shooting area 701a does not encompass the required shooting area 801a, the missing area 702 shown in Figure 7(c) occurs. In the example in Figure 8, the orientation of camera 111 is adjusted so that camera 111's shooting area 701a encompasses the required shooting area 801a, and the missing area 702 is eliminated. At this time, the imaging area calculation unit 602 may calculate a composite imaging area 802a by combining the imaging areas of the remaining cameras 111 excluding the imaging area 701a, using target values ​​for the position and orientation of the remaining cameras 111 other than the camera 111a that images the imaging area 701a. The imaging area calculation unit 602 may further calculate the required imaging area 801a based on the target imaging area 300 and the composite imaging area 802a.

[0067] The shooting area determination unit 603 determines whether the shooting area 701 of camera 111, whose position coordinates differ from those of the current value and the target value, includes the required shooting area 801a. In other words, the shooting area determination unit 603 also determines whether there are any missing areas in the total shooting area, which is a composite of the shooting areas of all cameras 111.

[0068] The correction target value calculation unit 604 calculates a correction target value to correct the orientation of the camera 111. For example, the correction target value calculation unit 604 receives a determination result from the shooting area determination unit 603. The correction target value calculation unit 604 obtains from the shooting area calculation unit 602 the range of the required shooting area 801a of the camera 111 that the shooting area determination unit 603 has determined does not include the required shooting area 801a. The correction target value calculation unit 604 obtains the current position value and target value of the camera 111 from the calibration unit 122 and the target value storage unit 123. Then, the correction target value calculation unit 604 calculates a correction target value for the orientation of the camera 111 based on the current position value of the camera 111. The correction target value may be a target value corrected so that the shooting area 701 of the camera 111 includes the required shooting area 801a. In other words, the correction target value may be a target value corrected to eliminate missing areas. The adjustment value calculation unit 124 calculates an attitude adjustment value for camera 111, where the current position coordinates and the target position coordinates are different, using a corrected target value instead of the target value. By changing the attitude of camera 111 according to the adjustment value, the shooting system 101 can achieve shooting without generating missing areas 702 in the target area 300.

[0069] The correction target value calculation unit 604 may calculate the correction target value in the order of attitude and focal length. For example, the correction target value calculation unit 604 may calculate the correction target value in the order of yaw angle, pitch angle, and focal length. If the coordinates of the current value and target value of the camera 111 are different with respect to the XY plane, the correction target value calculation unit 604 corrects the yaw angle. If the coordinates of the current value and target value of the camera 111 are different with respect to the Z axis (an example of a predetermined condition), the correction target value calculation unit 604 corrects the pitch angle. If the shooting area 701 of the camera 111 does not include the required shooting area 801a after correcting only the yaw angle and pitch angle, the correction target value calculation unit 604 may shorten the focal length of the camera 111 to shoot at a wider angle to include the shooting target area 300. In addition to the yaw angle and pitch angle, the correction target value calculation unit 604 may also correct the roll angle to include it. In this case, the correction target value calculation unit 604 may calculate the correction target values ​​in the order of yaw angle, pitch angle, roll angle, and focal length. The correction target value calculation unit 604 transmits the calculated correction target values ​​to the target value storage unit 123.

[0070] The target value storage unit 123 receives the corrected target value from the corrected target value calculation unit 604 and overwrites the target value of the camera 111's posture with the corrected target value.

[0071] The adjustment value calculation unit 124 obtains the current value from the calibration unit 122 and the target value from the target value storage unit 123 for the attitude and focal length of each camera 111. The target value here includes the value overwritten by the corrected target value. The adjustment value calculation unit 124 then calculates the adjustment value for each camera 111 from the difference between the current value and the target value. The adjustment value calculation unit 124 transmits the adjustment value for the attitude of the camera 111 to the respective attitude control unit 112. The adjustment value calculation unit 124 transmits the adjustment value for the focal length to each camera 111 via the attitude control unit 112.

[0072] Figure 9 is a flowchart showing the attitude adjustment process of the image processing device 102 in the second embodiment.

[0073] In step S901, the calibration unit 122 performs installation calibration in the same manner as in step S501 in the first embodiment.

[0074] In step S902, the position determination unit 601 obtains the current position of each camera 111 from the calibration unit 122.

[0075] In step S903, the position determination unit 601 obtains the target value for the position of each camera 111 from the target value storage unit 123.

[0076] In step S904, the position determination unit 601 compares the current position value with the target value for each camera 111 and determines whether the current position value and target value for each camera 111 are different. Specifically, the position determination unit 601 may determine whether the current position coordinates and target values ​​for each camera 111 are the same. If the position determination unit 601 determines that the positions are different, the process proceeds to step S905.

[0077] On the other hand, if the position determination unit 601 determines that the positions match, it skips steps S905 to S909 and returns to step S904 if the processing from step S904 onwards has not been completed for all cameras 111. If the processing has been completed for all cameras 111, it proceeds to step S910.

[0078] In step S905, the shooting area calculation unit 602 calculates the shooting area of ​​the camera 111, which has been determined to have different position coordinates. The shooting area calculation unit 602 may calculate the shooting area using the current position value and the target orientation value.

[0079] In step S906, the shooting area calculation unit 602 calculates the required shooting area of ​​the camera 111, which has been determined to have a different current position value than the target value.

[0080] In step S907, the shooting area determination unit 603 checks the inclusion relationship between the shooting area of ​​the camera 111 to be determined, whose current position value and target value are determined to be different, and the required shooting area, and determines whether or not a missing area exists. For example, the shooting area determination unit 603 may determine that a missing area exists if the shooting area of ​​the camera 111 to be determined does not cover the entire required shooting area. If the shooting area determination unit 603 determines that a missing area exists, it proceeds to step S908. On the other hand, if the shooting area determination unit 603 determines that no missing area exists, it skips steps S908 and S909, returns to step S904 if processing has not been completed for all cameras 111, and proceeds to step S910 if processing has been completed for all cameras 111.

[0081] In step S908, if the correction target value calculation unit 604 determines that a missing area exists, it calculates a correction target value for the posture in order to eliminate the missing area. Specifically, the correction target value calculation unit 604 may calculate the correction target value so that the necessary shooting area is included in the shooting area obtained by combining the shooting areas of the multiple cameras 111.

[0082] In step S909, the correction target value calculation unit 604 overwrites the target value of the camera 111's attitude stored in the target value storage unit 123 with the correction target value.

[0083] The processing in steps S904 to S909 is executed for each camera 111. Therefore, the processing in loop 1 continues until processing is completed for all cameras 111. Once the processing in loop 1 is complete, the process proceeds to step S910.

[0084] In step S910, the adjustment value calculation unit 124 calculates the difference between the target value and the current value for each component of the camera 111's attitude (yaw angle, pitch angle, roll angle), and calculates the attitude adjustment value necessary to match the target camera 111's attitude from this difference. The adjustment value calculation unit 124 also calculates the difference between the target value and the current value of the camera 111's focal length, and calculates the focal length adjustment value necessary to match the target camera 111's focal length from this difference.

[0085] In step S911, the adjustment value calculation unit 124 performs the same processing as in S505 in the first embodiment.

[0086] In this embodiment, if the installation position of the camera 111 differs from the target value, the target value of the camera 111's posture is corrected, and the posture is changed according to an adjustment value based on the difference between the corrected target value and the current value, thereby adjusting the camera 111 to the target posture. This makes it possible to achieve highly accurate and efficient posture adjustment even when the camera 111 cannot be installed in the position that matches the assumed target value.

[0087] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, this disclosure can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0088] The disclosures herein include the following information processing devices, systems, attitude adjustment methods, and programs. (Item 1) An information processing device that adjusts the orientation of multiple cameras that photograph a target area from different directions, A calibration means for calculating the current value of the camera's orientation based on the camera's image, A calculation means that obtains a target value for the camera's posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera's posture. An information processing device characterized by having the following features. (Item 2) The calculation means calculates the adjustment values ​​for each component of the Euler angle. The information processing device described in item 1, characterized by the features described herein. (Item 3) The system includes a determination means for obtaining a target value for the position of the camera and determining which camera's current position, calculated from that value, is different from the target value. An information processing device according to item 1 or item 2, characterized in that it is an information processing device according to item 1 or item 2. (Item 4) The camera has a region calculation means that calculates the camera's shooting area based on at least one of the current and target values ​​of the camera's position and orientation. The information processing device described in item 3, characterized by the features described herein. (Item 5) The aforementioned region calculation means calculates a composite shooting region by combining the shooting regions of the remaining cameras, excluding one camera. The information processing device described in item 4, characterized by the features described herein. (Item 6) The area calculation means calculates the necessary shooting area as the area within the target area, which is the area to be photographed by the multiple cameras, that does not overlap with the composite shooting area within the shooting area of ​​one camera. The information processing device described in item 5, characterized by the features described herein. (Item 7) The area to be photographed includes an area determination means for determining whether the required photographing area is encompassed by the photographing area of ​​the first camera. The information processing device described in item 6, characterized by the features described herein. (Item 8) If the shooting area of ​​the first camera does not include the required shooting area, the system has a correction means for calculating a corrected target value obtained by correcting the target value so that the shooting area of ​​the first camera includes the required shooting area. The information processing device described in item 7, characterized by the features described herein. (Item 9) The correction means calculates correction target values ​​for correcting the target values ​​for adjusting the posture in the order of yaw angle, pitch angle, and roll angle. The information processing device described in item 8, characterized by the features described herein. (Item 10) The correction means calculates a target value for the pitch angle if the target value for the yaw angle does not meet predetermined conditions. The information processing device according to item 9, characterized in that it is a processing device. (Item 11) The calibration means calculates the current value of the camera's focal length based on the camera's image, The calculation means obtains a target value for the focal length of the camera, compares the calculated current value of the focal length with the target value, and calculates an adjustment value for adjusting the focal length of the camera. An information processing device according to any one of items 1 to 11, characterized by the features described in item 1 to 11. (Item 12) The calibration means calculates the camera's orientation and focal length based on an image taken of a target area smaller than or equal to the size of the adjusted target area. The information processing device described in item 11, characterized by the features described herein. (Item 13) It has a correction means for calculating a correction target value to correct the target values ​​of posture and focal length, The correction means calculates the correction target value in the order of attitude and focal length. An information processing device according to item 11 or item 12, characterized in that it is an information processing device. (Item 14) The information processing device described in item 1, Multiple cameras, Multiple attitude control means for controlling the attitude of multiple cameras, A system characterized by comprising the following features. (Item 15) The attitude control means controls the camera's attitude based on the adjustment value. The system described in item 14, characterized by the features described herein. (Item 16) The camera controls the focal length based on the adjustment value obtained from the information processing device. The system described in item 14 or item 15, characterized by the features described herein. (Item 17) A method for adjusting the orientation of multiple cameras that photograph a target area from different directions, Based on the camera image, the current value of the camera's orientation is calculated. The system obtains a target value for the camera's posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera's posture. A method for adjusting posture characterized by the following features. (Item 18) A program that, when read and executed by a computer, causes the computer to perform the method described in item 17.

[0089] This disclosure is not limited to the embodiments described above, and various modifications and variations are possible. [Explanation of Symbols]

[0090] 10...Camera attitude adjustment system, 102...Image processing device, 111...Camera, 112...Attitude control unit, 122...Calibration unit, 124...Adjustment value calculation unit, 300...Shooting target area, 601...Position determination unit, 602...Shooting area calculation unit, 603...Shooting area determination unit, 604...Correction target value calculation unit, 801a...Required shooting area, 802a...Composite shooting area.

Claims

1. An information processing device that adjusts the orientation of multiple cameras that photograph a target area from different directions, A calibration means for calculating the current value of the camera's orientation based on the camera's image, A calculation means that obtains a target value for the camera's posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera's posture. An information processing device characterized by having the following features.

2. The calculation means calculates the adjustment values ​​for each component of the Euler angle. The information processing apparatus according to feature 1.

3. The system includes a determination means for obtaining a target value for the position of the camera and determining which camera's current position, calculated from that value, is different from the target value. The information processing apparatus according to feature 1.

4. The camera has a region calculation means that calculates the camera's shooting area based on at least one of the current and target values ​​of the camera's position and orientation. The information processing apparatus according to claim 3.

5. The aforementioned region calculation means calculates a composite shooting region by combining the shooting regions of the remaining cameras, excluding one camera. The information processing apparatus according to feature 4.

6. The area calculation means calculates the necessary shooting area as the area within the target area, which is the area to be photographed by the multiple cameras, that does not overlap with the composite shooting area within the shooting area of ​​one camera. The information processing apparatus according to feature 5.

7. The area to be photographed includes an area determination means for determining whether the required photographing area is encompassed by the photographing area of ​​the first camera. The information processing apparatus according to feature 6.

8. If the shooting area of ​​the first camera does not include the required shooting area, the system has a correction means for calculating a corrected target value obtained by correcting the target value so that the shooting area of ​​the first camera includes the required shooting area. The information processing apparatus according to feature 7.

9. The correction means calculates correction target values ​​for correcting the target values ​​for adjusting the posture in the order of yaw angle, pitch angle, and roll angle. The information processing apparatus according to feature 8.

10. The correction means calculates a target value for the pitch angle if the target value for the yaw angle does not meet predetermined conditions. The information processing apparatus according to feature 9.

11. The calibration means calculates the current value of the camera's focal length based on the camera's image, The calculation means obtains a target value for the focal length of the camera, compares the calculated current value of the focal length with the target value, and calculates an adjustment value for adjusting the focal length of the camera. The information processing apparatus according to feature 1.

12. The calibration means calculates the camera's orientation and focal length based on an image taken of a target area smaller than or equal to the size of the adjusted target area. The information processing apparatus according to feature 11.

13. It has a correction means for calculating a correction target value to correct the target values ​​of posture and focal length, The correction means calculates the correction target value in the order of attitude and focal length. The information processing apparatus according to feature 11.

14. The information processing apparatus according to claim 1, Multiple cameras, Multiple attitude control means for controlling the attitude of multiple cameras, A system characterized by comprising the following features.

15. The attitude control means controls the camera's attitude based on the adjustment value. The system according to feature 14.

16. The camera controls the focal length based on the adjustment value obtained from the information processing device. The system according to feature 14.

17. A method for adjusting the orientation of multiple cameras that photograph a target area from different directions, Based on the camera image, the current value of the camera's orientation is calculated. The system obtains a target value for the camera's posture, compares the current value of the posture with the target value, and calculates an adjustment value for adjusting the camera's posture. A method for adjusting posture characterized by the following features.

18. A program that, when read and executed by a computer, causes the computer to perform the method described in claim 17.

Images