Information processing apparatus, information processing method, and program

Abstract

To support adjustment of a plurality of cameras to appropriate positions.SOLUTION: The camera angles of the cameras that have captured the captured images in which the marker appears are respectively calculated for all the cameras that have captured the captured images, and the difference between the camera angles of the cameras that are in the adjacent positional relationship in the cameras in which the marker appears within the angle of view is calculated. Then, when there is an inter-camera angle exceeding the threshold among the calculated inter-camera angles, a camera whose installation position is to be adjusted is determined, an adjustment amount is calculated so that the inter-camera angle becomes equal to or less than the threshold, and an adjustment instruction image including the information is generated and displayed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, a technology called volumetric capture, which can generate a 3D model of a subject from images captured by multiple cameras, has been attracting attention. By generating a 3D model of the subject, it is possible to generate a virtual viewpoint image using a virtual camera that can be operated to any viewpoint, an image that cannot be seen with a physical camera. In order to generate such a high-quality virtual viewpoint image, the placement of the cameras is important. Therefore, Patent Document 1 discloses a technology for determining the placement of cameras so that the entire capture area is captured by two or more cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-191618 Summary of the Invention [Problem to be solved by the invention]

[0004] When generating textures to be pasted onto a 3D model, the optimal camera is selected depending on the position and orientation of the virtual camera, and the texture is generated using the image captured by that camera. However, when the virtual camera moves, the camera used for the texture changes, and the texture pasted onto the 3D model also changes. If the change in texture results in changes in color or brightness or a shift in the pasting position, the change in texture becomes noticeable, and there is a risk of degrading the quality of the virtual viewpoint image.

[0005] Therefore, in order to make the texture transition less noticeable, it is necessary to appropriately adjust the angle of each camera capturing the subject. In this case, since the 3D model may exist in any position within the captured area, it is necessary to make the texture transition less noticeable at any position. The technology described in Patent Document 1 does not take texture transitions into consideration, so the texture transitions are noticeable, which may result in a decrease in the quality of the virtual viewpoint image.

[0006] In view of the above-mentioned problems, the present disclosure aims to provide support for adjusting multiple cameras to appropriate positions. [Means for solving the problem]

[0007] The information processing device according to the present disclosure is characterized by having an acquisition means for acquiring captured images obtained by capturing images of a subject from multiple directions using multiple imaging devices; a calculation means for calculating, based on the captured images acquired by the acquisition means, angles between two lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject; and a determination means for determining, if any of the angles calculated by the calculation means exceeds a threshold value, an imaging device whose installation position is to be adjusted from among the imaging devices that captured the subject. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to assist in adjusting multiple cameras to appropriate positions. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image generation system that generates a virtual viewpoint image. [Figure 2] FIG. 2 is a diagram illustrating an example of the arrangement of cameras included in a camera group. [Figure 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a camera position confirmation device. [Figure 4]FIG. 2 is a block diagram illustrating an example of a functional configuration of a camera position confirmation device. [Figure 5] FIG. 10 is a diagram for explaining a method for calculating a camera angle. [Figure 6] 10 is a flowchart showing an example of the overall processing procedure up to displaying an instruction image for adjusting the installation positions of the cameras using the inter-camera angle. [Figure 7] 10 is a flowchart illustrating an example of a detailed processing procedure for calculating an inter-camera angle. [Figure 8] FIG. 10 is a diagram illustrating an example of a list of calculation results of inter-camera angles with respect to markers. [Figure 9] FIG. 10 is a diagram for explaining an inter-camera angle. [Figure 10] 10 is a flowchart illustrating an example of a detailed processing procedure for generating and displaying an adjustment instruction image based on an inter-camera angle. [Figure 11] FIG. 4 is a diagram showing an example of an adjustment instruction image in the first embodiment. [Figure 12] FIG. 10 is a diagram showing an example of adjustment instruction information after the installation position of the camera has been moved. [Figure 13] FIG. 10 is a diagram illustrating an example of an adjusted image. [Figure 14] FIG. 10 is a diagram showing another example of an adjustment instruction image in the first embodiment. [Figure 15] FIG. 10 is a diagram for explaining the inter-camera angle when multiple markers are installed. [Figure 16] FIG. 10 is a diagram illustrating an example of a list of calculation results of inter-camera angles with respect to markers. [Figure 17] FIG. 10 is a diagram showing an example of an adjustment instruction image in the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of an adjustment instruction image in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to the embodiments described below, and various forms within the scope of the gist of the invention are also included in the present disclosure. Furthermore, each embodiment described below merely represents one embodiment of the present disclosure, and each embodiment can be appropriately combined.

[0011] (First embodiment) In this embodiment, in a system for generating a virtual viewpoint image, in order to suppress degradation of the image quality of the virtual viewpoint image, it is determined whether there is a problem with the angle between the shooting directions of a camera shooting a subject and an adjacent camera (hereinafter referred to as the inter-camera angle). If there is a problem, an example will be described in which the adjustment amount for the camera to be adjusted is presented and the camera is easily moved to an appropriate position. A camera adjacent to a certain camera refers to the camera closest to the position of the certain camera among multiple cameras other than the certain camera. Furthermore, adjacent does not necessarily mean that the cameras are physically adjacent, but may also be spaced apart.

[0012] Fig. 1 is a diagram showing an example of the configuration of an image generation system 10 that generates virtual viewpoint images in this embodiment. As shown in Fig. 1, the image generation system 10 includes a group of cameras 100, a hub 150, an image generation device 160 that generates virtual viewpoint images, and a camera position confirmation device 170 that confirms the installation positions of the cameras.

[0013] The camera group 100 includes 28 imaging devices (cameras 111 to 138), each connected to the HUB 150 by wire or wirelessly in a star configuration. The configuration of the camera group 100 is not limited to this. For example, the number of cameras is not limited to 28 as long as a virtual viewpoint image can be generated, and the cameras may be daisy-chained and connected to the HUB 150. The camera group 100 also achieves high-precision synchronization by obtaining a master clock from a time server (not shown). Therefore, images captured by the camera group 100 are output as images captured at the same time.

[0014] The images captured by the 28 cameras 111 to 138 of the camera group 100 are sent to the image generation device 160 and the camera position confirmation device 170 via the HUB 150. The image generation device 160 has a UI (User Interface) unit that accepts user operations, such as a mouse, keyboard, operation buttons, and touch panel. A user viewing a virtual viewpoint image operates the UI unit to set the position and direction of the virtual viewpoint, and the image generation device 160 generates a virtual viewpoint image according to the setting. Specifically, the image generation device 160 reconstructs (models) the subject in a 3D space using the captured images acquired from the camera group 100. Then, the shape data (3D model) generated by the modeling is rendered according to the viewpoint position and direction of the virtual camera, thereby generating a virtual viewpoint image that represents the appearance from the virtual viewpoint.

[0015] The camera position confirmation device 170 is a control device interconnected with the camera group 100 via the HUB 150. The camera position confirmation device 170 receives captured images from the camera group 100, extracts cameras that capture markers in the captured images, and calculates the inter-camera angles between adjacent cameras among the multiple cameras that capture the markers. The camera position confirmation device 170 then checks the validity of the camera arrangement based on the calculated inter-camera angle information and presents adjustment information if necessary. Details of the camera position confirmation device 170 will be described later.

[0016] The configuration of the image generation system 10 is not limited to that shown in Fig. 1. The multiple cameras included in the camera group 100 may be directly connected to the image generation device 160 and the camera position confirmation device 170. The image generation device 160 may also be composed of multiple devices, and the image generation device 160 may have a UI unit built in. The devices included in the image generation system 10 may be connected by wire or wirelessly.

[0017] FIG. 2 is a diagram showing an example of the arrangement of cameras included in the camera group 100. As shown in FIG. 2, multiple cameras are arranged to surround a shooting area 200 to be shot, and these cameras capture images of the shooting area 200 from different directions to generate a virtual viewpoint image. Each camera included in the camera group 100 is, for example, a digital camera, and may be a camera that captures still images, a camera that captures video, or a camera that captures both still images and video. The image generation system 10 generates a virtual viewpoint image of the shooting area 200 using the images captured by the camera group 100. Cameras 111 to 138 are assigned camera numbers "1" to "28," respectively.

[0018] The marker 210 is a subject for calculating the inter-camera angle at the marker position. By capturing an image of the marker 210, the frequency components of the image can be calculated to detect the shape of the marker 210 captured in the image, and a three-dimensional coordinate system of the marker 210 can be derived based on the X-axis, Y-axis, and Z-axis, which are orthogonal to each other and have the world coordinate system as the origin. By using a special function of the marker 210, the camera angle in the reference direction of the marker can be calculated. By moving the marker to various positions within the capture area 200 and capturing images of it, the inter-camera angle can be confirmed at any position within the capture area 200. In the example of FIG. 2, the marker 210 is captured within the angle of view of eleven cameras: 111, 113, 115, 117, 120, 123, 128, 130, 132, 134, and 136. The combination of cameras that capture the marker 210 within the angle of view changes depending on the position where the marker 210 is placed.

[0019] 3 is a block diagram showing an example of the hardware configuration of a camera position confirmation device 170, which is an information processing device. The camera position confirmation device 170 has a CPU 2201, a ROM 2202, a RAM 2203, an auxiliary storage device 2204, a communication I / F 2205, an input unit 2206, and a display device 2207, each of which is connected to a bus 2208. The CPU 2201 controls the entire camera position confirmation device 170 using programs and data stored in the ROM 2202 or the auxiliary storage device 2204. Note that the camera position confirmation device 170 may have one or more dedicated hardware components different from the CPU 2201, and at least a portion of the processing by the CPU 2201 may be executed by the dedicated hardware components. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). Another example of the dedicated hardware is a DSP (Digital Signal Processor).

[0020] The ROM 2202 stores programs and various parameters that do not require modification. The RAM 2203 temporarily stores programs and data supplied from the auxiliary storage device 2204, as well as data supplied from the outside via the communication I / F 2205. The auxiliary storage device 2204 is configured, for example, with a hard disk drive, and stores various content data such as images and audio, programs, etc.

[0021] The communication I / F 2205 is used for communication with an external device such as the camera group 100. For example, if the camera position confirmation device 170 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 2205. If the camera position confirmation device 170 has a function for wireless communication with an external device, the communication I / F 2205 includes an antenna.

[0022] The input unit 2206 is an operation unit such as a mouse, keyboard, or touch panel. The display device 2207 is configured with, for example, a liquid crystal display or LED, and displays data such as images. Note that the camera position confirmation device 170 may not include the input unit 2206 or the display device 2207, but may be configured to be connected to an external operation unit or display device.

[0023] 4 is a block diagram showing an example of the functional configuration of the information processing device, camera position confirmation device 170. Camera position confirmation device 170 has an image receiving unit 301, a marker detection unit 302, a camera angle calculation unit 303, an inter-camera angle calculation unit 304, an adjustment determination unit 305, an adjustment camera determination unit 306, a target angle determination unit 307, an image generation unit 308, and a UI display unit 309.

[0024] The image receiving unit 301 acquires captured images from the camera group 100 and outputs the captured images to the marker detection unit 302. The marker detection unit 302 extracts captured images containing markers from the captured images acquired from the image receiving unit 301 and coordinate information of the markers, and outputs the extracted images to the camera angle calculation unit 303 and the image generation unit 308.

[0025] The camera angle calculation unit 303 detects markers from the captured image acquired from the marker detection unit 302 and calculates the camera angle. The camera angle is the rotation angle of the camera position when a specific orientation of the marker is used as a reference.

[0026] FIG. 5 is a diagram for explaining a method for calculating the camera angle. The angle of view 401 represents the camera's shooting range. The marker 402 represents the actual marker that is the subject, and the marker 403 represents the marker that appears in the captured image when the actual marker 402 is captured. The angle Aα 404 represents the pan angle with respect to the marker 403 when the optical axis of the camera is set to 0 degrees. The angle Aα 404 can be calculated from the camera's sensor size, the lens focal length, and the marker position within the angle of view. The angle Aβ 405 represents the rotation angle of the marker itself when a specific orientation of the marker is used as the reference. Using the above information, the camera angle A can be calculated using the following equation (1). Camera angle A = angle Aα + angle Aβ (1)

[0027] The range of possible values ​​for camera angle A calculated by equation (1) is from 0 to 360 degrees. Camera angle calculation unit 303 calculates the camera angle of the camera that captured each of all captured images that show the marker, and outputs the calculated camera angle to inter-camera angle calculation unit 304.

[0028] The inter-camera angle calculation unit 304 acquires the camera angles from the camera angle calculation unit 303, and calculates the inter-camera angle by calculating the difference between the camera angles of cameras that are adjacent to each other among the cameras that capture the marker within their angle of view. The process of calculating the inter-camera angle will be described in detail later. The inter-camera angle calculation unit 304 outputs the calculated inter-camera angle to the adjustment determination unit 305, the target angle determination unit 307, and the image generation unit 308.

[0029] The adjustment determination unit 305 checks whether the inter-camera angle acquired from the inter-camera angle calculation unit 304 exceeds a preset threshold value, and determines whether adjustment of the camera placement is necessary. The adjustment determination unit 305 then outputs the determination result to the camera to be adjusted determination unit 306 and the image generation unit 308. If the adjustment determination unit 305 determines that adjustment is necessary, the camera to be adjusted determines the camera whose installation position is to be adjusted from the inter-camera angle, and outputs this information to the target angle determination unit 307. The target angle determination unit 307 calculates the amount of adjustment required to make the inter-camera angle equal to or less than the threshold value, based on the inter-camera angle of the camera to be adjusted determined by the adjustment camera determination unit 306, and outputs the amount of adjustment to the image generation unit 308.

[0030] The image generation unit 308 acquires marker coordinate information from the marker detection unit 302 and generates an image in which the markers appearing in the captured image of the camera to be adjusted are highlighted. When multiple markers are present in the captured image, an image in which the markers are highlighted is generated to indicate which marker the inter-camera angle information belongs to. Note that highlighting is not essential when only one marker is detected, as in the example of FIG. 3. The image generation unit 308 also generates adjustment instruction information that indicates to the user which camera to adjust and by how much, using the inter-camera angle calculated by the inter-camera angle calculation unit 304, the determination result by the adjustment determination unit 305, and the adjustment amount calculated by the target angle determination unit 307. The image generation unit 308 then generates an adjustment instruction image in which the adjustment instruction information is superimposed on the captured image in which the markers are highlighted, and outputs the image to the UI display unit 309. Details of the adjustment instruction image will be described later.

[0031] The UI display unit 309 is configured by the display device 2207 and displays the adjustment instruction image acquired from the image generation unit 308 .

[0032] 6 is a flowchart showing an example of the overall processing procedure up to displaying an image for instructing adjustment of the camera installation positions using the inter-camera angle. Each flowchart described below is realized by the CPU 2201 reading out a program stored in the ROM 2202 or the auxiliary storage device 2204, expanding it in the RAM 2203, and executing the program. First, as shown in FIG. 2, the marker 210 is placed at a predetermined position in the shooting area 200, and the camera group 100 starts shooting, thereby starting the processing in FIG. 6.

[0033] First, in S501, the image receiving unit 301 acquires captured images from all cameras belonging to the camera group 100. Next, in S502, the marker detection unit 302 extracts cameras that capture images containing the marker 210. In the example shown in Fig. 2, as described above, 11 cameras have captured images containing the marker 210.

[0034] Next, in S503, the inter-camera angle is calculated using the procedure shown in Fig. 7 (described later). Then, in S504, it is confirmed whether the inter-camera angle exceeds a threshold value using the procedure shown in Fig. 10 (described later). An adjustment instruction image corresponding to the result is generated and displayed on the UI display unit 309.

[0035] FIG. 7 is a flowchart showing an example of a detailed processing procedure for calculating the inter-camera angle in S503 of FIG. First, in S601, the camera angle calculation unit 303 sorts the cameras extracted in S502 in ascending order of camera number, and selects the camera with the smallest camera number. In the example of Fig. 2, camera 111 with camera number "1" is selected.

[0036] Next, in S602, the processes of S603 and S604 (described later) are performed on all cameras that captured images showing the marker. In S603, the camera angle calculation unit 303 calculates the camera angle of the camera being processed when a specific orientation of the marker is used as a reference. In this process, the camera angle can be calculated using the method described above with reference to FIG. 5. Then, in S604, the camera angle calculation unit 303 selects the camera with the next smallest camera number.

[0037] After calculating the camera angles for all cameras extracted in S502, in S605 the inter-camera angle calculation unit 304 sorts the camera angles calculated in S603 in ascending order. Then, in S606, the inter-camera angle calculation unit 304 calculates the difference between the camera angles of adjacent cameras in the sorted order as the inter-camera angle. Note that the inter-camera angle can be said to be the angle between two lines connecting each of the adjacent cameras to the marker.

[0038] FIG. 8 is a diagram showing, as a list, an example of the calculation results of the inter-camera angle with respect to the marker 210 in the example arrangement of FIG. 2. In this embodiment, for example, the inter-camera angle of the camera 111 with camera number "1" represents the difference in camera angle with the camera 113 with camera number "3." In the example shown in FIG. 8, the inter-camera angle of the camera 111 with camera number "1" is 33.0 degrees, which is the difference between 51.0 degrees and 18.0 degrees. Note that the inter-camera angle of the camera 136 with camera number "26" is the difference in camera angle with the camera 111 with camera number "1." In the example shown in FIG. 8, the inter-camera angle of the camera 136 with camera number "26" is 27.0 degrees, which is the difference between 351.0 degrees and 378.0 degrees, which is obtained by adding 360 degrees to 18.0 degrees.

[0039] As described above, in S606 of Fig. 7, the inter-camera angles are calculated for all cameras that have detected markers. Thereafter, in S504 of Fig. 6, the adjustment determination unit 305 checks whether any inter-camera angles exceed the threshold, and if so, the camera to be adjusted determination unit 306 determines which cameras require adjustment of their installation positions. For example, if the threshold is set to 40 degrees, in the example of Fig. 8, the inter-camera angle of camera 123 with camera number "13" is 49.0 degrees, which exceeds the threshold. Therefore, the camera to be adjusted determination unit 306 determines that camera 123 with camera number "13" is the camera whose installation position requires adjustment.

[0040] In this embodiment, camera 123 is determined as the camera requiring adjustment, but the inter-camera angle of camera 123 with camera number "13" is the angle formed by camera 123 and camera 128 with camera number "18". Therefore, camera 128 may also be determined as the camera requiring adjustment. Also, although the processes of S601 and S605 are performed in ascending order of camera number, they may also be performed in descending order of camera number.

[0041] FIG. 9 is a diagram illustrating the inter-camera angle. Angle reference 701 is a reference for representing the camera angle, and is set in a predetermined direction with respect to the pattern of marker 210. Angle 702 represents the camera angle of camera 120, and is 153.0 degrees in the example of FIG. 8. Angle 703 represents the inter-camera angle between camera 120 and camera 123, and is 27.0 degrees in the example of FIG. 8. Angle 704 represents the inter-camera angle between camera 123 and camera 128, and is 49.0 degrees in the example of FIG. 8. Therefore, it is determined that the installation position of camera 123 needs to be adjusted so that the inter-camera angle of camera 123 is equal to or less than a threshold value.

[0042] FIG. 10 is a flowchart showing an example of detailed processing procedures for generating and displaying an adjustment instruction image based on the inter-camera angle in S504 of FIG. First, in S801, the adjustment determination unit 305 determines whether all inter-camera angles calculated in S606 of Fig. 7 are equal to or less than the threshold value. If the result of this determination is that all inter-camera angles are equal to or less than the threshold value, the process proceeds to S802. Then, in S802, the image generation unit 308 generates an adjusted image. Note that details of the adjusted image will be described later.

[0043] On the other hand, if the result of the determination in S801 is that there is an inter-camera angle that exceeds the threshold, the process proceeds to S803. Then, in S803, the adjustment camera determination unit 306 determines the camera whose inter-camera angle exceeds the threshold as the camera that needs adjustment, and the target angle determination unit 307 calculates the target camera angle for making the inter-camera angle equal to or less than the threshold. In the example of Fig. 8, the inter-camera angle of camera 123 with camera number "13" is 49.0, which exceeds the threshold, so the adjustment camera determination unit 306 determines it as the camera that needs adjustment.

[0044] Next, a method for calculating the target camera angle will be described with reference to Fig. 9. The inter-camera angle of camera 123 with camera number "13" is the angle between camera 123 and camera 128 on the right side, which captures the marker. Therefore, in order to make this inter-camera angle equal to or less than the threshold, it is necessary to move camera 123 in a direction closer to camera 128, that is, to the right.

[0045] Furthermore, regarding the amount of movement, since the inter-camera angle is 49.0 degrees, it is necessary to move camera 123 toward camera 128 so that the inter-camera angle is reduced by 9 degrees or more. On the other hand, the inter-camera angle of camera 120, camera number "10" on the left side of which the marker is captured, is 27.0 degrees. Therefore, if the inter-camera angle of camera 120 increases by more than 13.0 degrees by moving camera 123, the inter-camera angle of camera 120 will exceed the threshold value of 40.0 degrees. From the above, the adjustment amount of the inter-camera angle of camera 123 is between 9.0 degrees and 13.0 degrees, and the target camera angle of camera 123 can be calculated to be between 189.0 degrees and 193.0 degrees.

[0046] 10, next, in S804, the image generation unit 308 generates an adjustment instruction image. Details of the adjustment instruction image will be described later. In S805, the image generation unit 308 displays the generated image on the UI display unit 309, and the process ends.

[0047] FIG. 11 is a diagram showing an example of an adjustment instruction image. The adjustment instruction image 901 in FIG. 11(a) is displayed as an image in which information necessary for moving the installation position is added to the captured image captured by the camera 123 with camera number "13," whose inter-camera angle is 49.0, exceeding the threshold. A rectangle 902 is a frame for indicating to the user the marker used when calculating the inter-camera angle. The adjustment instruction information 903 is information for the user to adjust the installation position of the camera 123 while checking this information. Note that in the example shown in FIG. 11(a), the adjustment instruction information 903 is displayed superimposed on the lower left of the captured image, but the display position is not limited to the lower left. For example, the user may be able to specify the display position of the adjustment instruction information. The adjustment instruction information may be displayed at any position that does not overlap with the rectangle 902.

[0048] 11(b) is a diagram for explaining the details of the adjustment instruction information 903. The feasibility information 904 is information that changes depending on the determination result by the adjustment determination unit 305, and is displayed as "NG" in the adjustment instruction information 903. An icon 905 represents the camera that captured the image in which the marker 210 appears.

[0049] A dotted frame 906 indicates a camera whose installation position needs to be adjusted, and in the example of FIG. 11(b), it indicates the icon of camera 123 with camera number "13," whose inter-camera angle is 49.0, exceeding the threshold. Angle 907 indicates the inter-camera angle that exceeds the threshold. Accordingly, as shown in FIG. 11(b), the adjustment instruction information 903 displays the camera angle of camera 123, the inter-camera angle of camera 120 with camera number "10," and the inter-camera angle of camera 123 together. Note that in the example of FIG. 11(b), there is only one inter-camera angle that exceeds the threshold, but if there are multiple inter-camera angles that exceed the threshold, this information will be displayed in multiple positions.

[0050] An arrow 908 indicates the direction in which the installation position of the camera 123 is moved. In the example of Fig. 11(b), this suggests movement in a counterclockwise direction around the marker 210 so as to approach the camera 128 with camera number "18". A numerical range 909 indicates the amount of adjustment required for the inter-camera angle of the camera 123 to be equal to or less than a threshold value.

[0051] By following the above procedure, an adjustment instruction image is displayed on the UI display unit 309, allowing the user to actually move the installation positions of the cameras while referring to the adjustment instruction image. Thereafter, the inter-camera angle is calculated again using the same procedure, and the user checks the adjustment instruction image again and repeats adjusting the installation positions of the cameras until the inter-camera angle becomes equal to or less than the threshold. Note that when an adjustment instruction image is generated again according to the processing procedure in FIG. 6 after adjusting the installation positions of the cameras, it is not necessary to acquire images captured by all cameras as in the procedure in FIG. 6; it is also possible to acquire only images captured by the camera to be adjusted and the cameras on either side of it.

[0052] Fig. 12 is a diagram showing an example of adjustment instruction information after the installation position of camera 123 with camera number "13" has been moved. The adjustment instruction information in Fig. 12 shows the result of recalculating the camera angle by moving camera 123 after the adjustment instruction information in Fig. 11(b) was displayed. The camera angle is 185.0 degrees, which indicates that although camera 123 has been moved by 5 degrees, the inter-camera angle still exceeds the threshold, and further movement of camera 123 is required.

[0053] FIG. 13 shows an example of an adjustment completion image that is displayed when the installation position of camera 123 with camera number "13" is moved and the inter-camera angle becomes equal to or less than the threshold. The adjustment instruction image including the adjustment instruction information of FIG. 12 is displayed, and the user moves camera 123 another 4 degrees and recalculates the camera angle, resulting in the adjustment completion image of FIG. 13 being displayed. As a result of moving camera 123, the camera angle becomes 189.0 degrees, and the inter-camera angle of camera 123 becomes equal to or less than the threshold, so the adjustment completion image of FIG. 13 indicates that the adjustment is complete. The OK / NG information 1001 displays "OK" indicating that the adjustment is complete, and the angle 1002 is blank because the inter-camera angle is equal to or less than the threshold, indicating that the cameras are correctly positioned.

[0054] Furthermore, although examples of adjustment instruction information are described in FIGS. 11(b) and 12, other display modes may be used. FIG. 14 is a diagram showing another example of adjustment instruction information, illustrating an example in which the camera angles and inter-camera angles of all cameras are displayed. In the example of FIG. 11(b), only information related to camera 123 determined to require adjustment is displayed, but as in the example of FIG. 14, the camera angles and inter-camera angles of all cameras may be displayed. By using the display mode shown in FIG. 14, it becomes possible to simultaneously check the installation status of other cameras while simultaneously adjusting the other cameras, for example.

[0055] As described above, according to this embodiment, in a system for generating virtual viewpoint images, it is possible to determine the appropriateness of the camera installation position and whether adjustment is necessary, and to support moving to an appropriate position. This allows the user to adjust the camera installation position to prevent degradation of the image quality of the virtual viewpoint image.

[0056] In this embodiment, when one inter-camera angle exceeds the threshold and the camera with the smaller camera number of the two cameras that make up the inter-camera angle is selected as the camera whose installation position is to be adjusted. On the other hand, when two or more inter-camera angles exceed the threshold, the camera whose installation position is to be adjusted may be selected from the two cameras that make up the inter-camera angle depending on the situation. For example, in the example of FIG. 8 , the inter-camera angle of camera 120 (camera number "10") is 27 degrees, but it is possible that the inter-camera angles of camera 123 and camera 120 both exceed the threshold of 40 degrees. In this case, if the camera with the smaller camera number is selected as the camera whose installation position is to be adjusted according to the rules, the inter-camera angle of camera 117 (camera number "7") would need to be significantly increased, which may result in the inter-camera angle of camera 117 exceeding the threshold. Therefore, in such a case, the installation position of camera 123 may be fixed, and cameras 120 and 128 may be selected as the cameras to be adjusted.

[0057] (Second embodiment) In this embodiment, an example will be described in which a captured image is acquired using multiple markers, and whether or not adjustment of the camera installation position is necessary is determined, a camera requiring adjustment is selected, and the amount of adjustment is simultaneously derived for each marker. Note that the overall configuration of the image generation system for generating a virtual viewpoint image according to this embodiment and the internal configuration of the camera position confirmation device are the same as those in the first embodiment, and therefore their explanations will be omitted. Only the differences from the first embodiment will be described below.

[0058] FIG. 15 is a diagram illustrating the inter-camera angle when multiple markers are installed. FIG. 16 is a diagram showing, as a list, an example of the calculation results of the inter-camera angle for marker 211 in the arrangement example of FIG. 2. In the example of FIG. 15, two markers, marker 210 and marker 211, are arranged. When arranging multiple markers, the arrangement positions of the markers are determined in advance according to the focal length of the camera and the distance from the camera to the shooting area so that the multiple markers are captured within the camera's angle of view. As shown in FIG. 15, the cameras that capture images of marker 210 include camera 120 with camera number "10," camera 123 with camera number "13," and camera 128 with camera number "18." Furthermore, the cameras that capture images of marker 211 include camera 119 with camera number "9," camera 123 with camera number "13," and camera 128 with camera number "18." It is assumed that the marker 210 does not appear in the image captured by the camera 119 with the camera number "9," and the marker 211 does not appear in the image captured by the camera 120 with the camera number "10." Note that the angle reference 1101 is a reference for expressing the camera angle, similar to the angle reference 701, and is set in a predetermined direction with respect to the pattern of the marker 211.

[0059] Angle 1102 represents the camera angle of camera 119 relative to marker 211, and is 140.0 degrees in the example of FIG. 16. Angle 1103 represents the inter-camera angle between camera 119 and camera 123 relative to marker 211, and is 29.0 degrees in the example of FIG. 16. Angle 1104 represents the inter-camera angle between camera 123 and camera 128 relative to marker 211, and is 44.0 degrees in the example of FIG. 16. Therefore, it is determined that the installation position of camera 123 needs to be adjusted so that the inter-camera angles at both markers 210 and 211 are equal to or less than a threshold value.

[0060] The processing procedure in this embodiment is basically the same as that in the first embodiment. Specifically, the processing shown in Fig. 7 is performed for each marker, and the inter-camera angle is calculated for each marker. Then, in S801 of Fig. 10, it is determined whether the inter-camera angles for all markers are equal to or less than a threshold value, and if the inter-camera angles for all markers are equal to or less than the threshold value, an adjusted image is generated.

[0061] In the examples of FIGS. 8 and 16, the inter-camera angle of camera 123 with camera number "13" exceeds the threshold value. In this case, in S803, the adjustment camera determination unit 306 determines that camera 123 is a camera whose installation position needs to be adjusted, and the target angle determination unit 307 determines a target camera angle for making the inter-camera angle equal to or less than the threshold value. First, the adjustment amount when marker 210 is used as a reference is calculated by performing the same process as in the first embodiment, and the adjustment amount of the inter-camera angle of camera 123 is calculated to be between 9.0 degrees and 13.0 degrees so that camera 123 approaches camera 128. Therefore, the target camera angle of camera 123 is calculated to be between 189.0 degrees and 193.0 degrees.

[0062] 16, the inter-camera angle of camera 119 relative to marker 211 is 29.0 degrees. Therefore, if an adjustment of 12.0 degrees to 13.0 degrees is applied to the inter-camera angle of camera 123 relative to marker 210, out of the 9.0 degrees to 13.0 degrees that is the adjustment amount for the inter-camera angle of camera 123 relative to marker 210 described above, the inter-camera angle of camera 119 relative to marker 211 may exceed the threshold. Therefore, taking the inter-camera angle relative to marker 211 into consideration, the adjustment amount for the inter-camera angle of camera 123 is calculated to be between 9.0 degrees and 11.0 degrees so as to move closer to camera 128. As a result, the target camera angle of camera 123 is finally calculated to be between 189.0 degrees and 191.0 degrees.

[0063] Fig. 17 is a diagram showing an example of an adjustment instruction image based on the marker 210 in this embodiment. The adjustment instruction image 1201 in Fig. 17(a) is displayed as an image to which information necessary for moving the installation position is added to the image captured by the camera 123. The rectangle 1202 is a part for indicating to the user the marker used when calculating the inter-camera angle, and is displayed so as to surround the marker 210 in the example of Fig. 17(a). The adjustment instruction information 1203 is information for the user to adjust the installation position of the camera 123 while checking this information.

[0064] Fig. 17(b) is a diagram for explaining the details of the adjustment instruction information 1203. As shown in Fig. 17(b), the adjustment instruction information 1203 includes the inter-camera angles when the markers 210 and 211 are used as references. In the example of Fig. 17(b), the marker 1204 is positioned at the center, and the adjustment instruction information when the markers 210 and 211 are used as references is displayed superimposed on top of each other.

[0065] For cameras that include marker 210 in their captured images, icons representing the cameras are displayed arranged concentrically around marker 1204. The camera angle of camera 123, the inter-camera angle of camera 120, and the inter-camera angle of camera 123, all of which are based on marker 210, are also presented. On the other hand, for cameras that include marker 211 in their captured images, icons are displayed arranged outward from marker 1204, all of which are based on marker 211. The camera angle of camera 123, the inter-camera angle of camera 119, and the inter-camera angle of camera 123, all of which are based on marker 211, are also presented.

[0066] Dotted frame 1205 indicates a camera whose installation position needs to be moved. In the example of Fig. 17(a), rectangle 1202 is displayed to surround marker 210, so in Fig. 17(b), the inner icon of camera 123 is surrounded by dotted frame 1205. Angle 1206 represents the inter-camera angle based on marker 210, which exceeds the threshold, and also displays that the inter-camera angle is 49.0 degrees.

[0067] Arrow 1207 indicates the direction in which the installation position of camera 123 is moved. In the example of Fig. 17(b), this suggests movement in a counterclockwise direction around marker 210 so as to approach camera 128. Numerical range 1208 indicates the amount of adjustment required for both the inter-camera angle of camera 123 when marker 210 is used as the reference and the inter-camera angle of camera 123 when marker 211 is used as the reference to be equal to or less than a threshold value.

[0068] Furthermore, if the inter-camera angles based on the same marker exceed a threshold, the adjustment instruction information may be displayed in ascending order of camera numbers, and when adjustment for one camera is completed, the adjustment instruction information for the next camera may be displayed. In this case, the adjustment instruction information may be displayed in ascending order of camera numbers rather than in descending order.

[0069] 18 is a diagram showing an example of an adjustment instruction image based on the marker 211 in this embodiment. Note that the timing for switching the adjustment instruction image is not particularly limited, and the adjustment instruction image may be switched by a user operation, for example. Furthermore, when the user actually moves the installation position of the cameras and the inter-camera angle based on the marker 210 becomes equal to or smaller than the threshold value, while the inter-camera angle based on the marker 211 exceeds the threshold value, the adjustment instruction image may be switched to the adjustment instruction image shown in FIG. 18(a).

[0070] In the adjustment instruction image 1301 of Fig. 18(a), a rectangle 1302 is displayed so as to surround the marker 211. Fig. 18(b) is a diagram for explaining the details of the adjustment instruction information 1303 included in the adjustment instruction image 1301 of Fig. 18(a). In the example of Fig. 18(a), the rectangle 1202 is displayed so as to surround the marker 211, and therefore in Fig. 18(b), the outermost icons of the cameras 123 are surrounded by a dotted frame 1305. Furthermore, an angle 1306 represents the inter-camera angle based on the marker 211 that exceeds the threshold, and it is also displayed that the inter-camera angle is 44.0 degrees.

[0071] An arrow 1307 suggests counterclockwise movement around the marker 211 to approach the camera 128. A numerical range 1308 indicates the amount of adjustment required to make the inter-camera angle of the camera 123, when the marker 211 is used as the reference, equal to or less than a threshold.

[0072] As described above, according to this embodiment, in a system for generating a virtual viewpoint image, it is possible to determine the appropriateness of the camera installation position and whether adjustment is necessary, and to support moving to an appropriate position. Furthermore, by using multiple markers, for example, it is possible to prevent the inter-camera angle at another position from exceeding the threshold by setting the inter-camera angle at a certain position below a threshold. This reduces the need for readjustment of the camera installation position, enables adjustment of the camera installation position in a short time, and prevents degradation of the image quality of the virtual viewpoint image.

[0073] In this embodiment as well, all camera angles and inter-camera angles may be displayed as adjustment instruction information, as in the example shown in FIG.

[0074] (Other embodiments) In the above-described embodiment, an example in which the present disclosure is applied to an image generation system that generates a virtual viewpoint image has been described, but the present disclosure may also be applied to other systems. For example, the present disclosure may be applied so that the installation position of a camera can be adjusted when filming a game at a stadium, etc.

[0075] The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0076] The disclosure of this embodiment includes the following configuration, method, and program.

[0077] (Configuration 1) an acquisition means for acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation means for calculating an angle between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject based on the captured image acquired by the acquisition means; a determination means for determining, when any angle calculated by the calculation means exceeds a threshold value, an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject; and An information processing device comprising:

[0078] (Configuration 2) 2. The information processing device according to configuration 1, wherein the calculation means calculates an amount of adjustment for the installation position of the imaging device determined by the determination means so that the angle exceeding the threshold value becomes equal to or less than the threshold value. (Configuration 3) 3. The information processing apparatus according to configuration 2, wherein the calculation means calculates the adjustment amount based on the calculated angles so that all angles are equal to or smaller than the threshold value. (Configuration 4) The information processing device according to configuration 2 or 3, wherein the calculation means calculates, for each of two or more subjects, an angle between two straight lines connecting each of adjacent imaging devices and the subject, and calculates the adjustment amount for each of the two or more subjects so that all angles are equal to or less than the threshold value. (Configuration 5) The information processing device according to any one of configurations 2 to 4, further comprising an image generating means for generating an image including a captured image taken by the imaging device determined by the determining means and at least one of an angle exceeding the threshold value or the adjustment amount, and displaying the image on a display unit. (Configuration 6) The information processing device according to configuration 5, wherein the image generating means generates an image including an image captured by the imaging device determined by the determining means and a plurality of angles calculated by the calculating means, and displays the image on the display unit.

[0079] (method) an acquisition step of acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation step of calculating, based on the photographed images acquired in the acquisition step, angles between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject; a determination step of determining an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject, when any angle calculated in the calculation step exceeds a threshold value; An information processing method comprising:

[0080] (program) an acquisition step of acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation step of calculating, based on the photographed images acquired in the acquisition step, angles between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject; a determination step of determining an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject, when any angle calculated in the calculation step exceeds a threshold value; A program that causes a computer to execute the following. [Explanation of symbols]

[0081] 301 image receiving unit, 303 camera angle calculation unit, 304 inter-camera angle calculation unit, 305 adjustment determination unit, 306 adjustment camera determination unit

Claims

1. an acquisition means for acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation means for calculating an angle between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject based on the captured image acquired by the acquisition means; a determination means for determining, when any angle calculated by the calculation means exceeds a threshold value, an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject; and An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the calculation means calculates an amount of adjustment for the installation position of the imaging device determined by the determination means so that the angle exceeding the threshold value becomes equal to or smaller than the threshold value.

3. 3. The information processing apparatus according to claim 2, wherein the calculation means calculates the adjustment amount based on the calculated angles so that all angles are equal to or smaller than the threshold value.

4. The information processing device according to claim 2, characterized in that the calculation means calculates, for each of two or more subjects, the angle between two straight lines connecting each of adjacent imaging devices and the subject, and calculates the adjustment amount for each of the two or more subjects so that all angles are below the threshold value.

5. The information processing device according to any one of claims 2 to 4, further comprising an image generating means for generating an image including a captured image taken by the imaging device determined by the determining means and at least one of an angle exceeding the threshold value or the adjustment amount, and displaying the image on a display unit.

6. 6. The information processing device according to claim 5, wherein the image generating means generates an image including the captured image taken by the imaging device determined by the determining means and the plurality of angles calculated by the calculating means, and displays the image on the display unit.

7. an acquisition step of acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation step of calculating, based on the photographed images acquired in the acquisition step, angles between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject; a determination step of determining an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject, when any angle calculated in the calculation step exceeds a threshold value; An information processing method comprising:

8. an acquisition step of acquiring photographed images obtained by photographing a subject from a plurality of directions using a plurality of photographing devices; a calculation step of calculating, based on the photographed images acquired in the acquisition step, angles between two straight lines connecting the subject and each of adjacent imaging devices among the imaging devices that captured the subject; a determination step of determining an image capture device whose installation position is to be adjusted from among the image capture devices that have captured the subject, when any angle calculated in the calculation step exceeds a threshold value; A program that causes a computer to execute the following.

Images