Photographing system, photographing method, and program

Abstract

To provide an imaging system capable of easily calculating at least one of a blind area and a visible area from a subject, and provide an imaging method, and a program.SOLUTION: An imaging system S1 according to one embodiment includes: an imaging unit 11 which images a subject; an angle-of-view acquisition unit 126 (viewing angle acquisition unit) which acquires viewing angle information of the subject using the image captured by the imaging unit 11; a camera position / attitude acquisition unit 127 (subject information acquisition unit) which acquires position information and attitude information of the subject and position information of an object existing in a field-of-view area of the subject, using the image; and an image generation unit 128 (area calculation unit) which calculates at least one of a blind area in which the subject cannot be identified, and a visible area which is an area obtained by removing the blind area from the field-of-view area, on the basis of the viewing angle information of the subject, the position information and the attitude information, and the position information of the object existing in the field-of view area.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The present invention relates to a photography system, a photography method, and a program. [Background technology]

[0002] Techniques for implementing various types of control regarding cameras have been developed.

[0003] For example, Patent Document 1 discloses a technology in which, even when any interchangeable lens is attached to a camera body, the camera uses the model number information of the interchangeable lens to obtain power consumption information of the interchangeable lens via a network, thereby preventing the camera's power consumption from exceeding the available power supply of the power source. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-139689 A Summary of the Invention [Problem to be solved by the invention]

[0005] When a camera is shooting a specific space, in order to understand the range of the camera, it is necessary to actually shoot with the camera and analyze the captured image. Furthermore, depending on the location and number of cameras installed, there is a problem that shooting can be time-consuming.

[0006] An object of the present invention is to provide an imaging system, imaging method, and program that can easily calculate at least one of a blind spot area and a visible area from a subject. [Means for solving the problem]

[0007] An imaging system according to one aspect of the present invention includes an imaging unit that images a subject, a viewing angle acquisition unit that acquires viewing angle information of the subject using an image captured by the imaging unit, a subject information acquisition unit that acquires position information and posture information of the subject and position information of objects present within the viewing area of ​​the subject using the image, and an area calculation unit that calculates at least one of a blind spot area that the subject cannot grasp or a visible area, which is an area excluding the blind spot area from the viewing area, based on the viewing angle information, position information and posture information of the subject and the position information of objects present within the viewing area.

[0008] In one aspect of the present invention, a photography method includes a photography step of photographing a subject, a field of view acquisition step of acquiring field of view information of the subject using an image of the subject, a subject information acquisition step of acquiring position information and attitude information of the subject and position information of objects present within the field of view of the subject using the image, and an area calculation step of calculating at least one of a blind spot area that the subject cannot grasp, or a visible area which is an area excluding the blind spot area from the field of view, based on the field of view information, position information and attitude information of the subject and the position information of objects present within the field of view.

[0009] A program according to one aspect of the present invention causes a computer to execute the following steps: an imaging step of imaging a subject; a field of view acquisition step of acquiring field of view information of the subject using the image of the subject; a subject information acquisition step of acquiring position information and posture information of the subject and position information of objects present within the field of view of the subject using the image; and an area calculation step of calculating at least one of a blind spot area that the subject cannot grasp, or a visible area which is an area excluding the blind spot area from the field of view, based on the field of view information, position information and posture information of the subject and the position information of objects present within the field of view. Effect of the Invention

[0010] The present invention can provide an imaging system, imaging method, and program that can easily calculate at least one of a blind spot area and a visible area from a subject. [Brief description of the drawings]

[0011] [Figure 1A] 1 is a block diagram showing an example of the configuration of an imaging system according to a first embodiment. [Figure 1B] 4 is a block diagram showing a configuration example of a control unit according to the first embodiment; FIG. [Diagram 2] 2 is a schematic diagram illustrating a view frustum of a target camera according to the first embodiment; FIG. [Figure 3A] 11 is a flowchart illustrating an example of a process executed by a camera identifying unit according to the first embodiment. [Figure 3B] 11 is a flowchart illustrating an example of a process executed by a camera identifying unit according to the first embodiment. [Figure 4A] 5 is a flowchart showing an example of a process executed by a lens identification unit according to the first embodiment. [Figure 4B] 5 is a flowchart showing an example of a process executed by a lens identification unit according to the first embodiment. [Figure 5A] 4 is a flowchart showing an example of an overall process executed by the imaging system according to the first embodiment. [Figure 5B] 4 is a flowchart showing an example of an overall process executed by the imaging system according to the first embodiment. [Figure 5C] 4 is a flowchart showing an example of an overall process executed by the imaging system according to the first embodiment. [Figure 6] 13 is a schematic diagram illustrating the view frustums of a plurality of target cameras according to the second embodiment. FIG. [Figure 7A] 13 is a flowchart showing an example of an overall process executed by the imaging system according to the second embodiment. [Figure 7B] 13 is a flowchart showing an example of an overall process executed by the imaging system according to the second embodiment. [Figure 7C]13 is a flowchart showing an example of an overall process executed by the imaging system according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing a configuration example of an imaging system according to a third embodiment. [Figure 9A] FIG. 13 is a block diagram showing a configuration example of an imaging system according to a fourth embodiment. [Figure 9B] FIG. 13 is a block diagram showing a configuration example of a control unit according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] First embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In this embodiment, a photographing system is disclosed that uses an image photographed by a photographing unit to calculate at least one of a blind spot area of ​​the camera or a photographable area present in the image, and superimposes information on the area as a virtual image on the image photographed by the photographing unit, making it possible to display the information. Note that the "camera" that the photographing system photographs includes not only a camera having a normal camera body, but also any electronic device equipped with a camera function, such as a smartphone or a tablet terminal, and a robot equipped with a camera function. In addition, the camera may be one that photographs using not only visible light, but also light other than visible light, such as infrared light.

[0013] In the following explanation, the "angle of view" refers to the angle at which the camera in the image can capture light. Also, the "shooting area" refers to the area that the camera shoots when there is no area that the camera cannot grasp (blind area), while the "captureable area" refers to the area excluding the blind area from the shooting area when viewed from the camera.

[0014] Furthermore, as described below, the imaging system can also perform specific processing on general subjects other than cameras. For such general subjects, the angle at which the subject can capture light is referred to as the "viewing angle," and the area that the subject can see when there is no blind spot area as viewed from the subject is referred to as the "viewing area." Furthermore, the area excluding the blind spot area from the viewing area when viewed from the subject is referred to as the "visible area." The "viewing angle," "viewing area," and "visible area" are respectively used as higher-level concepts of the "angle of view," "shooting area," and "shootable area."

[0015] 1A is a block diagram showing a conceptual configuration example of a photography system S1. The photography system S1 includes a photography unit 11, a control unit 12, a camera DB (Database) 13, a lens DB 14, and a display unit 15. The photography system S1 may be a centralized system configured with a single computer, or may be a distributed system configured with multiple computers. Each component of the photography system S1 will be described below.

[0016] The photographing unit 11 photographs real images and outputs data of the photographed images to the control unit 12. The photographing unit 11 is configured to include, for example, one or more image pickup elements. The image photographed by the photographing unit 11 and used in the following processing is one or more images. The photographing unit 11 photographs multiple images, for example, by performing continuous photographing. When photographing continuously, the position of the target camera remains unchanged throughout each captured image. The photographing unit 11 functions as a stereo depth camera (3D camera) capable of measuring the distance between the photographing unit 11 and the target camera or object.

[0017] Using the image captured by the image capturing unit 11 and the data stored in the camera DB 13 and the lens DB 14, the control unit 12 calculates at least one of the blind spot area or the captureable area of ​​the camera (hereinafter also referred to as the target camera) present in the captured image, and displays the calculated information as a virtual image on the display unit 15. In the first embodiment, it is assumed that there is one target camera.

[0018] 1B is a block diagram of the control unit 12. The control unit 12 includes a memory 121, an I / O (Input / Output) unit 122, and an information processing unit 123.

[0019] The memory 121 is composed of a volatile memory, a non-volatile memory, or a combination of them. The memory 121 is not limited to one, and a plurality of memories may be provided. The volatile memory may be, for example, a RAM (Random Access Memory) such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). The non-volatile memory may be, for example, a PROM (Programmable ROM), an EPROM (Erasable Programmable Read Only Memory), or a Flash Memory.

[0020] The memory 121 is used to store one or more instructions. Here, the one or more instructions are stored in the memory 121 as a group of software modules. The information processing unit 123 can perform the following processes by reading and executing the one or more instructions from the memory 121.

[0021] The I / O unit 122 is a hardware interface that executes input and output of information to and from the outside of the control unit 12. In this embodiment, the control unit 12 is connected to the photographing unit 11, the camera DB 13, the lens DB 14, and the display unit 15, and appropriately executes input and output of information with these units via the I / O unit 122.

[0022] The information processing unit 123 is configured with any processor, such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), etc. The memory 121 may include a memory provided outside the information processing unit 123, as well as a memory built into the information processing unit 123.

[0023] The information processing unit 123 reads out and executes software (computer programs) from the memory 121, thereby realizing functions of a camera specifying unit 124, a lens specifying unit 125, an angle of view acquisition unit 126, a camera position / attitude acquisition unit 127, an image generation unit 128, etc. Each of these functions will be described below.

[0024] The camera identification unit 124 acquires an image captured by the image capture unit 11 and identifies the parameters of the camera housing of the target camera present in the image. The camera identification unit 124 identifies the camera housing by referring to the camera DB 13 using unique information of the camera housing shown in the captured image as a key. This unique information is at least one of the model number information of the camera housing or the housing shape of the target camera here, but is not limited to these.

[0025] Then, the camera identification unit 124 acquires and identifies the parameters stored in the camera DB 13 for that camera housing. In this example, the camera housing parameters are parameters that indicate the performance of the camera body, not the lens, such as the imaging element information of the camera housing, the lens mounting surface information, and the distance information of the imaging element. However, the camera housing parameters are not limited to this example as long as they are parameters for identifying the angle of view of the target camera.

[0026] The lens identification unit 125 acquires an image captured by the image capture unit 11, and identifies parameters of the lens attached to the target camera (hereinafter also referred to as the target lens) present in the image. The lens identification unit 125 identifies the lens by referring to the lens DB 14 using unique information of the target lens appearing in the captured image as a key. Here, this unique information is at least one of the lens model number information or the lens shape, but is not limited to these.

[0027] Then, the lens identification unit 125 acquires specification information about the lens stored in the lens DB 14. The acquired specification information is lens information such as the range of focal lengths that the lens can take and information about whether zooming is possible. The lens identification unit 125 identifies the focal length of the lens and whether zooming is possible by referring to the acquired specification information about the lens performance. If zooming is not possible with the lens to be identified, the focal length of the lens is a specific value, and the lens identification unit 125 acquires that value to identify the focal length of the target lens at the time of shooting.

[0028] On the other hand, when the target lens is capable of zooming, the lens identification unit 125 acquires specification information for identifying the zoom state of the target lens (in this example, information on the shape of the lens and information on the state of the zoom ring) from the lens DB 14. When zooming is possible, the focal length of the lens is a value within a predetermined range, and the focal length cannot be uniquely identified by simply identifying the lens. Therefore, the lens identification unit 125 compares the actual state of the target lens captured in the captured image with the specification information for identifying the zoom state to determine the zoom state of the target lens at the time of capture. In this way, the lens identification unit 125 uniquely identifies the focal length of the target lens at the time of capture from values ​​within a predetermined range. For example, the shape of the target lens and the state of its zoom ring are captured in the captured image, and the lens identification unit 125 acquires information on this state by analyzing an image of the side of the lens in the captured image.

[0029] In this manner, the lens identification unit 125 identifies the focal length of the lens at the time of shooting. However, the parameter identified by the lens identification unit 125 is not limited to the focal length of the lens at the time of shooting, so long as it is a parameter for identifying the angle of view of the target camera.

[0030] The angle of view acquisition unit 126 calculates and acquires the angle of view of the target camera at the time of shooting by the shooting unit 11 by using the parameters of the camera body identified by the camera identification unit 124 and the focal length of the lens at the time of shooting identified by the lens identification unit 125. Note that the angle of view may be any type of angle of view of at least one of horizontal, vertical, and diagonal, for example. In this example, the angle of view acquisition unit 126 calculates all the angles of view described above to calculate the view frustum (quadratic pyramid frustum) of the target camera.

[0031] The camera position / orientation acquisition unit 127 (subject information acquisition unit) acquires information on the coordinates (position) and orientation of the target camera appearing in the captured image using the image captured by the imaging unit 11. This allows the camera position / orientation acquisition unit 127 to identify the position and direction of the target camera at the time of capturing. Furthermore, the camera position / orientation acquisition unit 127 can also identify the coordinates (position information) of the outline of an object captured in the image.

[0032] The image generation unit 128 (area calculation unit) calculates the shooting area of ​​the target camera by integrating information on the viewing cone of the target camera calculated by the angle of view acquisition unit 126, information on the coordinates and attitude of the target camera acquired by the camera position / attitude acquisition unit 127, and position information of objects (obstacles) present within the field of view of the target camera.

[0033] Fig. 2 is a schematic diagram illustrating the view frustum of the target camera C from its side. In Fig. 2, the view frustum of the target camera C extends from the imaging element of the target camera C to infinity, and an obstacle O exists within the range of the view frustum. When there is no obstacle O, the imaging area A of the target camera C in the real space can be determined by applying the coordinate and orientation information of the target camera C to the imaging area A of the target camera C defined by the view frustum.

[0034] However, since an obstacle O (for example, a subject of the target camera C) exists in the real space of FIG. 2, an area B that is a blind spot for the target camera C exists behind the obstacle O as viewed from the target camera C. Therefore, the image generating unit 128 detects the presence of the obstacle O in the shooting area A of the target camera C in the captured image, and calculates at least one of a blind spot area B in the shooting area A from the target camera C originating from the obstacle O, or a shootable area D which is an area obtained by removing the blind spot area B from the shooting area A. Then, the image generating unit 128 generates a superimposed image in which information on at least one of the calculated blind spot area B or the shootable area D is superimposed on the captured image. Note that the image generating unit 128 generates a superimposed image, but is not limited to this, and at least one of information on the calculated blind spot area B or the shootable area D may be generated as an image.

[0035] The camera identification unit 124 and the lens identification unit 125 can determine the presence or absence of a camera housing or a lens, and identify model number information or shape by performing known image analysis processing. The lens identification unit 125 can also identify the zoom state of the lens by performing known image analysis processing. Similarly, the camera position / attitude acquisition unit 127 and the image generation unit 128 can calculate the coordinates and attitude of the target camera, detect the presence of an obstacle, and calculate a blind spot area or a captureable area by using known analysis processing. For example, AI (Artificial Intelligence) technology such as deep learning can be used for the analysis processing.

[0036] Moreover, each element of the control unit 12 may be realized by a plurality of computers connected by a network. Furthermore, any calculation process executed by each element may be executed by a computer other than the imaging system S1, and the control unit 12 may be configured to obtain the execution result.

[0037] Returning to FIG. 1A, the description will be continued. The camera DB 13 stores camera housing model number information, housing shape, and camera housing parameters in association with each other for a plurality of cameras. The camera DB 13 stores feature points indicating the characteristics of the housing in order to determine the shape and attitude of the housing. Furthermore, the camera DB 13 also stores general-purpose parameters for when a specific camera cannot be identified. The camera identification unit 124 accesses the camera DB 13 and acquires parameter information as described above.

[0038] The lens DB14 stores model number information, shape, and specification information of a plurality of lenses in association with each other. The lens DB14 stores feature points indicating lens characteristics in order to determine the shape and attitude of the lens. As described above, the specification information is lens information such as the range of focal lengths that the lens can take and information on whether zooming is possible. If zooming is possible, the specification information further includes specification information for identifying the zoom state. Furthermore, the lens DB14 also stores general-purpose parameters for when a specific lens cannot be identified. The lens identification unit 125 accesses the lens DB14 and acquires the parameter information as described above.

[0039] The camera DB 13 and the lens DB 14 may be connected to the control unit 12 via a network. For example, the camera DB 13 and the lens DB 14 may be a so-called cloud-type database, and specifically, the camera DB 13 and the lens DB 14 may be a HDD (Hard Disk Drive). In addition, when the target camera is a type such as a lens-integrated camera, the camera DB 13 and the lens DB 14 may be constructed as the same database so that the camera body parameters and the lens parameters can be acquired together with a single access by the control unit 12.

[0040] The display unit 15 is an interface such as a display that displays the superimposed image generated by the image generation unit 128, AR (Augmented Reality) glasses, etc. For example, when the display unit 15 is AR glasses and the image capture unit 11 captures the direction of the user's gaze while the user is wearing the AR glasses, the user can see the superimposed image through the AR glasses. As another example, the user may view the real space through AR glasses that are configured to be transparent. In this case, the display unit 15 can project a virtual image indicating at least one of a blind spot area or a captureable area at a position on the AR glasses corresponding to the position of the area superimposed in the above-mentioned superimposed image. In this way, the display unit 15 can visually emphasize at least one of a range that can be captured by the camera or a range that cannot be captured, and display it to the user. Note that the display unit 15 displays the superimposed image generated by the image generation unit 128, but is not limited to this, and may display at least one of information on the blind spot area B or the captureable area D as an image.

[0041] The photographing system S1 may also be provided with an input unit such as a button or a touch panel that accepts input from a user. The user inputs from the input unit whether to display one or both of the blind area and the imageable area from the target camera. The image generating unit 128 determines the target to be calculated for the blind area and the imageable area from the target camera based on the input, and calculates at least one of them. Then, the image generating unit 128 generates a superimposed image in which at least one of the calculated information of the blind area or the imageable area is superimposed on the photographed image. The display unit 15 displays the superimposed image. The user may also input information regarding the attitude of the target camera from the input unit.

[0042] 3A and 3B are flowcharts showing an example of a process in which the camera identifying unit 124 identifies the housing of the target camera by referring to the camera DB 13. Hereinafter, the details of the process of the camera identifying unit 124 will be described with reference to FIGS. 3A and 3B.

[0043] First, the camera identifying unit 124 acquires an image captured by the image capturing unit 11 (step S11). Next, the camera identifying unit 124 determines whether or not a camera is present in the captured image (step S12). If no camera is present in the captured image (No in step S12), the camera identifying unit 124 ends the process and waits until the next captured image is acquired.

[0044] If a camera is present in the captured image (Yes in step S12), the camera identification unit 124 determines whether the model number information of the camera is present as text information in the captured image, that is, whether the model number information is identifiable (step S13). If the model number information is not identifiable (No in step S13), the camera identification unit 124 acquires the shape of the camera housing present in the captured image from the captured image (step S14). Then, the camera identification unit 124 determines whether a camera housing corresponding to the shape exists in the camera DB 13 by referring to the camera DB 13 using the shape as a key (step S15). For example, the camera identification unit 124 can execute the determination in step S15 by comparing the feature points of the camera housing present in the captured image with the feature points of each housing stored in the camera DB 13.

[0045] If the model number information can be determined (Yes in step S13), or if the acquired shape of the camera housing exists in the camera DB 13 (Yes in step S15), the camera identifying unit 124 acquires parameters associated with the camera housing existing in the camera DB 13 as camera information (step S16). The camera identifying unit 124 thus ends the process.

[0046] If the acquired camera housing shape does not exist in the camera DB 13 (No in step S15), the camera identifying unit 124 acquires general-purpose parameters related to the camera housing as camera information from the camera DB 13 (step S17). The camera identifying unit 124 thus ends the process.

[0047] In addition, the camera DB 13 may roughly classify the camera housing into a plurality of types with respect to at least one of the size, shape, and structure of the camera housing, and may set general-purpose parameters for each of these types. For example, the camera DB 13 may set camera types such as digital cameras, single-lens reflex cameras, mirrorless cameras, and smartphones, and general-purpose parameters may be set for each of these types. As a result, even if the camera model cannot be identified, the camera identification unit 124 can obtain parameters that are relatively close to the parameters of the actual camera housing as camera information by comparing at least one of the size, shape, and structure of the camera housing in the captured image with such factors stored in the camera DB 13. Therefore, the calculation of the captureable area described later is performed with higher accuracy.

[0048] Furthermore, the photography system S1 may store the above-mentioned general-purpose parameters in advance in a storage unit (for example, memory 121) on the control unit 12 side. If the above-mentioned process by the camera identification unit 124 cannot be performed for some reason, such as an inability to access the camera DB 13, the camera identification unit 124 may access the storage unit and execute the same process as described above to obtain the stored general-purpose parameters as camera information. There may be one type of general-purpose parameter, or, as described above, a general-purpose parameter may be set for each type of camera housing.

[0049] 4A and 4B are flowcharts showing an example of a process in which the lens identification unit 125 identifies the lens of a target camera (hereinafter, also referred to as a target lens) by referring to the lens DB 14. Hereinafter, the details of the process of the lens identification unit 125 will be described with reference to FIGS. 4A and 4B.

[0050] First, the lens identification unit 125 acquires an image captured by the image capture unit 11 (step S21). Next, the lens identification unit 125 determines whether or not a lens is present in the captured image (step S22). If there is no lens in the captured image (No in step S22), the lens identification unit 125 ends the process and waits until the next captured image is acquired.

[0051] If a lens is present in the photographed image (Yes in step S22), the lens identification unit 125 determines whether model number information of the target lens is present as text information in the photographed image, that is, whether the model number information is identifiable (step S23). If the model number information is not identifiable (No in step S23), the lens identification unit 125 acquires the shape of the lens appearing in the photographed image from the photographed image (step S24). Then, the lens identification unit 125 determines whether a lens corresponding to the shape exists in the lens DB 14 by referring to the lens DB 14 using the shape as a key (step S25). For example, the lens identification unit 125 can execute the determination in step S25 by comparing the feature points of the shape of the lens appearing in the photographed image with the feature points of the shape of each lens stored in the camera DB 13.

[0052] If the model number information can be determined (Yes in step S23) or if the acquired lens shape exists in the lens DB 14 (Yes in step S25), the lens identification unit 125 acquires, as lens information, specification information on the lens performance associated with the lens existing in the lens DB 14 (step S26). Then, the lens identification unit 125 determines whether the target lens is capable of zooming based on the acquired specification information (step S27).

[0053] If the target lens is zoomable (Yes in step S27), the lens identification unit 125 compares the acquired specification information on the lens performance with the actual state of the lens captured as an image to determine the zoom state at the time of shooting (step S28). The lens identification unit 125 identifies the focal length of the lens at the time of shooting based on the determined zoom state at the time of shooting (step S29). The details are as described above. If the target lens is not zoomable (No in step S27), the lens identification unit 125 identifies the focal length of the lens included in the acquired specification information on the lens performance as the focal length of the lens (step S29).

[0054] If the acquired lens shape does not exist in the lens DB 14 (No in step S25), the lens identification unit 125 acquires general-purpose parameters as lens information from the lens DB 14 (step S30). Through this process, the lens identification unit 125 identifies the focal length of the lens. The lens identification unit 125 thus ends the process.

[0055] In the lens DB14, lenses may be roughly divided into a plurality of types with respect to at least one of the size, shape, and structure of the lens, and general-purpose parameters may be set for each of the types. For example, in the lens DB14, lens types (attributes) such as a zoom lens, a single focus lens, a standard lens, a wide-angle lens, and a telephoto lens may be set, and general-purpose parameters corresponding to each type may be set. In addition, a combination of a plurality of attributes such as a standard zoom lens, a wide-angle zoom lens, and a high-magnification zoom lens may be set as the lens type. As a result, even if the specific type of the lens cannot be identified, the lens identification unit 125 can obtain parameters relatively close to the parameters of the actual lens as lens information by comparing at least one of the size, shape, and structure of the lens in the captured image with such factors stored in the camera DB13. Therefore, the calculation of the imageable area described later is performed with higher accuracy.

[0056] Furthermore, the photographing system S1 may store the above-mentioned general-purpose parameters in advance in a storage unit (for example, memory 121) on the control unit 12 side. If the above-mentioned process by the lens identification unit 125 cannot be performed for some reason, such as inability to access the lens DB 14, the camera identification unit 124 may access the storage unit and execute the same process as described above to obtain the stored general-purpose parameters as camera information. There may be one type of general-purpose parameter, or, as described above, a general-purpose parameter may be set for each type of lens.

[0057] The determination process of steps S13 and S15 shown in Fig. 3A may be executed in reverse order. Similarly, the determination process of steps S23 and S25 shown in Fig. 4A may be executed in reverse order.

[0058] 5A to 5C are flowcharts showing an example of the overall processing executed by the photography system S1, and the overall processing of the photography system S1 will be described below with reference to FIGS. 5A to 5C.

[0059] First, the photographing unit 11 photographs an image of the target camera (step S31). Next, the camera identifying unit 124 acquires the image photographed by the photographing unit 11, thereby acquiring parameters related to the camera body of the target camera as camera information (step S32). Details of the process of this step S32 are as shown in Figs. 3A and 3B. Furthermore, the lens identifying unit 125 acquires the image photographed by the photographing unit 11, thereby acquiring focal length information related to the target lens as lens information (step S33). Details of the process of this step S33 are as shown in Figs. 4A and 4B.

[0060] Either of the processes in steps S32 and S33 may be executed first, or both may be executed in parallel. If desired information cannot be acquired in at least one of steps S32 and S33, the photography system S1 may determine that there is no camera capable of photographing in the photographed image, and may discontinue subsequent processes.

[0061] The angle of view acquisition unit 126 calculates the angle of view of the target camera at the time of shooting by using the parameters of the camera body identified by the camera identification unit 124 and the focal length of the lens at the time of shooting identified by the lens identification unit 125, thereby calculating the viewing frustum of the target camera (step S34).

[0062] In the photographing system S1, the coordinates and orientation of the photographing unit 11 are fixed in advance, and information on the coordinates and orientation, and information on the coordinate system are stored in the memory 121. The camera position / orientation acquisition unit 127 reads and acquires the information (step S35).

[0063] The camera position / attitude acquisition unit 127 acquires the coordinates of the target camera using the coordinates and attitude state (shooting direction) of the shooting unit 11 acquired in step S35 and the depth information of the image in which the target camera is captured (step S36). The coordinates of the target camera and the coordinates of the shooting unit 11 are coordinates in the same coordinate system. If the lens identification unit 125 has identified the target lens (that is, if the model number information of the target lens can be determined or if the acquired shape of the target lens exists in the lens DB 14), the camera position / attitude acquisition unit 127 may set the coordinates of the center of the identified lens as the coordinates of the target camera. If the lens identification unit 125 has not identified the target lens, the camera position / attitude acquisition unit 127 may set the coordinates of the part of the target camera closest to the shooting unit 11 as the coordinates of the target camera.

[0064] The camera position / attitude acquisition unit 127 acquires feature points of the shape information of the camera housing identified by the camera identification unit 124 and the lens identified by the lens identification unit 125 from the camera DB 13 and the lens DB 14, respectively. Then, the camera position / attitude acquisition unit 127 acquires the attitude information of the target camera captured in the captured image by comparing the acquired feature points of the camera housing and the lens with the feature points of the target camera captured in the captured image (step S37). If the feature points of the shape information cannot be acquired from at least one of the camera DB 13 and the lens DB 14, the camera position / attitude acquisition unit 127 acquires the attitude information of the target camera acquired from the input unit of the imaging system S1 in step S37. Any of the processes of step S37, steps S32 to S34, and steps S35 to S36 may be executed first, or any of these processes may be executed in parallel.

[0065] The image generation unit 128 integrates the information on the view frustum of the target camera calculated in step S34 with the coordinate and orientation information of the target camera acquired in steps S36 and S37, respectively, to calculate the shooting area of ​​the target camera (step S38).

[0066] The camera position / orientation acquisition unit 127 extracts the contour of an object (obstacle) in the shooting area of ​​the target camera from the image captured by the shooting unit 11 (step S39). For example, the camera position / orientation acquisition unit 127 detects edges in the captured video and divides the area by the edges to extract the contour of the object. A known technique can be applied to this contour extraction. In this way, the camera position / orientation acquisition unit 127 detects the presence of an obstacle in the shooting area of ​​the target camera in the captured image.

[0067] The camera position / attitude acquisition unit 127 uses the coordinate and attitude information of the photographing unit 11 acquired in step S35 and the depth information of the image in which the target camera is captured to identify the coordinates (position information) of the contour of the object extracted in step S39 (step S40). After that, the image generation unit 128 continuously draws straight lines passing through the coordinates of the contour of the identified object and the coordinates of the target camera (for example, the coordinates of the center of the lens of the target camera) so that the contour of the identified object passes through in its entirety. The image generation unit 128 determines the area surrounded by the straight lines behind the object as seen from the target camera as a blind spot area in the photographing area from the target camera originating from the object (obstacle) (step S41). The image generation unit 128 further calculates a photographable area, which is an area obtained by removing the blind spot area determined in step S41 from the photographing area (step S42). Then, the image generation unit 128 generates a superimposed image in which the calculated information of the blind spot area and the photographable area are superimposed on the photographed image (step S43). The image generating unit 128 outputs the generated superimposed image to the display unit 15, and the display unit 15 displays the superimposed image (step S44).

[0068] Recently, information devices have evolved significantly due to the evolution of technologies such as high-speed CPUs (Central Processing Units), high-definition screen display technology, small and lightweight batteries, as well as the spread and broadband of wireless network environments. As such information devices, not only smartphones, which are a representative example, but also so-called wearable devices have become widespread. As such wearable devices, many information devices have been developed that use glasses-type HMDs (head-mounted displays) to view a virtual reality world.

[0069] The photographing system S1 according to the present invention can be applied as such an information device, for example. As a specific process, the angle of view acquisition unit 126 acquires the angle of view (viewing cone) information of the camera, the camera position / attitude acquisition unit 127 acquires the position information and attitude information of the camera, and further the coordinates of the contour of the object, and the image generation unit 128 calculates at least one of the blind spot area or the photographable area based on these pieces of information, and generates a superimposed image in which the information of the area is emphasized. At this time, by simply photographing the target camera with the photographing system S1, the camera identification unit 124 and the lens identification unit 125 can respectively acquire information on the camera housing and the attached lens from the photographed image. Therefore, the user does not need to input information about the camera (for example, model number information), and the photographing system S1 can easily visualize the blind spot area and the visible area of ​​the camera.

[0070] For the user, the imaging system S1 presents the blind spot and visual recognition area of ​​the camera in an easy-to-understand manner in the virtual reality world, so the imaging system S1 can be used for various purposes. For example, the imaging system S1 can be used for photogrammetry purposes, which converts a captured image into a 3D (dimensional) model, and for crime prevention purposes, when a surveillance camera is installed in a certain space (e.g., a room), to search for the blind spot of the surveillance camera.

[0071] Embodiment 2 In the first embodiment, an example in which one camera is provided as a subject has been described, but multiple cameras may be provided as subjects. In the second embodiment, the processing performed by the photography system S1 in such a case will be described. The configuration of the photography system S1 is as shown in Figs. 1A and 1B. As for the processing performed by each unit of the photography system S1, differences from the first embodiment will be outlined, and details of other processing will be omitted since they are the same as those of the first embodiment.

[0072] The camera identification unit 124 acquires an image captured by the image capture unit 11, and identifies the camera housing parameters for each of the multiple target cameras present in the image. The camera identification unit 124 identifies each camera housing by referring to the camera DB 13 using unique information of each camera housing captured in the captured image as a key. The camera identification unit 124 then acquires and identifies the parameters stored in the camera DB 13 for each camera housing.

[0073] The lens identification unit 125 acquires an image captured by the image capture unit 11, and identifies the parameters of the lens attached to each of the multiple target cameras present in the image. The lens identification unit 125 identifies each lens by referring to the lens DB 14 using unique information of each lens captured in the captured image as a key.

[0074] Then, the lens identification unit 125 acquires the specification information stored in the lens DB 14 for each lens. For a target lens that cannot be zoomed, the focal length of the lens is a specific value, and the lens identification unit 125 acquires this value to identify the focal length of the lens at the time of shooting. On the other hand, for a target lens that can be zoomed, the lens identification unit 125 acquires specification information for identifying the zoom state of the target lens from the lens DB 14. The focal length of the lens included in this specification information is a value within a predetermined range, and the lens identification unit 125 determines the zoom state of the lens at the time of shooting to uniquely identify the focal length of the lens at the time of shooting from a value within the predetermined range. In this manner, the lens identification unit 125 identifies the focal length of each lens at the time of shooting.

[0075] The angle of view acquisition unit 126 calculates the angle of view of each target camera at the time of shooting, captured by the shooting unit 11, by using the parameters of each camera body identified by the camera identification unit 124 and the focal length of each lens at the time of shooting identified by the lens identification unit 125, thereby calculating the viewing frustum of each target camera.

[0076] The camera position / orientation acquisition unit 127 acquires information on the coordinates and orientation of each target camera appearing in the captured image using the image captured by the image capture unit 11. This allows the camera position / orientation acquisition unit 127 to identify the position and direction of each target camera at the time of capture. Furthermore, the camera position / orientation acquisition unit 127 also acquires position information of objects present within the viewing area of ​​each target camera.

[0077] The image generation unit 128 integrates information on the viewing cone (quadratic pyramid) of each target camera calculated by the angle of view acquisition unit 126, information on the coordinates and attitude of each target camera acquired by the camera position / attitude acquisition unit 127, and position information of objects (obstacles) present within the field of view of each target camera, to calculate the shooting area for each of the multiple target cameras.

[0078] 6 is a schematic diagram illustrating the view frustum of each of two target cameras from the side. In FIG. 6, the imaging area (view frustum) of the target camera C1 is represented by A1, and the imaging area (view frustum) of the target camera C2 is represented by A2. The imaging areas A1 and A2 are the same as the imaging area A in FIG. 2.

[0079] Here, since an obstacle O exists in the real space of FIG. 6, there are regions that are blind spots from the target cameras C1 and C2. The target cameras C1 and C2 have shooting areas A1 and A2 set so as to cover each other's blind spots to some extent, but there is still a region B12 that is blind from both cameras. The camera position / attitude acquisition unit 127 detects the presence of the obstacle O in the shooting areas A1 and A2, and the image generation unit 128 calculates the blind spot area in each shooting area. Then, the image generation unit 128 calculates at least one of four types of areas: the shooting area for each of the target cameras C1 and C2, the area A12 where the shooting areas of the target cameras C1 and C2 overlap, the area that can be shot by only one of the target cameras C1 and C2, or the area B12 that is blind from all of the target cameras C1 and C2. The image generation unit 128 generates a superimposed image in which information on the calculated area is superimposed on the captured image. Regarding the captureable areas of each of the target cameras C1 and C2, the image generation unit 128 may display the combined captureable areas of each camera as a single area in the superimposed image, or may display the captureable areas in the superimposed image so that they can be identified for each camera (for example, by color coding, etc.).

[0080] It should be noted that which of the above-mentioned four types of regions the image generating unit 128 calculates and displays can be switched by the image generating unit 128 judging the user's intended needs. For example, the image generating unit 128 may determine whether or not there are a predetermined threshold or more (e.g., three or more) cameras as subjects in the captured image. If there are a predetermined threshold or more cameras in the captured image, it is assumed that the cameras are installed to eliminate blind spots from the cameras for crime prevention reasons, etc., as a motivation for the user to install such a large number of cameras. Therefore, in this case, the image generating unit 128 calculates at least the region that is a blind spot from all the cameras (region B12 in FIG. 6) and displays it on the display unit 15.

[0081] As another example, the image generating unit 128 may calculate at least the area that is a blind spot from all the cameras and display it on the display unit 15 when the combined area of ​​the shooting areas of each camera in the captured image (shooting areas A1 and A2 in FIG. 6) is equal to or larger than a predetermined threshold or is equal to or larger than a predetermined percentage of the entire area of ​​the captured image.

[0082] The photographing system S1 may also be provided with an input unit such as a button or a touch panel that accepts input from a user. The user inputs one or more areas to be displayed from the above-mentioned four types of areas through the input unit. The image generating unit 128 determines which of the above-mentioned four types of areas to calculate based on the input, and calculates at least one of them. Then, the image generating unit 128 generates a superimposed image by superimposing information of the calculated area on the photographed image. The display unit 15 displays the superimposed image.

[0083] 7A and 7B are flowcharts showing an example of the overall processing executed by the photography system S1, and the overall processing of the photography system S1 will be described below with reference to Figs. 7A and 7B. Note that the description of the points already described in the description of Figs. 5A to 5C will be omitted as appropriate.

[0084] First, the photographing unit 11 photographs an image of the target camera (step S51). Next, the camera identifying unit 124 acquires the image photographed by the photographing unit 11, thereby acquiring parameters related to the camera housing of each target camera as camera information (step S52). In addition, the lens identifying unit 125 acquires the image photographed by the photographing unit 11, thereby acquiring focal length information related to each target lens as lens information (step S53).

[0085] The angle of view acquisition unit 126 calculates the angle of view of each target camera at the time of shooting by using the camera body parameters identified by the camera identification unit 124 and the focal length of the lens at the time of shooting identified by the lens identification unit 125. In this way, the angle of view acquisition unit 126 calculates the viewing frustum of each target camera (step S54).

[0086] The camera position / attitude acquisition unit 127 reads and acquires information related to the coordinate system, and information on the coordinates and attitude of the image capturing unit 11 for each image from the memory 121 (step S55). The camera position / attitude acquisition unit 127 acquires the coordinates of each target camera using the coordinates and attitude state (image capturing direction) of the image capturing unit 11 acquired in step S55 and depth information of the image in which the target camera appears (step S56).

[0087] The camera position / orientation acquisition unit 127 acquires orientation information of each target camera appearing in the captured image by comparing the acquired feature points of each camera housing and each lens with the feature points of each target camera appearing in the captured image (step S57).

[0088] The image generation unit 128 integrates the information of the view frustum of each target camera calculated in step S54 with the coordinate and orientation information of each target camera acquired in steps S56 and S57, respectively, to calculate the shooting area of ​​each target camera (step S58).

[0089] The camera position / attitude acquisition unit 127 extracts the contour of an object (obstacle) in the shooting area of ​​each target camera from the image captured by the shooting unit 11 (step S59). The camera position / attitude acquisition unit 127 identifies the coordinates (position information) of the contour of the object using the coordinate and attitude information of the shooting unit 11 acquired in step S55 and the depth information of the image captured by each target camera (step S60).

[0090] The image generating unit 128 draws straight lines passing through the coordinates of the contour of the identified object and the coordinates of each target camera (for example, the coordinates of the center of the lens of the target camera) continuously so that all contours of the identified object pass through the straight lines. The image generating unit 128 judges the area surrounded by the straight lines behind the object as a blind area in the shooting area of ​​each target camera originating from the object (obstacle) (step S61). The image generating unit 128 uses the judgment result of step S61 to specify the area that is a blind area from all target cameras, and generates an image in which the area is superimposed (step S62). Note that the area to be calculated and displayed may be at least one of the shooting area of ​​each target camera, the area where the shooting areas of two or more target cameras overlap, and the area that only one target camera among the multiple target cameras can shoot. Alternatively, the area to be calculated and displayed may include at least one of these areas and the area that is a blind area from all target cameras. The image generating unit 128 outputs the generated superimposed image to the display unit 15, and the display unit 15 displays the superimposed image (step S63).

[0091] As described above, the image generating unit 128 of the imaging system S1 can calculate at least any one of the following for multiple target cameras captured in a captured image: the imageable area for each target camera, the area where the imageable areas of the target cameras overlap, the area that can be captured by only one of the target cameras, or the blind spot area from all the target cameras. Then, a superimposed image in which information on the calculated area is superimposed on the image can be generated and displayed on the display unit 15. Therefore, the imaging system S1 can easily visualize the blind spot area and the visual recognition area of ​​each camera.

[0092] Third embodiment The photography system may take multiple images of a subject from at least one of different locations and different orientations, thereby more accurately and three-dimensionally identifying the photography area of ​​the target camera, and generating a three-dimensional superimposed image based on that photography area.

[0093] 8 is a block diagram showing a conceptual configuration example of the photography system S2. The photography system S2 further includes a photography information acquisition unit 16 in addition to the photography system S1. The photography system S2 also includes a plurality of photography units 11 (11A, 11B) as a configuration for taking a plurality of images of a camera as a subject from at least one of different locations and different orientations. The photography units 11A and 11B photograph the same subject from at least one of different locations and different orientations.

[0094] When the image capturing unit 11 captures images of a camera as a subject from a plurality of different locations, the image capturing information acquisition unit 16 acquires image capturing position information indicating where the image capturing unit 11 is located in real space at each image capturing time, and image capturing attitude information indicating what attitude state the image capturing unit 11 is in at each image capturing time. The image capturing position information includes, for example, values ​​indicating at least one of the coordinate position, speed, acceleration, or speed direction of the image capturing unit 11 in a three-dimensional rectangular coordinate system. Note that the change in the speed direction or attitude may be expressed as yaw, pitch, or roll. The image capturing attitude information may be expressed using a quaternion. The image capturing information acquisition unit 16 may acquire the image capturing position information using, for example, a satellite positioning system such as a GPS (Global Positioning System).

[0095] Each unit of the control unit 12 calculates the shooting area for each of the multiple target cameras by executing the same process as in the second embodiment for each of the multiple captured images captured by the imaging unit 11. Since the multiple target cameras do not move throughout the captured images, an image in which the position of each target camera is fixed is generated as a superimposed image. In step S35, the camera position / attitude acquisition unit 127 acquires shooting position information and shooting attitude information from the shooting information acquisition unit 16.

[0096] Each of the camera identification unit 124 to the camera position / attitude acquisition unit 127 of the control unit 12 may determine whether the target cameras shown in different captured images indicate the same camera or different cameras by comparing at least one of the positions and shapes of the target cameras. Based on the result of this determination, the image generation unit 128 can generate a superimposed image in which the target cameras are arranged in the same positions in space as they are in the captured images.

[0097] The image generating unit 128 calculates at least one of four types of areas: a captureable area for each target camera, an area where the captureable areas of the target cameras overlap, an area where only one of the multiple target cameras can capture, or a blind spot area from all the target cameras. The image generating unit 128 generates a superimposed image by superimposing the calculated area information on the captured image. The image generating unit 128 then generates a three-dimensional superimposed image by superimposing the calculated area information on an image that is three-dimensionally expressed by combining multiple captured images. The display unit 15 displays the superimposed image in at least one of the formats of VR (Virtual Reality) and AR (Augmented Reality). This superimposed image is displayed to the user in a state where the distance between the capturing unit 11 and the target camera or object is reflected.

[0098] As described above, the image generating unit 128 can generate a three-dimensional superimposed image by using the shooting position information and shooting attitude information acquired by the shooting information acquiring unit 16. In other words, the shooting system S2 can identify multiple target cameras in a space from multiple images captured continuously, combine information related to the identified target cameras, and present the information to the user as more three-dimensional information.

[0099] Each unit of the control unit 12 may calculate the shooting area of ​​one target camera by executing the same process as in the first embodiment for each of the captured images captured by the image capturing unit 11. After that, the image generating unit 128 calculates at least one of a blind spot area in the shooting area of ​​the target camera in the captured image, or a captureable area which is an area excluding the blind spot area from the shooting area. Then, the image generating unit 128 generates a three-dimensional superimposed image by superimposing information on the calculated area on an image which is three-dimensionally expressed by combining the multiple captured images. The image generating unit 128 outputs the generated superimposed image to the display unit 15, and the display unit 15 displays the superimposed image. Even in this way, the image capturing system S2 can calculate the area and generate the superimposed image.

[0100] In particular, when executing the process of step S39 in Fig. 5B (extraction of contours), the photographing system S2 needs to extract feature points necessary to identify the contour of the object as seen by the target camera. At this time, the photographing system S2 is provided with multiple photographing units 11, and photographs the object from different locations or different directions, so that all the contours of the object as seen by the target camera can be photographed. If it is not possible to extract all the required feature points in one photographing, the object is photographed from further different locations or different directions so that all the contours of the object as seen by the target camera can be photographed.

[0101] However, the photographing system S2 may have only one photographing unit 11, and the photographing unit 11 may take a plurality of photographs of the subject from at least one of different locations and different orientations. In this way, the photographing system S2 photographs the feature points necessary for identifying the above-mentioned object, and thereby the photographing system S2 can calculate the area and generate a superimposed image.

[0102] Fourth embodiment The imaging systems described in the first to third embodiments generate images that visualize blind spots and imaging areas of a camera as a subject, as a technology for visualizing things that cannot be seen in a virtual reality world. However, the subject may be something other than a camera, for example, a human being.

[0103] 9A is a block diagram showing a conceptual configuration example of an imaging system S3 according to embodiment 3. The imaging system S3 includes an imaging unit 31, a control unit 32, a person attribute DB (Database) 33, and a display unit 34. Hereinafter, explanations of the same points as in embodiment 1 will be omitted as appropriate.

[0104] Similar to the imaging unit 11, the imaging unit 31 captures one or more images and outputs data of the captured images to the control unit 32.

[0105] The control unit 32 uses the image captured by the imaging unit 31 and the data stored in the person attribute DB 33 to calculate at least one of the blind spot area or the field of view area of ​​a person (hereinafter also referred to as the target person) present in the captured image, and displays the calculated information as a virtual image on the display unit 34.

[0106] 9B is a block diagram of the control unit 32. The control unit 32 has a memory 321, an I / O unit 322, and an information processing unit 323. The memory 321 and the I / O unit 322 execute the same processes as the memory 121 and the I / O unit 122 according to the first embodiment, and therefore the description thereof will be omitted.

[0107] The information processing unit 323 is configured with an arbitrary processor or the like, as in the first embodiment, and realizes functions of a viewing angle acquisition unit 324, a person position / posture acquisition unit 325, an image generation unit 326, etc., by reading and executing software (computer program) from the memory 321. Each of these functions will be described below.

[0108] The viewing angle acquisition unit 324 acquires an image captured by the image capture unit 11, analyzes the image, and extracts information indicating attributes that determine the viewing angle of a person (hereinafter, a target person) present in the image from the captured image. This information is, for example, the facial shape and age of the target person. This image analysis can use publicly known technology such as AI. Then, the viewing angle acquisition unit 324 acquires the viewing angle of the target person by referring to the person attribute DB 33 using the information as a key. In this example, the viewing angle acquisition unit 324 acquires information on the viewing cone (corresponding to the viewing frustum of a camera) of the target person by acquiring the viewing angles in the horizontal, vertical, and diagonal directions.

[0109] The person position / orientation acquisition unit 325 uses the image captured by the imaging unit 31 to calculate the coordinates (position information) of the target person in the captured image and the posture of the target person. This allows the person position / orientation acquisition unit 325 to identify the position of the target person and the direction in which the target person is facing at the time of capturing the image. In particular, the person position / orientation acquisition unit 325 calculates the eye position of the target person as position information in order to calculate the visual field area of ​​the target person. Furthermore, the person position / orientation acquisition unit 325 can also identify the coordinates (position information) of the outline of an object reflected in the image.

[0110] The image generating unit 326 integrates the information on the visual cone of the target person calculated by the visual angle acquiring unit 324, the information on the coordinates and posture of the target person acquired by the person position / posture acquiring unit 325, and the position information of an object (obstacle) existing in the visual field of the target person to calculate the visual field of the target person. Then, the person position / posture acquiring unit 325 detects the presence of an obstacle in the visual field of the target person, and the image generating unit 326 calculates at least one of a blind spot area in the visual field of the target person caused by the obstacle, or a visual recognition area which is an area excluding the blind spot area from the visual field. Then, the image generating unit 326 generates a superimposed image by superimposing the information on the calculated area on the captured image. The details are as described in the first embodiment. The image generating unit 326 outputs the superimposed image to the display unit 34.

[0111] Returning to Fig. 9A, the description will be continued. In the person attribute DB 33, information indicating the attributes of a person is stored in association with information on the viewing angle of the person. The viewing angle acquisition unit 324 accesses the person attribute DB 33 and acquires the viewing angle information as described above. The display unit 34 is a display unit similar to the display unit 15, and the description will be omitted.

[0112] In this manner, the photographing system S3 can easily visualize the blind spot area and the visual recognition area of ​​a person. This photographing system S3 can also be applied to the second embodiment in the same manner. In this case, the image generating unit 326 can calculate at least one of the visual recognition area for each of the multiple target persons captured in the captured image, the area where the visual recognition areas of the target persons overlap, the visual recognition area of ​​only one of the target persons, or the blind spot area from all the target persons. Similarly, the photographing system S3 can also generate a three-dimensional superimposed image by applying it to the third embodiment in the same manner. The photographing system S3 can also be applied to animals other than humans.

[0113] The present invention is not limited to the above embodiment, and can be modified as appropriate within the scope of the present invention. For example, when a camera and a person are mixed in an image captured by the imaging system, the imaging system may calculate the blind spot area and imageable area of ​​each camera and the blind spot area and visibility area of ​​each person in the same manner as in the above embodiment, and display them in the superimposed image. In addition, when a plurality of subjects such as cameras or people are captured, the user may specify via the input unit whether or not each subject is to be subject to area calculation and display in the superimposed image. The imaging system executes the same process as in the above embodiment for one or more subjects specified as subjects to be subject to area calculation and display in the superimposed image, and displays the highlighted area in the superimposed image.

[0114] As described above, one or more processors included in the imaging system in the above-mentioned embodiments execute one or more programs including a set of instructions for causing a computer to execute the algorithms described with reference to the drawings. This process realizes the processes described in each embodiment.

[0115] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. The program may also take the form of, for example, an application program.

[0116] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0117] S1, S2, S3 Imaging Systems 11 imaging unit 12 control unit 13 Camera DB 14 Lens DB 15 Display unit 16 Shooting information acquisition unit 121 Memory 122 I / O section 123 Information processing unit 124 Camera identification unit 125 Lens identification unit 126 View angle acquisition unit 127 Camera position / attitude acquisition unit 128 Image generation unit 31 Imaging section 32 Control section 33 Person attribute DB 34 Display section 321 Memory 322 I / O section 323 Information processing unit 324 Viewing angle acquisition unit 325 Person position / posture acquisition unit 326 Image generation unit

Claims

1. An imaging unit that captures an image of a subject; a viewing angle acquisition unit that acquires viewing angle information of the subject using the image captured by the imaging unit; a subject information acquisition unit that acquires position information and orientation information of the subject and position information of objects present within a viewing area of ​​the subject using the image; and a region calculation unit that calculates at least one of a blind spot region where the subject cannot be recognized or a visual recognition region that is a region excluding the blind spot region from the visual field region, based on visual field angle information, position information, and attitude information of the subject and position information of an object present in the visual field region. Shooting system.

2. The photographing unit photographs a plurality of the subjects, the viewing angle acquisition unit acquires viewing angle information for each of the plurality of objects using images captured by the imaging unit; the subject information acquisition unit acquires position information and orientation information for each of the plurality of subjects, and position information of an object present within a field of view for each of the subjects, using the image; The area calculation unit calculates at least one of the visual recognition area for each subject, an area where the visual recognition areas of the subjects overlap, an area where only one of the subjects is visible, or the blind spot area from all of the subjects, based on the visual angle information, position information, and attitude information for each of the plurality of subjects and the position information of an object present in the visual recognition area for each of the subjects. The imaging system according to claim 1 .

3. A photographing step of photographing a subject; a viewing angle acquisition step of acquiring viewing angle information of the subject using an image of the subject; a subject information acquisition step of acquiring position information and orientation information of the subject and position information of an object present within a viewing area of ​​the subject using the image; a region calculation step of calculating at least one of a blind spot region where the subject cannot be grasped or a visual recognition region which is a region excluding the blind spot region from the visual field region, based on visual field angle information, position information, and attitude information of the subject and position information of an object present within the visual field region; The shooting system performs the shooting method.

4. A photographing step of photographing a subject; a viewing angle acquisition step of acquiring viewing angle information of the subject using an image of the subject; a subject information acquisition step of acquiring position information and orientation information of the subject and position information of an object present within a viewing area of ​​the subject using the image; a region calculation step of calculating at least one of a blind spot region where the subject cannot be grasped or a visual recognition region which is a region excluding the blind spot region from the visual field region, based on visual field angle information, position information, and attitude information of the subject and position information of an object present within the visual field region; A program that causes a computer to execute the following.

Images