Panoramic image generation device, panoramic image generation system, panoramic image generation method, and panoramic image generation program

Abstract

To provide a technique for changing the field angle of a panoramic image, without moving a plurality of imaging devices.SOLUTION: First acquisition means acquires an image imaged by a plurality of imaging devices having a PTZ function and second acquisition means acquires the drive information of the pan, tilt, and zoom of the imaging device. Panoramic image generation means generates the panoramic image on the basis of the image acquired by the first acquisition means and third acquisition means acquires a field angle change instruction for changing the field angle of the panoramic image generated by the panoramic image generation means. Drive amount decision means decides a PTZ drive amount for at least one imaging device on the basis of a PTZ drive position acquired by the second acquisition means and the field angle change instruction acquired by the third acquisition means and transmission means transmits an instruction for changing the drive position of the pan, tilt, and zoom of the imaging device on the basis of the PTZ drive amount decided by the drive amount decision means to the imaging device.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] A panoramic image generation system that generates a panoramic image using multiple imaging devices. [Background technology]

[0002] There is a technology that generates a panoramic image by synthesizing images captured by multiple imaging devices in order to capture a wide-angle range. When generating this panoramic image, the images captured by the different imaging devices are stitched together by using the overlapping parts of the images.

[0003] For example, Patent Document 1 proposes a panoramic image generation system that has multiple imaging units within a single imaging device and a control mechanism that can twist drive the imaging units, and that can change the angle of view of the panoramic image by moving the imaging units in response to a user's instruction to change the angle of view.

[0004] On the other hand, in an integrated type having multiple image capturing units in a single image capturing device, there is a problem that the image capturing range is limited when trying to create a panorama. Therefore, it is considered to generate a panoramic image using images captured by a separate image capturing device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-189214 Summary of the Invention [Problem to be solved by the invention]

[0006] In a technology for generating a panoramic image from a plurality of imaging devices each having a PTZ function, if the imaging devices are separate, the user must move the imaging device when changing the angle of view of the panoramic image.

[0007] An object of the present invention is to provide a panoramic image generating system that allows a user to change the angle of view of a panoramic image without moving multiple imaging devices. [Means for solving the problem]

[0008] In order to achieve the above object, the panoramic image generating device of the present invention has the following configuration: a first acquisition means for acquiring a plurality of captured images from a plurality of imaging devices each having a PTZ function, a second acquisition means for acquiring at least one of driving information of pan, tilt, and zoom of the imaging devices, a panoramic image generating means for generating a panoramic image based on the images acquired by the first acquisition means, a third acquisition means for acquiring an angle-of-view change instruction for changing the angle of view of the panoramic image generated by the panoramic image generating means, a drive amount determination means for determining a PTZ drive amount for at least one of the imaging devices based on the PTZ drive information acquired by the second acquisition means and the angle-of-view change instruction acquired by the third acquisition means, and a transmission means for transmitting an instruction to change at least one of the drive positions of pan, tilt, and zoom of the imaging devices based on the PTZ drive amount determined by the drive amount determination means, and when the third acquisition means acquires an angle-of-view change instruction, the transmission means transmits an instruction to change the PTZ drive position to the imaging device based on the PTZ drive amount determined by the drive amount determination means. Effect of the Invention

[0009] According to the present invention, a user can change the angle of view of a panoramic image without moving a plurality of imaging devices. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a hardware configuration diagram of a panoramic image generation system 100 according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram of an imaging device 110 according to a first embodiment. [Diagram 3]1A is a mechanical diagram of the image pickup device 110 according to the first embodiment, as viewed from above the vertical axis, and FIG. 1B is a mechanical diagram of the image pickup device 110 according to the first embodiment, as viewed from the side. [Figure 4] FIG. 2 is a functional block diagram of a panoramic image generating device 120 according to the first embodiment. [Diagram 5] FIG. 2 is a block diagram of an external monitor 130 according to the first embodiment. [Figure 6] A panoramic image 601 and a user interface screen in the first embodiment. [Figure 7] 6 is a flowchart showing PTZ control for a panoramic image 820 in the first embodiment. [Figure 8] 2 is a diagram showing a process of generating a panoramic image by the panoramic image generating device according to the first embodiment. [Figure 9] 8 is a diagram showing the processing of the system when a zoom control is instructed for a panoramic image 820 in the first embodiment. FIG. [Figure 10] 10 is a flowchart showing PTZ control on a panoramic image 820 in the second embodiment. [Figure 11] 10 is a flowchart showing a process for correcting distortion of a captured image 810 in the second embodiment. [Figure 12] 13 is a flowchart showing PTZ control on a panoramic image 820 in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment does not limit the present invention according to the claims, and not all of the combinations of features described in the embodiment are necessarily essential to the solution of the present invention. Note that the same components are described with the same reference numerals.

[0012] <First embodiment> <Overall configuration of imaging system> An example of the overall configuration of a panoramic image generation system 100 will be described with reference to Fig. 1. Fig. 1 shows the hardware configuration of a panoramic image generation system 150 according to the present invention. Note that the hardware configuration is merely an example and is not limited to this.

[0013] The panoramic image generation system 100 is composed of a plurality of imaging devices 110-A and 110-B, a panoramic image generation device 120, and an external monitor 130. The devices constituting the panoramic image generation system 100 are connected to each other via a network 140 so as to be able to communicate with each other.

[0014] The imaging device 110 is an imaging device equipped with an imaging section capable of changing the imaging range by performing pan, tilt, and zoom (hereinafter, PTZ) driving operations.

[0015] The panoramic image generating device 120 is an information processing device such as a personal computer, a server device, a tablet device, or a mobile terminal that communicates with the multiple imaging devices 110. The panoramic image generating device 120 transmits various control commands to the imaging devices 110, for example. The control commands are signals that instruct the execution of designated processes, and are, for example, commands to set shooting parameters for the imaging unit, drive control commands to a driving unit such as a motor, and commands to start / stop video streaming. The shooting parameters are parameters related to image capture. Furthermore, the panoramic image generating device 120 generates a panoramic image using multiple captured images acquired from the multiple imaging devices 110.

[0016] The external monitor 130 includes a display unit (for example, display means such as an LCD or an organic EL panel) and an input unit that accepts user operations.

[0017] The network 140 is a network such as the Internet or an IP network. The network 140 is composed of a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). The network 140 may also be composed of a wired local area network (LAN), a wireless LAN, a wide area network (WAN), etc. In this embodiment, the panoramic image generating system 100 that is capable of mutual communication via the network 140 is described, but the network 140 may also be connected by wire.

[0018] In this embodiment, the imaging device 110, the panoramic image generation device 120, and the external monitor 130 are described as being separate entities, but this is not limited thereto. For example, the panoramic image generation device 120 and the external monitor 130 may be integrated, or the imaging device 110-A and the panoramic image generation device 120 may be integrated. Also, in this embodiment, the number of imaging devices 110 is described as two, but there may be three or more.

[0019] <Configuration of the imaging device 110> Fig. 2 shows the functional configuration of the imaging device 110 constituting the panoramic image generating system 100. Furthermore, Fig. 3(a) is a mechanical diagram of the imaging device 110 as viewed directly above the vertical axis, and Fig. 3(b) is a mechanical diagram of the imaging device 110 as viewed from the side. The basic configuration of the imaging device 110 and the PTZ drive control of the drive unit 113, which will be described later, will be explained using Figs. 2 and 3. Note that the functional configuration is merely an example and is not limited to this.

[0020] The imaging device 110 is an imaging device that can change the imaging direction and angle of view by receiving a PTZ drive command from the panoramic image generation device 120 described later and performing PTZ. The imaging device 110 outputs captured images to the panoramic image generation device 120.

[0021] The imaging device 110 in this embodiment has an imaging section 201, an image processing section 202, a driving section 203, a control section 204, a recording section 205, a communication section 206, and an internal bus 207 that enables the above components to communicate with each other.

[0022] The imaging unit 201 is composed of a lens barrel equipped with imaging lenses including a focus lens, a zoom lens, and the like, an image sensor, and a mechanical drive system and circuits that drive them, and focuses light toward an imaging element (not shown). The imaging element is composed of a two-dimensional image sensor such as a CCD or CMOS, and photoelectrically converts the light focused on the light receiving surface and outputs the obtained electrical signal to an image processing unit 202.

[0023] The image processing unit 202 generates image data by performing image processing such as noise removal or gamma correction on the electrical signals converted by the imaging unit 201, and outputs the generated image data to the control unit 204. The image processing unit 202 according to this embodiment can also process control commands received from the control unit 204. For example, when the image processing unit 202 receives a control command from the control unit 204 instructing adjustment of image quality, it adjusts the image quality.

[0024] The driving unit 203 performs an operation to change the pan direction, tilt direction, and zoom magnification of the imaging device 110. The driving unit 203 is composed of a mechanical driving system (not shown), a driving motor, and a sensor for identifying a PTZ position (posture). The PTZ position is the PT position of the imaging device 110 and the position of the zoom lens. The driving unit 203 drives a driving system that moves the optical axis of the imaging device 110 in the pan direction and tilt direction, and drives a driving system that moves the zoom lens of the imaging unit 201 in the optical axis direction, based on a control command obtained from the control unit 204. Furthermore, the driving unit 203 identifies the PTZ position from a sensor for identifying the PTZ position (posture), and outputs it to the control unit 204. In this embodiment, a photointerrupter (hereinafter, PI) is used as a sensor for identifying the PTZ position, but the present invention is not limited to this.

[0025] The PTZ position is specified by the following method. First, the PT position is calculated from the PT drive amount and PI, and the Z position is specified from the drive amount of the zoom lens. Next, the specified PTZ position is output to the storage unit 205, which will be described later. The method of specifying the PT position will be described in detail later with reference to FIG. 3. In this embodiment, the drive unit 203 specifies the PTZ position, but the PTZ position may be a PTZ value.

[0026] In addition, in this embodiment, the imaging device 110 has been described as a device having a PT function, but it may be PT-driven using an external device such as an external camera platform. In this case, the driving unit 203 outputs a PT drive command to the external device, and the output command is stored in the storage unit 205.

[0027] The control unit 204 is a central processing unit that controls the entire imaging device 110. The control unit 204 acquires and analyzes various control commands via the communication unit 206. The control unit 204 further controls the imaging device 110 based on the analyzed various control commands. For example, when the control unit 204 acquires a control command related to PTZ drive control (PTZ drive command), the control unit 204 controls the drive unit 203. The control unit 204 also transmits information related to the PTZ position acquired from the drive unit 203 (PTZ information) to the panoramic image generating device 120 via the communication unit 206. The control unit 204 also stores the acquired various data in the storage unit 205.

[0028] The storage unit 205 has a function as a work area for the control unit 204 and a temporary storage area for data, and a function for storing various programs, various setting information, information on various parameters, images captured by the imaging unit 201, imaging parameters of each imaging unit, etc. The storage unit 205 acquires data such as PTZ position information at the time of capturing a predetermined captured image from the drive unit 203, a control command from the control unit 204, and a captured image from the image processing unit 202, and stores it together with system information within the imaging device.

[0029] The communication unit 206 acquires various control commands from the panoramic image generation device 120 via the network 140, and outputs them to the control unit 204. Furthermore, the communication unit 206 acquires PTZ position information and captured images from the storage unit 205, and outputs them to the panoramic image generation device 120. In this embodiment, the communication unit 206 acquires various control commands from the panoramic image generation device 120, but the communication unit 206 may also acquire control commands from the external monitor 130, which will be described later. For example, commands related to image quality and white balance may be acquired from the external monitor 130 via the network 140.

[0030] Here, the pan-tilt control by the drive unit 203 that drives the mechanical drive system in the imaging unit 201 will be explained with reference to Fig. 3. Fig. 3(a) is a diagram of the mechanical drive system as viewed from directly above the vertical axis, and Fig. 3(b) is a diagram of the imaging unit 201 as viewed from the side.

[0031] The mechanical drive system includes a bottom case 301, a turntable 302, a camera head support 303, and a camera head 304, and is composed of three pan heads 301 to 303. Sensors 305 to 308 for identifying a PTZ drive position (PTZ drive information) and an angular velocity sensor 309 for measuring angular velocity are arranged in the drive unit 113. Note that since the sensors 305 to 308 and the angular velocity sensor 309 are arranged inside the imaging device 110, they are basically not visible from the outside, but for the sake of convenience in the example of FIG. 3, they are illustrated as being visible.

[0032] In this embodiment, the clockwise direction as viewed from the top in FIG. 3(a) is the positive direction of the pan angle, and the counterclockwise direction is the negative direction of the pan angle. Also, the clockwise direction as viewed from the side in FIG. 3(b) is the positive direction of the tilt angle, and the counterclockwise direction is the negative direction of the tilt angle. The driving unit 203 drives in the pan direction (pan driving) by rotating the turntable 302 about the vertical axis as the central axis. Also, the driving unit 203 drives in the tilt direction (tilt driving) by rotating the camera head 304 about an axis perpendicular to the vertical axis as the central axis. In this embodiment, the camera can be driven from the initial position from -175 degrees to 175 degrees in the pan direction and from -25 degrees (diagonally downward) to 95 degrees (directly upward) in the tilt direction, but this driving range is not particularly limited to this. For example, the camera may be driven 360 degrees (endlessly) in the pan direction, or from -90 degrees to 90 degrees in the tilt direction. In this embodiment, the initial attitude in the tilt direction is an angle in the tilt direction (tilt angle) of 0 degrees, and the initial attitude in the pan direction is an angle in the pan direction (pan angle) of 0 degrees.

[0033] Here, the sensors 305 and 306 are disposed on a rotating portion on the turntable 302, and detect driving in the pan direction of the image capture device 110. In this example, the sensor 305 is disposed at a position rotated in the pan direction by more than 170 degrees (for example, 171 degrees) from the optical axis direction of the initial attitude, and the sensor 306 is disposed at a position rotated in the pan direction by more than -170 degrees (for example, -171 degrees) from the optical axis direction.

[0034] In this embodiment, the sensors 305 to 308 are configured with PI as described above. Here, the process of detecting the driving position in the pan direction of the imaging device 110 will be specifically described by taking the sensors 305 and 306 as an example. In the example of FIG. 3(a), a light-shielding wall (not shown) is installed on the fixed part of the bottom case 301. When the sensors 305 and 306 are not shielded by the light-shielding wall, the light receiving element receives light from the light emitting element, and the AD value takes a value close to a specified value (for example, 1023). Here, when the sensors 305 and 306 rotate to the position of the light-shielding wall by pan driving, the light receiving element is shielded, and the AD value of the PI becomes a value close to 0. In this embodiment, a threshold value is set to determine whether or not the light is shielded by the light-shielding wall, and if the AD value is equal to or greater than the threshold value (1023 or less), it is determined that the light is not shielded, and if not, it is determined that the light is shielded. This threshold value is not particularly limited, but here, the threshold value is 512. By detecting this light blocking, it is detected that the image capture device 110 has reached a specific drive position, and the drive position in the pan direction can be identified. Note that, in this example, the PI is configured such that the light emitting element and the light receiving element are disposed on the rotating part, and the light blocking wall is disposed on the fixed part, but the light blocking wall may be disposed on the rotating part, and each element may be disposed on the fixed part.

[0035] Further, sensors 307 and 308 are disposed on a fixed portion on camera head support 303, and detect driving of image capture device 110 in the tilt direction. In this example, sensor 307 is disposed to detect rotation of more than 90 degrees (e.g., 91 degrees) in the tilt direction from the optical axis direction of the initial posture, and sensor 308 is disposed to detect rotation of more than -20 degrees (e.g., -21 degrees) in the tilt direction from the optical axis direction of the initial posture. In this example, light-shielding walls corresponding to sensors 307 and 308 (light-emitting element and light-receiving element disposed on the fixed portion) are disposed on the rotating portion of camera head 304, but the sensors may be disposed on the rotating portion.

[0036] The angular velocity sensor 309 is disposed on the camera head 304 and detects the angular velocity in the pan direction and tilt direction of the camera head 304. The X, Y, and Z axes shown in Figures 3(a) and 3(b) are examples of output axes of the angular velocity sensor 309. For example, in the state where the pan angle is 0 degrees (initial attitude) in Figure 3(a), the angular velocity in the tilt direction can be detected by the X axis of the angular velocity sensor 309, and in the state where the tilt angle is 0 degrees (initial attitude) in Figure 3(b), the angular velocity in the pan direction can be detected by the Y axis of the angular velocity sensor 309.

[0037] In this way, the imaging device 110 of this embodiment can change the imaging direction by rotating the camera head 304 horizontally and vertically, and can change the imaging range (PTZ) by moving the zoom lens in the optical axis direction (PTZ function). Furthermore, the imaging device 110 can specify the PTZ position by being equipped with a sensor, for example, a PI.

[0038] <Configuration of the panoramic image generating device 120> 4 is a diagram showing the functional configuration within the panoramic image generating device 120. Note that this functional configuration is merely an example and is not limited to this.

[0039] The panoramic image generating device 120 is composed of an acquisition unit 401, a panoramic image generating unit 402, a control unit 403, a storage unit 404, a drive command generating unit 405, a communication unit 406, and an internal bus 407 that enables the above components to communicate with each other. The panoramic image generating device 120 generates a panoramic image from captured images acquired from a plurality of image capturing devices 110-A and 110-B, and outputs the generated image to the external monitor 130. The panoramic image generating device 120 also analyzes various control commands acquired from the external monitor 130. When the panoramic image generating device 120 acquires a control command (panoramic image control command) for changing the angle of view of the panoramic image from the external monitor 130, it outputs a drive command for instructing PTZ drive to at least one of the image capturing devices 110-A and 110-B.

[0040] The acquisition unit 401 acquires PTZ position information and captured images from the imaging device 110 via the communication unit 406. The acquisition unit 401 outputs the acquired PTZ position information to the storage unit 404. The acquisition unit 401 also outputs the acquired captured images to the panoramic image generation unit 402. Furthermore, the acquisition unit 401 acquires various control commands via the communication unit 406 and outputs them to the control unit 403.

[0041] The panoramic image generating unit 402 acquires captured images from the imaging devices 110-A and 110-B via the acquiring unit 401, and generates a panoramic image based on the captured images. Furthermore, the generated panoramic image is output to the control unit 403. In this embodiment, a panoramic image is generated by performing synthesis processing and correction processing on image information of overlapping parts of multiple images as a common feature point, but the method is not limited to this. For example, the panoramic image generating unit 402 may perform correction processing using AI or the like so that the feature points match.

[0042] The control unit 403 controls the entire panoramic image generating device 120. The panoramic image generated by the panoramic image generating unit 402 is output to the external monitor 130 via the communication unit 406. The control unit 403 also analyzes various control commands acquired from the acquisition unit 401, and controls the panoramic image generating device 120. For example, when the control unit 403 acquires a command related to driving via the communication unit 406, the control unit 403 outputs the command to the drive command generating unit 405.

[0043] The storage unit 404 functions as a work area for the control unit 404 and a temporary storage area for data, and stores various programs, various setting information, acquired images and user information, PTZ position information of each imaging device 110, information on the imaging device 110, etc. In this embodiment, the PTZ position information in an image acquired from the imaging device 110-A is stored as PTZ position information A, and the PTZ position information in an image acquired from the imaging device 110-B is stored as PTZ position information B, and when PTZ position information is acquired, it is stored in a manner of overwriting.

[0044] The drive command generating unit 405 (drive amount determining means) obtains a control command (panorama image control command) for changing the angle of view of the panorama image from the control unit 403, and obtains PTZ position information (PTZ drive information) from the storage unit 404. Based on the obtained panorama image control command and PTZ position information, a drive command is generated for instructing each of the imaging devices 110-A and 110-B to perform PTZ drive (determining the PTZ drive amount). In this embodiment, a case will be described in which the drive command generating unit 405 generates drive commands for the imaging devices 110-A and 110-B and outputs them to each imaging device 110 via the communication unit 406, but the number of drive commands to be output is not limited. It is sufficient to generate a drive command for at least one imaging device 110 of the multiple imaging devices 110, and output the generated drive command. For example, when a drive command for the image capture device 110-A is generated, the drive command may be output to the image capture device 110-A via the communication unit 406, and there is no need to output a drive command to the image capture device 110-B.

[0045] The communication unit 406 (transmission means) acquires various control commands from the external monitor 130 via the network 140, and outputs them to the control unit 403. Furthermore, the communication unit 206 acquires PTZ position information and captured images from the imaging device 110, and outputs them to the acquisition unit 401.

[0046] <External monitor 130 configuration> FIG. 5 shows the functional configuration of the external monitor 130 that constitutes the panoramic image generating system 100. As shown in FIG.

[0047] The external monitor 130 is composed of a display unit 501 such as a liquid crystal display, an input unit 502, an acquisition unit 503, a communication unit 504, and an internal bus 505 that enables the above components to communicate with each other. In this embodiment, the external monitor 130 and the panoramic image generation device 120 are described as separate entities, but the external monitor 130 and the panoramic image generation device 120 may be integrated. In that case, the components of the external monitor 130 and the panoramic image generation device 120 communicate with each other via the internal bus 505 (internal bus 407) that enables them to communicate with each other.

[0048] The display unit 501 displays a panoramic image acquired from the panoramic image generating device 120 and a setting screen for a user to change the angle of view of the panoramic image. The display unit 501 is, for example, a liquid crystal panel or an organic EL panel.

[0049] The input unit 502 is an interface for transmitting instructions received from a user to the acquisition unit 503, and examples of such interfaces include a controller, a dial, a joystick, and a touch panel. Information indicating an operation received by the input unit 502 is output to the acquisition unit 502 via an internal bus 505. In this embodiment, the user can input instructions regarding control of the angle of view of the panoramic image by operating the input unit 502 while checking the panoramic image displayed on the display unit 501. Furthermore, the input unit 502 converts the input user's operation into a signal and outputs it to the acquisition unit 503 as a control command.

[0050] The control unit 503 controls the entire external monitor 130. The control unit 503 acquires various control commands from the input unit 502 and controls the external monitor 130. The control unit 503 also acquires a panoramic image from the panoramic image generation device 120 and outputs a panoramic image control command inputted at the input unit to the panoramic image generation device 120.

[0051] The communication unit 504 is an interface capable of communicating with the panoramic image generating device 120 and the image capturing device 110 via the network 140, and communicates various control commands and data.

[0052] An example of the UI of the external monitor 130 will be described with reference to Fig. 6. Fig. 6 is a diagram showing a panoramic image 601 generated by the panoramic image generation device 120, and a UI screen that allows the user to PTZ control the panoramic image 601. The panoramic image 601 can be PTZ-operated using slider bars 602 to 604, and information on the slider bars operated by the user is converted into a control command by the input unit 502 and output to the panoramic image generation device 120.

[0053] A slider bar 602 is a zoom adjustment slider bar for controlling the zoom in the panoramic image 601, and when moved downward, zooms out, and when moved upward, zooms in.

[0054] The slider bar 603 is a tilt adjustment slider bar for controlling movement in the tilt direction in the panoramic image 601. When moved down, control is performed in the negative tilt direction, and when moved up, control is performed in the positive tilt direction.

[0055] The slider bar 604 is a pan adjustment slider bar for controlling movement in the pan direction in the panoramic image 601. Moving it to the left controls the pan in a negative direction, and moving it to the right controls the pan in a positive direction.

[0056] In this embodiment, the user inputs a PTZ control command using a slider bar for the panoramic image 601, but this is not limiting. A command to change the angle of view of the panoramic image does not need to be input separately for PTZ. A command to change the angle of view of the panoramic image may be input simultaneously with a PTZ control command by tilting a joystick, for example, or may be input by touching the center of the panoramic image displayed on a touch panel or the like.

[0057] 7 is a flowchart of an operation in which the panoramic image generating device 120 in this embodiment generates a panoramic image from videos acquired from multiple imaging devices 110, and PTZ controls the generated panoramic image through user operations. This control flow starts when the panoramic image generating device 120 receives a control command to generate a panoramic image via the network 140. Note that, in this embodiment, for simplicity of explanation, it is assumed that the PT angle of the drive unit 112 of each imaging device 110 is 0 degrees (initial attitude), but this is not limited to this. There is no problem if the images are captured at angles with different PT values.

[0058] In step S701, the control unit 403 determines whether a control command for generating a panoramic image has been received. If a control command for generating a panoramic image has been received, the process proceeds to step S702. If not, the process proceeds to step S715.

[0059] In step S702, the panoramic image generating unit acquires captured images from the imaging device 110-A and the imaging device 110-B and determines whether a panoramic image can be generated. If the overlapping portion of the captured images is equal to or less than a preset threshold, it is determined that a panoramic image cannot be generated, and the process proceeds to step S703. If the overlapping portion of the captured images is equal to or more than the preset threshold, the acquired captured images and the PTZ drive position are stored in the storage unit 404, and the process proceeds to step S704.

[0060] In step S703, the drive command generation unit 405 creates a PTZ drive command from the images acquired in S702 so that the overlapping portion of the captured images becomes equal to or greater than a set threshold value. The drive command created by the drive command generation unit 405 is transmitted to at least one of the image capture device 110-A and the image capture device 110-B via the communication unit 406, and the process proceeds to step S702.

[0061] In step S704, a panoramic image is generated based on the image acquired in step S702. Furthermore, PTZ position information of each imaging device in the image acquired in step S702 is acquired, and the generated panoramic image and the PTZ position information of each imaging device are stored in the storage unit 404. The control unit 403 acquires the panoramic image generated in step S704 from the storage unit 404, and outputs it to an external monitor. The processing of steps S702 and S704 will be described in detail with reference to FIG. 8.

[0062] In FIG. 8, the panoramic image generating device 120 obtains an image 810-A from the imaging device 110-A and an image 810-B from the imaging device 110-B. The panoramic image generating device 120 detects overlapping parts of the obtained images 810-A and 810-B using an overlapping range detection unit (not shown). When detecting overlapping parts of the images 810-A and 810-B, the overlapping range detection unit (not shown) performs a synthesis process (matching process) using image information of the overlapping parts of the images as feature points. At this time, the overlapping parts of the images are regarded as the overlapping range, but if feature points cannot be extracted from the obtained images, a correction process is performed on the obtained images. For example, if the obtained images have image distortion due to the lens, the overlapping range detection unit performs a distortion correction process. Furthermore, if the obtained images have different image sizes due to differences in zoom position or angle of view, the overlapping range detection unit performs a correction process so that the feature points match, and generates a panoramic image based on the corrected images. The position where the acquired or corrected images 810-A and 810-B overlap at the top and overlap without shifting in the horizontal direction is fixed, and the overlapping range detection unit sets the overlapping range as the overlapping range. Furthermore, it is determined whether the ratio of this overlapping range (area of ​​the overlapping part) to the area of ​​the panoramic image consisting of the images 810-A and 810-B is equal to or greater than a threshold value. If the ratio of the overlapping range to the area of ​​the panoramic image is equal to or greater than the threshold value, the panoramic image generating device 120 generates a panoramic image 820 from the acquired images 810-A and 810-B. At this time, the storage unit 404 stores the images 810-A and 810-B captured by the respective imaging devices 110, the overlapping range 830 set as the overlapping range, and the driving positions A and B (PTZ driving positions) in the images 810. The created panoramic image 601 is output to the external monitor 130, and the user can check the panoramic image 601. In this case, the center points of the images 810-A and 810-B are 840-A and 840-B, respectively.

[0063] In step S705, it is determined whether the acquisition unit 401 has acquired a control command (panoramic image control command) for changing the angle of view of the panoramic image, and if a panoramic image control command has been acquired, the process proceeds to step S706. If a panoramic image control command has not been acquired, the process proceeds to step S701. In this embodiment, the panoramic image control commands are a pan drive command, a zoom drive command, and a tilt drive command, and if at least one of them has been acquired, it is determined that a panoramic image control command has been acquired.

[0064] In step S706, it is determined whether or not a zoom drive command has been acquired for the panoramic image generated in step S704. Here, a zoom drive command is generated for a panoramic image when the slider bar 603 is slid up or down from the current position by the user. If a zoom drive command has been acquired, the process proceeds to step S707, and if a zoom drive command has not been acquired, the process proceeds to step S708.

[0065] In step S707, the control unit 403 outputs information about the zoom drive command to the drive command generating unit 405.

[0066] In step S708, it is determined whether a pan drive command has been acquired for the panoramic image generated in step S704. Here, a pan drive command is generated for a panoramic image when the slider bar 604 is slid to the right or left from the current position by the user. If a pan drive command has been acquired, the process proceeds to step S709, and if a pan drive command has not been acquired, the process proceeds to step S710.

[0067] In step S709, the control unit 403 outputs information about the pan drive command to the drive command generating unit 405.

[0068] In step S710, it is detected whether or not the user has acquired a control command for the tilt direction. If control in the tilt direction is not being performed, the process proceeds to step S702. If control in the tilt direction is being performed, the process proceeds to step S711.

[0069] In step S711, the control unit 403 outputs tilt drive command information to the drive command generating unit 405.

[0070] In step S712, the drive command generation unit 405 generates drive commands for the drive units 203 of the image capturing devices 110-A and 110-B based on the drive command for the acquired panoramic image and the PTZ position information acquired in step S704. Details of this process will be described with reference to FIG.

[0071] Fig. 9 is a diagram showing a case where panoramic image 920 is generated by controlling each imaging device so that the angle of view of panoramic image 820 is changed to the angle of view of panoramic image 920. In Fig. 9, the actual image sizes of panoramic image 820 and panoramic image 920 are the same, and for ease of explanation, they are represented as in Fig. 9. Panoramic image 920 is generated based on images 910-A and 910-B captured by imaging devices 110-A and 110-B, and the overlapping range of images 910-A and 910-B is overlapping range 930.

[0072] When performing PTZ control processing on the panoramic image 820, the ratio of the overlapping range 830 to the panoramic image 820 and the ratio of the overlapping range 930 to the panoramic image 920 are controlled to be equal. In other words, the aspect ratios of the panoramic image 820 and the panoramic image 920 are controlled to be the same. In this embodiment, the ratio of the overlapping range 830 to the panoramic image 820 and the ratio of the overlapping range 930 to the panoramic image 920 are controlled to be equal, but this is not limited thereto. For example, a drive command may be generated for the drive unit 203 of the image capturing device 110-A and the image capturing device 110-B so that the area of ​​the overlapping range 930 becomes equal to or larger than a predetermined threshold value set in advance.

[0073] Zoom processing is performed based on the user's control command for the panoramic image, and the overlapping range is detected by an overlapping range detection unit (not shown). Furthermore, drive commands to be output to the imaging device 110-A and the imaging device 110-B are calculated based on the overlapping range at this time. Specifically, the angle of view in the pan direction is 60 degrees, and the ratio of the overlapping range 830 is 20% of the image 810-A. In this case, the panoramic image 820 captures an image with a 100-degree range. At this time, if the coordinate positions of 840-B and 940-B are shifted in the positive pan direction by 5% of the imaging range, this corresponds to the imaging device 110-B being shifted 5 degrees in the positive pan direction. In this way, when a zoom drive command (angle of view change instruction) for the panoramic image is acquired, the zoom drive amount and pan drive amount for the imaging device 110-A and the imaging device 110-B are calculated.

[0074] Furthermore, when the user performs control in the pan direction, the ratio of overlapping range 830 for panoramic image 820 to overlapping range 930 for panoramic image 920 after zoom control becomes different. In this case, the zoom position of imaging device 110, which is located opposite to the moving pan direction, is operated so that the ratio of overlapping range 830 to the panoramic image matches the ratio of overlapping range 930 to the panoramic image, and the zoom drive amount is calculated. In this way, when a pan drive command (instruction to change the angle of view) for the panoramic image is acquired, the pan drive amount and zoom drive amount for imaging device 110-A and imaging device 110-B are calculated.

[0075] In this embodiment, when the feature points cannot be matched due to differences in the zoom positions of the image capturing device 110-A and the image capturing device 110-B, the image captured by the overlapping range detection unit (not shown) is corrected so that the feature points are matched. The drive amount is calculated by the above-mentioned method, but the method is not limited to this. For example, the zoom drive amount for the pan drive and the pan drive amount for the zoom drive may be defined in advance as a function and applied sequentially. The functions may be changeable.

[0076] In step S713, the drive command created in step S712 is transmitted to at least one of image capture device 110-A or image capture device 110-B.

[0077] In step S714, it is determined whether the generation of the panoramic image is complete. If the generation of the panoramic image is complete, this control flow ends. If the generation of the panoramic image is not complete, the process returns to step S701.

[0078] As described above, in this embodiment, a panoramic image is generated using images acquired from multiple imaging devices 110. Then, different PTZ controls are performed on each imaging device according to a user's PTZ control command for the panoramic image. This allows the user to change the angle of view of the panoramic image without moving the multiple imaging devices.

[0079] In this embodiment, the PTZ control is performed so as not to destroy the aspect ratio of the panoramic image 820, but the control method is not limited to this. For example, the control may be performed so that the imaging devices continue to capture images of the content that is a characteristic point within the overlapping range.

[0080] In addition, in this embodiment, the control from the user is explained with reference to Fig. 5 as controlling a slider bar on a UI screen, but this is not limited to this. For example, control may be possible from an external device such as an infrared remote control or controller capable of PTZ control. The controller may also be provided in the panoramic image generating device 120.

[0081] The panoramic image generation system 100 has the above configuration, and based on the PTZ drive command for the panoramic image input by the user, the panoramic image generation device 120 can transmit and control the PTZ drive command to the imaging device 110. Furthermore, it becomes possible to generate a panoramic image with a direction and angle of view desired by the user. In other words, it becomes possible for the user to change the angle of view of the panoramic image without moving the imaging device.

[0082] <Second embodiment> In the first embodiment, a method was described in which a PTZ drive command corresponding to each of the imaging devices 110-A and 110-B is transmitted in response to a user's instruction to change the angle of view of the panoramic image, and the angle of view of the panoramic image is changed without moving the imaging device 110. However, in this method, when PTZ control is performed on each imaging device, distortion may occur on the left and right of the panoramic image depending on the distance between the imaging devices 110. Therefore, a second embodiment in which distortion is corrected when distortion occurs on the left and right of the panoramic image will be described with reference to the flowcharts of Figs. 10 and 11 and Fig. 12. Note that the same components as those in the first embodiment will not be described in detail by using the same reference numerals as those in the first embodiment.

[0083] FIG. 10 is a flowchart showing a process of generating a panoramic image in accordance with the first embodiment, in which distortion correction is added.

[0084] The panoramic image generating device 120 used in the second embodiment further includes a correction unit (not shown) that detects and corrects image misalignment in the overlapping range.

[0085] In step S1001, a correction unit (not shown) corrects image shifts in the overlapping range that occur due to differences in zoom magnification and pan position between the image pickup device 110-A and the image pickup device 110-B.

[0086] FIG. 11 is a flowchart illustrating a method for correcting image distortion in step S1001.

[0087] FIG. 12 is an example of a panoramic image before distortion correction in which distortion has occurred on the left and right sides of the panoramic image due to the distance between the image capture devices 110.

[0088] Step S1101 is a process for identifying the imaging position of each imaging device in a panoramic image. Based on the feature points when the images are combined as shown in FIG. 7, the storage unit 404 stores which part of the panoramic image each imaging device has captured. Note that this process may be ignored if the information has been stored once. Also, this process may be ignored if the imaging position has been specified by a user operation or in advance.

[0089] In step S1102, the intermediate PTZ drive position of the image capturing device is calculated. For example, if the PTZ positions (P, T, Z) of the image capturing device 110-A and the image capturing device 110-B are (20, 0, 10) and (40, 0, 20), the intermediate PTZ position is (30, 0, 15).

[0090] In step S1103, the intermediate PTZ drive position between the imaging devices calculated in step S1102 for each imaging device is compared with the PTZ drive positions of imaging device 110-A and imaging device 110-B, and the difference between the intermediate PTZ drive position and the PTZ drive position is calculated for each imaging device.

[0091] Step S1104 is a process of correcting the image according to the difference in the PTZ drive position calculated in step S1103. For example, when comparing the intermediate PTZ drive position with the PTZ drive position, if the image appears to have been captured at an angle with respect to the positive pan direction, the image distortion is corrected by gradually increasing the pixels of the image with the intermediate PTZ drive position as the convergence point. Note that the correction method (correction content) is not limited to a method of calculation from the tilt angle with the intermediate PTZ drive position as the convergence point. The calculation method of the tilt angle or the correction method are not specified.

[0092] As described above, in this embodiment, the intermediate PTZ drive position is calculated from the difference between the intermediate PTZ drive position and the PTZ drive positions of the image capture device 110-A and the image capture device 110-B, and image distortion is corrected by gradually increasing the size of the pixels from the intermediate PTZ drive position as the convergence point. This makes it possible to change the angle of view of the panoramic image without moving the image capture device while suppressing the generation of a distorted panoramic image by correcting the left and right distortion in this embodiment.

[0093] In this embodiment, the PTZ control is performed so as not to destroy the aspect ratio of the panoramic image 820, but the control method is not limited to this. For example, the control may be performed so that the imaging devices continue to capture the content that is a characteristic point within the overlapping range. In addition, the zoom drive amount when pan control is performed and the pan drive amount when zoom control is performed may be determined in advance.

[0094] In addition, in this embodiment, the method of correcting the image distortion based on the intermediate PTZ drive position is shown, but the present invention is not limited to this. For example, the PTZ coordinates of the panoramic image in the virtual space may be calculated and used instead of the intermediate PTZ drive position, and the PTZ coordinates of the captured image may be calculated and used instead of the PTZ drive position of each imaging device 110 to obtain the amount of deviation.

[0095] <Third embodiment> In the first embodiment, a method for performing PTZ control on a panoramic image in real time by transmitting different PTZ control commands to each imaging device according to the user's PTZ control content for the panoramic image and video has been described.

[0096] However, depending on the installation positions of the multiple imaging devices 110, the PTZ driving range that can be changed by the user in the panoramic image differs, and a panoramic image cannot be generated after the PTZ control operation, which may cause the user to feel uncomfortable.

[0097] Therefore, using FIG. 12, we will explain a third embodiment, which is characterized in that the PTZ drivable range in a panoramic image, i.e., the range in which an instruction to change the angle of view for a panoramic image can be obtained, is specified when the panoramic image is first created.

[0098] In step S1201, the PTZ drive range in which a panoramic image can be generated is calculated based on the PTZ drive positions of each imaging device acquired in step S704, and the controllable range of the slide bars 602 to 604 is corrected. For example, when the zoom value range calculated from the position of the zoom lens of the imaging device 110 is 0 to 100, if the zoom value of the imaging device 110-A is 10 and the zoom value of the imaging device 110-B is 30, the range of the slider bar 602 is internally changed to 0 to 70. In addition, regarding the current zoom value at this time, the minimum zoom value of the imaging device 110-A or the imaging device 110-B is changed and set so as to correspond to the scale of the zoom value range of the current panoramic image. For example, in the case of the setting of the imaging device 110-A described above, the value is changed from 10 to 7.

[0099] The control range of the slider bar 604 is also corrected in the same manner with respect to the pan drive range corresponding to the slider bar 604. For example, assume that the pan driveable range in which a panoramic image corresponding to the slider bar 604 can be generated is ±60 degrees, and image capture devices 110-A and 110-B are driven and controlled in the positive pan direction. Then, if the pan angle of image capture device 110-A is 30 degrees and that of 110-B is 35 degrees, the value of the slider bar 604 is corrected to -35 degrees.

[0100] The user may be able to set the driving range of each of the slider bars 602 to 604. For example, when the driving range of the zoom adjustment slider bar 602 is 0 to 100, the user may set the driving range within the range of 0 to 100, and the pan / tilt driving range may be set in the same manner.

[0101] In addition, the user may be able to set an unimageable range. In this case, the control range of the adjustment sliders 602 to 604 may be determined so that PT drive control cannot be performed in the unimageable range.

[0102] As described above, in this embodiment, the PTZ drive range of the panoramic image is corrected based on the PTZ drive positions of the imaging device 110-A and the imaging device 110-B. This allows the user to understand the range of the panoramic image that can be PTZ controlled, thereby reducing the sense of discomfort felt by the user. [Explanation of symbols]

[0103] 110 Imaging device 120 Panoramic image generating device 203 Drive unit 401 Acquisition Department 402 Panoramic image generation means 405 Drive command generating means 406 Communications Department

Claims

1. a first acquisition means for acquiring a plurality of captured images from a plurality of imaging devices having a PTZ function; a second acquisition means for acquiring at least one of driving information of pan, tilt, and zoom of the imaging device; a panoramic image generating means for generating a panoramic image based on the image acquired by the first acquisition means; a third acquisition means for acquiring a view angle change instruction for changing the view angle of the panoramic image generated by the panoramic image generating means; a drive amount determination means for determining a PTZ drive amount for at least one of the image capture devices based on the pan, tilt, and zoom drive information acquired by the second acquisition means and the view angle change instruction acquired by the third acquisition means; a transmission means for transmitting to the imaging device an instruction to change at least one of a pan, tilt, and zoom drive position of the imaging device based on the PTZ drive amount determined by the drive amount determination means; having When the third acquisition means acquires an instruction to change the angle of view, the transmission means transmits an instruction to change a PTZ drive position to the imaging device based on the PTZ drive amount determined by the drive amount determination means. A panoramic image generating device comprising:

2. When creating a first panoramic image in the panoramic image generating means, the panoramic image generating means detects common feature points from the images acquired by the first acquisition means, and generates a first overlapping range by overlapping the feature points; the panoramic image generating means generates the first panoramic image such that the first overlapping range with respect to an area of ​​the first panoramic image is equal to or larger than a predetermined threshold value; 2. The panoramic image generating device according to claim 1,

3. When the angle of view is changed from the first panoramic image to the second panoramic image based on the angle of view change instruction acquired by the third acquisition means, The panoramic image generating device according to claim 2, characterized in that the drive amount determination means calculates the PTZ drive amount so that the ratio of the first overlap range to the area of ​​the first panoramic image is the same as the second overlap range to the area of ​​the second panoramic image.

4. The present invention further includes a distortion correction unit that corrects distortion of the image based on the PTZ drive position, The distortion correction means corrects the image acquired by the first acquisition means, The panoramic image generating means generates a panoramic image based on the corrected image.

2. The panoramic image generating device according to claim 1 .

5. the plurality of imaging devices includes a first imaging device and a second imaging device; the distortion correction means compares an intermediate PTZ drive position, which is an intermediate value between the PTZ drive positions of the first imaging device and the second imaging device, with the PTZ drive positions of the first imaging device and the second imaging device, and switches correction contents according to the difference between the intermediate PTZ drive positions.

5. The panoramic image generating apparatus according to claim 4.

6. a determination means for determining a range of the angle of view change instruction that can be acquired by the third acquisition means based on the image acquired by the first acquisition means; The apparatus further includes an input unit for a user to input an instruction to change the angle of view of the panoramic image, Based on the range of the angle of view change instruction determined by the determination means, Accepting an input to the input means; 2. The panoramic image generating device according to claim 1 .

7. A panoramic image generating system having a plurality of imaging devices each having a PTZ function and a panoramic image generating device for generating a panoramic image, The panoramic image generating device comprises: A control means for controlling the imaging directions of the plurality of imaging devices; a first acquisition means for acquiring captured images of the plurality of imaging devices; a second acquisition means for acquiring at least one of driving information of pan, tilt, and zoom of the imaging device; a panoramic image generating means for generating a panoramic image based on the image acquired by the first acquisition means; a third acquisition means for acquiring a view angle change instruction for changing the view angle of the panoramic image generated by the panoramic image generating means; a drive amount determination unit that determines a PTZ drive amount for at least one of the imaging devices based on the PTZ drive information acquired by the second acquisition unit and the angle of view change instruction acquired by the third acquisition unit; having When the third acquisition means acquires an instruction to change the angle of view, the control means changes a PTZ drive position of the imaging device based on the PTZ drive amount determined by the drive amount determination means. A panoramic image generating system comprising:

8. a control step of controlling the imaging directions of the plurality of imaging devices; a first acquisition step of acquiring captured images of the plurality of imaging devices; a second acquisition step of acquiring at least one PTZ driving position of the imaging device; a panoramic image generating step of generating a panoramic image based on the images acquired in the first acquisition step; a third acquisition step of acquiring a view angle change instruction for changing the view angle of the panoramic image generated in the panoramic image generating step; a drive amount determination step of determining a PTZ drive amount for at least one of the imaging devices based on the PTZ drive position acquired in the first acquisition step and the view angle change instruction acquired in the third acquisition step; having When an angle of view change instruction is acquired in the third acquisition step, the control step changes a PTZ drive position of the imaging device based on the PTZ drive amount determined in the drive amount determination step. A panoramic image generating method comprising:

9. A program for causing a computer to execute each step according to claim 8.

Images