Image processing system, image processing method, and program

Abstract

This technology provides highly accurate blur correction even when there are few features extracted from the image. [Solution] The image processing system comprises: an image acquisition means for acquiring a first captured image and a second captured image taken by a first camera; a feature extraction means for extracting features including a first feature from the first captured image and a second feature from the second captured image; a blur detection means for generating a first blur detection result, which is the result of detecting blur of the first camera by comparing the first feature and the second feature; a blur estimation means for generating a blur estimation result, which is the result of estimating the blur of the first camera, using a second blur detection result, which is the result of detecting blur of the second camera, and external parameters including coordinate information of the first camera and the second camera; and a blur correction means for correcting the blur based on at least one of the first blur detection result and the blur estimation result.

Description

Technical Field

[0001] The present disclosure relates to a technique for correcting image blur. Background Art

[0002] There is a demand for a technique for correcting image blur, for example, when generating a virtual viewpoint video from images captured by a plurality of cameras as disclosed in Patent Document 1.

[0003] Patent Document 2 discloses a method in which edges of linear subjects present in captured images are extracted as features (patches), and highly accurate alignment is performed using a plurality of patches between images captured at different times. Since this technique cannot detect movement parallel to the edges of the subject with respect to the patches, the amount of blur across the entire screen is obtained by combining a plurality of patches extracted from subject edges in different directions. Therefore, this technique needs to acquire patches of components in different directions from the entire image. Prior Art Documents Patent Documents

[0004] Patent Document 1 Japanese Unexamined Patent Application Publication No. 2020-188395 Patent Document 2 U.S. Patent No. 10121262 Specification Summary of the Invention Problems to be Solved by the Invention

[0005] However, when there are few feature points in a reference image, image blur caused by vibration or the like cannot be sufficiently detected, so blur correction cannot be performed with high accuracy.

[0006] Accordingly, the present disclosure provides a technique capable of correcting blur with high accuracy even when there are few features extracted from an image. Means for Solving the Problems

[0007] To solve this problem, for example, the image processing system disclosed herein has the following configuration: Image acquisition means for acquiring the first and second images captured by the first camera, A feature extraction means for extracting features including a first feature from the first captured image and a second feature from the second captured image, A shake detection means that generates a first shake detection result, which is the result of detecting shake of the first camera by comparing the first feature and the second feature, A shake estimation means generates a shake estimation result, which is the result of estimating the shake of the first camera, using a second shake detection result, which is the result of detecting shake of the second camera, and external parameters including coordinate information of the first camera and the second camera. A shake correction means for correcting the shake based on at least one of the first shake detection result and the shake estimation result, It is equipped with. [Effects of the Invention]

[0008] According to this disclosure, blur can be corrected with high accuracy even when there are few features extracted from the image. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic diagram showing an example of the overall configuration of an image processing system according to an embodiment. [Figure 2] A diagram showing an example of camera installation in the embodiment. [Figure 3] A block diagram illustrating the functions of the image processing system 101 of the embodiment. [Figure 4] A diagram illustrating the features extracted from the reference image in the first embodiment. [Figure 5A] Flowchart of the calibration mode in the first embodiment. [Figure 5B] A flowchart of the image processing device in the shooting mode of the first embodiment. [Figure 6]A diagram illustrating blur vectors based on features extracted from a reference image in the second embodiment. [Figure 7] A flowchart of an image processing apparatus in the imaging mode of the second embodiment. [Figure 8] A flowchart of an image processing apparatus in the imaging mode of the third embodiment. [Figure 9] A block diagram showing the hardware configuration of the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the present disclosure, and a plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and overlapping descriptions are omitted.

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. First, an overview of the embodiment will be described.

[0012] The embodiment is applied to, for example, a system in which a plurality of cameras are installed on structures at different positions in a field to perform synchronized shooting from multiple viewpoints, and a virtual viewpoint video is generated using images from the plurality of viewpoints obtained by shooting. The embodiment may generate a virtual viewpoint video for virtually viewing stadiums where games such as soccer and basketball are held, and concert halls where performances and plays are held from various angles. In the embodiment, for example, a subject in a captured image of each camera may be extracted as a foreground, and an object shape may be calculated using the foreground images of the same subject extracted from a plurality of captured images to generate a virtual viewpoint video. Thereby, the embodiment provides a user with an immersive video experience.

[0013] Here, in the embodiment, parameters including the positions, postures, focal lengths and the like of the plurality of cameras (also referred to as external parameters) may be calculated during calibration. Here, the cameras are installed on structures surrounding the imaging range. However, the posture of a camera changes from the posture at the time of calibration due to wind, vibration caused by movement of spectators near the structure, and the like. The embodiment corrects blur resulting from posture change and the like when generating virtual viewpoint video and the like. In particular, even if few features are extracted from a reference image captured at the time of reference calibration, the embodiment estimates blur based on the reference image and a current image, thereby improving the accuracy of blur correction. The reference image and the current image are an example of a first image and a second image. (First Embodiment) Hereinafter, the image processing system of the first embodiment will be described with reference to FIGS. 1 to 6.

[0014] (Overall Image Processing System) FIG. 1 is a schematic diagram showing an example of the overall configuration of the image processing system 101 according to the embodiment.

[0015] The image processing system 101 comprises a plurality of cameras 102, a plurality of image processing apparatuses 103, an image processing server 104, and a system control apparatus 105.

[0016] The plurality of cameras 102 are arranged so as to surround an imaging range 100 that is an imaging target. The camera 102 has an optical system such as a lens. The camera 102 images a subject in the imaging range 100, and outputs data of a captured image generated by the imaging to the image processing apparatus 103.

[0017] The image processing device 103 is connected to any of the multiple cameras 102 to send and receive data such as images. The multiple image processing devices 103 may be connected to each other in a daisy-chain configuration. For example, the multiple image processing devices 103 may be connected in a chain. For example, the terminal image processing device 103 may be connected to the image processing server 104. The image processing device 103 performs pre-processing on the captured images acquired from each camera 102. The pre-processing includes extracting the subject of the captured image as the foreground and generating a silhouette image of the foreground. The image processing device 103 outputs the processed captured images to the image processing server 104 for post-processing.

[0018] The image processing server 104 acquires pre-processed images from multiple image processing devices 103. The image processing server 104, for example, applies Visual Hull to multiple silhouette images to create and store a 3D model. The image processing server 104 generates a virtual viewpoint image operated by the user and outputs it to the system control device 105. The virtual viewpoint image may be an image viewed from the virtual viewpoint of a virtual camera.

[0019] The system control unit 105 displays the virtual viewpoint image generated by the image processing server 104 and accepts user input. For example, the user can operate the virtual camera's position and orientation (orientation) while viewing the virtual viewpoint image. The system control unit 105 controls changes to the settings of the camera 102 and the image processing device 103.

[0020] In the image processing system 101 shown in Figure 1, multiple image processing devices 103 are connected in a daisy-chain configuration, but the connection is not limited to this. For example, each image processing device 103 may be connected to the image processing server 104 in a star configuration. Alternatively, each of the multiple rows of linked image processing devices 103 may be connected to the image processing server 104. Also, in Figure 1, there are 10 cameras 102, but the number of cameras 102 can be any number, and the number may be changed as appropriate.

[0021] (Camera installation) Figure 2 shows an example of the installation of the camera 102 in this embodiment. The camera 102 in this embodiment is installed and fixed to a structure such as a pillar, wall, or fence at the venue. Figure 2 shows the camera 102 attached to the fence 106 of the competition venue. Figure 2(a) is an overall overhead view of the camera 102 installed on the fence 106. Figure 2(b) is an overhead view showing the area around the cameras from the opposite side of Figure 2(a). Figure 2(c) is a detailed enlarged view of the area around the cameras in Figure 2(a).

[0022] As shown in Figures 2(a), 2(b), and 2(c), the camera mounting unit 107 is attached to the ball portion of the fence 106, which is a structure, via clamps 107a and 107b. The two cameras 102a and 102b are attached to the camera mounting unit 107 via attitude adjustment mechanisms 108a and 108b, respectively. As a result, cameras 102a and 102b are fixed to the structure, and their relative positions are fixed. Cameras 102a and 102b are examples of a first camera and a second camera.

[0023] Coordinate 109a indicates the relative coordinate of camera 102a. Coordinate 109b indicates the relative coordinate of camera 102b. When there is no need to distinguish between camera 102a and camera 102b, they are referred to as camera 102. Similarly, for other identical components, the final letter is omitted when there is no need to distinguish between them.

[0024] The camera 102 is attached to the attitude adjustment mechanism 108. The attitude adjustment mechanism 108 has three rotation axes. Therefore, the attitude adjustment mechanism 108 adjusts the attitude of the camera 102 it holds using the pitch axis, yaw axis, and roll axis as rotation axes. The attitude adjustment mechanism 108 is fixed to the camera fixing part 107. Therefore, after adjusting the attitude of the camera 102, the attitude of the camera 102 is fixed by locking each axis of the attitude adjustment mechanism 108. As a result, the relative attitude of the camera 102 is fixed with respect to the camera fixing part 107. The mounting member in Figure 2 is an example, and the grounding member may have a different shape. Also, the mounting target may be a structure other than the fence 106.

[0025] (Functional configuration of the image processing system) Figure 3 is a block diagram illustrating the functions of the image processing system 101 of the embodiment.

[0026] Note that Figure 3 only shows two pairs of cameras 102 and image processing devices 103 for the sake of simplicity. However, in this embodiment, multiple (three or more) pairs of cameras 102 and image processing devices 103 may be connected to the image processing server 104. The processing and operation of cameras 102 and image processing devices 103 described below are performed by cameras 102 and image processing devices 103 respectively.

[0027] As shown in Figure 3, the camera 102 has an optical system consisting of a lens 121 and an image sensor 122.

[0028] Lens 121 has one or more lenses, including an imaging lens. Lens 121 forms an image of the light beam from the subject onto the light-receiving surface of the image sensor 122.

[0029] The image sensor 122 converts the light beam from the subject imaged by the lens 121 into an electrical signal. This allows the image sensor 122 to generate digital image data, which it then outputs to the image processing device 103. The term "image" may include both the image and the image data. The image sensor 122 may be, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0030] The image processing device 103 includes an image acquisition unit 130, a feature extraction unit 131, a reference image recording unit 132, a blur detection unit 133, a blur correction unit 134, a blur estimation unit 135, an image processing unit 136, and a communication unit 137.

[0031] The image acquisition unit 130 acquires the image output by the camera 102 and outputs it to the feature extraction unit 131.

[0032] The feature extraction unit 131 extracts features, including edges of subjects, within an image. The image in question includes a reference image captured in calibration mode and a current image captured in a mode other than calibration mode (e.g., shooting mode). The detailed operation of the feature extraction unit 131 will be described later. The current image and the reference image are images captured by the same camera 102, and may overlap in at least part of their shooting ranges. The reference image is, for example, an image captured before the current image. The feature extraction unit 131 outputs the image from which features have been extracted and the feature extraction results to the reference image recording unit 132 and the blur detection unit 133. Specifically, in calibration mode, the feature extraction unit 131 may output the image associated with the extracted features as the reference image to the reference image recording unit 132. On the other hand, in shooting mode, the feature extraction unit 131 may output the image associated with the extracted features as the current image to the blur detection unit 133.

[0033] The reference image recording unit 132 records the image acquired from the feature extraction unit 131 in calibration mode as a reference image to which the extracted features are linked.

[0034] The blur detection unit 133 compares the features of the current image indicated by the feature extraction results with the features of the reference image to detect blur in the camera 102 that captured both images. The blur detection unit 133 outputs the blur detection result, which is the result of detecting blur, and the acquired current image to the blur correction unit 134 and the communication unit 137.

[0035] The blur estimation unit 135 estimates the blur of camera 102. Specifically, if the image processing device 103 (referred to here as image processing device 103a) on which it is mounted is connected to camera 102a, the blur estimation unit 135 obtains the blur detection result of the other camera 102b detected by the blur detection unit 133 of the image processing device 103b via the communication unit 137. Based on the transformation matrix obtained from the coordinate information of cameras 102a and 102b obtained during the calibration mode described later, the blur estimation unit 135 estimates the blur of camera 102a from the blur detection result of the other camera 102b. The coordinate information includes, for example, information such as the attitude and position of camera 102. The coordinate information may also be either relative or absolute values. The blur estimation unit 135 outputs the estimated blur result to the blur correction unit 134.

[0036] The image blur correction unit 134 uses at least one of the blur detection results obtained from the blur detection unit 133 and the blur estimation results obtained from the blur estimation unit 135 to correct the blur in at least one of the blurring camera 102 and the current image (hereinafter also referred to as blur correction). Based on the blur detection results and the blur estimation results, the image blur correction unit 134 cuts out a range from the current image that matches the reference image. As a result, the image blur correction unit 134 generates an image in which the blur relative to the reference image has been canceled out by blur correction and outputs it to the image processing unit 136.

[0037] The image processing unit 136 extracts the foreground area, which is the subject to be modeled in 3D, from the current image that has been corrected for blur. The image processing unit 136 outputs the extraction result, which is the extracted foreground area, to the communication unit 137.

[0038] The communication unit 137 transmits the extraction results obtained from the image processing unit 136 to the image processing server 104. The communication unit 137 transmits the blur detection results obtained from the blur detection unit 133 to the communication unit 137 of the image processing device 103 connected to another camera 102 (for example, camera 102b).

[0039] The image processing server 104 includes a 3D model generation device 141, a 3D model storage device 142, and a virtual viewpoint image generation device 143.

[0040] The 3D model generation device 141 generates a 3D model using foreground images received from multiple synchronized image processing devices 103 and stores it in the 3D model storage device 142.

[0041] The virtual viewpoint image generation device 143 reads data such as 3D models from the 3D model storage device 142 in response to virtual camera control instructions from the system control device 105. The virtual viewpoint image generation device 143 generates a virtual viewpoint image based on the read data such as 3D models.

[0042] As described above, the system control device 105 issues control commands for the attitude, position, and operation of the virtual camera. The system control device 105 also has an external parameter calculation unit 151, which issues work commands for calibration at the start of shooting and calculates external parameters for the camera 102.

[0043] (Hardware configuration of the image processing device) The hardware configuration of the image processing device 103 will be described below. Figure 9 is a block diagram showing the hardware configuration of an image processing device 103 in an embodiment. The image processing device 103 is an example of a computer. The image processing device 103 includes a processor 901, a memory 902, a storage 903, a communication IF 904, an input IF 905, an output IF 906, and a bus 907. The processor 901, memory 902, storage 903, communication IF 904, input IF 905, and output IF 906 are connected to each other via the bus 907 so that they can send and receive information.

[0044] The processor 901 is an arithmetic processing unit, such as a CPU (Central Processing Unit). The image processing unit 103 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU. The processor 901 realizes various functions by reading programs stored in the storage 903 and loading them into the memory 902. For example, by reading programs, the processor 901 realizes some or all of the functions of the image acquisition unit 130, feature extraction unit 131, reference image recording unit 132, blur detection unit 133, blur correction unit 134, blur estimation unit 135, image processing unit 136, and communication unit 137. Furthermore, some or all of the functions of the image acquisition unit 130, feature extraction unit 131, reference image recording unit 132, blur detection unit 133, blur correction unit 134, blur estimation unit 135, image processing unit 136, and communication unit 137 may be implemented by one or more circuits such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0045] Memory 902 is a high-speed read / write storage device, such as RAM (Random Access Memory). Memory 902 functions as a work area when the processor 901 executes a program. Memory 902 temporarily stores the program and parameters necessary for program execution. These parameters include the attitude and position coordinates of the camera 102, as well as information about camera shake.

[0046] Storage 903 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Storage 903 retains programs, parameters necessary for program execution, and the results of program execution even when power is not supplied. Storage 903 stores, for example, image data output by camera 102.

[0047] The communication interface 904 is an interface for enabling communication with external devices via a wired or wireless network. External devices include, for example, a camera 102, an image processing server 104, and a system control device 105.

[0048] The Input IF905 is an interface for receiving information input from a user via an input device. Examples of input devices include a mouse, keyboard, and touch panel.

[0049] Output IF906 is an interface for outputting information to an output device. An output device is, for example, a display device such as a monitor.

[0050] Furthermore, the control device, image processing server 104, and system control device 105 mounted on camera 102 have the same hardware configuration as the image processing device 103.

[0051] (Shake detection) Next, the operation of the feature extraction unit 131 and the blur detection unit 133 will be explained using Figure 4. Figure 4 is a diagram illustrating the features extracted from the reference image in the embodiment. Figure 4(a) shows a reference image of one of the cameras 102. Figure 4(b) shows a reference image of a different camera 102 than that of Figure 4(a).

[0052] Figure 4 shows images of the finish line and the course lines when photographing a track and field event. Calibration is performed when there are no people or objects in the shooting range, including the track. Therefore, the image includes only objects corresponding to the background and the ground, and does not include runners on the track. On the other hand, the current image, which is the target of blur correction, is an image taken during the competition and includes subjects such as athletes. Therefore, for example, the objects and living subjects such as people included in the reference image are fewer than in the current image. Also, the reference image may be taken before the current image. Figure 4 shows the features (patches) of the extracted feature components with rectangular dotted lines.

[0053] Figure 4(a) shows a horizontal edge, as indicated by the course line 212. Image 201 shows multiple vertical edges, such as the pole 211 and the finish board 215. Figure 4(b) shows image 202, which shows horizontal edges such as the course line 221, but no vertical edges. Note that the vertical and horizontal directions intersect (e.g., orthogonal). Also, the vertical and horizontal directions intersect (e.g., orthogonal).

[0054] The feature extraction unit 131 of this embodiment extracts linear edges, which are features of the subject, as patches from the reference image and the current image, and saves the patches in association with the image. In Figure 4, the rectangle represented by the dotted line indicates the range of the extracted patches. Figure 4 shows only a portion of the extracted patches for the sake of simplicity. In image 201 of Figure 4(a), patch 213 is a vertical component patch where the angle of the edge with respect to the horizontal is 45° or more. Patch 213 is extracted as a component that can be used for detecting horizontal blur. Patch 214 is a horizontal component patch where the angle of the edge with respect to the horizontal is less than 45°. Patch 214 is extracted as a component that can be used for detecting vertical blur. In image 202 of Figure 4(b), patch 222 is a horizontal component patch where the angle of the edge with respect to the horizontal is less than 45°. Patch 222 is extracted as a component that can be used for detecting vertical blur.

[0055] In image 201 of Figure 4(a), the vertical component patch 213 and the horizontal component patch 214 have been extracted. On the other hand, in image 202 of Figure 4(b), the horizontal component patch 222 has been extracted, but the vertical component patch has not. Although Figure 4 describes feature extraction for the reference image, the feature extraction unit 131 performs the same feature extraction on the current image to extract patches.

[0056] The blur detection unit 133 searches for patches linked to a reference image that matches the extracted patch and determines the displacement within the patch's image. The search for matching patches with the reference image is performed for multiple patches, and the amount of image blur is detected by statistical processing of the results. For example, the blur detection unit 133 aggregates the results of only vertical component patches to detect horizontal blur, and aggregates the results of only horizontal component patches to detect vertical blur. As shown in Figure 4(b), if vertical component patches cannot be detected, it is difficult for the blur detection unit 133 to detect horizontal blur. Figure 4(b) illustrates an extreme example with no vertical component patches, but even if vertical components are present, if they are few in number, the blur detection unit 133 may misdetect patches during the search, and the background portion that constitutes a vertical component patch may be hidden by a foreground subject in the current image. As a result, it may become difficult to detect horizontal camera blur.

[0057] (Conversion of blur) Therefore, in this embodiment, two cameras 102 are mounted on the same camera fixing unit 107, and the camera 102 transmits and shares the shake detection results of each camera 102. The shake estimation unit 135 then performs a coordinate transformation on the shake detection result of one camera 102 using a transformation matrix obtained from the coordinate information of both cameras 102, and estimates the shake of the other camera 102.

[0058] Camera shake estimation will be explained using two cameras 102a and 102b mounted on the same camera fixing unit 107 as an example. The external parameter calculation unit 151 of the system control device 105 calculates the external parameters of each camera 102 during calibration. The external parameters include coordinate information such as the attitude and position coordinates of the camera 102. The attitude of camera 102a is represented by the rotation matrix R1 and the translation matrix t1. The attitude of camera 102b is represented by the rotation matrix R2 and the translation matrix t2. At this time, the shake estimation unit 135 calculates the transformation matrix R to convert the coordinate system of camera 102a to the coordinate system of camera 102b. 12 , and the translation vector t 12 This is calculated based on equations (1) and (2).

number

[0059] Next, let's assume that the shake detection unit 133 has detected shake in camera 102a and has detected a shake vector V1, which is a vector of shake. At this time, camera 102b is mounted on the same camera fixing unit 107 as camera 102a. Therefore, each camera 102 can be considered to vibrate as a single unit. As a result, the shake estimation unit 135 of camera 102b can calculate the shake vector V2 of camera 102b from the shake vector V1 of camera 102a, as shown in equation (3).

number

[0060] In this way, the camera 102b's shake estimation unit 135 outputs the calculated shake vector V2 as the shake estimation result.

[0061] On the other hand, the camera 102a's shake estimation unit 135 can calculate the camera 102a's shake vector V1 from the shake vector V2 detected by the camera 102b's shake detection unit 133 by using the camera 102a's rotation matrix R1 and translation matrix t1, and the camera 102b's rotation matrix R2 and translation matrix t2.

[0062] (flow) The image processing system 101 of this embodiment has a calibration mode and a shooting mode. The system mode can be switched by instruction from a user operating the system control device 105. The calibration mode is a mode that includes the work of determining camera external parameters, including the coordinate system of the camera orientation and position of each camera, and the work of acquiring a reference image. The calibration mode is performed, for example, when the system is started up, and at a time when there are no foreground subjects in the shooting range on the field. After calibration is performed when the system is started for the first time, the image processing system 101 may recall and use the camera external parameters and reference image saved during the calibration. In addition, the image processing system 101 may perform calibration again if a certain period of time has passed since the previous calibration. This allows the image processing system 101 to take into account slight changes in camera orientation over time and changes in the background.

[0063] Next, the flowcharts in each mode of the image processing system 101 in this embodiment will be described using the flowchart in Figure 5. Figure 5 shows the flowcharts for each mode. Figure 5A is the flowchart for the calibration mode in the first embodiment. Figure 5B is the flowchart for the shooting mode in the first embodiment.

[0064] First, the operation in calibration mode will be explained using Figure 5A. The flowchart on the left of Figure 5A is the flowchart for the system control device 105. The flowchart on the right of Figure 5A is the flowchart for the image processing device 103. The calibration operation will be performed after all cameras 102 have been adjusted to the desired field of view.

[0065] In step S101, the external parameter calculation unit 151 of the system control device 105 performs a calibration process for acquiring external parameters of the camera using markers or the like. In the calibration process, the external parameter calculation unit 151 acquires images of the shooting range including the markers captured by each camera 102, for example, with multiple markers placed in the shooting area.

[0066] In step S102, the external parameter calculation unit 151 calculates external parameters using the images taken by each camera in step S101, and calculates coordinate information such as the attitude and position of each camera 102 in the coordinate system of each camera 102 relative to world coordinates. Note that the method for calculating position and attitude is a known technique, so it will not be explained here. The external parameter calculation unit 151 transmits the calculated coordinate information to the camera 102.

[0067] In S100, the communication unit 137 of the image processing device 103 acquires coordinate information related to the orientation and coordinates of the camera 102 transmitted by the system control device 105. The communication unit 137 may also acquire position coordinates, etc., via the image processing server 104. Furthermore, the communication unit 137 may acquire position coordinates, etc., of two cameras 102 installed on the same camera fixing unit 107.

[0068] In step S103, the blur estimation unit 135 calculates a transformation matrix between the two cameras 102 installed on the same camera fixing unit 107 based on the camera position coordinates acquired in step S100. Alternatively, this transformation matrix may be calculated by the system control unit.

[0069] In step S104, the image acquisition unit 130 acquires a reference image from the camera 102. For example, the camera 102 may capture the shooting range while excluding the marker used in step S101 and subjects such as people from the shooting range, generate a reference image, and transmit it to the image processing device 103. Therefore, the image acquisition unit 130 may acquire the image captured in step S101 as a reference image from the camera 102 or the system control device 105.

[0070] In step S105, the feature extraction unit 131 extracts patches as features from the reference image, associates the extracted patches with the reference image, and outputs them to the reference image recording unit 132.

[0071] In step S106, the reference image recording unit 132 records the reference image together with the associated patch. The reference image recording unit 132 may transmit the reference image with the associated patch to the system control device 105 via the communication unit 137.

[0072] In step S107, the system control unit 105 acquires a reference image from the image processing device 103. Alternatively, the system control unit 105 may acquire the reference image via the image processing server 104.

[0073] In step S108, the system control unit 105 selects a camera 102 from each reference image whose number of features (patches) is insufficient as the camera 102 for which the blur estimation results will be used in blur correction. For example, the system control unit 105 may select a camera 102 whose number of vertical component patches and horizontal component patches extracted from the reference image are equal to or greater than a threshold as the camera 102 for which blur correction will be performed using the blur detection results. On the other hand, the system control unit 105 may select a camera 102 whose number of vertical component patches or horizontal component patches extracted from the reference image is less than or equal to a threshold as the camera 102 for which blur correction will be performed using the blur estimation results. In this case, if the number of patches extracted from both cameras 102 installed in the same camera fixing unit 107 is insufficient, the system control unit 105 may display a warning to the user. The operation when a warning is issued will be explained later. Alternatively, the user may select a camera 102 by visually checking each reference image and patch. Alternatively, the image stabilization unit 134 may select a camera 102 that uses image stabilization estimation for image stabilization.

[0074] Next, the flow in shooting mode will be explained using Figure 5B. Figure 5B is a flowchart of the image processing device in shooting mode according to the first embodiment. The flowchart on the left of Figure 5B is the flowchart of one image processing device 103 (here, image processing device 103a). The flowchart on the right of Figure 5B is the flowchart of the other image processing device 103 (here, image processing device 103b). Image processing devices 103a and 103b perform almost the same processing. Therefore, in the explanation of Figure 5B, the processing of one image processing device 103 will be explained, and the explanation of the similar processing of the other will be omitted or simplified. As described above, the camera 102a of image processing device 103a is installed in the same camera fixing unit 107 as the camera 102b of image processing device 103b.

[0075] When the shooting mode starts, in step S121, the image acquisition unit 130 of the image processing device 103a acquires the image captured by the camera 102a as the current image.

[0076] In step S122, the feature extraction unit 131 extracts patches as features from the acquired current image.

[0077] In step S123, the blur detection unit 133 compares the extracted patch with the reference image patch of the reference image recording unit 132 and detects the amount of blur (also called simple blur), which is the amount of displacement of the current image blur relative to the reference image. The blur detection unit 133 outputs the blur detection result, including the blur, to the blur correction unit 134 and the communication unit 137.

[0078] In step S124, the communication unit 137 of the image processing device 103a of camera 102a transmits the blur detection result detected in step S123 to the communication unit 137 of the image processing device 103b of the other camera 102b, which is mounted on the same camera fixing unit 107. After this, the image processing device 103a executes the processing described later from S125 onwards.

[0079] Next, the processing of the image processing device 103b connected to the camera 102b from step S125 onwards will be described. Note that the image processing device 103b executes the processes from steps S121 to S124 described above before processing step S125.

[0080] In step S125, the communication unit 137 of the image processing device 103b acquires the blur detection result of the other camera 102a from the image processing device 103a and outputs it to the blur estimation unit 135.

[0081] In step S126, the blur estimation unit 135 performs a coordinate transformation on the blur detection result of the other camera 102a received in step S125 by the image processing device 103a. Based on the coordinate-transformed blur detection result, the blur estimation unit 135 estimates the blur of camera 102b and outputs the estimated blur to the blur correction unit 134.

[0082] In step S127, the image stabilization unit 134 determines whether or not to perform image stabilization using the image stabilization estimation result. For example, based on the selection result in S108, the image stabilization unit 134 may determine whether or not to use the image stabilization estimation result based on whether or not the image processing device 103b on which it is mounted is connected to the camera 102 that uses the image stabilization estimation result. The image stabilization unit 134 may also perform the selection in S108 in step S127. If the image stabilization unit 134 determines to perform image stabilization using the image stabilization estimation result, it proceeds to step S129. On the other hand, if the image stabilization unit 134 determines to perform image stabilization without using the image stabilization estimation result, it proceeds to step S128. The image stabilization unit 134 may also perform the processing in steps S128 and S129 based on the determination result based on the number of patches extracted from the current image.

[0083] In step S128, the image stabilization unit 134 performs image stabilization based on the image stabilization detection results of its own image stabilization unit 133, since a sufficient number of patches have been detected from the reference image of the camera 102b by feature extraction.

[0084] On the other hand, in step S129, the blur correction unit 134 determines that a sufficient number of patches have not been detected from the reference image of the camera 102 by feature extraction, and therefore performs blur correction based on the blur estimation result estimated by the blur estimation unit 135.

[0085] In step S130, the image stabilization unit 134 of each image processing device 103 determines whether a command to end shooting has been issued from the system control device 105. If the image stabilization unit 134 determines that it has not received a command to end shooting, it returns to step S121 or step S125 and repeats the above flow for the image of the next frame. Note that the image processing device 103 may perform the operations from steps S121 to S130 within the period of one frame of a video in shooting mode. On the other hand, if the image stabilization unit 134 determines that it has received a command to end shooting, it terminates the flow.

[0086] As described above, the first embodiment estimates the camera 102's blur based on the blur detection results of the two cameras 102. As a result, the first embodiment can correct the blur based on the estimated blur estimation result, even when there are few features extracted from the image. Consequently, the first embodiment can correct the blur with high accuracy even when there are few features extracted from the image.

[0087] In the first embodiment, two image processing devices 103 transmit and share the blur detection results of two cameras 102 installed on the same camera fixing unit 107. Then, a blur estimation unit 135 performs a coordinate transformation based on the blur detection results of the two cameras 102 and coordinate information such as the orientation of each camera 102, and estimates the blur of the other camera 102. As a result, this embodiment can correct the blur of camera 102 with high accuracy even in images of camera 102 with few extracted features.

[0088] In the first embodiment, if the number of extracted features is greater than or equal to a threshold, the blur is corrected based on the detected blur, thus enabling blur correction with high accuracy. On the other hand, even when the number of extracted features is small, the first embodiment can achieve blur correction with high accuracy by estimating the blur.

[0089] (Other examples) In this embodiment, the case where the camera mounting part 107 is attached to a fence is used as an example, but the mounting situation is not limited to this. For example, if multiple camera mounting parts 107 are attached to a rigid body such as concrete, and the attachment positions are sufficiently close together, the multiple camera mounting parts 107 may be considered as the same rigid body (or the same member).

[0090] For example, if each of the two camera mounting units 107 holds two cameras 102 and is mounted adjacent to the same concrete wall surface, the two cameras 102 can be considered to be fixed to the same rigid body. In this case, the shake estimation unit 135 may estimate the shake of the camera mounted on the other camera mounting unit 107 based on the shake detection result of the camera mounted on one of the camera mounting units 107.

[0091] Furthermore, this embodiment describes the case where two cameras 102 are installed on the same camera fixing unit 107. However, this embodiment may also be applied when three or more cameras 102 are installed on the camera fixing unit 107.

[0092] Furthermore, in this embodiment, one image processing device 103 is connected to one camera 102. However, this embodiment may also be applied to a system in which one image processing device is connected to multiple cameras 102. In that case, one image processing device 103 can perform blur detection on multiple cameras 102, and blur estimation can be performed without communication between multiple image processing devices 103.

[0093] (How to set up a camera for motion blur estimation) In this embodiment, the camera 102 that uses the blur estimation results is selected by the user during calibration mode, but the selection method is not limited to this method. For example, when the feature extraction unit 131 detects patches during calibration mode, the camera 102 whose number of detected patches is less than (or less than or equal to) a preset threshold may use the blur estimation results of the blur estimation unit 135.

[0094] Furthermore, if the number of patches detected from the images of the two cameras 102 installed on the same camera fixing unit 107 is less than a preset threshold, the system control device 105 may output a warning message on the screen. This allows the user to recognize from the warning message that the number of patches detected from the images of the two cameras 102 is less than the threshold. In this case, the user can change the field of view of camera 102 so that the number of patches extracted from the image of at least one of the cameras 102 is equal to or greater than the threshold. For example, the user can adjust the posture adjustment mechanism 108 so that the desired shooting range is captured while including subjects that are extracted as patches during feature extraction, including at least one of the vertical and horizontal components. Furthermore, the user can adjust the focal length of the lens of camera 102 to the wide-angle side so that subjects that are extracted as patches during feature extraction are captured.

[0095] Furthermore, when designing the installation position of the cameras 102, the designer may set the placement and orientation of the cameras 102 in advance, anticipating what will be captured within the field of view of each camera 102. For example, if two cameras 102 installed on the same camera fixing unit 107 have a small number of patches extracted from the image of one camera 102 during feature extraction, the designer can set the field of view of the other camera 102 so that the number of extracted patches is equal to or greater than a threshold.

[0096] Furthermore, the image stabilization unit 134 may perform image stabilization using the image stabilization estimation result when the number of patches extracted from the current image acquired in shooting mode is less than (or below) a threshold, rather than the number of patches extracted from the reference image. For example, the user pre-sets the orientation and field of view of the two cameras 102 installed on the same camera fixing unit 107 so that the number of patches extracted from the reference images of both cameras is greater than or equal to a threshold. In shooting mode under these conditions, if the number of patches extracted from the current image is less than a threshold due to factors such as a moving subject overlapping the range within the reference image from which the patches are extracted, the image stabilization unit 134 may perform image stabilization using the image stabilization estimation result.

[0097] Furthermore, in this embodiment, the image stabilization unit 134 performed image stabilization using either the image stabilization detection result or the image stabilization estimation result, but it may also be used to determine the reliability of the image stabilization detection result as follows. For example, in two cameras 102 installed on the same camera fixing unit 107, the user pre-sets the orientation and field of view of the cameras 102 such that the number of patches extracted from the reference images of both cameras 102 is equal to or greater than a threshold. The image stabilization unit 134 compares the image stabilization detection result and the image stabilization estimation result of one of the cameras 102, and if the absolute value of the difference is less than or equal to a threshold, it may consider the image stabilization detection result to be highly reliable and perform image stabilization using the image stabilization detection result. On the other hand, if the absolute value of the difference is equal to or greater than a threshold, the image stabilization unit 134 may consider the image stabilization detection result to be unreliable and send a message to the image processing server 104 and the system control device 105 indicating that image stabilization cannot be performed. In this case, the image processing server 104 may generate a 3D model without using the foreground image captured by the camera 102 that image stabilization cannot be performed. When a 3D model is generated using a foreground image with low blur correction accuracy, the quality of the 3D model will be lower than when an image with low blur correction accuracy is not used. Therefore, the image processing server 104 can suppress the deterioration of the 3D model quality by performing the processing as described above. The absolute value of the difference used for reliability determination may be the absolute value of the scalar difference between the blur detection result vector and the blur estimation result vector.

[0098] (If no cutting is performed) In this embodiment, the image stabilization unit 134 extracted a range of the current image that matched the reference image, but the processing may be performed as follows. For example, the image stabilization unit 134 calculates the amount of change in the camera 102's posture based on the amount of image displacement detected by the image stabilization unit 133, and calculates the posture of the camera 102 at that time. The image stabilization unit 134 then transmits the current image without extracting the current image, and also transmits the calculated camera 102 posture information to the image processing server 104. The 3D model generation device 141 of the image processing server 104 may update the posture of the camera 102 based on the current image and camera 102 posture information transmitted from the image processing device 103 to generate a 3D model.

[0099] (Second Embodiment) Next, the image processing system 101 in the second embodiment will be described using Figure 6.

[0100] The configuration of the image processing system 101 in the second embodiment is the same as that shown in Figures 1 and 2 of the first embodiment, so a description will be omitted. In the second embodiment, when the number of vertical component patches or horizontal component patches, which are feature extraction results of the reference image, is less than a threshold, blur correction is performed using both the blur detection result and the blur estimation result.

[0101] Figure 6 illustrates the blur vectors obtained from features extracted from the reference image in the second embodiment. Figures 6(a) and 6(b) show the respective reference images acquired by two cameras 102 installed on the same camera fixing unit 107. Specifically, Figure 6(a) shows the reference image of camera 102a. Figure 6(b) shows the reference image of the other camera 102b. Figures 6(c), 6(d), 6(e), and 6(f) show the blur vectors.

[0102] The reference image in Figure 6(a) shows that the feature extraction unit 131 has sufficiently extracted both vertical and horizontal component patches. On the other hand, in the reference image in Figure 6(b), the feature extraction unit 131 has sufficiently extracted horizontal component patches, but has not extracted vertical component patches.

[0103] As described above, in the reference image from camera 102b, the horizontal component patches are sufficiently extracted, so the blur detection unit 133 can accurately detect vertical blur. On the other hand, in the reference image from camera 102b, the blur detection unit 133 cannot detect horizontal blur. Therefore, in this embodiment, if only the vertical or horizontal component patches are sufficiently extracted, the blur correction unit 134 performs blur correction using the blur detection result and the blur estimation result, as described below.

[0104] In the camera 102b that captured Figure 6(b), the image stabilization unit 134 performs vertical image stabilization using the image stabilization detection result. On the other hand, the image stabilization unit 134 performs horizontal image stabilization using only the horizontal component from the image stabilization estimation result. The detected vector 601 in Figure 6(c) is the image stabilization vector detected by the image stabilization unit 133 from the image 6(a). The detected vector 601 can be decomposed into a horizontal component 601a and a vertical component 601b. The image stabilization vector 602 in Figure 6(d) is the image stabilization vector of camera 102b estimated by the coordinate transformation of equation (3) in the first embodiment using the image stabilization vector of camera 102a. The image stabilization vector 602 can be decomposed into a horizontal component 602a and a vertical component 602b. The detected vector 603 in Figure 6(e) is the image stabilization vector detected by the image stabilization unit 133 from the image 6(b). The detected vector 603 indicates that only vertical image stabilization has been detected. At this time, the image stabilization unit 134 corrects the image stabilization of the camera 102b based on the vertical detection vector 603, which is the blur detection result, and the horizontal component 602a of the blur vector 602, which is the blur estimation result. The blur vector 604 in Figure 6(f) is a vector obtained by combining the horizontal component 602a of the blur estimation result and the vertical detection vector 603 of the blur detection result. The image stabilization unit 134 uses this blur vector 604 to correct the image stabilization of the camera 102b.

[0105] (flowchart) Next, the flow of the second embodiment will be described using Figure 7. Figure 7 is a flowchart of the image processing device in the shooting mode of the second embodiment. The calibration mode of the second embodiment is almost the same as Figure 5A of the first embodiment, so the explanation will be omitted.

[0106] The image stabilization unit 134 may set the image stabilization method for the camera 102 selected in step S108 of the second embodiment to perform image stabilization using the image stabilization estimation results as follows: The image stabilization unit 134 determines whether the number of patches of the vertical component and the number of patches of the horizontal component extracted as features from the reference image in step S105 are less than (or less than or equal to) a threshold. If the image stabilization unit 134 determines that the number of patches of both components is less than the threshold, it sets the camera 102 to perform image stabilization using only the image stabilization estimation results. If the image stabilization unit 134 determines that the number of patches of both components is greater than (or equal to) the threshold, it sets the camera 102 to perform image stabilization using the image stabilization detection results. If the image stabilization unit 134 determines that only the number of patches of either the vertical component or the horizontal component is less than the threshold, it sets the camera 102 to perform image stabilization using the image stabilization detection results for the component that is not less than the threshold, and using the image stabilization estimation results for the component that is less than the threshold. The system control device 105 may also set the image stabilization method for the camera 102. The image stabilization unit 134 may perform the above determination based on the number of features extracted from the current image, or the number of features extracted from both the reference image and the current image.

[0107] Next, the flow in the shooting mode of the second embodiment will be explained using Figure 7. Figure 7 is a flowchart of the shooting mode of the second embodiment. Image processing devices 103a and 103b perform almost the same processing. Therefore, in the explanation of Figure 7, the processing of one image processing device 103 will be explained, and the explanation of similar processing of the other image processing device 103 will be omitted or simplified.

[0108] Steps S211 to S216 are the same as steps S121 to S126 in Figure 5B of the first embodiment, so their explanation will be omitted.

[0109] In step S217, the blur correction unit 134 determines whether or not to use the blur estimation result to correct the blur. If the blur correction unit 134 determines, based on the number of features as described above, that it will not use the blur estimation result, it proceeds to S218. On the other hand, if the blur correction unit 134 determines, based on the number of features, that it will use the blur estimation result, it proceeds to S219.

[0110] In step S218, since the number of patches extracted from the reference image of camera 102b by the feature extraction unit 131 is sufficient for both the vertical and horizontal components, the blur correction unit 134 performs blur correction using the blur detection result of the blur detection unit 133.

[0111] In step S219, if the number of vertical or horizontal component patches extracted from the reference image by the feature extraction unit 131 during calibration is less than the threshold, the blur correction unit 134 determines whether or not to perform blur correction using the blur vectors in both the horizontal and vertical directions of the blur estimation result. If the blur correction unit 134 determines that the number of features in both the vertical and horizontal components is less than the threshold, it decides to use the blur vectors in both directions and proceeds to S220. On the other hand, if the blur correction unit 134 determines that the number of features in either the vertical or horizontal component is not less than the threshold, it decides to use the blur vector in one direction and proceeds to S221.

[0112] In step S220, since both the vertical and horizontal component patches have not been sufficiently extracted by the feature extraction unit 131, the blur correction unit 134 performs blur correction using the blur estimation results from the blur estimation unit 135 in both directions.

[0113] In step S221, since at least one of the vertical and horizontal component patches has been sufficiently extracted by the feature extraction unit 131, the blur correction unit 134 determines whether or not to correct the blur detection result using only the horizontal blur vector. If the blur correction unit 134 determines that the horizontal component patch has not been sufficiently extracted and that blur correction should be performed using only the horizontal blur vector of the blur detection result, the process proceeds to step S222. On the other hand, if the blur correction unit 134 determines that the vertical component patch has not been sufficiently extracted and that blur correction should be performed using only the vertical blur vector of the blur detection result, the process proceeds to step S223.

[0114] In step S222, the blur correction unit 134 performs blur correction using the horizontal blur vector from the blur detection result by the blur detection unit 133 and the vertical blur vector from the blur estimation result which has been decomposed into vertical and horizontal components, since a sufficient number of patches for the vertical component have been extracted by the blur detection unit 133.

[0115] In step S223, the blur correction unit 134 performs blur correction using the vertical blur vector from the blur detection result by the blur detection unit 133 and the horizontal blur vector from the blur estimation result, since a sufficient number of patches for the lateral component have been extracted by the blur detection unit 133.

[0116] In step S224, the image stabilization unit 134 of each image processing device 103 determines whether or not a command to end shooting has been issued from the system control device 105. If the image stabilization unit 134 determines that it has not received a command to end shooting, it returns to step S211 and repeats the above flow for the image of the next frame. On the other hand, if the image stabilization unit 134 determines that it has received a command to end shooting, it terminates the flow.

[0117] As described above, the second embodiment performs blur correction using the blur detection results for components where a sufficient number of patches have been extracted from either the vertical or horizontal component and blur detection has been performed. The second embodiment then decomposes the blur estimation results into vertical and horizontal components and performs blur correction using the blur estimation results only for the direction in which the extraction of patches from either the vertical or horizontal component is insufficient. As a result, the second embodiment can improve the accuracy of blur correction even when it is not possible to extract sufficient patches from either the vertical or horizontal direction.

[0118] (Third embodiment) The image processing system 101 in the third embodiment will be described below.

[0119] The configuration of the image processing system 101 in the third embodiment is the same as that shown in Figures 1 and 2 of the first embodiment, so a description will be omitted. In the third embodiment, the user configures the cameras 102 such that the number of patches in the feature extraction results of the reference image captured by each camera 102 is equal to or greater than a threshold. For example, the user configures the two cameras 102 installed on the same camera fixing unit 107 to have different focal lengths.

[0120] Here, if the focal length of camera 102 is different, a longer focal length amplifies the blur on the screen, thus improving the accuracy of blur detection by feature extraction. On the other hand, because a longer focal length results in greater blur on the screen, the search range for the position of the current image patch that matches the reference image patch is limited, making it difficult to detect blur beyond the search range. Therefore, the blur correction unit 134 of the third embodiment switches between using the blur detection result and the blur estimation result depending on the magnitude of the detected blur.

[0121] (flowchart) Next, the flow of the third embodiment will be described using Figure 8. Figure 8 is a flowchart of the image processing device in the shooting mode of the third embodiment. In the calibration mode of the third embodiment, it is the same as the flowchart of Figure 5A of the first embodiment, except for step S108. In the third embodiment, no camera 102 is set to use the blur estimation result during calibration, and the focal length of the lens of one camera 102 installed on the same camera fixing unit 107 is set to be either telephoto or wide-angle relative to the focal length of the lens of the other camera 102.

[0122] Next, the flow in the shooting mode of the third embodiment will be explained using Figure 8. Figure 8 is a flowchart of the shooting mode of the third embodiment. Image processing devices 103a and 103b perform almost the same processing. Therefore, in the explanation of Figure 8, the processing of one image processing device 103 will be explained, and the explanation of the other similar processing will be omitted or simplified.

[0123] The operation of steps S311 to S316 is the same as steps S121 to S126 in the flow chart of Figure 5B of the first embodiment, so we will omit the explanation.

[0124] In step S317, the blur correction unit 134 of the image processing device 103b determines whether the amount of blur detected in the previous frame is less than (or smaller than) a threshold. The amount of blur detected in the previous frame is the amount of blur indicated by the blur detection result detected in step S313 of the previous frame, assuming that the flow from steps S311 to S323 is processed as one frame. The blur threshold may be set in advance to be smaller than the range used when searching for the position of the patch in the current image that matches the patch in the reference image. If the blur correction unit 134 determines that the amount of blur detected in the previous frame is less than the threshold, the process proceeds to step S318. On the other hand, if the blur correction unit 134 does not determine that the amount of blur detected in the previous frame is less than the threshold, that is, if it determines that the amount of blur detected in the previous frame is greater than or equal to the threshold, the process proceeds to step S321.

[0125] In step S318, the image stabilization unit 134 determines whether the focal length of the lens of camera 102b is longer than the focal length of the other camera 102a, that is, whether it is on the telephoto side or not. If the image stabilization unit 134 determines that it is on the telephoto side, it proceeds to step S319. On the other hand, if the image stabilization unit 134 determines that the focal length is shorter and it is on the wide-angle side, it proceeds to step S320.

[0126] In step S319, the image stabilization unit 134 performs image stabilization using the image stabilization detection result of the image stabilization detection unit 133 of the telephoto camera 102b. This is because, since the amount of blur is small, correcting the blur using the image stabilization detection result of the telephoto camera 102b is more accurate than correcting the blur using the image stabilization estimation result of the image stabilization detection result of the wide-angle camera 102a.

[0127] In step S320, the image stabilization unit 134 performs image stabilization using the image stabilization estimation result. Specifically, the image stabilization unit 134 uses the image stabilization estimation result of the image stabilization unit 135, which is converted from the image stabilization detection result of the other camera 102a on the telephoto side to the image stabilization detection result of its own camera 102b, to perform image stabilization. This is because, since the amount of blur is small, performing image stabilization using the image stabilization estimation result estimated using the image stabilization detection result of the telephoto camera 102a results in better accuracy of image stabilization.

[0128] In step S321, the image stabilization unit 134 determines whether the focal length of the lens of camera 102b is longer than that of camera 102a, i.e., whether it is on the telephoto side. If the image stabilization unit 134 determines that it is on the telephoto side, it proceeds to step S323. On the other hand, if the image stabilization unit 134 determines that the focal length is shorter and it is on the wide-angle side, it proceeds to step S322.

[0129] In step S322, the image stabilization unit 134 performs image stabilization using the image stabilization detection result from the image stabilization detection unit 133 of the wide-angle camera 102b. This is because, due to the large amount of blur, it is difficult to detect blur at the telephoto end, and detecting blur at the wide-angle end provides better accuracy for image stabilization.

[0130] Meanwhile, in step S323, the image stabilization unit 134 performs image stabilization using the image stabilization estimation result. Specifically, the image stabilization unit 134 uses the image stabilization detection result of the other camera 102a on the wide-angle side to convert it to the image stabilization detection result of its own camera 102b, and then performs image stabilization using the image stabilization estimation result. This is because, since the amount of blur is large, image stabilization using image stabilization estimation based on image stabilization detection on the wide-angle side is more accurate.

[0131] In step S324, the image stabilization unit 134 of each image processing device 103 determines whether or not a command to end shooting has been issued from the system control device 105. If the image stabilization unit 134 determines that it has not received a command to end shooting, it returns to step S311 and repeats the above flow for the image of the next frame. On the other hand, if the image stabilization unit 134 determines that it has received a command to end shooting, it terminates the flow.

[0132] As described above, the third embodiment switches between using the shake detection result and shake estimation result of the telephoto camera or the shake detection result and shake estimation result of the wide-angle camera, depending on the amount of shake detected in the previous frame.

[0133] Furthermore, in the configuration of the third embodiment, the blur estimation results may be used as follows. When the blur detection unit 133 performs blur detection, it searches for a position that matches a patch in the reference image from the current image, but the range that the blur detection unit 133 searches may be changed. For example, a normal blur detection unit 133 searches for a position that matches a patch in the reference image, and the search range is a predetermined number of pixels (for example, 30 pixels each) in the horizontal and vertical directions. If the amount of blur exceeds the search range, the blur detection unit 133 cannot detect the blur. Therefore, the blur detection unit 133 may shift the range in which it searches for a patch when detecting blur in the telephoto camera 102, based on the blur estimation results using the blur detection results of the wide-angle camera. For example, suppose the blur estimation results using the blur detection results of the wide-angle camera are 40 pixels in the horizontal direction and 45 pixels in the vertical direction. In this case, the shake detection unit 133 of the telephoto camera 102 may be shifted 40 pixels horizontally and 45 pixels vertically, with the center being a patch search range of 40 pixels in both the horizontal and vertical directions. This allows the shake detection unit 133 to detect large shakes even in the telephoto camera 102.

[0134] (Other embodiments) The embodiments described above may be combined. When embodiments are combined, the image processing system may select which embodiment to correct the blur according to based on pre-set conditions, or the user may select any of the embodiments.

[0135] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, this disclosure can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0136] The disclosures herein include the following image processing systems, image processing methods, and programs. (Item 1) Image acquisition means for acquiring the first and second images captured by the first camera, A feature extraction means for extracting features including a first feature from the first captured image and a second feature from the second captured image, A shake detection means that generates a first shake detection result, which is the result of detecting shake of the first camera by comparing the first feature and the second feature, A shake estimation means generates a shake estimation result, which is the result of estimating the shake of the first camera, using a second shake detection result, which is the result of detecting shake of the second camera, and external parameters including coordinate information of the first camera and the second camera. A shake correction means for correcting the shake based on at least one of the first shake detection result and the shake estimation result, An image processing system characterized by comprising the following features. (Item 2) The first image was captured before the second image. The blur correction means corrects the blur by selecting either the first blur detection result or the blur estimation result based on the result of comparing at least one of the number of features with a preset threshold. The image processing system described in item 1, characterized by the features described herein. (Item 3) The blur correction means corrects the blur using the blur estimation result if at least one of the number of features is less than or equal to a preset threshold. The image processing system described in item 2. (Item 4) The feature extraction means extracts the features of the vertical and horizontal components of the feature, The blur correction means compares the number of features in the vertical component and the number of features in the horizontal component with a preset threshold, and selects either the first blur detection result or the blur estimation result to correct the blur. An image processing system according to any one of items 1 to 3, characterized by the features described above. (Item 5) The aforementioned image stabilization means is If either the number of features in the vertical component or the number of features in the horizontal component is less than or equal to a predetermined threshold, For components where the number of features is equal to or greater than a threshold, the blur is corrected using the first blur detection result. For components where the number of features is less than or equal to the threshold, the deviation is corrected using the deviation estimation results. The image processing system described in item 4, characterized by the features described herein. (Item 6) The blur correction means corrects the blur using the vertical and horizontal vectors of the blur estimation result if both the number of features in the vertical component and the number of features in the horizontal component are less than or equal to a preset threshold. The image processing system according to item 4 or item 5, characterized in that it is the same as described in item 5. (Item 7) The blur correction means determines that blur correction is not possible if the absolute value of the difference between the first blur detection result and the blur estimation result is equal to or greater than a preset threshold. An image processing system according to any one of items 1 to 6, characterized by the features described above. (Item 8) The image stabilization means compares the first focal length of the first camera with the second focal length of the second camera, and selects either the first blur detection result or the blur estimation result to correct the blur. An image processing system according to any one of items 1 to 7, characterized by the features described in item 1 to 7. (Item 9) The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is less than or equal to a preset threshold, If the first focal length is more telephoto than the second focal length, The blur is corrected based on the first blur detection result. The image processing system described in item 8, characterized by the features described above. (Item 10) The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is less than or equal to a preset threshold, When the first focal length is wider angle than the second focal length, The blur is corrected based on the aforementioned blur estimation results. The image processing system according to item 8 or item 9, characterized by the above. (Item 11) The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is greater than or equal to a preset threshold, If the first focal length is more telephoto than the second focal length, The blur is corrected based on the aforementioned blur estimation results. An image processing system according to any one of items 8 to 10, characterized by the features described above. (Item 12) The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is greater than or equal to a preset threshold, When the first focal length is wider angle than the second focal length, The blur is corrected based on the first blur detection result. An image processing system according to any one of items 8 to 11, characterized by the features described above. (Item 13) A calculation means for calculating the external parameters based on captured images acquired from a plurality of cameras, including the first camera and the second camera. An image processing system according to any one of items 1 to 12, characterized by comprising: (Item 14) Front camera 1, The second camera, Fixing means installed on the structure, A first posture adjustment means is installed on the aforementioned fixing means and holds the first camera and adjusts its posture, A second posture adjustment means is installed on the aforementioned fixing means and holds the second camera and adjusts its posture, Equipped with, The relative positions of the first camera and the second camera are fixed. An image processing system according to any one of items 1 to 13, characterized by the features described in item 1 to 13. (Item 15) The blur estimation means estimates the blur by performing a coordinate transformation on the second blur detection result based on the transformation matrix obtained from the poses of the first and second cameras indicated by the coordinate information. An image processing system according to any one of items 1 to 14, characterized by the features described in item 1 to 14. (Item 16) Image acquisition process to acquire the first and second images captured by the first camera, A feature extraction step of extracting features including a first feature from the first captured image and a second feature from the second captured image, A shake detection step that generates a first shake detection result, which is the result of detecting shake of the first camera by comparing the first feature and the second feature, A shake estimation step generates a shake estimation result, which is the result of estimating the shake of the first camera, using a second shake detection result, which is the result of detecting shake of the second camera, and external parameters including coordinate information of the first camera and the second camera. A shake correction step that corrects the shake based on at least one of the first shake detection result and the shake estimation result, An image processing method characterized by having the following features. (Item 17) A program that, when read and executed by a computer, causes the computer to perform each step of the image processing method described in item 16.

[0137] This disclosure is not limited to the embodiments described above, and various modifications and variations are possible. [Explanation of symbols]

[0138] 101...Image processing system, 102...Camera, 103...Image processing device, 104...Image processing server, 105...System control device, 107...Camera fixing unit, 108...Attitude adjustment mechanism, 130...Image acquisition unit, 131...Feature extraction unit, 133...Shake detection unit, 134...Shake correction unit, 135...Shake estimation unit, 151...External parameter calculation unit.

Claims

1. Image acquisition means for acquiring the first and second images captured by the first camera, A feature extraction means for extracting features including a first feature from the first captured image and a second feature from the second captured image, A shake detection means that generates a first shake detection result, which is the result of detecting shake of the first camera by comparing the first feature and the second feature, A shake estimation means generates a shake estimation result, which is the result of estimating the shake of the first camera, using a second shake detection result, which is the result of detecting shake of the second camera, and external parameters including coordinate information of the first camera and the second camera. A shake correction means for correcting the shake based on at least one of the first shake detection result and the shake estimation result, An image processing system characterized by comprising the following features.

2. The first image was taken before the second image. The blur correction means corrects the blur by selecting either the first blur detection result or the blur estimation result based on the result of comparing at least one of the number of features with a preset threshold. The image processing system according to feature 1.

3. The blur correction means corrects the blur using the blur estimation result if at least one of the number of features is less than or equal to a preset threshold. The image processing system according to claim 2.

4. The feature extraction means extracts the features of the vertical and horizontal components of the feature, The blur correction means compares the number of features in the vertical component and the number of features in the horizontal component with a preset threshold, and selects either the first blur detection result or the blur estimation result to correct the blur. The image processing system according to feature 1.

5. The aforementioned image stabilization means is If either the number of features in the vertical component or the number of features in the horizontal component is less than or equal to a predetermined threshold, For components where the number of features is equal to or greater than a threshold, the blur is corrected using the first blur detection result. For components where the number of features is less than or equal to the threshold, the deviation is corrected using the deviation estimation results. The image processing system according to feature 4.

6. The blur correction means corrects the blur using the vertical and horizontal vectors of the blur estimation result if both the number of features in the vertical component and the number of features in the horizontal component are less than or equal to a preset threshold. The image processing system according to feature 4.

7. The shake correction means determines that shake correction is not possible if the absolute value of the difference between the first shake detection result and the shake estimation result is equal to or greater than a preset threshold. The image processing system according to feature 1.

8. The image stabilization means compares the first focal length of the first camera with the second focal length of the second camera, and selects either the first blur detection result or the blur estimation result to correct the blur. The image processing system according to feature 1.

9. The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is less than or equal to a preset threshold, If the first focal length is more telephoto than the second focal length, The blur is corrected based on the first blur detection result. The image processing system according to feature 8.

10. The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is less than or equal to a preset threshold, When the first focal length is wider angle than the second focal length, The blur is corrected based on the aforementioned blur estimation results. The image processing system according to feature 8.

11. The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is greater than or equal to a preset threshold, If the first focal length is more telephoto than the second focal length, The blur is corrected based on the aforementioned blur estimation results. The image processing system according to feature 8.

12. The aforementioned image stabilization means is If the amount of blur indicated by the first blur detection result of the previous frame of the first captured image is greater than or equal to a preset threshold, When the first focal length is wider angle than the second focal length, The blur is corrected based on the first blur detection result. The image processing system according to feature 8.

13. A calculation means for calculating the external parameters based on captured images acquired from a plurality of cameras, including the first camera and the second camera. The image processing system according to claim 1, characterized by comprising the following features.

14. The first camera, The aforementioned second camera, Fixing means installed on the structure, A first posture adjustment means is installed on the aforementioned fixing means and holds the first camera and adjusts its posture, A second posture adjustment means is installed on the aforementioned fixing means and holds the second camera and adjusts its posture, Equipped with, The relative positions of the first camera and the second camera are fixed. The image processing system according to feature 1.

15. The blur estimation means estimates the blur by transforming the second blur detection result based on the transformation matrix obtained from the orientations of the first and second cameras indicated by the coordinate information. The image processing system according to feature 1.

16. Image acquisition process to acquire the first and second images captured by the first camera, A feature extraction step of extracting features including a first feature from the first captured image and a second feature from the second captured image, A shake detection step that generates a first shake detection result, which is the result of detecting shake of the first camera by comparing the first feature and the second feature, A shake estimation step generates a shake estimation result, which is the result of estimating the shake of the first camera, using a second shake detection result, which is the result of detecting shake of the second camera, and external parameters including coordinate information of the first camera and the second camera. A shake correction step that corrects the shake based on at least one of the first shake detection result and the shake estimation result, An image processing method characterized by having the following features.

17. A program that, when read and executed by a computer, causes the computer to perform each step of the image processing method described in claim 16.

Images