Imaging processing system

Abstract

This imaging processing system (100) is provided with a first mobile body (1), a second mobile body (2), and an image processing device (3). The first mobile body (1) comprises: a first position information reception unit (12) that receives three-dimensional position information from a three-dimensional position information transmission device; and a gyro sensor (13) that acquires the posture of the first mobile body (1). The second mobile body (2) comprises a second imaging unit (20) that images a scene including a subject (60) and a marker (61) of the first mobile body (1) to acquire an overall image (6). The image processing device (3) comprises an external parameter estimation unit (32) that estimates an external parameter of the second imaging unit (20) on the basis of the two-dimensional coordinates of a feature point of the marker (61) included in the overall image (6) acquired by the second imaging unit (20), and the three-dimensional coordinate position and orientation in a world coordinate system of the marker (61) identified by the first position information reception unit (12) and the gyro sensor (13).

Description

Photography and Processing System

[0001] The present disclosure relates to an imaging processing system.

[0002] Conventionally, there has been known a photography processing system that reconstructs three-dimensional visual information of a subject from a full image of the subject including a marker captured by a photography device and external parameters of the photography device when the full image was captured, and generates a free viewpoint video using the three-dimensional visual information. The external parameters of the photography device include, for example, the position and orientation of the photography device. The external parameters of the photography device are estimated using the two-dimensional coordinates of the markers captured in the full image, the three-dimensional coordinates of the markers in three-dimensional space, and internal parameters of the photography device. For example, Patent Document 1 discloses a configuration in which a moving object having a marker is moved within the photography area of ​​a photography device arranged around a subject in order to calibrate the internal parameters of the photography device.

[0003] Patent No. 7122694

[0004] However, while the technology disclosed in Patent Literature 1 can calibrate the internal parameters of the camera, it is configured to fix the camera around the subject and capture the subject and markers. In other words, with the technology disclosed in Patent Literature 1, the subject can only be captured from the location where the camera is installed, making it difficult to capture the subject and markers over a wide area. As a result, there is a risk that the subject image and external parameters of the camera required to generate a free viewpoint video cannot be obtained.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a photography processing system that can photograph a subject and markers necessary to estimate external parameters of the photography unit over a wide area.

[0006] In order to solve the above-mentioned problems and achieve the object, the present disclosure provides an image processing system including at least one first moving body having one or more markers of different shapes attached to its outer surface and moving around a subject, at least one second moving body having the function of photographing the subject while moving around the subject, and an image processing device that estimates extrinsic parameters of the second moving body. The first moving body includes a first position information receiving unit that receives three-dimensional position information from a three-dimensional position information transmitting device that transmits three-dimensional position information of the moving body, and a gyro sensor that acquires its own attitude. The second moving body includes a second image capturing unit that captures an entire image by capturing a scene including the subject and the markers of the first moving body. The image processing device includes an extrinsic parameter estimating unit that estimates the extrinsic parameters of the second image capturing unit based on the two-dimensional coordinates of feature points of the markers included in the entire image captured by the second image capturing unit and the three-dimensional coordinate position and attitude of the marker in a world coordinate system identified by the first position information receiving unit and the gyro sensor.

[0007] The photographing processing system according to the present disclosure has the advantage of being able to photograph a subject and markers necessary for estimating external parameters of the photographing unit over a wide area.

[0008] FIG. 1 is an overall configuration diagram schematically illustrating an imaging processing system according to a first embodiment; FIG. 2 is a block diagram illustrating an imaging processing system according to the first embodiment; FIG. 3 is a flowchart illustrating a processing procedure of an external parameter estimation unit included in an image processing device of the imaging processing system according to the first embodiment; FIG. 4 is an explanatory diagram illustrating an example of processing by the external parameter estimation unit included in the image processing device of the imaging processing system according to the first embodiment; FIG. 5 is a flowchart illustrating a different processing procedure of the external parameter estimation unit included in the image processing device of the imaging processing system according to the first embodiment;

[0009] Hereinafter, an image capture processing system according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] First Embodiment. FIG. 1 is a schematic diagram illustrating an overall configuration of an image capture processing system 100 according to the first embodiment. The image capture processing system 100 according to the first embodiment reconstructs a three-dimensional shape or three-dimensional visual information of a subject 60 from a plurality of captured images of the subject, and generates a free-viewpoint video using the reconstructed three-dimensional visual information, etc. As shown in FIG. 1 , the image capture processing system 100 according to the first embodiment includes a first moving body 1, a second moving body 2, an image processing device 3, and a three-dimensional space reconstruction device 4. As indicated by the white arrows in FIG. 1 , the image capture processing system 100 is configured such that processing is performed in the order of the first moving body 1 and the second moving body 2, the image processing device 3, and the three-dimensional space reconstruction device 4, and the free-viewpoint video generated by the three-dimensional space reconstruction device 4 is displayed on the display device 5. In the image capture processing system 100 according to the first embodiment, the first moving body 1, the second moving body 2, the image processing device 3, and the three-dimensional space reconstruction device 4 are each configured as a standalone device.

[0011] As shown in FIG. 1 , the first moving body 1 is, for example, a drone, and has the function of photographing a subject 60 while moving around the subject 60. The first moving body 1 has one or more markers 61 of different shapes provided on its outer surface. If multiple markers 61 are provided on the outer surface of the first moving body 1, the markers 61 should all have different shapes in order to distinguish between them. The second moving body 2 is, for example, a drone, and has the function of photographing a subject 60 while moving around the subject 60. FIG. 1 shows a configuration having one first moving body 1 and three second moving bodies 2. The number of first moving bodies 1 is not limited to the number shown in the figure and may be two or more. The number of second moving bodies 2 is also not limited to the number shown in the figure and may be one or more. The movement of the first moving body 1 and the second moving body 2 may be manually operated or automatically controlled.

[0012] First, the configurations of the first moving body 1 and the second moving body 2 will be described. Fig. 2 is a block diagram showing an image capture processing system 100 according to the first embodiment. As shown in Fig. 2, the first moving body 1 has a first image capture unit 10, a first data communication unit 11, a first position information receiving unit 12, and a gyro sensor 13. Each second moving body 2 has a second image capture unit 20 and a second data communication unit 21.

[0013] As shown in FIG. 1 , the first image capture unit 10 captures an image of a subject 60. The first data communication unit 11 transmits information acquired by the first moving body 1 to the image processing device 3 via an external network. The first position information receiving unit 12 receives three-dimensional position information from a three-dimensional position information transmission device that transmits the three-dimensional position information of the moving body, and stores the three-dimensional position information in association with a timestamp indicating the date and time of reception. The three-dimensional position information transmission device is used to calculate the three-dimensional coordinates of the first moving body 1 in a world coordinate system, such as a Global Navigation Satellite System (GNSS) satellite or a base station in a Real Time Kinematic (RTK) system that improves the accuracy of signals from GNSS satellites. The first moving body 1 can acquire its three-dimensional coordinate position in a world coordinate system by receiving the three-dimensional position information via the first position information receiving unit 12. The gyro sensor 13 acquires the attitude of the first moving body 1. The first moving body 1 can acquire a three-dimensional coordinate position and attitude in the world coordinate system, which are external parameters of the first imaging unit 10, from the three-dimensional coordinate position acquired by the first position information receiving unit 12 and the attitude acquired by the gyro sensor 13. The first moving body 1 transmits to the image processing device 3 the subject image acquired by the first imaging unit 10, the three-dimensional coordinate position acquired by the first position information receiving unit 12, and the attitude of the first moving body 1 acquired by the gyro sensor 13, in association with an ID (Identifier) ​​of the first moving body 1.

[0014] Furthermore, the markers 61 provided on the outer surface of the first moving body 1 have portions that serve as feature points and are used to calculate the external parameters of the second moving body 2, such as the position and posture. For example, when a rectangular marker 61 is used, the four corners of the outer periphery can be used as feature points, as shown in FIG. 1 . For example, the markers 61 may be designated markers such as ArUco markers or QR markers, or may be markers with simple shapes such as linear, circular, star-shaped, or rectangular. The corners of the markers 61 that serve as feature points may be colored or equipped with lights. The markers 61 provided on multiple first moving bodies 1 may be of the same type and shape, or may be of different types and shapes.

[0015] 2 , each second moving body 2 has a second photographing unit 20 and a second data communication unit 21. The second photographing unit 20 photographs a scene including a subject 60 and a marker 61 provided on the outer surface of the first moving body 1 to obtain an entire image 6, and stores the entire image 6 in association with the photographing time. The second data communication unit 21 transmits information obtained by the second moving body 2 to the image processing device 3 via an external network. Each second moving body 2 transmits the entire image 6 obtained by the second photographing unit 20 to the image processing device 3 in association with the ID of the second moving body 2.

[0016] Next, a description will be given of the configuration of the image processing device 3. As shown in Fig. 2, the image processing device 3 has an information communication unit 30, a marker information storage unit 31, and an external parameter estimation unit 32.

[0017] The information communication unit 30 communicates with the first data communication unit 11 via an external network to receive information acquired by the first mobile object 1. The information communication unit 30 also communicates with the second data communication unit 21 via the external network to receive information acquired by the second mobile object 2. The information communication unit 30 also transmits data processed by the image processing device 3 to the three-dimensional space reconstruction device 4 via the external network.

[0018] The marker information storage unit 31 stores information about the marker 61. The information about the marker 61 is, for example, the type and shape of the marker 61, and the difference in coordinate values ​​between feature points in three-dimensional space stored by the marker 61 in x, y, and z, which are defined as three-dimensional axes of the world coordinate system. As described above, the type and shape of the marker 61 are designated markers such as ArUco markers and QR markers, free markers, etc. This information is stored in advance in the marker information storage unit 31 as prior information about the marker 61.

[0019] The external parameter estimation unit 32 estimates the external parameters of the second imaging unit 20 based on the two-dimensional coordinates of the feature points of the marker 61 included in the entire image 6 acquired by the second imaging unit 20, the three-dimensional coordinate position and orientation of the marker 61 in the world coordinate system identified by the first position information receiving unit 12 and the gyro sensor 13, and the internal parameters and distortion parameters of the second imaging unit 20. Note that the entire image 6 acquired by the second imaging unit 20 and the three-dimensional coordinate position and orientation of the marker 61 acquired by the first position information receiving unit 12 and the gyro sensor 13 need to be acquired at the same time. Furthermore, the information of the marker 61 identified by the first position information receiving unit 12 and the gyro sensor 13 may be composed of, for example, six parameters representing the three-dimensional coordinate values ​​and orientation values ​​of the marker 61, or may be composed of pairs of three-dimensional coordinate values ​​of four or more feature points on the marker 61.

[0020] 3 is a flowchart showing the processing procedure of the external parameter estimation unit 32 included in the image processing device 3 of the image capture system 100 according to the first embodiment. As shown in Fig. 3, the external parameter estimation unit 32 acquires the entire image 6 acquired by the second image capture unit 20 of each second moving body 2 (step S11), and extracts the markers 61 included in the entire image 6 (step S12). Note that the external parameter estimation unit 32 acquires the entire image 6 separately from each second moving body 2 and performs parallel processing.

[0021] Next, the external parameter estimation unit 32 calculates the three-dimensional coordinate position in the world coordinate system of the marker 61 identified by the first position information receiving unit 12 and the gyro sensor 13 (step S13). Finally, the external parameter estimation unit 32 calculates the external parameters of the second image capture unit 20 based on the two-dimensional pixel coordinates of the extracted feature points of the marker 61 in the entire image 6, the three-dimensional coordinate position of the marker 61 in the world coordinate system, and the internal parameters and distortion parameters of the second image capture unit 20 (step S14). The external parameters here are the three-dimensional coordinate position and orientation of the second image capture unit 20 in the world coordinate system.

[0022] Here, a method for calculating the external parameters of the second image capturing unit 20 in the world coordinate system by the external parameter estimation unit 32 will be described. FIG. 4 is an explanatory diagram showing an example of processing by the external parameter estimation unit 32 included in the image processing device 3 of the image capturing processing system 100 according to the first embodiment. As shown in FIG. 4 , the external parameter estimation unit 32 estimates the external parameters of the second image capturing unit 20 using the two-dimensional coordinates of the feature points of the marker 61 in the overall image 6, the three-dimensional coordinates of the marker 61 in three-dimensional space, which is real space, and the internal parameters and distortion parameters of the second image capturing unit 20. Examples of the internal parameters of the second image capturing unit 20 include the focal length or the coordinates of the image center. Examples of the distortion parameters of the second image capturing unit 20 include lens distortion. Specifically, as shown in FIG. 4 , the external parameter estimation unit 32 estimates the external parameters of the second image capturing unit 20 using the two-dimensional coordinates (u, v) of the feature point P1 of the marker 61 in the overall image 6 and the three-dimensional coordinates (X ｗ , Y ｗ , Z ｗThe external parameters of the second imaging unit 20 are estimated by solving a so-called Perspective-n-Point Problem (PnP problem) based on the correspondence between the marker 61 and the 3D points in the real space. When solving the PnP problem, if the marker 61 is placed two-dimensionally on a plane, at least four feature points corresponding to three-dimensional points in the real space and two-dimensional points on the image are required. When solving the PnP problem, if the marker 61 is placed three-dimensionally, at least six feature points corresponding to three-dimensional points in the real space and two-dimensional points on the image are required.

[0023] The processing procedure of the external parameter estimation unit 32 included in the image processing device 3 of the imaging processing system 100 according to the first embodiment is not limited to the above configuration. Fig. 5 is a flowchart showing a different processing procedure of the external parameter estimation unit 32 included in the image processing device 3 of the imaging processing system 100 according to the first embodiment.

[0024] As shown in FIG. 5 , the external parameter estimation unit 32 acquires the entire image 6 acquired by the second image capture unit 20 of the second moving object 2 (step S21) and extracts the markers 61 included in the entire image 6 (step S22). Next, the external parameter estimation unit 32 estimates the external parameters of the second image capture unit 20 when a specific one of the feature points of the marker 61 is set as the origin (step S23). Hereinafter, a coordinate system with a specific one of the feature points of the marker 61 as the origin will be referred to as a marker coordinate system. The external parameters of the second image capture unit 20 in the marker coordinate system can be calculated based on the two-dimensional pixel coordinates of the extracted feature points of the markers 61 in the entire image 6, the coordinate positions of the feature points in the marker coordinate system of the markers 61, and the internal parameters and distortion parameters of the second image capture unit 20.

[0025] Next, the external parameter estimation unit 32 calculates a conversion formula for converting the marker coordinate system into the world coordinate system (step S24) based on the first position information receiving unit 12 and the gyro sensor 13. Finally, using the calculated conversion formula, the external parameters of the second imaging unit 20 in the marker coordinate system are converted into external parameters of the second imaging unit 20 in the world coordinate system (step S25).

[0026] The three-dimensional space reconstruction device 4 includes a three-dimensional space model generation unit 40 that reconstructs the object 60 included in the object image into three-dimensional visual information using techniques such as Neural Radiance Fields (NeRF) or volumetric video, and generates a free-viewpoint video using the reconstructed three-dimensional visual information. The three-dimensional space model generation unit 40 acquires, from the image processing device 3, an object image acquired by the first moving object 1 and external parameters of the first moving object 1 in association with each other. The three-dimensional space model generation unit 40 also acquires, from the image processing device 3, an object image acquired by the second moving object 2 and external parameters of the second moving object 2 in association with each other. The three-dimensional space model generation unit 40 reconstructs the object 60 into three-dimensional visual information based on the information acquired from the image processing device 3, and generates a free-viewpoint video using the reconstructed three-dimensional visual information. The free-viewpoint video generated by the three-dimensional space model generation unit 40 is transmitted to the display device 5 via an external network in response to a request from the display device 5, and is output by the display device 5. Examples of the display device 5 include a smartphone, a tablet terminal, VR (Virtual Reality) goggles with a VR image distribution function, AR (Augmented Reality) glasses, etc. The three-dimensional space reconstruction device 4 can also request information necessary for restoring three-dimensional visual information from the image processing device 3.

[0027] As described above, the image processing system 100 according to the first embodiment includes at least one first moving body 1 having at least one marker 61 of different shapes provided on its outer surface and moving around a subject 60, at least one second moving body 2 having a function of photographing the subject 60 while moving around the subject 60, and an image processing device 3 that estimates external parameters of the second moving body 2. The first moving body 1 includes a first position information receiving unit 12 that receives three-dimensional position information from a three-dimensional position information transmission device and a gyro sensor 13 that acquires its own attitude. The second moving body 2 includes a second imaging unit 20 that acquires an entire image 6 by photographing a scene including the subject 60 and the marker 61 of the first moving body 1. The image processing device 3 includes an external parameter estimation unit 32 that estimates external parameters of the second imaging unit 20 based on two-dimensional coordinates of feature points of the marker 61 included in the entire image 6 acquired by the second imaging unit 20 and the three-dimensional coordinate position and attitude of the marker 61 in the world coordinate system identified by the first position information receiving unit 12 and the gyro sensor 13. Therefore, the photography processing system 100 according to the first embodiment uses the first moving body 1 and the second moving body 2 that move around the subject 60, and therefore can photograph the subject 60 and the markers 61 necessary for estimating the external parameters of the second photography unit 20 over a wide area. Furthermore, since it is not necessary to equip the second moving body 2 with a position information receiving unit that receives three-dimensional position information from the three-dimensional position information transmission device and a gyro sensor in order to acquire the external parameters of the second moving body 2, it is possible to reduce the cost of the equipment.

[0028] FIG. 6 is a block diagram illustrating a first modification of the image capture processing system 100 according to the first embodiment. The image capture processing system 100A illustrated in FIG. 6 is configured such that the first moving body 1 receives and aggregates information acquired by each second moving body 2 via the first data communication unit 11. The first moving body 1 aggregates information acquired by the first moving body 1 and information acquired by each second moving body 2, and transmits the aggregated information to the image processing device 3. The information acquired by the first moving body 1 includes the subject image acquired by the first image capture unit 10, the three-dimensional coordinate position of the first moving body 1 in the world coordinate system acquired by the first position information receiving unit 12, the attitude of the first moving body 1 acquired by the gyro sensor 13, and the ID of the first moving body 1 associated with these. The information acquired by the second moving body 2 includes the entire image 6 acquired by the second image capture unit 20 and the ID of the second moving body 2 associated with the entire image 6.

[0029] FIG. 7 is a block diagram illustrating a second variation of the image capture processing system 100 according to the first embodiment. In the image capture processing system 100B illustrated in FIG. 7 , the image processing device 3 includes a three-dimensional space model generation unit 40. The three-dimensional space model generation unit 40 acquires an object image acquired by a first moving body 1 in association with external parameters of the first moving body 1. The three-dimensional space model generation unit 40 also acquires an object image acquired by a second moving body 2 in association with external parameters of the second moving body 2 from the image processing device 3. The three-dimensional space model generation unit 40 then reconstructs the object 60 into three-dimensional visual information based on the acquired information and generates a free-viewpoint video using the reconstructed three-dimensional visual information. The free-viewpoint video generated by the three-dimensional space model generation unit 40 is transmitted to the display device 5 by the information communication unit 30 and output by the display device 5. The image capture processing system 100B illustrated in FIG. 7 may also be configured such that the first moving body 1 receives and aggregates information acquired by each second moving body 2 via the first data communication unit 11, as illustrated in FIG. 6 .

[0030] FIG. 8 is a block diagram showing a third modification of the image capture processing system 100 according to the first embodiment. The image capture processing system 100C shown in FIG. 8 is configured such that the marker information holding unit 31 and the external parameter estimating unit 32 included in the image processing device 3 in the image capture processing systems 100, 100A, and 100B are provided in each second moving body 2. Each second moving body 2 acquires the three-dimensional coordinate position and orientation in the world coordinate system of the marker 61 identified by the first position information receiving unit 12 and the gyro sensor 13 from the first moving body 1, estimates the external parameters using the external parameter estimating unit 32, and then transmits the subject image and the external parameters associated with each other to the three-dimensional space reconstruction device 4 via the second data communication unit 21. Note that the image capture processing system 100C shown in FIG. 8 may also be configured such that the first moving body 1 receives and aggregates information acquired by each second moving body 2 via the first data communication unit 11, as shown in FIG. 6.

[0031] Furthermore, in the image capturing and processing systems 100, 100A, 100B, and 100C according to the first embodiment, the first moving body 1 has the first image capturing unit 10, but this is not limiting, and the first moving body 1 may not have the first image capturing unit 10. That is, the first moving body 1 may have markers 61 on its outer surface, move around the subject 60, and be used only to estimate the external parameters of the second image capturing unit 20.

[0032] Second Embodiment Next, an image capture processing system 101 according to the second embodiment will be described. FIG. 9 is a block diagram showing the image capture processing system 101 according to the second embodiment. As shown in FIG. 9, in the image capture processing system 101 according to the second embodiment, in addition to the configuration of the first embodiment, each second moving body 2 has a second position information receiving unit 22. The second position information receiving unit 22 receives three-dimensional position information from the three-dimensional position information transmission device and stores the three-dimensional position information in association with a timestamp, which is information on the date and time at which the three-dimensional position information was received. By receiving the three-dimensional position information using the second position information receiving unit 22, each second moving body 2 can acquire its three-dimensional coordinate position in the world coordinate system.

[0033] For example, if the signal received from the three-dimensional position information transmission device by the first position information receiving unit 12 of the first moving body 1 is weak, an error may occur in the three-dimensional coordinate position in the world coordinate system of the first moving body 1. Therefore, the external parameter estimation unit 32 uses the second moving body 2 that received a strong signal from the three-dimensional position information transmission device among the multiple second moving bodies 2, and redefines the information based on the signal received by the first position information receiving unit 12 based on the three-dimensional position information received by the second position information receiving unit 22 and the external parameters of the second imaging unit 20 in the marker coordinate system whose origin is a specific one of the feature points of the marker 61.

[0034] Specifically, first, the external parameter estimation unit 32 estimates the external parameters of the second image capture unit 20 based on the marker 61 in the overall image 6 acquired by the second image capture unit 20 of the second moving body 2 that has received a strong signal from the three-dimensional position information transmission device. The external parameter estimation unit 32 solves the PnP problem using a specific feature point of the marker 61 as the origin, and determines the relative coordinates and relative orientation of the second image capture unit 20 with respect to the first image capture unit 10. The external parameter estimation unit 32 then determines the three-dimensional coordinates of the first image capture unit 10 in the world coordinate system from the relative coordinates of the second image capture unit 20 with respect to the first image capture unit 10 and the three-dimensional coordinate position of the second image capture unit 20 in the world coordinate system based on the three-dimensional position information of the second image capture unit 20, and redefines the three-dimensional coordinate position of the first moving body 1. The external parameter estimation unit 32 estimates the other external parameters of the second moving body 2 based on the redefined three-dimensional coordinate position of the first moving body 1. In addition, in the photography processing system 101 of embodiment 2, when the first moving body 1 receives a strong signal from the three-dimensional position information transmission device, the external parameters of each second moving body 2 are estimated based on the three-dimensional coordinate position of the first moving body 1 in the world coordinate system, as in embodiment 1 above.

[0035] In addition, in the photography processing system 101 of embodiment 2, as shown in Figure 6, the first mobile body 1 may receive and consolidate the information acquired by each second mobile body 2 via the first data communication unit 11.

[0036] 7, the image processing system 101 according to the second embodiment may also be configured such that the image processing device 3 includes a three-dimensional space model generation unit 40. In this case, too, the first mobile object 1 may receive and consolidate information acquired by each second mobile object 2 via the first data communication unit 11.

[0037] 8, the image capturing system 101 according to the second embodiment may also be configured such that the marker information storage unit 31 and the external parameter estimation unit 32 of the image processing device 3 in the image capturing systems 100, 100A, and 100B are provided in each second moving body 2. In this case, too, the first moving body 1 may be configured to receive and aggregate information acquired by each second moving body 2 via the first data communication unit 11.

[0038] Third Embodiment Next, an image capturing system 102 according to a third embodiment will be described. FIG. 10 is a schematic diagram illustrating the overall configuration of the image capturing system 102 according to the third embodiment. As shown in FIG. 10, in the image capturing system 102 according to the third embodiment, one or more markers 61 of different shapes are provided on the outer surface of each second moving body 2, and no markers are provided on the first moving body 1. That is, the first image capturing unit 10 of the first moving body 1 captures the markers 61 of each second moving body 2 to acquire marker images. Note that when multiple markers 61 are provided on the outer surface of the second moving body 2, the shapes of the markers 61 are all different in order to distinguish between them. The other configurations are the same as those of the first embodiment.

[0039] The first moving body 1 transmits to the image processing device 3 the marker image acquired by the first photographing unit 10, the three-dimensional coordinate position acquired by the first position information receiving unit 12, and the attitude of the first moving body 1 acquired by the gyro sensor 13, in association with the ID of the first moving body 1. Each second moving body 2 transmits to the image processing device 3 the subject image acquired by the second photographing unit 20, in association with the ID of each second moving body 2.

[0040] The external parameter estimation unit 32 estimates the external parameters of the second imaging unit 20 based on the two-dimensional coordinates of the feature points of the marker image acquired by the first imaging unit 10 and the three-dimensional coordinate position and attitude of the first imaging unit 10 in the world coordinate system identified by the first position information receiving unit 12 and the gyro sensor 13.

[0041] 11 is a flowchart showing the processing procedure of the external parameter estimation unit 32 included in the image processing device 3 of the image capture processing system 102 according to the third embodiment. As shown in FIG. 11 , the external parameter estimation unit 32 acquires a marker image acquired by the first image capture unit 10 of the first moving body 1 (step S31) and extracts a marker 61 included in the marker image (step S32). Next, the external parameter estimation unit 32 calculates the three-dimensional coordinate position and orientation of the second moving body 2 in a marker coordinate system having a specific one of the feature points of the marker 61 as the origin, based on the three-dimensional coordinate position and orientation of the first moving body 1 in the world coordinate system and the two-dimensional pixel coordinates of the feature points of the marker image (step S33). Once the relative distance between the first moving body 1 and the feature point of the marker image is determined, the three-dimensional coordinate position and orientation in the world coordinate system, which are external parameters of the second image capture unit 20 of the second moving body 2, can be calculated from the three-dimensional coordinate position and orientation in the world coordinate system of the first moving body 1 (step S34). In this way, when the second photographing unit 20 of the second moving body 2 acquires a subject image of the subject 60, the external parameter estimation unit 32 can estimate the external parameters of the second photographing unit 20 by having the first moving body 1 photograph the marker 61 provided on the outer surface of the second moving body 2.

[0042] In addition, in the photography processing system 102 of embodiment 3, as shown in Figure 6, the first mobile body 1 may receive and consolidate the information acquired by each second mobile body 2 via the first data communication unit 11.

[0043] 7, the image processing system 102 according to the third embodiment may also be configured such that the image processing device 3 includes a three-dimensional space model generation unit 40. In this case, too, the first mobile object 1 may receive and consolidate information acquired by each second mobile object 2 via the first data communication unit 11.

[0044] Fig. 12 is a block diagram showing a modified example of the image capture processing system 102 according to the third embodiment. In the image capture processing system 102A shown in Fig. 12, the marker information holding unit 31 and the external parameter estimating unit 32 included in the image processing device 3 in the image capture processing systems 100, 100A, and 100B are provided in the first moving object 1. The first moving object 1 transmits the external parameters of the second image capture unit 20 estimated by the external parameter estimating unit 32 to the three-dimensional space reconstruction device 4 in association with the subject image acquired by the second moving object 2.

[0045] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0046] 1 First moving body, 2 Second moving body, 3 Image processing device, 4 Three-dimensional space reconstruction device, 5 Display device, 6 Whole image, 10 First imaging unit, 11 First data communication unit, 12 First position information receiving unit, 13 Gyro sensor, 20 Second imaging unit, 21 Second data communication unit, 22 Second position information receiving unit, 30 Information communication unit, 31 Marker information storage unit, 32 External parameter estimation unit, 40 Three-dimensional space model generation unit, 60 Subject, 61 Marker, 100, 100A, 100B, 100C, 101, 102, 102A Photography processing system.

Claims

1. A photography processing system comprising: at least one first moving body having one or more markers of different shapes attached to its outer surface and moving around a subject; at least one second moving body having the function of photographing while moving around the subject; and an image processing device that estimates external parameters of the second moving body, wherein the first moving body has a first position information receiving unit that receives three-dimensional position information from a three-dimensional position information transmission device that transmits three-dimensional position information of the moving body, and a gyro sensor that acquires its own attitude; the second moving body has a second photographing unit that photographs a scene including the subject and the markers of the first moving body to acquire an overall image; and the image processing device has an external parameter estimation unit that estimates the external parameters of the second photographing unit based on the two-dimensional coordinates of feature points of the markers included in the overall image acquired by the second photographing unit, and the three-dimensional coordinate position and attitude of the marker in a world coordinate system identified by the first position information receiving unit and the gyro sensor.

2. The photographing processing system described in claim 1, characterized in that the second moving body further comprises a second position information receiving unit that receives three-dimensional position information from the three-dimensional position information transmission device, and the external parameter estimation unit redefines the information based on the three-dimensional position information received by the first position information receiving unit based on the three-dimensional position information received by the second position information receiving unit and the external parameters of the second photographing unit.

3. The photographing processing system according to claim 1 or 2, characterized in that the first moving body further has a first photographing unit that photographs the subject and acquires a subject image.

4. The photographing processing system according to any one of claims 1 to 3, characterized in that the image processing device is provided on the second moving body.

5. An image processing system comprising: at least one first moving body having the function of photographing a subject while moving around; at least one second moving body having one or more markers of different shapes attached to its outer surface and having the function of photographing a subject while moving around; and an image processing device that estimates external parameters of the second moving body, wherein the first moving body has a first imaging unit that photographs the marker of the second moving body to acquire a marker image, a first position information receiving unit that receives three-dimensional position information from a three-dimensional position information transmission device that transmits three-dimensional position information of the moving body, and a gyro sensor that acquires its own attitude; the second moving body has a second imaging unit that photographs the subject to acquire a subject image; and the image processing device has an external parameter estimation unit that estimates the external parameters of the second imaging unit based on the two-dimensional coordinates of feature points of the marker image acquired by the first imaging unit and the external parameters of the first imaging unit identified by the first position information receiving unit and the gyro sensor.

6. The photographing processing system according to claim 5, wherein the image processing device is provided on the first moving body.

7. The photography processing system described in claim 1, 2, 3 or 5, characterized in that the first mobile body further has a first data communication unit that transmits the acquired information to the image processing device, and the second mobile body further has a second data communication unit that transmits the acquired information to the image processing device.

8. The photographing processing system described in claim 1, 2, 3, 5 or 7, characterized in that the first mobile body further has a first data communication unit that transmits the acquired information to the image processing device, the second mobile body further has a second data communication unit that transmits the acquired information to the first mobile body, and the first mobile body receives and aggregates the information acquired by the second mobile body and transmits the aggregated information to the image processing device using the first data communication unit.

9. The photographing processing system described in claim 1, 2, 3, 5, 7 or 8, characterized in that the image processing device has a three-dimensional space model generation unit that restores three-dimensional visual information of the subject using the image of the subject acquired by the second photographing unit and the external parameters of the second photographing unit estimated by the external parameter estimation unit.

10. A photography processing system as described in any one of claims 1 to 8, further comprising a three-dimensional space reconstruction device that reconstructs three-dimensional visual information of the subject using an image of the subject acquired by the second photography unit and the external parameters of the second photography unit estimated by the external parameter estimation unit.

Images