Imaging processing system, imaging processing device, imaging processing method, and imaging processing program

Abstract

An imaging processing system (100) comprises an imaging device (1) and a video processing device (2). The imaging device has: an imaging unit (10) that acquires an overall image (5); a first image processing unit (11) that associates, with a subject image (5A), a marker image (5B) cut out from the overall image and a two-dimensional pixel coordinate position of a marker (51) in the overall image; and a first information communication unit (13) that transmits, to the video processing device, the subject image, the marker image, and the pixel coordinate position of the marker. The video processing device has: a second information communication unit (20) that receives the subject image, the marker image, and the pixel coordinate position of the marker from the imaging device; a second image processing unit (21) that calculates two-dimensional coordinates of a feature point of the marker in the overall image; and an external parameter estimation unit (23) that estimates an external parameter of the imaging unit using the calculated two-dimensional coordinates of the feature point of the marker and three-dimensional coordinates of the marker in a three-dimensional space.

Description

Photographing processing system, photographing processing device, photographing processing method, and photographing processing program

[0001] The present disclosure relates to an image capture processing system, an image capture processing device, an image capture processing method, and an image capture processing program.

[0002] Conventionally, as disclosed in, for example, Patent Document 1, a photography processing system is known that reconstructs three-dimensional visual information of a subject from an entire image of the subject including a marker captured by a photography device and external parameters of the photography device when the entire image was captured, and generates a free viewpoint video using the three-dimensional visual information. The external parameters of the photography device include, for example, the position and orientation of the photography device. The external parameters of the photography device are estimated using the two-dimensional coordinates of the markers captured in the entire image, the three-dimensional coordinates of the markers in three-dimensional space, and the internal parameters of the photography device. In order to improve the estimation accuracy of the external parameters of the photography device, it is necessary to accurately calculate the two-dimensional coordinates of the markers.

[0003] Japanese Patent Application Laid-Open No. 2023-21368

[0004] However, in the case of the technology disclosed in Patent Document 1, when an entire image including a subject and a marker is captured, changing the image quality of either the subject image or the marker image also changes the image quality of the other image. For example, reducing the subject image to a resolution that matches the communication bandwidth of the network reduces the image quality of the marker image. If the image quality of the marker image is reduced, it may not be possible to accurately calculate two-dimensional coordinates, which may reduce the accuracy of estimating the external parameters of the imaging device. Furthermore, if the image format is changed from raster to vector format, for example, to reduce the data size of the entire image, the image quality of the subject image may be reduced, which may prevent sufficient restoration of three-dimensional visual information.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a photography processing system that can suppress deterioration in the image quality of marker images even when the image quality of the overall image is deteriorated by, for example, setting different image qualities for the subject image and the marker image.

[0006] In order to solve the above-mentioned problems and achieve the object, the photography processing system according to the present disclosure includes a photography device having a function of photographing a subject, and a video processing device that analyzes images photographed by the photography device. The photography device includes: a photography unit that photographs a scene including a subject and at least one marker arranged around the subject to acquire an overall image; a first image processing unit that detects and cuts out a marker image from the overall image acquired by the photography unit, and associates the cut-out marker image and the two-dimensional pixel coordinate position of the marker in the overall image with the subject image; and a first information communication unit that transmits the subject image, marker image, and two-dimensional pixel coordinate position of the marker in the overall image processed by the first image processing unit to the video processing device. The video processing device has a second information communication unit that receives the subject image, the marker image, and the two-dimensional pixel coordinate positions of the marker in the overall image from the imaging device; a second image processing unit that calculates two-dimensional coordinates of feature points of the marker in the overall image based on the marker image and the two-dimensional pixel coordinate positions of the marker in the overall image; and an external parameter estimation unit that estimates external parameters of the imaging unit using the two-dimensional coordinates of the feature points of the marker calculated by the second image processing unit and the three-dimensional coordinates of the marker in three-dimensional space.

[0007] According to the photography processing system of the present disclosure, by setting different image qualities for the subject image and the marker image, it is possible to suppress degradation in the image quality of the marker image even if the image quality of the overall image is degraded.

[0008] FIG. 1 is an overall configuration diagram schematically showing an imaging processing system according to a first embodiment; FIG. 2 is a block diagram showing an imaging processing system according to the first embodiment; FIG. 3 is an explanatory diagram showing an example of processing by a first image processing unit included in an imaging device of the imaging processing system according to the first embodiment; FIG. 4 is a flowchart showing a processing procedure of a first image processing unit included in an imaging device of the imaging processing system according to the first embodiment; FIG. 5 is an explanatory diagram showing an example of processing by an external parameter estimating unit included in a video processing device of the imaging processing system according to the first embodiment;

[0009] Hereinafter, an image capture processing system, an image capture processing device, an image capture processing method, and an image capture processing program according to embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] First Embodiment. Fig. 1 is a schematic diagram illustrating an overall configuration of an image capture processing system 100 according to the first embodiment. The image capture processing system 100 according to the first embodiment reconstructs a three-dimensional shape or three-dimensional visual information of a subject from a plurality of images captured by an image capture device 1, and generates a free-viewpoint video using the three-dimensional visual information, etc. As shown in Fig. 1, the image capture processing system 100 according to the first embodiment includes an image capture device 1, a video processing device 2, and a three-dimensional space reconstruction device 3. As indicated by the white arrows in Fig. 1, the image capture processing system 100 is configured such that processing is performed in the image capture device 1, the video processing device 2, and the three-dimensional space reconstruction device 3 in that order, and the free-viewpoint video generated by the three-dimensional space reconstruction device 3 is displayed on the display device 4. In the image capture processing system 100 according to the first embodiment, the image capture device 1, the video processing device 2, and the three-dimensional space reconstruction device 3 are each configured as a standalone device.

[0011] As shown in Fig. 1, the photographing device 1 is, for example, a camera, and has a function of photographing a subject 50. A plurality of photographing devices 1 are arranged around the subject 50, each installed at a different photographing location. For convenience of illustration, Fig. 1 shows only one of the plurality of photographing devices 1, and the other photographing devices are omitted.

[0012] First, a description will be given of the configuration of the photographing device 1. Fig. 2 is a block diagram showing the photographing processing system 100 according to the first embodiment. As shown in Fig. 2, the photographing device 1 includes a photographing unit 10, a first image processing unit 11, a first marker information holding unit 12, and a first information communication unit 13.

[0013] As shown in FIG. 1 , the image capturing unit 10 captures a scene including a subject 50 and multiple markers 51 arranged around the subject 50 to acquire an entire image 5. Note that for convenience of illustration, FIG. 1 shows only one of the multiple markers 51, and the other markers are omitted. The marker 51 has a feature point and is used to calculate external parameters of the image capturing unit 10, such as its position and orientation. As shown in FIG. 1 , for example, when a rectangular marker 51 is used, the four corners of the periphery can be used as feature points. The multiple markers 51 may be arranged at regular intervals around the subject 50, or may be arranged randomly around the subject 50. For example, the markers 51 may be designated markers such as ArUco markers or QR markers, or may be markers with simple shapes such as linear, circular, star-shaped, or rectangular. The markers 51 may be colored or have lights attached to their feature point corners. The multiple markers 51 may be of the same type and shape or different types and shapes. Furthermore, the markers 51 are not limited to being arranged in multiple numbers around the subject 50, but may be arranged in a single number.

[0014] FIG. 3 is an explanatory diagram illustrating an example of processing by the first image processing unit 11 included in the image capturing device 1 of the image capturing processing system 100 according to the first embodiment. As illustrated in FIG. 3 , the first image processing unit 11 detects a marker 51 from the entire image 5 captured by the image capturing unit 10 and cuts out the marker 51 as a marker image 5B from the entire image 5. The first image processing unit 11 then outputs the two-dimensional pixel coordinate position of the marker 51 in the entire image 5. For example, in the case of a rectangular marker, the two-dimensional pixel coordinate position of the marker 51 is the pixel coordinate of the characteristic point at each of the four corners. The first image processing unit 11 also reduces the entire image 5 to a resolution that matches the communication bandwidth used for communication between the image capturing device 1 and the video processing device 2, thereby generating a subject image 5A. The first image processing unit 11 then associates the cut-out marker image 5B and the two-dimensional pixel coordinate position of the marker 51 in the entire image 5 with the subject image 5A with reduced resolution. As a method of associating the marker image 5B and the subject image 5A, for example, a camera ID (Identifier) ​​that is a camera identifier and the shooting time are added to the header of each image data of the marker image 5B and the subject image 5A. The video processing device 2 determines which marker image 5B and which subject image 5A are associated based on the information on the camera ID and the shooting time.

[0015] Next, an example of the processing performed by the first image processing unit 11 will be described with reference to a flowchart. FIG. 4 is a flowchart illustrating the processing procedure of the first image processing unit 11 included in the image capturing device 1 of the image capturing processing system 100 according to the first embodiment. As illustrated in FIG. 4 , the first image processing unit 11 defines the maximum image size that can be delivered to the video processing device 2 in accordance with the communication bandwidth between the image capturing device 1 and the video processing device 2 (step S101). Next, the first image processing unit 11 receives the entire image 5 captured by the image capturing unit 10 (step S102) and performs a process of detecting a marker 51 from the entire image 5 (step S103). The first image processing unit 11 determines whether the marker 51 has been detected from the entire image 5 (step S104). If it determines that the marker 51 has been detected (step S104: Yes), it performs a process of cropping a marker image 5B from the entire image 5. The first image processing unit 11 also outputs the two-dimensional pixel coordinate position of the marker 51 in the entire image 5 (step S105). On the other hand, if the first image processing unit 11 determines in step S104 that the marker 51 has not been detected (step S104: No), the processing returns to step S102, and a new overall image 5 captured by the imaging unit 10 is input.

[0016] In step S105, the first image processing unit 11 cuts out the marker image 5B from the entire image 5, and then reduces the entire image 5 to the maximum image size defined in step S101 to generate the subject image 5A (step S106). Finally, the first image processing unit 11 associates the cut-out marker image 5B with the two-dimensional pixel coordinate position of the marker 51 in the entire image 5, with the subject image 5A (step S107). This makes it possible to transmit the marker image 5B and the subject image 5A from the imaging device 1 to the video processing device 2 in real time without degrading the image quality of the marker image 5B.

[0017] If the marker image 5B is in a raster format, the first image processing unit 11 may convert the marker image 5B into a vector format. By converting the marker image 5B into a vector format, the amount of data can be reduced, and therefore the amount of communication required when transmitting the marker image 5B from the imaging device 1 to the video processing device 2 can be reduced.

[0018] The first marker information storage unit 12 stores information about the markers 51. The information about the markers 51 includes, for example, the type and shape of the markers 51, and the distances between feature points in three-dimensional space stored by the markers 51. The information about the markers 51 also includes position information in three-dimensional space of a specific feature point in the marker 51 that serves as the origin in three-dimensional space, and of each feature point in each of the other markers 51 relative to the origin marker 51. As described above, the type and shape of the marker 51 include designated markers such as ArUco markers and QR markers, free markers, and the like. The position information about the markers 51 in three-dimensional space includes the three-dimensional coordinates of the feature points of the marker 51 in three-dimensional space and the distances between the multiple markers 51. The distances between the markers 51 are, for example, the distances between the feature points of the marker 51 that serves as the origin and each feature point of the other markers 51. This information is stored in advance in the first marker information storage unit 12 as advance information about the markers 51.

[0019] The first information communication unit 13 transmits the subject image 5A, the marker image 5B, and the two-dimensional pixel coordinate positions of the markers 51 in the entire image 5 processed by the first image processing unit 11 to the video processing device 2 via an external network. The first information communication unit 13 also receives information on the markers 51 held in the video processing device 2 from the video processing device 2. The information on the markers 51 received from the video processing device 2 is stored in the first marker information holding unit 12.

[0020] Next, we will explain the configuration of the video processing device 2. As shown in Fig. 2, the video processing device 2 has a second information communication unit 20, a second image processing unit 21, a second marker information storage unit 22, an external parameter estimation unit 23, and an information data transmission unit 24.

[0021] The second information communication unit 20 communicates with the first information communication unit 13 via an external network, and receives the subject image 5A, the marker image 5B, and the two-dimensional pixel coordinate positions of the markers 51 in the entire image 5. The second information communication unit 20 also transmits information about the markers 51 stored in the second marker information storage unit 22 to the photographing device 1.

[0022] The second image processing unit 21 performs image analysis of the marker image 5B. Specifically, the second image processing unit 21 performs image quality improvement processing on the marker image 5B and detects two-dimensional coordinates of feature points of the marker 51 from the marker image 5B. The image quality improvement processing, for example, is processing to convert the raster-format marker image 5B captured by the image capture unit 10 into vector format. Converting the marker image 5B into vector format can improve the image quality of the marker image 5B. Note that, if the first image processing unit 11 of the image capture device 1 converts the raster-format marker image 5B into vector format, the second image processing unit 21 performs image processing such as enlarging the marker image 5B. Alternatively, the second image processing unit 21 may detect two-dimensional coordinates of feature points of the marker 51 from the marker image 5B without performing image quality improvement processing on the marker image 5B. Note that the image quality improvement processing is not limited to processing to convert the raster-format marker image 5B into vector format, and may be, for example, sharpening processing or super-resolution technology.

[0023] Then, the second image processing unit 21 calculates the two-dimensional coordinates of the feature points of the markers 51 in the overall image 5 based on the two-dimensional coordinates of the feature points of the markers 51 detected from the marker image 5B after image quality enhancement processing and the two-dimensional pixel coordinate position of the markers 51 in the overall image 5. The two-dimensional coordinates of the feature points of the markers 51 in the overall image 5 are the two-dimensional pixel coordinates before the marker image 5B was cut out from the overall image 5. In this way, by performing image quality enhancement processing on the marker image 5B, the two-dimensional coordinates of the feature points of the markers 51 in the overall image 5 can be corrected and accurately calculated, and as a result, the estimation accuracy of the external parameters of the imaging unit 10 can be improved.

[0024] The second marker information storage unit 22 stores information about the marker 51. The information about the marker 51 is the same as the information stored in the first marker information storage unit 12. The first marker information storage unit 12 and the second marker information storage unit 22 are synchronized via the first information communication unit 13 and the second information communication unit 20 and store the same information.

[0025] 5 is an explanatory diagram showing an example of processing by the external parameter estimation unit 23 included in the video processing device 2 of the image capture processing system 100 according to the first embodiment. As shown in FIG. 5 , the external parameter estimation unit 23 estimates external parameters of the image capture unit 10 based on the marker image 5B obtained by image analysis by the second image processing unit 21. Specifically, the external parameter estimation unit 23 estimates the external parameters of the image capture unit 10 using the two-dimensional coordinates of the feature points of the markers 51 in the overall image 5 calculated by the second image processing unit 21, the three-dimensional coordinates of the markers 51 in the three-dimensional space that is the real space, and the internal parameters of the image capture device 1. The internal parameters of the image capture device 1 are, for example, internal features of the image capture device 1, such as focal length, lens distortion, or the coordinates of the image center. Specifically, the external parameter estimation unit 23 estimates the external parameters of the imaging unit 10 by solving a so-called PnP problem (Perspective-n-Point Problem) based on the correspondence between the two-dimensional coordinates (u, v) of the feature point P1 of the marker 51 in the entire image 5 calculated by the second image processing unit 21 and the three-dimensional coordinates (Xw, Yw, Zw) of the feature point P2 of the marker 51 in real space corresponding to the feature point P1. When solving the PnP problem, if the marker is placed two-dimensionally on a plane, at least four feature points corresponding to three-dimensional points in real space and two-dimensional points on the image are required. When solving the PnP problem, if the marker is placed three-dimensionally, at least six feature points corresponding to three-dimensional points in real space and two-dimensional points on the image are required.

[0026] The information data transmitting unit 24 transmits the subject image 5A and the external parameters of the imaging unit 10 estimated by the external parameter estimating unit 23 to the three-dimensional space reconstructing device 3 via an external network.

[0027] The three-dimensional space reconstruction device 3 includes a three-dimensional space model generation unit 30 that uses techniques such as Neural Radiance Fields (NeRF) or volumetric video to reconstruct the subject 50 included in the subject image 5A into three-dimensional visual information and generates a free-viewpoint video using the three-dimensional visual information. The three-dimensional space model generation unit 30 uses external parameters of the imaging unit 10 and the subject image 5A to reconstruct the subject 50 included in the subject image 5A into three-dimensional visual information and generates a free-viewpoint video using the three-dimensional visual information. The free-viewpoint video generated by the three-dimensional space model generation unit 30 is transmitted to the display device 4 via an external network in response to a request from the display device 4 and output by the display device 4. Examples of the display device 4 include a smartphone, a tablet device, VR goggles with a VR (Virtual Reality) image distribution function, and AR (Augmented Reality) glasses. The three-dimensional space reconstruction device 3 can also request information necessary for reconstructing the three-dimensional visual information from the video processing device 2.

[0028] Incidentally, when transmitting the entire image 5 captured by the imaging device 1 to the video processing device 2 in real time, it is necessary to reduce the entire image 5 to a resolution that matches the communication bandwidth of the external network. However, if the entire image 5 including the subject 50 and the marker 51 is reduced, the image quality of the image portion of the marker 51 will be reduced. If the image quality of the image portion of the marker 51 is reduced, it will be impossible to accurately calculate the two-dimensional coordinates of the feature points of the marker 51 from the image, and there is a risk that the estimation accuracy of the external parameters of the imaging unit 10 will be reduced.

[0029] Therefore, in the image capturing and processing system 100 according to the first embodiment, the first image processing unit 11 detects and cuts out the marker image 5B from the entire image 5 captured by the image capturing unit 10, and performs processing to associate the cut-out marker image 5B with the two-dimensional pixel coordinate position of the marker 51 in the entire image 5 with the object image 5A. That is, in the image capturing and processing system 100 according to the first embodiment, by setting different image qualities for the object image 5A and the marker image 5B, degradation of the image quality of the marker image 5B can be suppressed even when the image quality of the entire image 5 is degraded. This allows only the object image 5A to be reduced in accordance with the resolution of the communication bandwidth, and the associated marker image 5B and object image 5A can be transmitted to the video processing device 2 in real time without degrading the image quality of the marker image 5B. As a result, the two-dimensional coordinates of the feature points of the marker 51 in the entire image 5 can be calculated based on the marker image 5B with high image quality, thereby improving the estimation accuracy of the external parameters of the image capturing unit 10.

[0030] Furthermore, in the photography processing system 100 according to the first embodiment, by performing high-quality image processing on only the marker image 5B, it is possible to accurately calculate the two-dimensional coordinates of the feature points of the marker 51 in the overall image 5, thereby improving the estimation accuracy of the external parameters of the photography unit 10.

[0031] The photography processing system 100 according to the first embodiment is not limited to a configuration in which a plurality of photography devices 1 are installed at a photography location around the subject 50, but may be configured such that the photography device 1 is configured as a moving object such as a drone, and the photography device 1 moves around the subject 50 while the photography unit 10 captures an entire image 5 including the marker 51 and the subject 50. In this case, the movement of the photography device 1 may be manually operated or automatically controlled.

[0032] Fig. 6 is a block diagram showing a modified example of the image capture processing system 100 according to the first embodiment. In the image capture processing system 100A shown in Fig. 6, the video processing device 2 includes a three-dimensional space model generation unit 30. The three-dimensional space model generation unit 30 reconstructs the subject 50 into three-dimensional visual information using external parameters of the image capture unit 10 and the subject image 5A, and generates a free-viewpoint video using the three-dimensional visual information. The free-viewpoint video generated by the three-dimensional space model generation unit 30 is transmitted to the display device 4 by the information data transmission unit 24 and output by the display device 4.

[0033] Second Embodiment Next, an image capturing system 101 according to the second embodiment will be described. As described above, the first marker information storage unit 12 and the second marker information storage unit 22 store information on the markers 51 in advance. However, for example, while capturing an image of the subject 50, the positions of the placed markers 51 may shift. Also, new markers 51 may be placed around the subject 50. In such cases, it is necessary to update the position information of the markers 51 in three-dimensional space. Therefore, the image capturing system 101 according to the second embodiment has a configuration capable of calculating the three-dimensional coordinates of the feature points of each placed marker 51 based on the image captured by the image capturing unit 10.

[0034] Fig. 7 is a schematic diagram illustrating an overall configuration of an image capturing system 101 according to the second embodiment. As shown in Fig. 7, the image capturing system 101 according to the second embodiment includes an image capturing device 1, a video processing device 2, and a three-dimensional space reconstruction device 3. In the image capturing system 101 according to the second embodiment, the image capturing device 1 is a moving object such as a drone, and while the image capturing device 1 moves around a subject 50, the image capturing unit 10 captures an image of a scene including the subject 50 and a marker 51, thereby acquiring an entire image 5.

[0035] Fig. 8 is a block diagram showing an image capture processing system 101 according to the second embodiment. As shown in Fig. 8, the image capture device 1 further includes a moving path calculation unit 14 in addition to an image capture unit 10, a first image processing unit 11, a first marker information holding unit 12, and a first information communication unit 13. The image capture unit 10, the first image processing unit 11, the first marker information holding unit 12, and the first information communication unit 13 have the same configurations as those of the image capture processing system 100 according to the first embodiment, and therefore detailed description thereof will be omitted.

[0036] The movement path calculation unit 14 calculates a movement path so that a specific marker 51 among the plurality of markers 51 appears on the photographed screen 10a photographed by the photographing unit 10. The photographing device 1 moves based on the movement path calculated by the movement path calculation unit 14.

[0037] 9 is an explanatory diagram illustrating an example of the function of the movement path calculation unit 14 included in the image capturing device 1 of the image capturing processing system 101 according to the second embodiment. As illustrated in FIG. 9 , in order to accurately estimate the relative distance between the markers 51a and 51b, the movement path calculation unit 14 calculates the movement path of the image capturing device 1 so that the size ratio of the two markers 51a and 51b displayed on the image capturing screen 10a captured by the image capturing unit 10 approaches the size ratio of the actual markers 51a and 51b from a state where the size ratio is different from the size ratio of the actual markers 51a and 51b. FIG. 9A illustrates a state where the size ratio of the two markers 51a and 51b displayed on the image capturing screen 10a is different from the size ratio of the actual markers 51a and 51b. FIG. 9B illustrates a state where the size ratio of the two markers 51a and 51b displayed on the image capturing screen 10a approaches the size ratio of the actual markers 51a and 51b. The size ratio of the actual markers 51a and 51b is stored in advance in the first marker information storage unit 12. By matching the sizes of the markers 51a and 51b in the image captured by the imaging unit 10, it is possible to match the estimation accuracy for each marker 51a and 51b when estimating external parameters for each marker 51a and 51b.

[0038] FIG. 10 is an explanatory diagram showing a method for calculating the three-dimensional distance between multiple markers 51 a and 51 b arranged in a three-dimensional space in the image capture processing system 101 according to the second embodiment. As shown in FIG. 10 , the external parameter estimation unit 23 first estimates external parameters of the image capture unit 10 based on each of the two markers 51 a and 51 b displayed on the image capture screen 10 a captured by the image capture unit 10. For the right-side marker 51 a shown in FIG. 10 , if the corner A0, which is one of the feature points, is set as the origin (0,0,0), the three-dimensional coordinates of the other feature points, corners A1, A2, and A3, are determined from the shape and size of the marker 51 a. This allows the external parameters of the image capture unit 10 to be estimated based on the corner A0 as the origin for the right-side marker 51 a. The external parameters of the photographing unit 10 can be estimated by solving the PnP problem based on the internal parameters of the photographing device 1 and the correspondence between the two-dimensional coordinates of the feature points of the markers 51a and the three-dimensional coordinates between the feature points of the markers 51a.

[0039] 10, the corner B0, which is one of the feature points, is set as the origin (0,0,0), and the three-dimensional coordinates of the other feature points, corners B1, B2, and B3, are calculated. This makes it possible to estimate the external parameters of the imaging unit 10 for the left marker 51b, with corner B0 as the origin.

[0040] The external parameter estimation unit 23 then calculates the three-dimensional distance between the imaging unit 10 and each of the markers 51a and 51b based on the external parameters of the imaging unit 10 for each of the markers 51a and 51b. When the external parameters of the imaging unit 10 for each of the markers 51a and 51b are estimated, a vector v1 of the feature point A0 relative to the imaging unit 10 and a vector v2 of the feature point B0 relative to the imaging unit 10 are obtained. From vector v1 and vector v2, a vector v3 between the feature point A1 and the feature point B1 can be obtained. In this manner, by estimating the external parameters of each of the markers 51a and 51b displayed on the imaging screen 10a captured by the imaging unit 10, the three-dimensional distance between the markers 51a and 51b can be calculated. Once the three-dimensional distance between the markers 51a and 51b can be calculated, the three-dimensional distance of the feature point of each of the other markers 51 relative to the feature point of a specific marker 51 set as the origin among the multiple markers 51 can be determined, and therefore the three-dimensional coordinates of the feature points of each of the other markers 51 can be calculated.

[0041] The movement path calculation unit 14 can calculate the movement path of the imaging device 1 when multiple markers 51 arranged around the subject 50 are located at approximately the same height. This is because if multiple markers 51 are located at different heights by a certain amount or more, it becomes difficult to adjust the size ratio of the multiple markers 51 displayed on the imaging screen 10a of the imaging unit 10. Furthermore, the number of markers 51 displayed on the imaging screen 10a of the imaging unit 10 is not limited to two as shown in FIG. 9 , and may be three or more.

[0042] Next, other functions of the movement path calculation unit 14 will be described. In a conventional photography processing system, for example, a specific marker 51 that appears on the photography screen 10a of the photography unit 10 at a certain time N-a and at a time N that is +a hours after the time N-a may not be present on the photography screen 10a at a time N+b that is +b hours after the time N. This is because the photography unit 10 is unable to capture the marker 51 due to movement of the photography device 1.

[0043] Therefore, in the movement path calculation unit 14 in embodiment 2, when a specific marker 51 that appears on the shooting screen 10a of the shooting unit 10 at a certain time N-a appears on the shooting screen 10a of the shooting unit 10 at a time N after +a hours have elapsed, the movement path calculation unit 14 calculates the movement path of the shooting device 1 and its posture on the movement path so that the same marker 51 will also appear on the shooting screen 10a of the shooting unit 10 at a time N+b after +b hours have elapsed.

[0044] 11 is an explanatory diagram showing an example of other functions of the movement path calculation unit 14 of the image capturing device 1 of the image capturing processing system 101 according to the second embodiment. In Fig. 11, (A) shows the image capturing screen 10a of the image capturing unit 10 at time Na, (B) shows the image capturing screen 10a of the image capturing unit 10 at time N that is +a hours after time Na, and (C) shows the image capturing screen 10a of the image capturing unit 10 at time N+b that is +b hours after time N.

[0045] The movement path calculation unit 14 calculates the amount of change in the position and attitude of the imaging unit 10 when +a time has passed, based on information about the position and attitude of the imaging unit 10 relative to a specific marker 51 that appears on the imaging screen 10a of the imaging unit 10 at time Na, and information about the position and attitude of the imaging unit 10 relative to the same marker 51 that appears on the imaging screen 10a of the imaging unit 10 at time N, which is +a time after time Na, and the movement speed of the imaging unit 10. From the calculated amount of change in the position and attitude of the imaging unit 10, the movement path calculation unit 14 determines the position and attitude of the imaging unit 10 at time N+b, which is +b time after time N, so that the same marker 51 appears on the imaging screen 10a of the imaging unit 10 at time N+b, and calculates the movement path of the imaging device 1 and the attitude on the movement path so that the imaging unit 10 is in that position and attitude at time N+b. As a result, a specific marker 51 that appears on the photographing screen 10a of the photographing unit 10 at a certain time N can be displayed on the photographing screen 10a at time N+b, which is +b time later. Note that time a and time b may be the same length or different lengths. The position and orientation of the photographing unit 10 are the same as the external parameters of the photographing unit 10.

[0046] The image capturing system 101 according to the second embodiment may have a configuration in which the video processing device 2 includes a three-dimensional space model generating unit 30, as shown in FIG.

[0047] As described above, in the photography processing system 101 according to the second embodiment, even if the positions of the placed markers 51 shift or new markers 51 are placed around the subject 50 while photographing the subject 50, for example, it is possible to calculate the three-dimensional coordinates of the feature points of each marker 51 based on the image photographed by the photography unit 10, and to update the accurate position information of the markers 51. The first marker information holding unit 12 and the second marker information holding unit 22 are synchronized via the first information communication unit 13 and the second information communication unit 20, and hold the updated position information of the markers 51.

[0048] Third Embodiment Next, an image capturing and processing device 200 according to the third embodiment will be described. Fig. 12 is a block diagram showing the image capturing and processing device 200 according to the third embodiment. As shown in Fig. 12, the image capturing and processing device 200 according to the third embodiment is configured as a single device that combines some of the functions of the image capturing device 1 described in the first embodiment and some of the functions of the video processing device 2. The image capturing and processing device 200 according to the third embodiment includes an image capturing unit 10, an image processing unit 15, a marker information holding unit 16, an external parameter estimating unit 23, and an information data transmitting unit 24.

[0049] The imaging unit 10 has the same functions as the imaging unit 10 of the first embodiment. The image processing unit 15 has a combined function of the first image processing unit 11 and the second image processing unit 21 of the first embodiment. The marker information holding unit 16 has a combined function of the first marker information holding unit 12 and the second marker information holding unit 22 of the first embodiment. The external parameter estimation unit 23 has a similar function to the external parameter estimation unit 23 of the first embodiment. The information data transmission unit 24 has a similar function to the information data transmission unit 24 of the first embodiment. Although detailed description will be omitted, the imaging processing device 200 of the third embodiment does not include the first information communication unit 13 and the second information communication unit 20 described in the first embodiment, and therefore does not have functions related to the first information communication unit 13 and the second information communication unit 20.

[0050] The image capturing and processing device 200 according to the third embodiment may be configured such that a plurality of image capturing devices 1 are arranged around the subject 50, each installed at a different image capturing location, or the image capturing device 1 is a moving body such as a drone, which moves around the subject 50 and captures a scene including the subject 50 and the markers 51 with the image capturing unit 10 to obtain the entire image 5. When the image capturing and processing device 200 according to the third embodiment is configured as a moving body such as a drone, it may be configured to include the movement path calculation unit 14 described in the second embodiment.

[0051] Fig. 13 is a block diagram showing a modified example of the imaging processing device 200 according to the third embodiment. The imaging processing device 200A shown in Fig. 13 has a configuration including a three-dimensional space model generation unit 30. The three-dimensional space model generation unit 30 reconstructs the subject 50 into three-dimensional visual information using the external parameters of the imaging unit 10 and the subject image 5A, and generates a free-viewpoint video using the three-dimensional visual information. The free-viewpoint video generated by the three-dimensional space model generation unit 30 is transmitted to the display device 4 by the information data transmission unit 24 and output by the display device 4.

[0052] 14 is an explanatory diagram showing an example of the configuration of a computer system that realizes the image capture processing systems 100, 100A, 101 and the image capture processing devices 200, 200A according to this embodiment. The image capture processing systems 100, 100A, 101 and the image capture processing devices 200, 200A according to this embodiment function as the image capture processing systems 100, 100A, 101 and the image capture processing devices 200, 200A by executing a computer program on the computer system that describes the processes in the image capture processing systems 100, 100A, 101 and the image capture processing devices 200, 200A. As shown in FIG. 14 , this computer system includes a processor 300, a memory 301, a storage 302, and a communication device 303, which are connected via a system bus 304.

[0053] The processor 300, memory 301, storage 302, and communication device 303 can transmit and receive information to and from each other via a system bus 304. The processor 300 is an example of a processing circuit and includes one or more of a central processing unit (CPU), a digital signal processor (DSP), and a system large-scale integration (LSI). The memory 301 includes one or more of a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The memory 301 also includes a recording medium on which a computer-readable program is recorded. Such a recording medium includes one or more of a non-volatile or volatile semiconductor memory, a magnetic disk, a flexible memory, an optical disk, a compact disk, and a digital versatile disc (DVD). The memory 301 stores programs to be executed by the processor 300, necessary data obtained during the processing, etc. The memory 301 is also used as a temporary storage area for programs. The communication device 303 is a receiver and transmitter that perform communication processing. Note that the computer system is not limited to the configuration shown in FIG. 14 and may include other components.

[0054] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above configuration, for example, a computer program is installed in storage 302 from a medium such as a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). When the program is executed, the program is read from storage 302 and stored in the main storage area of ​​memory 301. In this state, processor 300 executes the processes of image capture processing systems 100, 100A, 101 and image capture processing devices 200, 200A according to this embodiment in accordance with the program stored in memory 301.

[0055] In the above description, a program describing the processing in the photography processing systems 100, 100A, 101 and the photography processing devices 200, 200A is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this. Depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet via the communication device 303.

[0056] The program in this embodiment causes, for example, a computer system to execute the following steps: acquiring an overall image 5 of a captured scene including a subject 50 and at least one marker 51 placed around the subject 50; detecting and cutting out a marker image 5B from the overall image 5, associating the cut-out marker image 5B and the two-dimensional pixel coordinate position of the marker 51 in the overall image 5 with the subject image 5A, and calculating two-dimensional coordinates of feature points of the marker 51 in the overall image 5 based on the marker image 5B and the two-dimensional pixel coordinate position of the marker 51 in the overall image 5; and estimating external parameters of the imaging unit 10 that captures the overall image 5 using the calculated two-dimensional coordinates of the feature points of the marker 51 and the three-dimensional coordinates of the marker 51 in three-dimensional space.

[0057] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0058] 1 Shooting device, 2 Video processing device, 3 Three-dimensional space reconstruction device, 4 Display device, 5 Overall image, 5A Subject image, 5B Marker image, 10 Shooting unit, 10a Shooting screen, 11 First image processing unit, 12 First marker information storage unit, 13 First information communication unit, 14 Movement path calculation unit, 15 Image processing unit, 16 Marker information storage unit, 20 Second information communication unit, 21 Second image processing unit, 22 Second marker information storage unit, 23 External parameter estimation unit, 24 Information data transmission unit, 30 Three-dimensional space model generation unit, 50 Subject, 51, 51a, 51b Marker, 100, 100A, 101 Shooting processing system, 200, 200A Shooting processing device, 300 Processor, 301 Memory, 302 Storage, 303 Communication device, 304 System bus.

Claims

1. A system comprising: an imaging device having a function of photographing a subject; and a video processing device that analyzes images photographed by the imaging device, wherein the imaging device comprises: an imaging unit that photographs a scene including the subject and at least one marker placed around the subject to obtain an overall image; a first image processing unit that detects and cuts out a marker image from the overall image obtained by the imaging unit, and associates the cut-out marker image with a two-dimensional pixel coordinate position of the marker in the overall image with an image of the subject; and a first information communication unit that transmits the subject image, the marker image, and the two-dimensional pixel coordinate position of the marker in the overall image processed by the first image processing unit to the video processing device, wherein the video processing device comprises: a second information communication unit that receives the subject image, the marker image, and the two-dimensional pixel coordinate position of the marker in the overall image from the imaging device; and a second image processing unit that calculates two-dimensional coordinates of feature points of the marker in the overall image based on the marker image and the two-dimensional pixel coordinate position of the marker in the overall image. an external parameter estimation unit that estimates external parameters of the imaging unit using two-dimensional coordinates of the feature points of the markers calculated by the second image processing unit and three-dimensional coordinates of the markers in three-dimensional space.

2. The photographing processing system of claim 1, characterized in that the first image processing unit generates the subject image by reducing the overall image to a resolution that matches the communication bandwidth used by the first information communication unit and the second information communication unit.

3. The photography processing system according to claim 1 or 2, characterized in that the second image processing unit performs processing to improve the image quality of the cut-out marker image.

4. The photography processing system according to any one of claims 1 to 3, characterized in that the photography device is configured to capture the entire image by photographing the subject with the photography unit while moving around the subject.

5. The photography processing system of claim 4, wherein the external parameter estimation unit estimates external parameters of the photography unit based on each marker for at least two markers photographed by the photography unit, and the second image processing unit calculates the three-dimensional distance between the photography unit and each of the markers based on the estimated external parameters of the photography unit, calculates the three-dimensional distance between each of the markers from the three-dimensional distance between the photography unit and each of the markers, and calculates the three-dimensional coordinates of feature points of multiple markers other than the feature point of a specific marker that serves as the origin, among the multiple markers arranged around the subject, based on the three-dimensional distance between each of the markers.

6. The photography processing system according to claim 4 or 5, characterized in that the photography device further comprises a movement path calculation unit that calculates a movement path so that the specific marker appears on the photography screen photographed by the photography unit, and moves based on the movement path calculated by the movement path calculation unit.

7. The photography processing system of claim 6, wherein at least two or more of the markers are arranged around the subject, and the movement path calculation unit calculates the movement path of the photography device so that the size ratio of the at least two markers displayed on the photography screen photographed by the photography unit approaches the size ratio of the actual markers.

8. The photography processing system according to claim 6 or 7, characterized in that the movement path calculation unit calculates the amount of change in the position and attitude of the photography unit when +a time has passed based on information on the position and attitude of the photography unit relative to a specific marker that appears on the photography screen of the photography unit at time N-a, information on the position and attitude of the photography unit relative to the same marker that appears on the photography screen of the photography unit at time N, which is +a time after time N-a, and the movement speed of the photography unit, and calculates, from the calculated amount of change in the position and attitude of the photography unit, a movement path and an attitude on the movement path so that the same marker will appear on the photography screen of the photography unit at time N+b, which is +b time after time N.

9. The imaging processing system according to any one of claims 1 to 8, characterized in that the video processing device has a three-dimensional space model generation unit that reconstructs three-dimensional visual information of the subject using the subject image and the external parameters of the imaging unit estimated by the external parameter estimation unit.

10. A photography processing system as described in any one of claims 1 to 9, further comprising a three-dimensional space reconstruction device that reconstructs three-dimensional visual information of the subject using the subject image and the external parameters of the photography unit estimated by the external parameter estimation unit.

11. An imaging processing device comprising: an imaging unit that captures an image of a scene including a subject and at least one marker placed around the subject to obtain an overall image; an image processing unit that detects and cuts out a marker image from the overall image obtained by the imaging unit, associates the cut-out marker image and the two-dimensional pixel coordinate position of the marker in the overall image with an image of the subject, and calculates two-dimensional coordinates of feature points of the marker in the overall image based on the marker image and the two-dimensional pixel coordinate position of the marker in the overall image; and an external parameter estimation unit that estimates external parameters of the imaging unit using the two-dimensional coordinates of the feature points of the marker calculated by the image processing unit and the three-dimensional coordinates of the marker in three-dimensional space.

12. The imaging processing device according to claim 11, wherein the image processing unit performs processing to improve the image quality of the extracted marker image.

13. The photographing and processing device according to claim 11 or 12, characterized in that the photographing unit is configured to photograph the subject while moving around the subject to obtain the entire image.

14. The photography processing device described in claim 13, characterized in that the external parameter estimation unit estimates external parameters of the photography unit based on each marker for at least two markers photographed by the photography unit, and the image processing unit calculates the three-dimensional distance between the photography unit and each of the markers based on the estimated external parameters of the photography unit, calculates the three-dimensional distance between each of the markers from the three-dimensional distance between the photography unit and each of the markers, and calculates the three-dimensional coordinates of feature points of multiple markers arranged around the subject with respect to a feature point of a specific marker that serves as the origin based on the three-dimensional distance between each of the markers.

15. An imaging processing device as described in claim 13 or 14, further comprising a movement path calculation unit that calculates a movement path so that the specific marker appears on the imaging screen captured by the imaging unit, and moves based on the movement path calculated by the movement path calculation unit.

16. The photographing processing device according to claim 15, characterized in that at least two or more of the markers are arranged around the subject, and the movement path calculation unit calculates the movement path so that the size ratio of the at least two markers displayed on the photographed screen photographed by the photographing unit approaches the size ratio of the actual markers.

17. The photography processing device described in claim 15 or 16, characterized in that the movement path calculation unit calculates the amount of change in position and attitude of the photography unit when +a time has passed based on information on the position and attitude of the photography unit relative to a specific marker that appears on the photography screen of the photography unit at time N-a, information on the position and attitude of the photography unit relative to the same marker that appears on the photography screen of the photography unit at time N, which is +a time after time N-a, and the movement speed of the photography unit, and calculates, from the calculated amount of change in position and attitude of the photography unit, a movement path and an attitude on the movement path so that the same marker appears on the photography screen of the photography unit at time N+b, which is +b time after time N.

18. A photography processing device as described in any one of claims 11 to 17, further comprising a three-dimensional space model generation unit that restores three-dimensional visual information of the subject using the subject image and the external parameters of the photography unit estimated by the external parameter estimation unit.

19. A photography processing method comprising the steps of: acquiring an overall image of a scene including a subject and at least one marker placed around the subject; detecting and cutting out a marker image from the overall image, associating the cut-out marker image and the two-dimensional pixel coordinate position of the marker in the overall image with the subject image, and calculating two-dimensional coordinates of a feature point of the marker in the overall image based on the marker image and the two-dimensional pixel coordinate position of the marker in the overall image; and estimating external parameters of a photography unit that photographs the overall image using the calculated two-dimensional coordinates of the feature point of the marker and the three-dimensional coordinates of the marker in three-dimensional space.

20. A photography processing program that causes a computer to execute the following steps: acquiring an overall image of a scene including a subject and at least one marker placed around the subject; detecting and cutting out a marker image from the overall image, associating the cut-out marker image and the two-dimensional pixel coordinate position of the marker in the overall image with the subject image, and calculating two-dimensional coordinates of a feature point of the marker in the overall image based on the marker image and the two-dimensional pixel coordinate position of the marker in the overall image; and estimating external parameters of a photography unit that photographs the overall image using the calculated two-dimensional coordinates of the feature point of the marker and the three-dimensional coordinates of the marker in three-dimensional space.

Images