Information processing device, information processing method, and program

Abstract

This makes it possible to reduce the processing load involved in generating synchronized images. [Solution] The information processing device acquires a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and which is asynchronous to the first image. It detects feature points of the subject from these images, and further obtains the amount of shift in the feature points through matching based on the feature points. Based on the amount of shift in the feature points, it corrects the second image to a synchronized image that is synchronized with the first image. Here, the information processing device decides whether to perform the correction on the second image using a first correction process or a second correction process which is more accurate and has a higher processing load than the first correction process, based on the amount of shift in the feature points.

Description

Technical Field

[0001] The present invention relates to information processing technology using a plurality of images.

Background Art

[0002] There is a three-dimensional reconstruction system that synthesizes a plurality of images to create a free viewpoint image or a three-dimensional image. In the three-dimensional reconstruction system, it is necessary to synchronize the imaging timings of the plurality of images used for synthesis with high precision. If each image is not synchronized, the synthesized three-dimensional image will suffer from image quality degradation such as blurred contours. Therefore, conventionally, imaging has been performed by synchronizing the times between a plurality of imaging devices to synchronize the imaging timings. Thus, in order to generate a high-quality three-dimensional image, all images need to be synchronized. However, when communication is performed between devices via a wireless network, the delay time due to communication is not always constant, and it may become impossible to completely synchronize the imaging timings. Also, in an environment where the processing time is not constant, such as software processing within a device, it may sometimes become impossible to completely synchronize the imaging timings. Further, a large amount of computing resources are required for the process of generating a three-dimensional image from a plurality of images. If there is a positional shift in each image due to a difference in imaging timings, an even larger amount of computing resources will be required. Therefore, image processing is required to detect the positional shift between a plurality of images after imaging with respect to other images and correct that positional shift.

[0003] On the other hand, Patent Document 1 discloses a technique for performing a transformation process on an image using parameters calculated based on feature points detected between a reference image and a transformation target image in a plurality of non-synchronized cameras. Also, Patent Document 2 discloses a technique for generating high-precision three-dimensional points when the similarity of the fields of view of a plurality of imaging devices is high and generating low-precision three-dimensional points when the similarity is low.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2017-17607 [Patent Document 2] International Publication No. 2019 / 230813 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, the technology disclosed in Patent Document 1 mentioned above may result in a high processing load if the feature points are significantly misaligned. Furthermore, while Patent Document 2 mentioned above can reduce the processing load when feature points are significantly misaligned, the accuracy of the three-dimensional image will be low if the similarity is low.

[0006] Therefore, the present invention aims to reduce the processing load associated with the generation of synchronized images. [Means for solving the problem]

[0007] The information processing apparatus of the present invention includes: an image acquisition means for acquiring a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and asynchronous with the first image; a feature point detection means for detecting feature points of a subject from the image; a matching means for acquiring the amount of displacement of the feature points by matching based on the feature points; and an image correction means for correcting the second image to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, wherein the image correction means determines, based on the amount of displacement of the feature points, whether to perform the correction on the second image by a first correction process or a second correction process which is more accurate and has a higher processing load than the first correction process. [Effects of the Invention]

[0008] According to the present invention, the processing load related to the generation of synchronized images can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of a three-dimensional reconstruction system. [Figure 2] This figure shows an example configuration of an information processing device according to the first embodiment. [Figure 3] This is a conceptual diagram illustrating the feature point matching process. [Figure 4] This is an explanatory diagram of the image correction process performed by the first and second correction units. [Figure 5] This is a flowchart of the information processing according to the first embodiment. [Figure 6] This figure shows an example configuration of an information processing device according to the second embodiment. [Figure 7] This is a flowchart of the information processing according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all of the features described in these embodiments are essential to the solution of the present invention; these features may be combined in any way. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). In addition, in the following embodiments, redundant explanations of the same or similar configurations or processing steps will be omitted.

[0011] Figure 1 shows an overview of the three-dimensional reconstruction system according to this embodiment. The three-dimensional reconstruction system of this embodiment is a system that creates free-viewpoint images and three-dimensional images by combining multiple images captured by multiple imaging devices. As shown in Figure 1, the three-dimensional reconstruction system according to this embodiment is configured to include a plurality of imaging devices (four imaging devices 101 to 104 in the example of Figure 1) and an information processing device 120. The information processing device 120 is configured to include an image synchronization device 100 and an image synthesis device 105.

[0012] The imaging devices 101 to 104 and the image synchronization device 100 of the information processing device 120 communicate information such as image data via a communication network. The image synchronization device 100 performs correction processing on the images received from imaging devices 101 to 104, corresponding to the shift in feature points caused by the difference in imaging timing of each imaging device, and sends the images to the image synthesis device 105. The image synthesis device 105 combines the multiple corrected images received from the image synchronization device 100 to generate, for example, a three-dimensional image. In this embodiment, a three-dimensional reconstruction system comprising imaging devices 101 to 104 and an information processing device 120 including an image synchronization device 100 and an image synthesis device 105 is given as an example, but the system is not limited to this. For example, the imaging devices may be smartphones or tablet terminals equipped with imaging functions. Also, although this embodiment gives an example with four imaging devices, the system is not limited to this, and many more imaging devices may be used. Furthermore, the system is not limited to the example in which the image synchronization device 100 and the image synthesis device 105 are included in the information processing device 120, and the image synchronization device 100 and the image synthesis device 105 may be separate information processing devices. In addition, the functions of the image synchronization device 100 and the image synthesis device 105 may be included in each imaging device.

[0013] <First Embodiment> Figure 2 is a diagram showing an example configuration of the information processing device 120 according to the first embodiment. The image synchronization device 100 in Figure 2(a) is the same image synchronization device 100 shown in Figure 1, and the image synthesis device 105 is the same image synthesis device 105 shown in Figure 1. Figure 2(a) is a diagram showing an example of the internal functional configuration of the image synchronization device 100 in particular. Note that in Figure 2(a), any two of the imaging devices 101 to 104 shown in Figure 1 are shown as the first imaging device 201 and the second imaging device 202.

[0014] The first imaging device 201 and the second imaging device 202 each capture an area that includes the same subject. It is assumed that the first imaging device 201 and the second imaging device 202 perform imaging at asynchronous timings. The first imaging device 201 outputs the captured image to the reference image acquisition unit 203 of the image synchronization device 100, and the second imaging device 202 outputs the captured image to the asynchronous image acquisition unit 204 of the image synchronization device 100.

[0015] The reference image acquisition unit 203 acquires the captured image sent from the first imaging device 201 as a reference image for three-dimensional image generation. The asynchronous image acquisition unit 204 acquires the captured image sent from the second imaging device 202 as an asynchronous image captured at a time asynchronous to the reference image. That is, the image synchronization device 100 uses the captured image from the first imaging device 201 as the reference image for three-dimensional image synthesis, and acquires the captured image from the second imaging device 202 as an asynchronous image synchronized with the reference image. In this embodiment, an example is given in which the captured image from the first imaging device 201 is used as the reference image and the captured image from the second imaging device 202 is used as the asynchronous image, but this is not the only example. For example, the captured image from any other imaging device may be used as the reference image, and the captured image from any other imaging device may be used as the asynchronous image. For example, if the asynchronous image acquisition unit 204 uses an image captured by one of the imaging devices as a reference image, it selects and acquires an image captured by another imaging device that has the timestamp value closest to the timestamp value of the reference image as an asynchronous image. In other words, the second imaging device 202 illustrated in this embodiment is an imaging device that captures an image with the timestamp value closest to the timestamp value of the reference image captured by the first imaging device 201. The reference image acquisition unit 203 outputs the reference image acquired from the first imaging device 201 to the feature point detection unit 205a, and the asynchronous image acquisition unit 204 outputs the asynchronous image acquired from the second imaging device 202 to the feature point detection unit 205b.

[0016] The feature point detection unit 205a detects feature points of a subject or the like from the reference image sent from the reference image acquisition unit 203, and outputs the detected feature point information to the matching unit 206. Similarly, the feature point detection unit 205b detects feature points of a subject or the like from the asynchronous image sent from the asynchronous image acquisition unit 204, and outputs the detected feature point information to the matching unit 206. Since the process of detecting feature points of a subject or the like is an existing process, its description is omitted.

[0017] The matching unit 206 performs feature point matching processing using the feature point information detected by the feature point detection unit 205a from the reference image and the feature point information detected by the feature point detection unit 205b from the asynchronous image, and derives pairs of similar feature points between the reference image and the asynchronous image. Since the feature point matching processing is an existing process, its description is omitted. Further, the matching unit 206 calculates the deviation amount of the similar feature points of the asynchronous image with respect to the feature points of the reference image from the pairs of similar feature points between the reference image and the asynchronous image. That is, the matching unit 206 performs a process of acquiring the deviation amount between similar feature points of the reference image and the asynchronous image as the deviation amount of the feature points due to the deviation of the imaging timings of the reference image and the asynchronous image, and notifies the image correction unit 207 of the deviation amount of the feature points.

[0018] The image correction unit 207 includes a first correction unit 208, a second correction unit 209, and a correction determination unit 210. The image correction unit 207 performs image correction processing on the asynchronous image according to the deviation amount of the feature points detected by the matching unit 206, and generates an image synchronized with the reference image (referred to as a synchronized image). The first correction unit 208 and the second correction unit 209 each take the reference image and the asynchronous image as inputs, and generate a synchronized image synchronized with the reference image by correcting the asynchronous image according to the deviation amount of the feature points between the reference image and the asynchronous image.

[0019] Here, the first correction unit 208 performs a first correction process that generates a synchronized image with a lower processing load and less processing time than the second correction unit 209, but with lower accuracy in correcting the amount of deviation from the reference image. The second correction unit 209 performs a second correction process that generates a synchronized image with a higher processing load and more processing time than the first correction unit 208, but with high accuracy in correcting the amount of deviation from the reference image. Details of the high-processing-load, high-precision image correction processing by the second correction unit 209 and the low-processing-load, low-precision image correction processing by the second correction unit 208 will be described later.

[0020] The correction determination unit 210 determines whether to perform image correction processing using the first correction unit 208 or the second correction unit 209 based on the information of the feature point misalignment detected by the matching unit 206. In this embodiment, if the feature point misalignment is below a predetermined misalignment threshold, the correction determination unit 210 decides to perform image correction using the first correction unit 208, which has low processing load but low correction accuracy. On the other hand, if the feature point misalignment exceeds a predetermined misalignment threshold, the correction determination unit 210 decides to perform image correction using the second correction unit 209, which has high correction accuracy but high processing load.

[0021] Furthermore, the correction determination unit 210 may determine whether to perform image correction processing in the first correction unit 208 or the second correction unit 209, based on the feature point displacement amount as well as the geometric optical system parameters and shooting position of the imaging device. Here, the geometric optical system parameters of the imaging device include the angle of view of the lens and the shutter speed. In the three-dimensional reconstruction system of this embodiment, multiple imaging devices are arranged at different positions, so the positions of the feature points in the reference image and the asynchronous image change depending on the shooting position of each imaging device. For this reason, the correction determination unit 210 determines whether to perform image correction processing in the first correction unit 208 or the second correction unit 209, based on the feature point displacement amount detected by the matching unit 206, the geometric optical system parameters and shooting position of the imaging device. The correction determination unit 210 may use the geometric optical system parameters and shooting position of either the first imaging device 201 or the second imaging device 202, or it may use the geometric optical system parameters and shooting position of both.

[0022] Furthermore, the correction determination unit 210 may determine which image correction process, the first correction unit 208 or the second correction unit 209, to perform, by referring not only to the aforementioned feature point displacement amount, the geometric optical system parameters of the imaging device, and the shooting position, but also to the computational resources of the information processing device 120.

[0023] Figure 2(b) shows an example of the hardware configuration of the information processing device 120 shown in Figure 1. Note that if the image synchronization device 100 and the image synthesis device 105 in Figure 1 are separate devices, these image synchronization device 100 and image synthesis device 105 may each have the hardware configuration shown in Figure 2(b). The CPU (Central Processing Unit) 221 controls various devices connected to the bus 228 and performs information processing related to the information processing device of this embodiment. In the case of the image synchronization device 100, the CPU 221 performs information processing related to each functional unit of the image synchronization device shown in Figure 2(a) above. ROM (Read Only Memory) 222 stores the BIOS program and boot program. RAM (Random Access Memory) 223 is used as the main memory of CPU 221. The large-capacity memory 224 stores the aforementioned images, feature points, images after image correction processing, and the information processing program according to this embodiment. The information processing program stored in the large-capacity memory 224 is loaded into the RAM 223 and executed by the CPU 221. This realizes the information processing by the information processing device 120 of this embodiment. In the case of the image synchronization device 100, the CPU 221 executes the information processing program according to this embodiment to realize the information processing related to each functional unit of the image synchronization device shown in Figure 2(a).

[0024] The input unit 225 is a keyboard, mouse, touch panel, etc., and processes information input from the user. The display unit 226 displays images and various processing results. I / O (Input / Output) 227 is connected to, for example, an imaging device, an external information processing device (not shown), a network, or an external display device, and communicates with them. Bus 228 connects the CPU 221, ROM 222, RAM 223, large-capacity memory 224, input unit 225, display unit 226, and I / O 227 in a manner that enables them to communicate with each other.

[0025] Figure 3 is a conceptual diagram showing an example of the feature point displacement amount of a subject obtained by the matching unit 206 of the image synchronization device 100 shown in Figure 2 through feature point matching processing. In Figure 3, the first imaging device 201 shown in Figure 2 is shown as the first imaging device 301, and the second imaging device 202 in Figure 2 is shown as the second imaging device 302. Also in Figure 3, an example is shown where the feature points are shifted because the imaging timing of the second imaging device 302 is earlier than that of the first imaging device 301. Furthermore, Figure 3(a) shows an example where the feature point displacement amount is small, and Figure 3(b) shows an example where the feature point displacement amount is large.

[0026] In Figure 3(a), image 311 is a reference image captured by the first imaging device 301, and image 312 is an asynchronous image captured by the second imaging device 302. In the case of Figure 3(a), the subject person 320 is stationary and the ball 321 is moving, so the matching unit 206 in Figure 2 derives the feature point displacement amount of the ball 321 as a feature point displacement amount using pairs of similar feature points from the reference image and the asynchronous image with different imaging timings. However, the feature point displacement amount of the ball 321 shown in Figure 3(a) is small even when the imaging timings of the first imaging device 301 and the second imaging device 302 are different.

[0027] On the other hand, in Figure 3(b), image 331 is a reference image captured by the first imaging device 301, and image 332 is an asynchronous image captured by the second imaging device 302. In the example of Figure 3(b), the posture of the subject, person 320, has changed significantly. Here, since the second imaging device 302 captures images earlier than the first imaging device 301, image 332 from the second imaging device 302 is an image of the subject captured earlier in time than image 331 from the first imaging device 301. Therefore, in the example of Figure 3(b), the matching unit 206 in Figure 2 detects a large shift amount due to the change in the posture of person 320 as the amount of feature point shift corresponding to the pair of similar feature points of the reference image and the asynchronous image with different capture timings.

[0028] Figure 4 is a schematic diagram showing an example of image correction processing by the first correction unit 208 and the second correction unit 209 shown in Figure 2. Note that the first correction unit 408 shown in Figure 4(a) is the same as the first correction unit 208 shown in Figure 2, and the second correction unit 409 shown in Figure 4(b) is the same as the second correction unit 209 shown in Figure 2. Also, the captured images 401 and 402 shown in Figure 4(a) are the same as images 311 and 312 in Figure 3(a), and the captured images 411 and 412 shown in Figure 4(b) are the same as images 331 and 332 in Figure 3(a).

[0029] As explained in Figure 3(a), if the feature point misalignment is small, for example, if the feature point misalignment is below a predetermined misalignment threshold, the correction determination unit 210 decides to perform image correction processing by the first correction unit 408 (the first correction unit 208 in Figure 2), as described above. Therefore, as shown in Figure 4(a), image correction is performed by the first correction unit 408, which has a lower processing load and lower correction accuracy than the second correction unit 409, and the corrected synchronized image 410 is generated.

[0030] On the other hand, as explained in Figure 3(b), if the feature point misalignment is large, for example, if the feature point misalignment exceeds a predetermined misalignment threshold, the correction determination unit 210 decides to perform image correction processing by the second correction unit 409 (the second correction unit 209 in Figure 2), as described above. Therefore, as shown in Figure 4(b), image correction is performed by the second correction unit 409, which has a higher processing load and higher correction accuracy than the first correction unit 408, and the corrected synchronized image 419 is generated.

[0031] The image correction algorithms used by the first correction unit 408 and the second correction unit 409 can be existing deformation processing methods, such as linear methods like projection transformation or nonlinear methods like the TPS (Thin-plate spline) method. In image correction processing algorithms using these deformation processing methods, deformation processing is performed on the image based on motion vectors obtained from feature points detected from the image. When linear methods such as projection transformation or nonlinear methods such as the TPS method are used as image correction processing algorithms in the first correction unit 408 and the second correction unit 409, the second correction unit 409 uses a transformation method that is more accurate and has a higher processing load than the first correction unit 408.

[0032] Furthermore, the image correction processing algorithms used by the first correction unit 408 and the second correction unit 409 may also be image processing methods using neural network inference. The neural network according to this embodiment has a learning model that takes a pre-prepared reference image and an asynchronous image as input and outputs a corrected synchronous image. When the first correction unit 408 and the second correction unit 409 use neural network inference as the image correction processing algorithm, the second correction unit 409 uses a neural network with higher accuracy and processing load than the first correction unit 408.

[0033] In this embodiment, as shown in Figure 4, the first correction unit 408 uses a projection transformation process 420 as the image correction processing algorithm, and the second correction unit 409 uses a neural network inference process 421. The neural network inference process 421 used in the second correction unit 409 is a process that enables image correction processing with higher accuracy and processing load than the projection transformation process 420 used in the first correction unit 408.

[0034] Figure 5 is a flowchart showing the information processing flow in the image synchronization device 100 shown in Figure 2 of the information processing device 120 according to this embodiment, from the generation of a synchronized image synchronized with the reference image from the asynchronous image based on the reference image and the asynchronous image. First, in step S501, the reference image acquisition unit 203 acquires and holds the image captured by the first imaging device 201 as a reference image, and the asynchronous image acquisition unit 204 acquires and holds the image captured by the second imaging device 202 as an asynchronous image. Based on the timestamp value assigned to the reference image, the asynchronous image acquisition unit 204 acquires the image with the closest acquisition time, that is, the image with the timestamp value closest to the timestamp value of the reference image, as an asynchronous image. After step S501, the image synchronization device 100 proceeds to step S502, which is performed by the feature point detection unit 205a and the feature point detection unit 205b.

[0035] In step S502, the feature point detection unit 205a performs feature point detection processing on the reference image, and the feature point detection unit 205B performs feature point detection processing on the asynchronous image. After step S502, the image synchronization device 100 proceeds to step S503, which is performed by the matching unit 206.

[0036] In step S503, the matching unit 206 performs feature point matching based on the feature point information output from the feature point detection unit 205a and the feature point detection unit 205b, and detects and stores the amount of feature point misalignment of the asynchronous image relative to the reference image. Any existing method can be used for feature point matching, and it is particularly desirable to use a method that enables detection with sub-pixel accuracy. For example, the matching unit 206 uses methods such as the phase-limited correlation (POC) method, the NCC (normalized cross correlation) method, or the SAD (sum of absolute differences) method, which enable detection with sub-pixel accuracy. After step S503, the image synchronization device 100 proceeds to step S504, which is performed by the correction determination unit 210.

[0037] When the process proceeds to step S504, the correction determination unit 210 determines whether the feature point misalignment detected by the matching unit 206 is less than or equal to a predetermined misalignment threshold. If the correction determination unit 210 determines that the feature point misalignment is less than or equal to the misalignment threshold, the image synchronization device 100 proceeds to step S505, which is performed by the first correction unit 208. On the other hand, if the correction determination unit 210 determines that the feature point misalignment exceeds the misalignment threshold, the image synchronization device 100 proceeds to step S506, which is performed by the second correction unit 209.

[0038] If the process proceeds to step S505, the first correction unit 208 performs image correction on the asynchronous image according to the amount of feature point displacement relative to the reference image. This correction is low in processing load but has low accuracy, thereby generating a synchronized image that is synchronized with the reference image. After that, the image synchronization device 100 completes the processing shown in the flowchart of Figure 5. On the other hand, if the process proceeds to step S506, the second correction unit 209 performs image correction on the asynchronous image according to the amount of feature point displacement relative to the reference image. This correction is more accurate but has a higher processing load than that of the first correction unit 208, and generates a synchronized image that is synchronized with the reference image. After that, the image synchronization device 100 completes the processing shown in the flowchart of Figure 5.

[0039] As mentioned above, the image synchronization device 100 generates a synchronized image by switching between two correction processes with different processing loads based on the amount of feature point misalignment of the subject obtained by feature point matching between the reference image and the asynchronous image. Then, the image synthesis device 105 generates a free-viewpoint image or a three-dimensional image using the reference image and the synchronized image output from the image synchronization device 100. The process of generating free-viewpoint images and three-dimensional images using multiple images is an existing technology, so its explanation will be omitted.

[0040] As described above, the information processing device 120 according to this embodiment can reduce the processing load required for generating a high-precision synchronized image from an asynchronous image by appropriately switching between two correction units with different processing loads to generate a synchronized image from an asynchronous image. As a result, the information processing device 120 of this embodiment can generate high-quality free-viewpoint images and three-dimensional images while reducing the processing required for three-dimensional reconstruction.

[0041] In the embodiments described above, an example was given in which two correction units with different processing loads and accuracy, such as a first correction unit and a second correction unit, are used. However, the system is not limited to two correction units, and a configuration using three or more correction units, each with a different processing load and accuracy, is also possible. Furthermore, in the embodiment described above, the matching unit 206 calculates the feature point misalignment amount using the reference image acquired from the reference image acquisition unit 203 and the asynchronous image acquired from the asynchronous image acquisition unit 204. However, the images used for feature point matching are not limited to these. The asynchronous image acquisition unit 204 may acquire the asynchronous image as the first asynchronous image and another frame that is continuous with the first asynchronous image on the time axis as the second asynchronous image, and the matching unit 206 may calculate the feature point misalignment amount by feature point matching of the first and second asynchronous images.

[0042] <Second Embodiment> In the first embodiment described above, an example was explained in which two image correction units, the first correction unit 208 and the second correction unit 209, are switched according to the amount of feature point displacement. In contrast, the second embodiment describes an example in which a time synchronization function is added to each imaging device, and in the image synchronization device 600, the control unit 613 controls each functional unit, thereby reducing the processing load required for generating synchronized images. Note that the outline of the three-dimensional reconstruction system according to the second embodiment is the same as in Figure 1 described above, and the hardware configuration of the information processing device 120 is also the same as in Figure 2(b), so their illustration and explanation are omitted.

[0043] Figure 6 shows an example configuration of the information processing device 120 according to the second embodiment, where the image synchronization device 600 is the same as the image synchronization device 100 shown in Figure 1, and the image synthesis device 105 is the same as the image synthesis device 105 shown in Figure 1. Figure 6 is a diagram that shows an example of the functional configuration of the image synchronization device 600 in particular according to the second embodiment. In Figure 6, any two of the imaging devices 101 to 104 shown in Figure 1 are shown as the first imaging device 601 and the second imaging device 602. In addition, in the image synchronization device 600, the reference image acquisition unit 603, the asynchronous image acquisition unit 604, the feature point detection unit 605a, the feature point detection unit 605b, the matching unit 606, and the image correction unit 607 are the same as the corresponding functional units in Figure 2(a), so their explanations are omitted.

[0044] The first imaging device 601 and the second imaging device 602 each capture an area that includes the same subject. In the second embodiment, as in the first embodiment described above, the first imaging device 601 outputs the captured image to the reference image acquisition unit 603 of the image synchronization device 600, and the second imaging device 602 outputs the captured image to the asynchronous image acquisition unit 604 of the image synchronization device 600.

[0045] In the second embodiment, the first imaging device 601 has a first synchronization unit 611, and the second imaging device 602 has a second synchronization unit 612. The first synchronization unit 611 and the second synchronization unit 612 each have a clock circuit for synchronizing time with other imaging devices via a network, and an imaging synchronization circuit for aligning imaging timing with other imaging devices based on the time information from the clock circuit. The clock circuit is a hardware counter that manages time by counting up at predetermined intervals. The clock circuit also has a function to correct the counter value in order to synchronize time with other imaging devices via a network. The imaging synchronization circuit controls the imaging timing of the imaging device by triggering imaging when the count value from the clock circuit exceeds a predetermined count threshold. Furthermore, the imaging synchronization circuit adds highly accurate imaging time information to the captured image as metadata by embedding the time represented by the count value from the clock circuit as a timestamp value in the captured image.

[0046] The control unit 613 of the image synchronization device 600 acquires imaging time information, which is attached as metadata to the reference image, from the reference image acquisition unit 603, and also acquires imaging time information, which is attached as metadata to the asynchronous image, from the asynchronous image acquisition unit 204. Based on the imaging time information acquired from the reference image and the asynchronous image, the control unit 613 controls the feature point detection unit 605a, the feature point detection unit 605b, the matching unit 606, and the image correction unit 607.

[0047] For example, if the difference in time between the acquisition times of the reference image and the asynchronous image is less than or equal to a predetermined time threshold, the control unit 613 determines that the acquisition timings of the first imaging device 601 and the second imaging device 602 are synchronized and controls the system to omit the execution of subsequent processes. These subsequent processes include those performed by the feature point detection unit 605a, the feature point detection unit 605b, the matching unit 606, and the image correction unit 607. On the other hand, if the difference in time between the acquisition times of the reference image and the asynchronous image is greater than a predetermined time threshold, the control unit 613 determines that the acquisition timings of the imaging device 601 and the imaging device 602 are not synchronized and controls the system to execute subsequent processes. In other words, in this case, the control unit 613 controls the system to execute the processes performed by the feature point detection unit 605a, the feature point detection unit 605b, the matching unit 606, and the image correction unit 607, similar to the first embodiment described above.

[0048] The predetermined time threshold may be a fixed value, but it may also be changed dynamically as follows. For example, the control unit 613 may detect the movement of a subject using time-series images captured by any imaging device, and may increase the time threshold if the subject's movement is slow. For example, the control unit 613 may increase the time threshold by a predetermined value if the subject's movement is slower than a predetermined movement threshold, or it may increase the time threshold as the subject's movement becomes slower. In other words, if the subject's movement is slow, the amount of feature point displacement of the subject is unlikely to change significantly even if the imaging timing is off, for example, so the control unit 613 may change to increase the time threshold.

[0049] Furthermore, for example, the control unit 613 may change the time threshold according to the field of view of the first imaging device 601 and the second imaging device 602. For example, the control unit 613 may obtain the field of view from the geometric optical system parameters of the imaging device and increase the time threshold if the field of view is greater than a predetermined field of view threshold, or increase the time threshold by a predetermined value as the field of view increases. In other words, when the field of view is large, the amount of feature point displacement of the subject is unlikely to change significantly even if the imaging timing is off, so the control unit 613 may change the time threshold to be larger.

[0050] For example, the control unit 613 may change the time threshold based on the distance from the imaging device to the subject (shooting distance). For example, the control unit 613 may obtain the shooting distance to the subject based on the parallax when two imaging devices positioned differently capture the same subject, and if the shooting distance is greater than a predetermined distance threshold, it may increase the time threshold by a predetermined value. Alternatively, the control unit 613 may increase the time threshold as the shooting distance increases. That is, when the shooting distance is large, the amount of feature point displacement of the subject is unlikely to change significantly even if the imaging timing is off, so the control unit 613 may change the time threshold to be larger.

[0051] In the above explanation, examples were given of changing the time threshold according to the movement of the subject, the field of view of the imaging device, and the shooting distance. However, the control unit 613 may also change the time threshold according to two or more combinations of the movement of the subject, the field of view of the imaging device, and the shooting distance.

[0052] Figure 7 is a flowchart showing the information processing flow in the image synchronization device 600 shown in Figure 6 of the information processing device 120 according to the second embodiment, from the generation of a synchronized image synchronized with the reference image from the asynchronous image based on the reference image and the asynchronous image. In the flowchart of Figure 7, steps S701 to S706 correspond to steps S501 to S506 in Figure 5, and since they are the same processes as described above, their explanations will be omitted. In the second embodiment, after step S701, the processing of the image synchronization device 600 is performed by the control unit 613, proceeding to step S707, and then further to step S708.

[0053] In the flowchart of Figure 7, when the process proceeds to step S707, the control unit 613 acquires the imaging time information attached to the reference image from the reference image acquisition unit 603, and also acquires the imaging time information attached to the asynchronous image from the asynchronous image acquisition unit 204. After step S707, the processing of the control unit 613 proceeds to step S708.

[0054] When the process proceeds to step S708, the control unit 613 determines, based on the imaging time information obtained from the reference image and the asynchronous image in step S707, whether the difference in imaging time between the first imaging device 601 and the second imaging device 602 is greater than a predetermined time threshold. If the control unit 613 determines that the difference in imaging time is greater than the predetermined time threshold, the image synchronization device 600 proceeds to step S702, which is performed by the feature point detection unit 605a and the feature point detection unit 605b. The processing from step S702 onward is the same as the processing from step S502 onward in Figure 5 described above, so its explanation is omitted. On the other hand, if the control unit 613 determines in step S708 that the difference in imaging time is less than or equal to the predetermined time threshold, the process in the flowchart of Figure 7 is terminated. That is, if the difference in imaging time is determined to be less than or equal to the predetermined time threshold, the image synchronization device 600 does not perform the processing related to the feature point detection unit 605a, the feature point detection unit 605b, the matching unit 606, and the image correction unit 607.

[0055] As described above, in the information processing device 120 according to the second embodiment, the imaging time and imaging timing are synchronized in each imaging device. Furthermore, in the information processing device 120 of the second embodiment, the synchronization of the imaging timing is determined based on the imaging time information from each imaging device, and the device controls whether or not to perform the processing from feature point detection to image correction processing described in the first embodiment according to the determination result. As a result, according to the second embodiment, the processing load can be reduced even further than in the first embodiment.

[0056] In the embodiments described above, it was assumed that the imaging devices 101 to 104 were fixed, but the placement and shooting direction of the imaging devices 101 to 104 may be changed. When the placement and shooting direction of the imaging devices 101 to 104 are changed, the coordinates of the real space being captured by each imaging device are associated with the coordinates of the shooting range of each imaging device. This clarifies the correspondence between the imaging devices 101 to 104 and the subject, and even if the placement and shooting direction of the imaging devices 101 to 104 are changed, it is possible to realize information processing that generates a synchronous image from a reference image and an asynchronous image, as described in the embodiments above.

[0057] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0058] A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). It may also include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0059] This embodiment includes the following configurations, methods, and programs. (Composition 1) Image acquisition means for acquiring a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and asynchronous with the first image, A feature point detection means for detecting feature points of a subject from an image, A matching means for obtaining the amount of deviation of the feature points by matching based on the feature points, Image correction means for correcting the second image to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, It has, The image correction means is characterized in that it determines, based on the amount of displacement of the feature points, whether to perform the correction on the second image by a first correction process or a second correction process that is more accurate and has a higher processing load than the first correction process. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the image correction means performs correction by the second correction process on the second image when the amount of displacement of the feature points is greater than a predetermined displacement threshold. (Composition 3) The feature point detection means detects feature points of the subject from the first image and the second image, respectively. The information processing apparatus according to configuration 1 or 2, characterized in that the matching means derives pairs of similar feature points between the first image and the second image by matching the feature points detected from the first image with the feature points detected from the second image, and obtains the amount of shift of the feature points from the pair of feature points. (Composition 4) The image acquisition means acquires the second image captured by the second imaging device and a third image that is continuous with respect to the second image in the time axis. The feature point detection means detects feature points of the subject from the second image and the third image, respectively. The information processing apparatus according to configuration 1 or 2, characterized in that the matching means derives pairs of similar feature points from the second image and the third image by matching the feature points detected from the second image with the feature points detected from the third image, and obtains the amount of shift of the feature points from the pair of feature points. (Composition 5) The information processing apparatus according to any one of configurations 1 to 4, characterized in that the image acquisition means acquires, as the second image, an image from among a plurality of images captured by a plurality of imaging devices different from the first imaging device, which has the timestamp closest to the timestamp assigned to the first image. (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that the image correction means determines whether to perform the correction on the second image by the first correction process or the second correction process, based on at least one of the following: the amount of displacement of the feature points, the geometric optical system parameters and shooting position of the first imaging device, and the geometric optical system parameters and shooting position of the second imaging device. (Composition 7) The first imaging device and the second imaging device are imaging devices that synchronize their imaging timing based on time information to perform imaging. The information processing apparatus according to any one of configurations 1 to 6, further comprising control means for causing the feature point detection means, the matching means, and the image correction means to execute when the time difference between the time of acquisition of the first image by the first imaging device and the time of acquisition of the second image by the second imaging device exceeds a predetermined time threshold. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the control means changes the predetermined time threshold in accordance with at least one of the following: the movement of the subject, the field of view of the first imaging device and the second imaging device, and the shooting distance from the first imaging device and the second imaging device to the subject. (Composition 9) The information processing apparatus according to configuration 8, characterized in that the control means changes the predetermined time threshold by a predetermined value when the movement of the subject is slower than a predetermined motion threshold. (Composition 10) The information processing apparatus according to configuration 8, characterized in that the control means changes the predetermined time threshold as the movement of the subject slows down. (Composition 11) The information processing apparatus according to any one of configurations 8 to 10, characterized in that the control means changes the predetermined time threshold by a predetermined value when the field of view is greater than a predetermined field of view threshold. (Composition 12) The information processing apparatus according to any one of the configurations 8 to 10, characterized in that the control means changes the predetermined time threshold as the field of view increases. (Composition 13) The information processing apparatus according to any one of configurations 8 to 12, characterized in that the control means changes the predetermined time threshold by a predetermined value when the shooting distance is greater than a predetermined distance threshold. (Composition 14) The information processing apparatus according to any one of configurations 8 to 12, characterized in that the control means changes the predetermined time threshold as the shooting distance increases. (Composition 15) The first correction process in the image correction means is a first transformation process that transforms the second image into a synchronized image synchronized with the first image, based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image. The information processing apparatus according to any one of configurations 1 to 14, wherein the first correction process in the image correction means is a process of transforming the second image into a synchronized image synchronized with the first image based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image, and is a transformation process that is more accurate and has a higher processing load than the first transformation process. (Composition 16) The first correction process in the image correction means is a process of transforming the second image into a synchronized image synchronized with the first image based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image. The information processing apparatus according to any one of configurations 1 to 14, characterized in that the second correction process in the image correction means is a neural network inference process that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image. (Composition 17) The first correction process in the image correction means is an inference process of a first neural network that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image. The information processing apparatus according to any one of configurations 1 to 14, characterized in that the second correction process in the image correction means is a neural network inference process that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image, and is a second neural network inference process that is more accurate and has a higher processing load than the first neural network inference process. (Composition 18) The information processing apparatus according to any one of configurations 1 to 17, further comprising an image synthesis means for generating a free-viewpoint image or a three-dimensional image by combining the first image and the second image after correction by the image correction means. (Method 1) An image acquisition step that acquires a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and which is asynchronous with the first image, A feature point detection process for detecting feature points of a subject from an image, A matching step to obtain the amount of deviation of the feature points by matching based on the feature points, An image correction step in which the second image is corrected to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, It has, The image correction step is characterized by determining, based on the amount of displacement of the feature points, whether to perform the correction on the second image by a first correction process or a second correction process that is more accurate and has a higher processing load than the first correction process. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 through 18. [Explanation of Symbols]

[0060] 100: Image synchronization device, 201, 202: Imaging device, 203: Reference image acquisition unit, 204: Asynchronous image acquisition unit, 205a, 205b: Feature point detection unit, 206: Matching unit, 207: Image correction unit, 208: First correction unit, 209: Second correction unit, 210: Correction determination unit

Claims

1. Image acquisition means for acquiring a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and asynchronous with the first image, A feature point detection means for detecting feature points of a subject from an image, A matching means for obtaining the amount of deviation of the feature points by matching based on the feature points, Image correction means for correcting the second image to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, It has, The image correction means is characterized in that it determines, based on the amount of displacement of the feature points, whether to perform the correction on the second image by a first correction process or a second correction process that is more accurate and has a higher processing load than the first correction process.

2. The information processing apparatus according to claim 1, characterized in that the image correction means performs the correction by the second correction process on the second image when the amount of displacement of the feature points is greater than a predetermined displacement threshold.

3. The feature point detection means detects feature points of the subject from the first image and the second image, respectively. The information processing apparatus according to claim 1, wherein the matching means derives pairs of similar feature points between the first image and the second image by matching the feature points detected from the first image and the feature points detected from the second image, and obtains the amount of shift of the feature points from the pair of feature points.

4. The image acquisition means acquires the second image captured by the second imaging device and a third image that is continuous with respect to the second image in the time axis. The feature point detection means detects feature points of the subject from the second image and the third image, respectively. The information processing apparatus according to claim 1, wherein the matching means derives pairs of similar feature points from the second image and the third image by matching the feature points detected from the second image and the feature points detected from the third image, and obtains the amount of shift of the feature points from the pair of feature points.

5. The information processing apparatus according to claim 1, characterized in that the image acquisition means acquires, as the second image, an image from among a plurality of images captured by a plurality of imaging devices different from the first imaging device, which has the timestamp closest to the timestamp assigned to the first image.

6. The information processing apparatus according to claim 1, wherein the image correction means determines whether to perform the correction on the second image by the first correction process or the second correction process, based on at least one of the geometric optical system parameters and shooting position of the first imaging device and the geometric optical system parameters and shooting position of the second imaging device, in addition to the amount of displacement of the feature points.

7. The first imaging device and the second imaging device are imaging devices that synchronize their imaging timing based on time information and perform imaging accordingly. The information processing apparatus according to claim 1, further comprising control means for causing the feature point detection means, the matching means, and the image correction means to be executed when the time difference between the time of acquisition of the first image by the first imaging device and the time of acquisition of the second image by the second imaging device exceeds a predetermined time threshold.

8. The information processing apparatus according to claim 7, characterized in that the control means changes the predetermined time threshold in accordance with at least one of the following: the movement of the subject, the field of view of the first imaging device and the second imaging device, and the shooting distance from the first imaging device and the second imaging device to the subject.

9. The information processing apparatus according to claim 8, characterized in that the control means increases the predetermined time threshold by a predetermined value when the movement of the subject is slower than a predetermined motion threshold.

10. The information processing apparatus according to claim 8, characterized in that the control means changes the predetermined time threshold as the movement of the subject slows down.

11. The information processing apparatus according to claim 8, characterized in that the control means changes the predetermined time threshold by a predetermined value when the field of view is greater than a predetermined field of view threshold.

12. The information processing apparatus according to claim 8, characterized in that the control means changes the predetermined time threshold as the field of view increases.

13. The information processing apparatus according to claim 8, characterized in that the control means changes the predetermined time threshold by a predetermined value when the shooting distance is greater than a predetermined distance threshold.

14. The information processing apparatus according to claim 8, characterized in that the control means changes the predetermined time threshold as the shooting distance increases.

15. The first correction process in the image correction means is a first transformation process that transforms the second image into a synchronized image synchronized with the first image, based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image. The information processing apparatus according to any one of claims 1 to 14, wherein the first correction process in the image correction means is a process of transforming the second image into a synchronized image synchronized with the first image based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image, and is a transformation process that is more accurate and has a higher processing load than the first transformation process.

16. The first correction process in the image correction means is a process of transforming the second image into a synchronized image synchronized with the first image based on a motion vector obtained from feature points detected from the first image and feature points detected from the second image. The information processing apparatus according to any one of claims 1 to 14, characterized in that the second correction process in the image correction means is a neural network inference process that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image.

17. The first correction process in the image correction means is an inference process of a first neural network that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image. The information processing apparatus according to any one of claims 1 to 14, wherein the second correction process in the image correction means is a neural network inference process that takes the first image and the second image as inputs and outputs a synchronized image in which the second image is synchronized with the first image, and is a second neural network inference process that is more accurate and has a higher processing load than the first neural network inference process.

18. The information processing apparatus according to claim 1, further comprising an image synthesis means for generating a free-viewpoint image or a three-dimensional image by combining the first image and the second image after correction by the image correction means.

19. An image acquisition step that acquires a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and which is asynchronous with the first image, A feature point detection process for detecting feature points of a subject from an image, A matching step to obtain the amount of deviation of the feature points by matching based on the feature points, An image correction step in which the second image is corrected to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, It has, The information processing method is characterized in that, in the image correction step, it is determined whether to perform the correction on the second image by a first correction process or a second correction process which is more accurate and has a higher processing load than the first correction process, based on the amount of displacement of the feature points.

20. Computers, Image acquisition means for acquiring a first image captured by a first imaging device and a second image captured by a second imaging device different from the first imaging device and asynchronous with the first image, A feature point detection means for detecting feature points of a subject from an image, A matching means for obtaining the amount of deviation of the feature points by matching based on the feature points, Image correction means for correcting the second image to a synchronized image synchronized with the first image based on the amount of displacement of the feature points, It has, The image correction means is a program that functions as an information processing device that determines, based on the amount of displacement of the feature points, whether to perform the correction on the second image by a first correction process or a second correction process that is more accurate and has a higher processing load than the first correction process.

Images