Learning systems, learning methods, imaging devices

Abstract

By ensuring that the training of the three-dimensional reconstruction model is performed appropriately according to the image acquisition scene used for training, it is possible to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the training process. [Solution] The learning system comprises an imaging unit that captures images for training a three-dimensional reconstruction model, the imaging unit being configured to acquire multiple types of captured images as images, a scene information acquisition unit that acquires scene information indicating the type of scene the imaging unit is to capture, a selection processing unit that selects the type of three-dimensional reconstruction model to be used for training and the type of captured image based on the scene information acquired by the scene information acquisition unit, and a learning processing unit that performs training of the three-dimensional reconstruction model according to the model type selected by the selection processing unit using the type of captured image selected by the selection processing unit.

Description

Technical Field

[0001] The present technology relates to a learning system and method for training a three-dimensional reconstruction model, which is an AI model for generating free-viewpoint images, and an imaging device for capturing images used in the training.

Background Art

[0002] For example, there are AI (Artificial Intelligence) models such as NeRF (Neural Radiance Fields) and 3D-GS (3Dimension Gaussian Splatting) that generate free-viewpoint images. Specifically, it is an AI model that performs machine learning using a plurality of captured images of a target subject captured from various viewpoints, and renders a two-dimensional image when viewing the subject from an arbitrary viewpoint and line-of-sight direction. It is also called a three-dimensional reconstruction model.

[0003] Here, the following Non-Patent Document 1 can be cited as a related prior art. This Non-Patent Document 1 discloses a technique for performing three-dimensional reconstruction by Gaussian splatting from a blurred captured image using event information detected by an event sensor such as an EVS (Event-based Vision Sensor). By using the event information in the error calculation during learning, it is possible to improve the accuracy of three-dimensional reconstruction even when the input image during learning is blurred.

Prior Art Documents

Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

[0005] However, if event information is used for error calculation as described in Non-Patent Document 1 above, there is a problem in that the color quality of the rendered image deteriorates when the input image is not blurred. Furthermore, if the input image is not blurred in the first place, (considering the issues mentioned above) there is no need to use event information, and it can be said that error calculations based on event information are being performed unnecessarily.

[0006] This technology was developed in light of the above circumstances, and aims to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the learning process by ensuring that the learning of the three-dimensional reconstruction model is performed appropriately according to the image acquisition scene of the images used for learning. [Means for solving the problem]

[0007] The learning system relating to this technology comprises an imaging unit that captures images for training a three-dimensional reconstruction model, the imaging unit being configured to acquire multiple types of captured images as the images; a scene information acquisition unit that acquires scene information indicating the type of scene to be captured by the imaging unit; a selection processing unit that selects the type of three-dimensional reconstruction model to be used for training and the type of captured image based on the scene information acquired by the scene information acquisition unit; and a learning processing unit that performs training of the three-dimensional reconstruction model according to the model type selected by the selection processing unit using the type of captured image selected by the selection processing unit. This makes it possible to train the three-dimensional reconstruction model appropriately according to the image capture scene used for training. [Brief explanation of the drawing]

[0008] [Figure 1] This figure illustrates an example configuration of a learning system as a first embodiment of this technology. [Figure 2]This is an explanatory diagram of the image acquisition process for obtaining training images for a three-dimensional reconstruction model. [Figure 3] This is a block diagram showing an example configuration of an imaging device as a first embodiment. [Figure 4] This block diagram shows an example of the hardware configuration of the server device included in the learning system. [Figure 5] This is a functional block diagram illustrating the various functions of the imaging device as a first embodiment. [Figure 6] This diagram illustrates the variance of depth values ​​in depth images when the shape of the subject is simple. [Figure 7] This diagram illustrates the variance of depth values ​​in depth images when the shape of the subject is complex. [Figure 8] This diagram shows a comparison of edge images for subjects with simple shapes and subjects with complex shapes. [Figure 9] This is a flowchart of the processing corresponding to the pre-imaging in the first embodiment. [Figure 10] This is a flowchart of the processing corresponding to the imaging in the first embodiment. [Figure 11] This is a flowchart of the processing when using depth in the first embodiment. [Figure 12] This is a flowchart of the processing corresponding to the learning phase in the server device. [Figure 13] This figure shows an example of a distance histogram generated using the dToF method. [Figure 14] This is a functional block diagram illustrating the various functions of the imaging device as a second embodiment. [Figure 15] This is a flowchart of the processing corresponding to the pre-imaging in the second embodiment. [Figure 16] This is a flowchart of the processing corresponding to the imaging in the second embodiment. [Figure 17] This is a flowchart of the processing for handling the use of depth in the second embodiment. [Figure 18]It is a diagram showing a configuration example of a learning system when the server device has a position and orientation estimation unit. [Figure 19] It is a functional block diagram for explaining the functions of an imaging device as a modification example for presenting guide information on moving speed. [Figure 20] It is a diagram showing an example display of guide information. [Figure 21] It is a flowchart diagram of the process corresponding to pre-imaging in the modification example. [Figure 22] It is a flowchart diagram of the process to be executed during main imaging in the modification example.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, referring to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. First Embodiment> (1-1. System Overview) (1-2. Configuration Example of Imaging Device) (1-3. Configuration Example of Server Device) (1-4. Learning Method as an Embodiment) (1-5. Processing Procedure) <2. Second Embodiment> <3. Modification Example> <4. Summary of Embodiments> <5. The Present Technology>[

[0010] <1. First Embodiment> (1-1. System Overview) FIG. is a diagram for explaining a configuration example of a learning system 1 as the first embodiment according to the present technology. The learning system 1 of this embodiment is configured as a system for learning a three-dimensional reconstruction model, which is an AI (Artificial Intelligence) model that generates free-viewpoint images. The three-dimensional reconstruction model is an AI model that renders a two-dimensional image of a subject as seen from an arbitrary viewpoint and line of sight by performing machine learning using multiple captured images of the subject taken from various viewpoints. Examples of three-dimensional reconstruction models include three-dimensional reconstruction models using NeRF (Neural Radiance Fields) and 3D-GS (3-Dimensional Gaussian Splatting).

[0011] In the learning system 1 of this embodiment, the device for obtaining captured images to be used as training images for the three-dimensional reconstruction model and the device for performing training processing on the three-dimensional reconstruction model using the training images are separate devices. Specifically, the learning system 1 of this embodiment includes an imaging device 2 as the former device and a server device 4 as the latter device. These imaging devices 2 and server devices 4 are capable of communicating data with each other via a communication network, such as the Internet, or network NT.

[0012] The imaging device 2 is configured to acquire multiple types of images for use as images for training a three-dimensional reconstruction model. Here, "imaging" in this specification broadly means obtaining image data that captures a subject. The image data referred to here is a general term for data consisting of multiple pixel data, and the pixel data is a broad concept that includes not only data indicating the magnitude of the amount of light received from the subject, but also, for example, distance (depth) to the subject, polarization information of the subject, temperature information, etc. In other words, the "image data" obtained by "imaging" (imaging image data) includes data as a grayscale image that shows information on the magnitude of the amount of light received for each pixel, data as a depth image that shows depth information to the subject for each pixel, data as a polarization image that shows polarization information of incident light for each pixel, and data as a thermal image that shows temperature information for each pixel. Furthermore, the "image data" also includes data as an event image obtained by an event sensor (for example, an EVS (Event-based Vision Sensor)) in which multiple event detection pixels that detect changes in the amount of light received above a predetermined amount as an event are arranged in two dimensions. This event image data can be described as image data that indicates whether or not an event occurred for each pixel, and can be expressed as a type of image data that captures the movement of a subject.

[0013] Here, there are various possible specific configurations for the imaging device 2. For example, it could be a smartphone, tablet terminal, or portable personal computer with imaging capabilities, or a camera-shaped device such as a mirrorless digital camera or compact digital camera. It is not limited to any particular configuration.

[0014] As shown in Figure 2, the imaging device 2 is operated by the user to obtain multiple images of the target subject 3 from multiple viewpoints in order to acquire training images for a three-dimensional reconstruction model. For example, one could perform an operation to capture multiple still images by sequentially operating the shutter (release) while changing the position of the imaging device 2 relative to the subject 3. Alternatively, if the imaging device 2 is configured to acquire video images, one could start recording and then change the position of the imaging device 2 relative to the subject 3. In this case, each frame image in the video image data obtained by imaging would correspond to the multiple captured images described above.

[0015] In Figure 1, the server device 4 has a learning processing unit F41 that performs learning processing on a three-dimensional reconstruction model. The learning processing unit F41 performs learning processing on a three-dimensional reconstruction model using training images captured by the imaging device 2.

[0016] The pre-trained three-dimensional reconstruction model is used for rendering free-viewpoint images. There are two possible methods for rendering free-viewpoint images: one that accepts only the viewpoint and line of sight as specified information, and another that accepts not only the viewpoint and line of sight but also the time. In the latter embodiment, it is possible to specify a time within the acquisition period of the training images, and the rendered image is generated as an image that reproduces the state of movement of subject 3 at the specified time. For example, if subject 3 is a dog and was wagging its tail during the acquisition period, a free-viewpoint image is generated that reproduces the state of movement at the specified time from among the series of tail-wagging movements.

[0017] Furthermore, the display of free-viewpoint images generated using a pre-trained three-dimensional reconstruction model can be performed on the imaging device 2, especially when the imaging device 2 is configured as a smartphone or tablet terminal. In this case, the user specifies the viewpoint and line of sight (and the time information if necessary) by inputting commands to the imaging device 2. In response to this specification, either the server device 4 or the imaging device 2 generates free-viewpoint images using the pre-trained three-dimensional reconstruction model.

[0018] In the learning system 1 shown in Figure 1, the learning process for the three-dimensional reconstruction model is performed on the server device 4 side because, currently, it takes a lot of time to perform the learning process on the imaging device 2 side. However, it is not mandatory for the server device 4 to perform the learning process; it is also possible to adopt a configuration where the learning process is performed on the imaging device 2 side.

[0019] (1-2. Example of imaging device configuration) Figure 3 is a block diagram showing an example of the configuration of the imaging device 2. As shown in the figure, the imaging device 2 comprises an imaging unit 21, a control unit 22, a storage unit 23, an input unit 24, an output unit 25, an IMU (Inertial Measurement Unit) 26, and a communication unit 27. As shown in the figure, each of these units is connected via a bus 28, enabling them to communicate data with one another.

[0020] The imaging unit 21 is an imaging unit that captures images for training a three-dimensional reconstruction model, and is configured to acquire multiple types of captured images. Specifically, the imaging unit 21 in this example has a grayscale image capturing unit 211, an event image capturing unit 212, and a depth image capturing unit 213, as shown in the figure, and is configured to acquire grayscale images, event images, and depth images as captured images.

[0021] The grayscale image acquisition unit 211 has a grayscale sensor 211a and is configured to capture grayscale images. Here, a grayscale sensor refers to a sensor that has a pixel array section comprising a group of grayscale pixels, each having multiple grayscale pixels that detect the amount of light received by a light-receiving element according to a predetermined number of grayscale values. In this example, the grayscale sensor 211a is configured to acquire a color image using RGB as the grayscale image.

[0022] The event image acquisition unit 212 has an event sensor 212a as an EVS and is configured to acquire event images. An event sensor refers to a sensor that has a pixel array section comprising a group of event detection pixels, each having multiple event pixels that detect changes in the amount of light received by a light-receiving element that exceed a predetermined amount as an event. In this example, the event sensor 212a is configured to detect both "positive polarity events," which are increases in the amount of light received, and "negative polarity events," which are decreases in the amount of light received, exceeding a predetermined amount.

[0023] The event sensor 212a generates "event information" for each detected event, which includes at least "event detection address" information indicating the detection address (address indicating the position in the pixel coordinate system) and "event detection time" information indicating the time the event was detected. In this example, since both positive and negative polarity events can be detected, the event information generates information that includes the event detection address and event detection time information mentioned above, as well as "event type information" indicating the type of detected event (whether it is a positive or negative polarity event).

[0024] In this case, the event sensor 212a is defined with a frame period that is tens or hundreds of times shorter than the frame period of the grayscale sensor 211a, and the event sensor 212a outputs "event information" about events detected within one frame period as data of the event image for that frame period.

[0025] The depth image acquisition unit 213 has a depth sensor 213a and is configured to acquire depth images. A depth sensor is a sensor that has a pixel array section in which pixels having light-receiving elements are arranged in two dimensions, and is configured to obtain information indicating the distance (depth) to the subject for each pixel based on the light-receiving signal of the light-receiving elements. In this example, the depth image acquisition unit 213 has a depth sensor 213a that performs distance measurement using the dToF (direct Time of Flight) method, and correspondingly the depth image acquisition unit 213 has a light-emitting unit 213b for illuminating the subject with light for distance measurement.

[0026] As is well known, in the dToF method, the distance to an object is measured based on the time it takes for the reflected light from the object to be received, using the time the ranging light is emitted as a reference. Specifically, in the dToF method, ranging light is pulsed at a very short period, such as several hundred MHz (megahertz), and the distance is measured for each pulse based on the time taken as described above. Such distance measurements for each pulse are performed tens of thousands to hundreds of thousands of times to obtain a single depth image. The dToF method then generates a distance histogram, which is statistical information of these tens of thousands to hundreds of thousands of distance measurements. The distance measured for each pulse will vary due to the influence of noise, etc. Therefore, a histogram showing the frequency of each distance is generated as the distance histogram, and the distance with the highest frequency in this histogram is derived as the final distance measurement result. For clarification, it should be noted that the derivation of the distance measurement result based on the distance histogram described above is performed for each pixel.

[0027] In the imaging device 2, the control unit 22 is configured with a microcomputer equipped with, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU performs various control and calculation processes to realize the operation of the imaging device 2 by executing processing based on programs stored in the ROM and memory unit 23, and programs loaded into the RAM. In particular, the control unit 22 performs various processes to realize the learning method as described later, which will be explained separately.

[0028] The storage unit 23 is composed of a non-volatile storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive), and stores various types of information. For example, the storage unit 23 can be used to store programs for the control unit 22 to execute various processes, or to store images captured by the imaging unit 21.

[0029] The input unit 24 conceptually represents a device for inputting various types of information to the imaging device 2. For example, the input unit 24 could be various types of controls or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. The input unit 24 detects user operations, and the control unit 22 interprets the signals corresponding to the input operations. Furthermore, the input unit 24 may be configured to allow information input via various removable recording media.

[0030] The output unit 25 conceptually represents a device that provides various types of information to the user. For example, it could be a display unit for presenting visual information to the user, or an audio output unit for presenting auditory information. For instance, the display unit could be composed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel. The audio output unit could be composed of a speaker or an amplifier. The display unit in the output unit 25 displays various information on the display screen based on instructions from the control unit 22. In particular, the display unit is capable of displaying various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the control unit 22. Furthermore, the audio output unit in the output unit 25 performs various audio outputs based on instructions from the control unit 22.

[0031] The IMU26 has at least an acceleration sensor and an angular velocity sensor, and detects the three-dimensional inertial motion (translational motion and rotational motion in the orthogonal three-axis directions) of the imaging device 2. Hereafter, the detection information regarding the three-dimensional inertial motion detected by IMU26 will be referred to as "IMU information."

[0032] The communications unit 27 performs communication processing via transmission lines such as the Internet, and communicates with various devices via wired / wireless communication, bus communication, etc. In the learning system 1 of this example, the control unit 22 is able to communicate data with the server device 4 via the communication unit 20.

[0033] In the imaging device 2 configured as described above, the software for processing in this embodiment can be installed, for example, by network communication via the communication unit 27 or by information input via the input unit 24. Alternatively, the software may be stored in advance in the ROM of the control unit 22 or the storage unit 23, etc. The control unit 22 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing for the imaging device 2.

[0034] (1-3. Example of Server Device Configuration) Figure 4 is a block diagram showing an example of the hardware configuration of server device 4. As shown in the figure, the server device 4 is equipped with a processor 41. The processor 41 is configured to have at least a CPU and executes various processes according to programs stored in ROM 42 or programs loaded from storage 419 into RAM 43. In addition to the CPU, the processor 41 may also be configured to have a GPU (Graphics Processing Unit) for the execution of various image processing tasks.

[0035] As shown in the diagram, the aforementioned learning processing unit F41 is located within the processor 41. In other words, the learning process for the three-dimensional reconstruction model is performed by the processor 41.

[0036] RAM 43 also stores data necessary for the processor 41 to perform various processes. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 53. An input / output interface (I / F) 45 is also connected to this bus 53.

[0037] An input section 46, consisting of controls or operating devices, is connected to the input / output interface 45. For example, the input section 46 can be various controls or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. The input unit 46 detects user operations, and the signal corresponding to the input operation is interpreted by the processor 41.

[0038] Furthermore, a display unit 47, such as an LCD or organic EL panel, and an audio output unit 48, such as a speaker, are connected to the input / output interface 45, either as an integrated unit or as separate components. The display unit 47 is used for displaying various types of information and is composed of, for example, a display device provided on the casing of a computer device, or a separate display device connected to a computer device.

[0039] The display unit 47 displays images for various image processing purposes, such as videos to be processed, on the display screen based on instructions from the processor 41. In particular, the display unit 47 displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the processor 41.

[0040] The input / output interface 45 may also be connected to a storage unit 49 consisting of non-volatile memory devices such as SSDs or HDDs, or a communication unit 50 consisting of a modem or the like.

[0041] The communications unit 50 performs communication processing via transmission lines such as the Internet, and communicates with various devices via wired / wireless communication, bus communication, etc.

[0042] A drive 51 is also connected to the input / output interface 45 as needed, and a removable recording medium 52 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted therein.

[0043] The drive 51 can read data (including computer programs, etc.) used for each process from the removable recording medium 52. The read data is stored in the storage unit 49, or, if the read data is image data or audio data, the image or audio is output by the display unit 47 or the audio output unit 48. The computer program read from the removable recording medium 52 is installed in the storage unit 49 as needed.

[0044] Furthermore, the server device 4 is not limited to being a single computer device as shown in Figure 4, but may be configured as a system of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet, etc. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0045] (1-4. Learning Methods as Implementations) Here, with respect to the three-dimensional reconstruction model, improved learning accuracy is required to enhance the accuracy of free-viewpoint image generation, and at the same time, efficiency improvements in learning are needed to reduce the time required to generate free-viewpoint images. In view of these points, the objective of this embodiment is to achieve both improved accuracy in free-viewpoint image generation and efficient learning processing by ensuring that the learning of the three-dimensional reconstruction model is performed appropriately according to the imaging scene of the images used for learning. To achieve the above objective, in the learning system 1 of the embodiment, the control unit 22 in the imaging device 2 has the functions of a scene information acquisition unit F1 and a selection processing unit F2, which will be described below.

[0046] Figure 5 is a functional block diagram illustrating various functions of the control unit 22 of the imaging device 2 as an embodiment. As shown in the figure, the control unit 22 functions as a scene information acquisition unit F1 and a selection processing unit F2. In addition, the control unit 22 in this embodiment also functions as an imaging control unit F3, a position and orientation estimation unit F4, and a deblur (blur removal) processing unit F5.

[0047] The scene information acquisition unit F1 acquires scene information indicating which scene the imaging unit 21 is targeting for imaging. Specifically, in this example, the scene information acquisition unit F1 determines the scene based on the image captured by the imaging unit 21 and acquires scene information.

[0048] Here, imaging for scene determination is performed prior to the imaging of images used to train the three-dimensional reconstruction model. Hereafter, the latter imaging, i.e., the imaging of images used to train the three-dimensional reconstruction model, will be referred to as "main imaging," and the imaging of images for scene determination performed prior to main imaging will be referred to as "pre-imaging." In this example, since capturing at least an event image and a depth image is required to realize scene determination, the imaging unit 21 is configured to capture these event images and depth images during pre-imaging. Specifically, in this example, pre-imaging includes capturing not only event images and depth images, but also grayscale images.

[0049] In this example, the scene information acquisition unit F1 determines whether or not subject 3 is a moving subject as part of the scene determination. In addition, the scene information acquisition unit F1 in this example determines whether or not subject 3 has a complex shape as part of the scene determination. Various methods can be considered for determining whether subject 3 is a moving subject based on the captured image, but in this example, the determination of whether or not it is a moving subject is made based on the event image obtained by the imaging unit 21. For the method of determining whether or not it is a moving subject based on the event image, for example, either of the methods in the following references 1 and 2 can be adopted. ·Reference 1: “Event-based Moving Object Detection and Tracking” Anton Mitrokhin, Cornelia Fermuller, Chethan M Parameshwara, Yiannis Aloimonos:arXiv 2020 / 01 / 12 ·Reference 2: “Event-based Real-time Moving Object Detection Based On IMU Egomotion Compensation” Chunhui Zhao, Yakun Li, Yang Lyu:2023 IEEE International Conference on Robotics and Automation (ICRA) References 1 and 2 disclose techniques for separating egomotion (events associated with camera movement) from subject movement. Specifically, Reference 1 discloses a technique for estimating a warp function (affine transformation) so that events overlap well using warp, and for representing egomotion with the warp function, while Reference 2 discloses a technique for estimating the warp function using angular velocity obtained from an IMU (IMU26 in this example).

[0050] Furthermore, in the scene information acquisition unit F1, the determination of whether or not the subject 3 has a complex shape is made based on the depth image obtained by the imaging unit 21 in this example. Specifically, the depth image is divided into multiple small regions (multiple pixels × multiple pixels), and the variance of the depth value is calculated for each small region. The larger this variance value, the more complex the shape of the subject can be considered to be.

[0051] Figures 6 and 7 illustrate the variance of depth values ​​in sub-regions. Figure 6 shows an example of a depth image when the object's shape is simple, while Figure 7 shows an example when the object's shape is complex. In depth images, the variability in depth values ​​mainly occurs at the edges of the object. In the case of a simple shape as shown in Figure 6, the number of edges included in a sub-region tends to be small, so the variance of depth values ​​in that sub-region is small. On the other hand, in the case of a complex shape like that shown in Figure 7, the number of edges included within a small region tends to increase, resulting in a larger variance of depth values ​​within that small region.

[0052] In this example, the scene information acquisition unit F1 calculates the variance of the depth values ​​for each sub-region in the depth image obtained from pre-imaging, and determines whether the shape of the subject is complex based on the number of sub-regions whose variance is greater than or equal to a threshold. For example, it is conceivable to determine whether the shape is complex by determining whether the number of sub-regions whose variance is greater than or equal to a threshold is greater than or equal to a threshold.

[0053] Alternatively, the determination of whether or not the shape is complex could be based on the grayscale image rather than the depth image. For example, Figure 8 shows examples of edge images for a simple object (Figure 8A) and a complex object (Figure 8B). Comparing these, it is clear that the more complex the shape, the more edges are detected. Therefore, it is conceivable to perform edge detection on the grayscale image and determine whether or not the shape is complex based on the number of detected edges. Specifically, the determination of whether or not the shape is complex could be based on whether or not the number of edges is above a threshold. Furthermore, instead of generating edge images from grayscale images, it is also possible to acquire them by installing edge sensors.

[0054] In Figure 5, the selection processing unit F2 selects the type of three-dimensional reconstruction model to be used for learning and the type of captured image based on the scene information acquired by the scene information acquisition unit F1. In this example, the selection processing unit F2 selects the type of three-dimensional reconstruction model from among multiple three-dimensional reconstruction models that use different types of images for error calculation during training.

[0055] To clarify, the learning of the three-dimensional reconstruction model is performed using the grayscale images obtained in the above-mentioned imaging, that is, multiple grayscale images captured while changing the viewpoint for learning purposes, and information indicating the imaging viewpoint and line of sight for each of those grayscale images, that is, information indicating the position and orientation of the imaging device 2, as learning input data. Therefore, in this embodiment, the three-dimensional reconstruction models considered as candidates for selection by the selection processing unit F2 all share the common feature of using these grayscale images and information indicating the position and orientation of the imaging device 2 for learning. Hereinafter, the multiple grayscale images obtained by this imaging process, which are captured while changing the viewpoint for learning purposes, will be referred to as "learning grayscale images," and the information indicating the position and orientation of the imaging device 2 will be referred to as "position and orientation information."

[0056] With this in mind, we will now explain an example of model selection by the selection processing unit F2 in this example. In this example, if the scene information indicates a moving subject, the selection processing unit F2 selects the type of image to be used for training: a grayscale image and an event image. It also selects the type of three-dimensional reconstruction model that includes a time-deformed field and uses the event image for error calculation. If subject 3 is a moving subject, the generation of the free-viewpoint image is performed in a manner that accepts not only the viewpoint and line of sight direction as described above, but also the time. For this reason, the model type to select in this case is the model type with time deformation, as described above.

[0057] Further details about the three-dimensional reconstruction model that uses event images for error calculation will be explained later.

[0058] Furthermore, if the scene information indicates a complex shape, the selection processing unit F2 selects the type of image to be used for training, specifically a grayscale image and a depth image, and also selects the type of three-dimensional reconstruction model that uses the depth image for error calculation.

[0059] Here, the selection processing unit F2 determines that subject 3 is neither a moving subject nor has a complex shape (indicating that the scene information does not apply to any of these), and selects a type of normal three-dimensional reconstruction model. Specifically, it selects a type of three-dimensional reconstruction model that does not use a time-deformed field and uses only grayscale images for error calculation (only grayscale images among grayscale images, event images, and depth images).

[0060] The imaging control unit F3 controls the imaging unit 21 so that only images of the type selected by the selection processing unit F2 are acquired from among the types of images that the imaging unit 21 can acquire. In other words, the imaging unit 21 is designed to capture only images of the type selected by the selection processing unit F2 in response to the current imaging. Here, if the selection processing unit F2 does not select a depth image as the image to be used for learning (in this example, when the subject is "moving" or when it is neither "moving" nor "complex in shape"), the imaging control unit F3 controls the depth image acquisition unit 213 to a non-operating state during this imaging, and the light emission unit 213b does not emit light.

[0061] The control of the imaging control unit F3 as described above prevents the capture of images that are unnecessary for training during the actual imaging process. Therefore, it is possible to improve the efficiency of learning.

[0062] In this example, the control unit 22 performs imaging control for the current imaging by the imaging control unit F3 as described above, in accordance with the selection by the selection processing unit F2, and also performs the process of transmitting information indicating the type of three-dimensional reconstruction model selected by the selection processing unit F2 to the server device 4. This allows the learning processing unit F41 of the server device 4 to perform learning using an appropriate type of three-dimensional reconstruction model according to the imaging scene.

[0063] The position and orientation estimation unit F4 estimates the position and orientation information of the imaging device 2 (information indicating the viewpoint position and direction of the line of sight during imaging) for each learning grayscale image, based on the grayscale image obtained by the imaging unit 21 through this imaging, i.e., the learning grayscale image described above.

[0064] The position and orientation estimation unit F4 estimates position and orientation information based on training grayscale images using the colmap technique. As is well known, colmap is a technique that estimates position and orientation information based on the results of detecting corresponding points between training grayscale images.

[0065] In this example, the position and orientation estimation unit F4 performs position and orientation information estimation using the depth image when the scene is "complex in shape" and the depth image is selected by the selection processing unit F2. Specifically, it estimates position and orientation information using only the grayscale image and the depth image among the grayscale image, depth image, and event image. More specifically, in this case, the position and orientation estimation unit F4 estimates position and orientation information using the grayscale image, the depth image, and IMU information from IMU26 (IMU information detected during this imaging). At this time, the IMU information is used to resolve the scale uncertainty of the depth image (used when normalizing to a specific scale). Hereafter, the estimation of position and orientation information performed using grayscale images, depth images, and IMU information in this manner will be referred to as "depth-referenced position and orientation estimation." Furthermore, if the scene is "complex in shape" and a depth image is selected by the selection processing unit F2, the position and orientation estimation unit F4 will perform position and orientation information estimation based on IMU information (IMU information detected during this imaging) as a reference, separately from the depth reference position and orientation estimation described above (hereinafter referred to as "IMU reference position and orientation estimation"). Examples of IMU reference position and orientation estimation include position and orientation estimation by SLAM (Simultaneous Localization and Mapping).

[0066] Here, when using depth images to train a three-dimensional reconstruction model, the reason for performing both depth-referenced position and orientation estimation and IMU-referenced position and orientation estimation as position and orientation information estimation, as described above, is to switch whether or not to use IMU-referenced error information (absolute error) in training the three-dimensional reconstruction model, depending on the reliability of the IMU information. Hereafter, the position and orientation information obtained by depth reference position and orientation estimation will be referred to as "depth reference position and orientation information." Similarly, the position and orientation information obtained by IMU reference position and orientation estimation will be referred to as "IMU reference position and orientation information."

[0067] Depth reference position and orientation information obtained through depth reference position and orientation estimation has an uncertainty in scale. On the other hand, since the absolute value of the amount of movement between images can be determined by using IMU information, IMU reference position and orientation estimation makes it possible to estimate position and orientation information at the actual scale. Considering this point, it is desirable to use IMU reference position and orientation information for error calculation when training a three-dimensional reconstruction model. However, there is no guarantee that IMU information can always be obtained with high reliability, and if low-reliability information is used as IMU reference position and orientation information for error calculation, it will lead to a decrease in the training accuracy of the three-dimensional reconstruction model.

[0068] Therefore, in this embodiment, the reliability of the IMU information is estimated from the degree of agreement between the depth reference position and orientation information and the IMU reference position and orientation information, both of which are estimated using IMU information. If it is determined that the reliability of the IMU information is high, a three-dimensional reconstruction model that uses the absolute value error based on the IMU reference position and orientation information is used for training. If it is determined that the reliability of the IMU information is not high, a three-dimensional reconstruction model that uses the relative value error (rank error) based on the depth reference position and orientation information is used for training.

[0069] Specifically, in this example, the control unit 22 calculates a value indicating the degree of agreement between the depth reference position and attitude information and the IMU reference position and attitude information as the reliability of the IMU information, and determines whether the calculated reliability is above a predetermined threshold. If it is determined that the reliability is high, the selection processing unit F2, as described above, selects a three-dimensional reconstruction model that calculates the absolute error as the three-dimensional reconstruction model used for error calculation of the depth image. On the other hand, if it is determined that the reliability is not high, it selects a three-dimensional reconstruction model that calculates the rank error as the three-dimensional reconstruction model used for error calculation of the depth image.

[0070] Note that the absolute error is,

number

number

[0071] Furthermore, the position and orientation estimation unit F4 performs position and orientation information estimation using a normal colmap when the scene information indicates a "moving subject" and the selection processing unit F2 selects an event image, or when the scene information is neither a "complex shape" nor a "moving subject" and the selection processing unit F2 selects only a grayscale image. In other words, it estimates position and orientation information based on a training grayscale image without using depth images or event images.

[0072] In this example, the position and orientation estimation unit F4 estimates the position and orientation information based on the deblurred image, which is a learning grayscale image deblurred by the deblurred processing unit F5, when the scene information indicates a "moving subject" and the selection processing unit F2 selects an event image.

[0073] The deblurring unit F5 performs deblurring on the training grayscale image based on the event image. For information on the technique of performing deblurring of grayscale images based on event images, please refer to Reference 3 below. ·Reference 3: “Event-Based Fusion for Motion Deblurring with Cross-Modal Attention” Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, Luc Van Gool: European Conference on Computer Vision, 2022 Reference 3 discloses a technique for deblurring a grayscale image (RGB image) by inputting events within the exposure time of the grayscale image into a neural network.

[0074] In this example, the deblur processing unit F5 performs deblurring on the training grayscale images on the condition that the magnitude of motion of the subject 3, estimated based on the image captured by the imaging unit 21, is greater than a reference value. Specifically, the deblur processing unit F5 calculates a motion index value indicating the magnitude of motion of the subject 3 based on the event image obtained in this imaging, and determines whether or not blurring occurs based on whether or not this motion index value is greater than or equal to a predetermined threshold (reference value). In this example, this determination of whether or not blurring occurs is performed for each training grayscale image. The deblur processing unit F5 then performs deblurring only on the training grayscale images for which blurring has been determined to have occurred.

[0075] This makes it possible to perform deblurring only on training grayscale images where subject 3 has significant movement, thus preventing the deblurring process from being performed unnecessarily. Therefore, the processing required to train the three-dimensional reconstruction model can be made more efficient.

[0076] Here, the deblur processing unit F5 assigns a timestamp indicating the time of acquisition to the deblurred grayscale image. As described above, in this example, when deblurring is performed, the position and orientation estimation unit F4 estimates the position and orientation information using the deblurred image as input. In this case, the timestamp attached to the estimated position and orientation information will be the same as the timestamp attached to the deblurred image. From this perspective, if the timestamp attached to the deblur image is inappropriate, it becomes impossible to ensure time consistency between the positional information and the event information indicated by the event image during the training of the three-dimensional reconstruction model, leading to a decrease in training accuracy.

[0077] Therefore, in this example, assuming that the deblur processing unit F5 generates a deblurred image with a predetermined timing within the exposure period of the learning grayscale image to be deblurred as the image generation reference time, the position and orientation estimation unit F4 adds a timestamp indicating the image generation reference time of the deblurred image to the position and orientation information estimated for the deblurred image. Specifically, in this example, the deblur processing unit F5 generates a deblurred image, using the timing of the exposure center of the training grayscale image to be deblurred as the image generation reference time, and the position and orientation estimation unit F4 adds to the position and orientation information estimated using the deblurred image, directly inheriting the timestamp indicating the image generation reference time, which indicates the timing of the exposure center, attached to the deblurred image. This prevents situations where, for example, a deblurred image is generated using the exposure center of the target training grayscale image as the image generation reference time, while the position and orientation information estimated using the deblurred image as input is given a timestamp indicating the exposure start timing. This prevents inconsistencies in timing between position and orientation information and event information during the training of a three-dimensional reconstruction model, thereby improving training accuracy.

[0078] Furthermore, in this example, when the scene information indicates a "moving subject," the training of the three-dimensional reconstruction model is performed using a training grayscale image that has not been deblurred. In other words, even when deblurring is performed by the deblurring processing unit F5 in response to the scene information indicating a "moving subject," and the deblurred image is used for estimating the position and orientation information by the position and orientation estimation unit F4, the grayscale image that has not been deblurred is used for training the three-dimensional reconstruction model.

[0079] In this case, the three-dimensional reconstruction model (a three-dimensional reconstruction model that uses event images for error calculation) is a model that performs error calculation considering blur, as described in Reference 4 below. ·Reference 4 “BAD-NeRF:Bundle Adjusted Deblur Neural Radiance Fields” Peng Wang, Lingzhe Zhao, Ruijie Ma, Peidong Liu:arXiv 2022 / 11 / 23 Specifically, it is a three-dimensional reconstruction model that calculates error information for training by comparing the training grayscale image with the sum of rendering images at multiple timings within the exposure period of the training grayscale image.

[0080] Here, since deblurring does not completely remove blur, using a deblurred image as a training image may lead to a decrease in training accuracy. As described above, by using an unblurred image as the training image, it is possible to prevent a decrease in training accuracy caused by the incompleteness of the deblurring process, and to improve the training accuracy of the three-dimensional reconstruction model when subject 3 is a moving subject. In particular, by using a three-dimensional reconstruction model that performs error calculations considering blur as described above, it becomes possible to ensure that the three-dimensional reconstruction model is properly trained to handle cases where grayscale images that have not been deblurred are used, thereby improving the training accuracy of the three-dimensional reconstruction model.

[0081] (1-5. Processing Procedure) Referring to the flowcharts in Figures 9 to 12, an example of a processing procedure for realizing the learning method as the first embodiment described above will be explained. Figures 9 to 11 show examples of processing procedures to be executed by the control unit 22 in the imaging device 2, and Figure 12 shows an example of processing procedures to be executed by the processor 41 in the server device 4. The processes shown in Figures 9 to 11 are executed by the CPU of the control unit 22 based on the programs stored in the ROM and storage unit 23 mentioned above, while the process shown in Figure 12 is executed by the processor 41 based on the programs stored in the ROM 42 and storage unit 49. In the flowcharts shown in Figures 9 to 11, the control unit 22 is represented as the entity that executes the processing for explanatory purposes.

[0082] Figure 9 is a flowchart of the processing corresponding to the pre-imaging described above. First, in step S101, the control unit 22 performs a process to wait for the start of pre-imaging. That is, it waits until a predetermined condition is met, such as a predetermined operation input from the user, which is the condition for starting pre-imaging.

[0083] If the control unit 22 determines in step S101 that pre-imaging should begin, it proceeds to step S102 and executes the pre-imaging process. In this example, the imaging unit 21 is instructed to capture a grayscale image, an event image, and a depth image.

[0084] In step S103, following step S102, the control unit 22 determines whether or not the shape is complex. In other words, in this example, the control unit 22 determines whether or not the shape of the subject 3 is complex based on the depth image obtained in the pre-imaging process in step S102. Note that a specific method for determining whether or not the shape of the subject 3 is complex has already been explained, so a redundant explanation will be avoided.

[0085] If it is determined in step S103 that the shape is not complex, the control unit 22 proceeds to step S104 to determine whether or not the subject is moving. In other words, in this example, it is determined whether or not subject 3 is moving based on the event image obtained in the pre-imaging process in step S102. Note that a specific method for determining whether or not subject 3 is moving has already been explained, so a redundant explanation will be avoided.

[0086] In step S104, if it is determined that subject 3 is not a moving subject (i.e., it is neither a complex shape nor a moving subject), the control unit 22 proceeds to step S105 and executes a process to select only the grayscale image to be used. That is, it executes a process to select only the grayscale image from among the images that the imaging unit 21 can acquire (grayscale image, event image, and depth image) as the image to be used for training the three-dimensional reconstruction model.

[0087] Furthermore, if the control unit 22 determines in step S103 that the subject 3 has a complex shape, it proceeds to step S106 and executes the depth image selection process. That is, it executes a process to select a grayscale image and a depth image from the images that the imaging unit 21 can acquire as images to be used for training the three-dimensional reconstruction model.

[0088] Furthermore, if the control unit 22 determines in step S104 that subject 3 is a moving subject, it proceeds to step S107 and executes the event image selection process. That is, it executes a process to select grayscale images and event images from the images that the imaging unit 21 can acquire as images to be used for training the three-dimensional reconstruction model.

[0089] The control unit 22 completes the series of processes shown in Figure 9 depending on whether it has executed any of the processes described in steps S105 to S107 above.

[0090] Figure 10 is a flowchart of the processing corresponding to this imaging. In step S201, the control unit 22 performs a process to wait for the start of the actual imaging. That is, it waits until a predetermined condition is met, such as a predetermined operation input from the user, which indicates that the actual imaging should be started.

[0091] If it is determined in step S201 that this imaging should begin, the control unit 22 proceeds to step S202 to determine whether or not to use an event. That is, it determines whether or not the event image usage selection process was executed in the preceding step S107, which corresponds to the pre-imaging. If, in step S202, the event image usage selection process has not been executed and it is determined that it is not an event, the control unit 22 proceeds to step S203 to determine whether or not it is a depth image usage, that is, whether or not the depth image usage selection process was executed in the previous step S106.

[0092] If the depth image usage selection process in step S106 has not been executed and it is determined that depth image usage is not required (i.e., the grayscale image usage selection process in step S105 has been executed), the control unit 22 proceeds to step S204 and executes the grayscale image acquisition process. That is, for the main acquisition, the imaging unit 21 is instructed to capture only the grayscale image.

[0093] In step S205, following step S204, the control unit 22 performs position and orientation estimation processing based on the grayscale image. That is, it takes the grayscale image obtained from the imaging process in step S204 (a grayscale image of the subject 3 captured from multiple viewpoints) as input and performs position and orientation information estimation (estimation by colmap).

[0094] In step S206, following step S205, the control unit 22 transmits to the server device 4 the grayscale image, position and orientation information, and usage model instruction information that specifies a model that uses only the grayscale image. Specifically, it transmits to the server device 4 the grayscale image (learning grayscale image) obtained in the imaging process of step S204, the position and orientation information for each grayscale image obtained in the estimation process of step S205, and usage model instruction information (hereinafter referred to as "type A") that specifies a three-dimensional reconstruction model that uses only the grayscale image among the grayscale image, event image, and depth image for error calculation.

[0095] Furthermore, if the control unit 22 determines in step S202 that an event is being used, it proceeds to step S207 to perform the imaging process for the grayscale image and the event image. That is, for the main imaging, the imaging unit 21 is instructed to perform the imaging of the grayscale image and the event image.

[0096] In step S208, following step S207, the control unit 22 determines whether or not there is a blurred image. Specifically, based on the event image obtained by the imaging process in step S207, the control unit 22 determines whether or not there is blur for each grayscale image. If there is at least one grayscale image that is determined to have blur, the control unit 22 determines that there is a blurred image. If there are no grayscale images that are determined to have blur, the control unit 22 determines that there is no blurred image.

[0097] If the control unit 22 determines in step S208 that a blurred image exists, it proceeds to step S209 and performs deblurring of the blurred image. In this example, deblurring is performed based on the event image, but the details have already been explained, so a redundant explanation will be avoided. In the deblurring process of step S209, a timestamp indicating the aforementioned image generation reference time is added to the generated deblurred image.

[0098] The control unit 22 proceeds to step S210 in accordance with the processing performed in step S209.

[0099] Furthermore, if the control unit 22 determines in step S208 that there is no blurred image, it skips the deblurring process in step S209 and proceeds to step S210.

[0100] In step S210, the control unit 22 performs position and orientation estimation processing based on the grayscale image. Specifically, if there is a grayscale image that has been deblurred in step S209, the control unit 22 estimates the position and orientation information based on the grayscale image as a deblurred image. If the deblurring process in step S209 was not performed, the control unit 22 estimates the position and orientation information based on the grayscale image that has not been deblurred. In step S210, the position and orientation information estimated using the deblurred image as input is assigned the timestamp attached to the deblurred image as is.

[0101] In step S211, following step S210, the control unit 22 transmits to the server device the undeblurred grayscale image, position and orientation information, event image, and model instruction information that instructs the model to be used for error calculation based on the event information. The undeblurred image and event image transmitted in step S211 are the grayscale image and event image obtained by the imaging process in step S207. The model instruction information transmitted in step S211 instructs the use of a three-dimensional reconstruction model that uses the grayscale image (undeblurred grayscale image) as the learning input image, uses event information for error calculation, and calculates the error between the learning grayscale image and the sum of rendering images at multiple timings within the exposure period of the learning grayscale image as error information for learning, as described in Reference 4. Hereafter, the model instruction information transmitted in step S211 will be referred to as "typeB".

[0102] Furthermore, if the control unit 22 determines in step S203 that depth is being used, it executes the depth usage handling process in step S212.

[0103] Figure 11 is a flowchart of the processing for when depth is used in step S212. In the depth image handling process in step S212, the control unit 22 first performs the imaging process for the grayscale image and the depth image in step S221. That is, for the main imaging, it causes the imaging unit 21 to perform imaging of the grayscale image and the depth image. Also in step S221, in order to enable the switching of the usage model according to the IMU reliability described earlier, the IMU 26 is caused to perform detection of IMU information during the main imaging.

[0104] In step S222, following step S221, the control unit 22 performs position and orientation estimation processing based on the grayscale image and depth image. Specifically, it performs position and orientation information estimation processing using the grayscale image, depth image, and IMU information, as described above as "depth reference position and orientation estimation".

[0105] In step S223, following step S222, the control unit 22 performs position and attitude estimation processing based on IMU information. Specifically, it performs position and attitude information estimation processing using IMU information (for example, position and attitude estimation by SLAM), which is the "IMU reference position and attitude estimation" mentioned above.

[0106] Note that the estimation processes in steps S222 and S223 may be performed in any order.

[0107] In step S224, following step S223, the control unit 22 performs IMU reliability estimation processing. Specifically, as illustrated earlier, it calculates a value indicating the degree of agreement between the depth reference position and attitude information (position and attitude information estimated in step S222) and the IMU reference position and attitude information (position and attitude information estimated in step S223) as the reliability of the IMU information.

[0108] In step S225, following step S224, the control unit 22 determines whether the IMU reliability is high or not. Specifically in this example, it determines whether the reliability of the IMU information calculated in step S224 is above a predetermined threshold.

[0109] In step S225, if the reliability of the IMU information is above a threshold and the control unit 22 determines that the IMU reliability is high, the control unit 22 proceeds to step S226 and transmits the grayscale image, IMU reference position and orientation information, depth image, and usage model instruction information (hereinafter referred to as "type C") which instructs the depth usage model that uses absolute error to the server device 4. Here, "depth usage model" refers to a three-dimensional reconstruction model that uses the depth image for error calculation.

[0110] On the other hand, if in step S225 the control unit 22 determines that the reliability of the IMU information is not above a threshold and that the IMU reliability is not high, the control unit 22 proceeds to step S227 and transmits the grayscale image, position and orientation information, depth image, and usage model instruction information (hereinafter referred to as "type D") which instructs the depth usage model using rank error to the server device 4.

[0111] The control unit 22 completes the depth usage handling process in step S212 depending on whether it has performed the processing in step S226 or S227.

[0112] In Figure 10, the control unit 22 completes the series of processes shown in Figure 10 depending on whether it has executed any of the processes in steps S206, S211, or S212.

[0113] Figure 12 is a flowchart of the processing corresponding to the learning phase in server device 4. It should be assumed that, for the processing shown in this figure to be executed, the imaging device 2 has already performed one of the transmission processes in steps S206, S211, S226, or S227 (at least the transmission of the model instruction information).

[0114] In step S301, the processor 41 branches the processing according to the type of model instruction information used. Specifically, it branches the processing according to the aforementioned types of model instruction information used: "typeA", "typeB", "typeC", and "typeD".

[0115] If the model instruction information is "typeA", the processor 41 proceeds to step S302 and performs a learning process using a three-dimensional reconstruction model that uses only grayscale images, with the grayscale image and position / orientation information as training data.

[0116] Furthermore, if the model instruction information is "type B", the processor 41 proceeds to step S303 and performs learning processing using a three-dimensional reconstruction model that uses event information for error calculation, with the undeblurred grayscale image, position and orientation information, and event image as training data. Specifically, based on the aforementioned "type B" model instruction information, the processor 41 uses the event image for error calculation and, as shown in Reference 4, uses a three-dimensional reconstruction model that calculates the error between the training grayscale image and the sum of rendering images at multiple timings within the exposure period of the training grayscale image as error information for learning, with the undeblurred grayscale image as the training input image.

[0117] If the model instruction information is "type C", the processor 41 proceeds to step S304 and performs a learning process using a depth-dependent three-dimensional reconstruction model that uses absolute error, with the grayscale image, IMU reference position and orientation information, and depth image as training data. Here, "depth-dependent three-dimensional reconstruction model" refers to a three-dimensional reconstruction model that uses the depth image for error calculation.

[0118] If the model instruction information is "typeD", the processor 41 proceeds to step S305 and performs a learning process using a depth-dependent three-dimensional reconstruction model that uses rank error, with the grayscale image, depth reference position and orientation information, and depth image as training data.

[0119] The processor 41 completes the series of processes shown in Figure 12 depending on whether it has executed any of the processes from steps S302 to S305.

[0120] In the example above, the decision of whether to use absolute error or relative error (rank error) was based on the IMU confidence level. However, the decision of whether to use absolute or relative error can also be based on the confidence level of the depth information shown in the depth image. The reliability of depth information can be estimated by the magnitude of the peak value (frequency peak value) in the distance histogram when the dToF method is used as the distance measurement method to obtain depth images, as in this example.

[0121] Figure 13 shows an example of a distance histogram generated using the dToF method. The larger the peak value of this distance histogram, the more reliable the depth information can be estimated to be.

[0122] Here, if the peak value of the distance histogram is used as the confidence level of the depth information, this confidence level can be calculated on a pixel-by-pixel basis within the depth sensor 213a. In this case, the confidence level of the depth information can be output by the depth sensor 213a as metadata for the depth image.

[0123] Here, when determining whether to use absolute error or relative error depending on the reliability of the depth information as described above, the selection processing unit F2 can be described as follows. That is, the selection processing unit F2, based on the reliability estimation result of the depth information shown by the depth image, selects between a three-dimensional reconstruction model that uses the relative error of the depth information for error calculation and a three-dimensional reconstruction model that uses the absolute error of the depth information for error calculation.

[0124] Furthermore, the reliability of depth information varies depending on the distance measurement method, so it is advisable to switch the error calculation method according to the distance measurement method. For example, in the stereo distance measurement method, absolute value errors due to calibration errors occur, so it may be advisable to use rank error.

[0125] Furthermore, while the above example shows selecting a three-dimensional reconstruction model that uses event images for error calculation in response to scenes with "moving subjects," it is also possible to select a three-dimensional reconstruction model that uses depth images for error calculation in response to scenes with "moving subjects."

[0126] <2. Second Embodiment> Next, a second embodiment will be described. In the second embodiment, the frequency of acquiring captured images is adjusted based on the characteristic quantities of the subject 3, such as the magnitude of its movement and the degree of complexity of its shape.

[0127] Figure 14 is a functional block diagram illustrating the functions of the control unit 22A included in the imaging device 2 as a second embodiment. The configuration of the imaging device 2 in the second embodiment is the same as in the first embodiment, except that it is equipped with a control unit 22A instead of a control unit 22, so it is not shown in the illustration. Furthermore, in the following explanation, parts that are the same as those already explained will be given the same reference numerals and step numbers, and the explanation will be omitted.

[0128] In Figure 14, the control unit 22A differs from the control unit 22 in that it has the added function of a frequency adjustment unit F6. The frequency adjustment unit F6 adjusts the imaging frequency for grayscale images that the imaging unit 21 can acquire, based on the feature quantities of the subject 3 estimated based on the image captured by the imaging unit 21. Specifically, it performs the following first and second frequency adjustment processes. That is, the first frequency adjustment process adjusts the imaging frequency for grayscale images that the imaging unit 21 can acquire, based on the magnitude of the movement of the subject 3 estimated based on the image captured by the imaging unit 21. The second frequency adjustment process adjusts the imaging frequency for grayscale images that the imaging unit 21 can acquire, based on the degree of complexity of the shape of the subject 3 estimated based on the image captured by the imaging unit 21.

[0129] Specifically, in the first frequency adjustment process, the frequency of capturing grayscale images is increased when the movement of subject 3 is large, and conversely, the frequency of capturing grayscale images is decreased when the movement of subject 3 is small. Similarly, in the second frequency adjustment process, the frequency of capturing grayscale images is increased when the complexity of the shape of subject 3 is high, and conversely, the frequency of capturing grayscale images is decreased when the complexity of the shape of subject 3 is low. When subject 3 is moving large or has a high degree of shape complexity, it is desirable to use more training samples (training images) to improve the learning accuracy. By performing the first and second frequency adjustment processes described above, the frequency of capturing training grayscale images can be increased to accommodate cases where a large number of training samples are required, thereby improving the accuracy of generating free-viewpoint images by improving the learning accuracy of the three-dimensional reconstruction model. Furthermore, when a large number of training samples are not necessary, the frequency of capturing grayscale images can be decreased, thus improving the efficiency of the learning process.

[0130] In this example, the first frequency adjustment process is performed in response to the case where subject 3 is a moving subject. Specifically, when the scene information acquisition unit F1 determines that subject 3 is a "moving subject," the frequency adjustment unit F6 determines whether the value indicating the magnitude of subject 3's movement (hereinafter referred to as the "motion amount index value"), calculated based on the event image obtained in the pre-imaging, is above a predetermined threshold. If the motion amount index value is above the threshold, a determination result is obtained that subject 3 is moving significantly; if the motion amount index value is below the threshold, a determination result is obtained that subject 3 is not moving significantly. If it is determined that subject 3 is moving significantly, the imaging frequency of the grayscale image in this imaging (which is the frame rate in the case of video) is set to be higher than when it is determined that subject 3 is not moving significantly. Here, the motion index values ​​mentioned above can be calculated based on the techniques described in the aforementioned references 1 and 2.

[0131] Furthermore, in this example, if the frequency adjustment unit F6 determines in the first frequency adjustment process that the movement of subject 3 is large, it sets the frequency of event image acquisition to be increased (higher than when it is determined that the movement of subject 3 is not large), in addition to the frequency of grayscale image acquisition in this imaging.

[0132] Furthermore, in this example, the second frequency adjustment process is performed in response to the case where the shape of subject 3 is complex. Specifically, when the scene information acquisition unit F1 determines that the shape of subject 3 is complex, the frequency adjustment unit F6 determines whether the value indicating the degree of complexity of the shape of subject 3 (hereinafter referred to as the "complexity index value") calculated based on the depth image obtained in the pre-imaging is above a predetermined threshold. If the complexity index value is above the threshold, a determination result is obtained that the degree of complexity of the shape of subject 3 is high; if the complexity index value is below the threshold, a determination result is obtained that the degree of complexity of the shape of subject 3 is not high. If it is determined that the degree of complexity of the shape of subject 3 is high, the imaging frequency of the grayscale image in the main imaging is set to be higher than when it is determined that the degree of complexity of the shape of subject 3 is not high. Here, the complexity index value can be calculated as the number of subregions whose variance of depth values ​​is greater than or equal to a threshold, as explained with reference to Figure 6 above. Alternatively, the complexity index value can be calculated based on the number of edges of the subject, as explained with reference to Figure 7 above.

[0133] In this example, the frequency adjustment unit F6, in the second frequency adjustment process, if it determines that the complexity of the shape of subject 3 is high, sets not only the frequency of capturing grayscale images in this imaging but also the frequency of capturing depth images to be higher than when it is determined that the complexity of the shape of subject 3 is not high. Here, when increasing the frequency of capturing depth images, not only the frame rate of the depth sensor 213a is adjusted but also the emission period of the light-emitting unit 213b is adjusted.

[0134] Referring to the flowcharts in Figures 15 to 17, specific processing procedure examples for realizing the frequency adjustment method as the second embodiment described above will be explained. These processes shown in Figures 15 to 17 are executed by the control unit 22A based on programs stored in the ROM or storage unit 23 within the control unit 22A. In the second embodiment, the processing performed by the processor 41 of the server device 4 is the same as that described in Figure 12, so a diagrammatic explanation is omitted.

[0135] Figure 15 is a flowchart of the processing corresponding to the pre-imaging in the second embodiment. The difference from the process in the first embodiment shown in Figure 9 is that steps S111 to S114 have been added.

[0136] If the control unit 22A determines in step S103 that the shape is complex, it proceeds to step 111 to determine whether the complexity is high or not. Specifically, it calculates the complexity index value mentioned above and determines whether the complexity index value is above a predetermined threshold.

[0137] In step S111, if the complexity index value is above a threshold and the complexity is determined to be high, the control unit 22A proceeds to step S112 and executes a high-frequency selection process. That is, it performs a process to select a high frequency as the imaging frequency for the grayscale image and depth image in this imaging. Then, the control unit 22A proceeds to step S106 (depth image usage selection process).

[0138] On the other hand, if in step S111 the complexity index value is not above the threshold and the control unit 22A determines that the complexity is not high, the control unit 22A skips the selection process in step S112 and proceeds to step S106.

[0139] Furthermore, if the control unit 22A determines in step S104 that the subject is moving, it proceeds to step S113 to determine whether the movement is large or not. Specifically, it calculates the aforementioned motion amount index value and determines whether the motion amount index value is above a predetermined threshold.

[0140] In step S113, if the motion amount index value is above a threshold and it is determined that the motion is large, the control unit 22A proceeds to step S114 and executes a high-frequency selection process. That is, it performs a process to select a high frequency as the imaging frequency for the grayscale image and event image in this imaging. Then, the control unit 22A proceeds to step S107 (event image usage selection process).

[0141] On the other hand, if in step S113 the motion amount index value is not above the threshold and the control unit 22A determines that the motion is not large, the control unit 22A skips the selection process in step S114 and proceeds to step S107.

[0142] Figure 16 is a flowchart of the processing corresponding to the actual imaging in the second embodiment. The differences from the processing in the first embodiment shown in Figure 10 are that steps S251 and S252 are added, and the depth usage processing in step S212A is executed instead of the depth usage processing in step S212.

[0143] If the control unit 22A determines in step S202 that an event is being used, it proceeds to step S251. In step S251, the control unit 22A determines whether or not high-frequency selection is performed, that is, whether or not the high-frequency selection process in step S114 has been executed.

[0144] In step S251, the high-frequency selection process in step S114 is executed, and if it is determined that a high-frequency selection is made, the control unit 22A proceeds to step S252 and executes the high-frequency setting process. That is, it sets the imaging unit 21 to have a high frequency for capturing grayscale images and event images in this imaging. Then, the control unit 22A proceeds to step S207 (grayscale image and event image capturing process).

[0145] On the other hand, if step S251 determines that there is no high-frequency selection, the control unit 22A skips the high-frequency setting process in step S252 and proceeds to step S207.

[0146] Furthermore, if the control unit 22A determines in step S203 that depth is being used, it executes the depth usage handling process shown in step S212A in Figure 17.

[0147] The depth usage handling process shown in Figure 17 differs from the depth usage handling process shown in Figure 11 in that steps S253 and S254 have been added. Specifically, in step S253, the control unit 22A determines whether or not high-frequency selection is performed, that is, whether or not the high-frequency selection process in step S112 has been executed.

[0148] In step S253, the high-frequency selection process in step S112 is executed, and if it is determined that a high-frequency selection is made, the control unit 22A proceeds to step S254 and executes the high-frequency setting process. That is, it sets the imaging unit 21 to have a high frequency for capturing grayscale images and depth images in this imaging. Then, the control unit 22A proceeds to step S221 (image capture process for grayscale images and depth images).

[0149] On the other hand, if it is determined in step S253 that there is no high-frequency selection, the control unit 22A skips the high-frequency setting process in step S254 and proceeds to step S221.

[0150] In the above example, the imaging frequency was adjusted in two stages: an increased frequency and a decreased frequency. However, it is also possible to adjust it in three or more stages. In that case, the imaging frequency should be increased in stages according to the magnitude of the feature quantities of subject 3, such as the magnitude of motion and the complexity of its shape.

[0151] In this case, when the movement of subject 3 is large or the degree of shape complexity is high, these can be said to be one of the imaging scenes, and the imaging frequency adjustment as the second embodiment described above can be rephrased as imaging frequency adjustment according to the imaging scene. The above example shows how to adjust the imaging frequency and select the model and images to be used depending on the imaging scene. However, it is also possible to adjust only the imaging frequency without selecting the model and images to be used, as a processing method depending on the imaging scene.

[0152] Furthermore, while the above example illustrates how the imaging frequency is adjusted based on the scene determination result, it is also conceivable that the imaging frequency could be adjusted based on factors other than the scene determination result (degree of motion or degree of shape complexity), such as user operation.

[0153] <3. Variant> The specific examples described so far are merely examples, and this technology can take on a variety of configurations as variations. For example, the above example shows selecting an image to perform the main imaging based on the scene determination result. However, it is also conceivable to perform the main imaging on all images that the imaging unit 21 can acquire, and then select the image to send to the server device 4 for learning based on the scene determination result. This prevents unnecessary image transmission processing, thereby improving learning efficiency. Furthermore, it reduces the processing load on the imaging device 2 and lowers power consumption.

[0154] Furthermore, while the above example illustrates a configuration in which the imaging unit 21 acquires various types of images using separate sensors, it is also possible to adopt a configuration in which at least two types of images are acquired by a single sensor. For example, a configuration using a mixed-mount sensor that combines pixels for obtaining grayscale images and pixels for obtaining event images, as described in Reference 5 below, is conceivable. • Reference 5: Japanese Patent Publication No. 2024-106169

[0155] For example, when adopting such a mixed-sensor configuration, it is also conceivable to adopt a configuration in which scene determination (whether or not it is a "moving subject") and deblurring are performed within the sensor. Thus, the scene information acquisition unit F1 can also be installed within the sensor device that obtains the captured image.

[0156] Furthermore, while the above examples of scenes to be judged include "moving subjects" and "complex shapes," it is also possible to judge whether or not the scene is "low-light." If the scene is judged to be low-light, it is possible to select a three-dimensional reconstruction model that does not use a time-deformed field and uses grayscale images, event images, and depth images for error calculation. If the scene is not low-light, a model that does not use a time-deformed field and uses only the grayscale image from the three images mentioned above for error calculation is selected.

[0157] Furthermore, although the above example shows that the position and orientation estimation unit F4 is located on the imaging device 2 side, it is also possible to adopt a configuration in which a server device 4B having the position and orientation estimation unit F4 is provided instead of the server device 4, as shown in the learning system 1B in Figure 18. As shown in the figure, the server device 4B includes a processor 41B having a learning processing unit F41 and a position and orientation estimation unit F4, instead of the processor 41. Furthermore, the learning system 1B is provided with an imaging device 2B instead of the imaging device 2, and the imaging device 2B differs from the imaging device 2 in that it is provided with a control unit 22B in which the position and orientation estimation unit F4 is omitted, instead of a control unit 22 (or control unit 22A) having a position and orientation estimation unit F4.

[0158] Furthermore, while the above example uses event images to calculate the error during training of a three-dimensional reconstruction model when dealing with moving subjects, it is also possible to use models that use depth images instead of event images for error calculation.

[0159] Furthermore, while the above example shows how scene information is obtained through scene determination processing based on captured images, it is also possible to adopt a configuration in which the user performs a scene selection operation, and the information of the scene selected in that operation is obtained as scene information.

[0160] Furthermore, although not specifically mentioned above, it is also conceivable that a GUI for issuing instructions to start pre-imaging and subsequent main imaging could be displayed on the display unit of the output unit 25.

[0161] In this imaging process, the user moves the imaging device (2 or 2B) to capture images of subject 3. However, in order for the three-dimensional reconstruction model to be properly trained, the movement speed of the imaging device must be appropriate. For example, if the movement speed is excessively fast, the required number of images (images taken at the required movement intervals) cannot be obtained, leading to a decrease in training accuracy. Conversely, if the movement speed is excessively slow, unnecessary images will be captured, leading to a decrease in training efficiency.

[0162] Therefore, it is conceivable to provide the user with guide information to guide the movement speed of the imaging device during actual imaging. Figure 19 is a functional block diagram of the control unit 22C that should be included in an imaging device as a modified example for presenting guide information regarding such movement speed. As shown in the figure, the control unit 22C differs from the control unit 22 (or control unit 22B) in that a guide presentation processing unit F7 is added.

[0163] The guide presentation processing unit F7, while the imaging unit 21 is capturing images to be used for learning (i.e., during the actual imaging), performs a process to present guide information related to the movement speed of the imaging viewpoint to the user based on the error between the target movement speed of the imaging viewpoint determined based on the images captured by the imaging unit 21 and the measured movement speed of the imaging viewpoint.

[0164] In this example, the guidance information is presented when subject 3 is a moving subject, and is provided in a way that guides the user to a movement speed appropriate to the speed of subject 3's movement. When subject 3 is moving quickly, it is desirable to acquire images taken at the shortest possible movement intervals as training images (grayscale images and event images) in order to improve training accuracy. Therefore, the guide presentation processing unit F7 in this example calculates a value indicating the speed of movement of the subject 3 (hereinafter referred to as the "motion velocity index value") based on the event image obtained by pre-imaging, and determines the target movement speed of the imaging device 2 during the actual imaging (i.e., the target movement speed of the imaging viewpoint) according to the magnitude of the motion velocity index value. This target movement speed can be determined, for example, based on table information or a function that shows the correspondence between the motion velocity index value and the target movement speed. Here, the motion velocity index value can be calculated based on the techniques described in References 1 and 2, similar to the motion volume index value mentioned above.

[0165] Then, during imaging, the guide presentation processing unit F7 performs the necessary guide information presentation processing based on the comparison result between the movement speed of the imaging device 2 (imaging viewpoint) and the target movement speed described above. Specifically, in this example, if the movement speed of the imaging device 2 exceeds the target movement speed, it performs guide information presentation processing indicating that the movement speed should be slowed down, and conversely, if the movement speed of the imaging device 2 is lower, it performs guide information presentation processing indicating that the movement speed should be increased. In this example, this guide information is presented by displaying message information on the display unit of the output unit 25.

[0166] Figure 20 shows an example of how guide information is displayed. Figure 20 shows an example where, in response to the imaging device 2's movement speed being below the target movement speed, a message prompting acceleration, such as "Please move the camera a little faster," is displayed. However, if the imaging device 2's movement speed exceeds the target movement speed, a message prompting deceleration, such as "Please move the camera a little slower," is displayed.

[0167] The flowcharts in Figures 21 and 22 show specific processing procedures that the control unit 22C should execute to realize the method for presenting guide information related to the movement speed described above. Figure 21 is a flowchart of the processing corresponding to the pre-imaging stage. The difference from the processing shown in Figure 9 is the addition of step S121.

[0168] Specifically, if the control unit 22C determines in step S104 that the subject is moving, it proceeds to step S121 to perform the target movement speed determination process. That is, the target movement speed is determined based on the motion speed index value calculated based on the event image obtained by the pre-imaging process in step S102. Note that the specific method for determining the target movement speed has already been explained above, so a redundant explanation will be avoided. The control unit 22C proceeds to step S107 in response to having performed the decision process in step S121.

[0169] Figure 22 is a flowchart of the processes to be performed during the imaging process. In step S401, the control unit 22C determines whether the error between the current moving speed and the target moving speed is greater than or equal to a predetermined value. Here, the current moving speed is the moving speed of the imaging device 2, which is calculated in real time. The moving speed of the imaging device 2 can be calculated, for example, based on IMU information detected by the IMU 26.

[0170] In step S401, if it is determined that the error between the current movement speed and the target movement speed is not greater than or equal to a predetermined value, the control unit 22C proceeds to step S402 to determine whether the imaging is complete or not. In step S402, if it is determined that the imaging is not complete, the control unit 22C returns to step S401. In other words, the control unit 22C is configured to wait until the error between the current movement speed and the target movement speed becomes greater than or equal to a predetermined value, or until the imaging is complete, based on the processing in steps S401 and S402.

[0171] In step S401, if the control unit 22C determines that the error between the current speed and the target speed is greater than or equal to a predetermined value, it proceeds to step S403 to determine whether the current speed is less than the target speed (current speed < target speed). If the current speed is less than the target speed, the control unit 22C proceeds to step S404 to execute the acceleration guide display process. That is, it performs a process to display guide information that encourages acceleration of the speed, such as the message information that encourages acceleration as illustrated in Figure 20 above, on the display unit of the output unit 25.

[0172] On the other hand, if in step S403 it is determined that the current travel speed is not less than the target travel speed, the control unit 22C proceeds to step S405 and executes deceleration guide display processing. That is, it processes the display of the output unit 25 to display guide information that encourages deceleration, such as the message information that encourages deceleration as exemplified above.

[0173] The control unit 22C returns to step S401 depending on whether it has performed the processing in step S404 or S405. As a result, during imaging, guide information is displayed according to the deviation between the current movement speed and the target movement speed that exceeds a predetermined value.

[0174] If the control unit 22C determines in step S402 that the imaging process is complete, it terminates the series of processes shown in Figure 22.

[0175] In addition to visual presentations using displays as exemplified above, guide information can also be presented using other methods, such as auditory presentations using speakers or tactile presentations using vibrations.

[0176] In the explanation so far, grayscale images, event images, and depth images have been given as examples of images that the imaging unit 21 can acquire. However, it is also conceivable that the imaging unit 21 be configured to acquire polarization images in addition to grayscale images. In that case, when a scene with "objects that are highly glossy or reflective" is selected by the user, the selection processing unit F2 selects a three-dimensional reconstruction model to be used for learning and an image to be captured. This includes selecting a three-dimensional reconstruction model that calculates errors based on the polarization information shown by the polarized image, and selecting a grayscale image and a polarized image.

[0177] <4. Summary of Embodiments> As described above, the learning system as an embodiment (1, 1B) comprises an imaging unit that captures images for training a three-dimensional reconstruction model, an imaging unit (21) configured to acquire multiple types of captured images, a scene information acquisition unit (F1) that acquires scene information indicating the type of scene the imaging unit is targeting, a selection processing unit (F2) that selects the type of three-dimensional reconstruction model to be used for training and the type of captured image based on the scene information acquired by the scene information acquisition unit, and a learning processing unit (F41) that performs training of the three-dimensional reconstruction model according to the model type selected by the selection processing unit using the type of captured image selected by the selection processing unit. This makes it possible to train the three-dimensional reconstruction model appropriately according to the image capture scene used for training. Therefore, it is possible to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the learning process.

[0178] Furthermore, in the learning system as an embodiment, there is an imaging control unit (F3) that controls the imaging unit so that only images of the type selected by the selection processing unit are acquired from among the types of images that the imaging unit can acquire. This helps prevent the capture of images that are unnecessary for learning. Therefore, it is possible to improve the efficiency of learning.

[0179] Furthermore, in the learning system as an embodiment, the selection processing unit selects the type of three-dimensional reconstruction model from among multiple three-dimensional reconstruction models that use different types of images for error calculation in learning. This makes it possible to train a model that performs error calculations using the appropriate image depending on the scene. Therefore, it is possible to improve the learning accuracy of the three-dimensional reconstruction model and improve the accuracy of generating free-viewpoint images.

[0180] Furthermore, in the learning system as an embodiment, the scene information acquisition unit determines the scene based on the image captured by the imaging unit and acquires scene information. As a result, the device determines the scene based on the captured image, eliminating the need for the user to manually specify the scene. Therefore, it is possible to reduce the user's operational burden required for learning.

[0181] Furthermore, in the learning system as an embodiment, the scene information acquisition unit determines whether or not the subject is a moving subject as part of the scene determination. This makes it possible to use different types of three-dimensional reconstruction models and images for training, depending on whether the subject is moving or not. Therefore, it is possible to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the learning process.

[0182] Furthermore, in the learning system as an embodiment, if the scene information indicates a moving subject, the selection processing unit selects a type of three-dimensional reconstruction model with a time-deformed field as the type of three-dimensional reconstruction model. This makes it possible to use a three-dimensional reconstruction model suitable for situations where the subject in a free-viewpoint image is moving during training. Therefore, it is possible to improve the accuracy of generating free-viewpoint images.

[0183] Furthermore, in the learning system as an embodiment, the imaging unit is configured to acquire an event image as one of several types of captured images, and the selection processing unit, if the scene information indicates a moving subject, selects the type of three-dimensional reconstruction model that uses the event image for error calculation. This makes it possible to use a three-dimensional reconstruction model in training that performs error calculations suitable for cases where the subject in a free-viewpoint image is moving. Therefore, it is possible to improve the accuracy of generating free-viewpoint images.

[0184] Furthermore, in the learning system as an embodiment, the scene information acquisition unit determines whether or not the subject has a complex shape as part of the scene determination. This makes it possible to use different types of three-dimensional reconstruction models and images for training, depending on whether the subject's shape is complex or not. Therefore, it is possible to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the learning process.

[0185] Furthermore, in the learning system as an embodiment, the imaging unit is configured to acquire a depth image as one of several types of captured images, and the selection processing unit, if the scene information indicates a complex shape, selects a type of three-dimensional reconstruction model that uses the depth image for error calculation. This makes it possible to use a three-dimensional reconstruction model for training that performs error calculations suitable for cases where the shape of the target object in a free-viewpoint image is complex. Therefore, it is possible to improve the accuracy of generating free-viewpoint images.

[0186] Furthermore, in the learning system as an embodiment, the selection processing unit, based on the confidence estimation result of the depth information shown by the depth image, selects between a three-dimensional reconstruction model that uses the relative error of the depth information for error calculation and a three-dimensional reconstruction model that uses the absolute error of the depth information for error calculation. This makes it possible to perform learning using relative error when the reliability of the depth information is low, and learning using absolute error when the reliability of the depth information is high. Therefore, it is possible to ensure that appropriate learning is performed according to the reliability of the depth information, thereby improving the accuracy of generating free-viewpoint images.

[0187] Furthermore, in the learning system as an embodiment, the imaging unit is configured to acquire grayscale images and event images, and includes a position and orientation estimation unit (F4) that estimates position and orientation information, which is information indicating the viewpoint position and direction of gaze during imaging, for each learning grayscale image, which is a plurality of grayscale images captured while changing the viewpoint for learning, and a deblurring processing unit (F5) that performs deblurring processing on the learning grayscale images based on the event images, and the position and orientation estimation unit estimates position and orientation information based on the deblurred image, which is the learning grayscale image deblurred by the deblurring processing unit. As a result, even in scenes where the subject is moving, a deblurred training image is used to estimate position and orientation information, thereby improving the accuracy of position and orientation information estimation used in training the three-dimensional reconstruction model. Therefore, even when the target subject is moving, the three-dimensional reconstruction model can be trained with high accuracy, thereby improving the accuracy of generating free-viewpoint images.

[0188] Furthermore, in the learning system as an embodiment, the deblur processing unit generates a deblurred image, with a predetermined timing within the exposure period of the learning grayscale image to be deblurred as the image generation reference time, and the position and orientation estimation unit assigns a timestamp indicating the image generation reference time of the deblurred image to the position and orientation information estimated using the deblurred image. This ensures time consistency between positional information and event information shown in event images. Therefore, positional information and event information with guaranteed temporal consistency can be used to train the three-dimensional reconstruction model, thereby improving the training accuracy of the three-dimensional reconstruction model and improving the accuracy of generating free-viewpoint images.

[0189] Furthermore, in the learning system as an embodiment, the deblurring unit performs deblurring on the condition that the magnitude of the subject's movement estimated based on the image captured by the imaging unit is greater than a reference value. This makes it possible to perform deblurring only when the subject is moving significantly, thus preventing unnecessary deblurring. Therefore, the processing required to train the three-dimensional reconstruction model can be made more efficient.

[0190] Furthermore, in the learning system as an embodiment, when the selection processing unit selects the type of three-dimensional reconstruction model that uses event images for error calculation, the learning processing unit uses a grayscale image that has not undergone deblurring by the deblurring processing unit as the grayscale image for learning. As a result, when training a three-dimensional reconstruction model for a moving subject, an undeblurred grayscale image is used as the training grayscale image. Since deblurring does not completely remove blur, using a deblurred image as a training image may lead to a decrease in training accuracy. As described above, by using an unblurred image as a training image, it is possible to prevent a decrease in training accuracy caused by the incompleteness of deblurring, and to improve the training accuracy of the three-dimensional reconstruction model when the subject is moving.

[0191] Furthermore, in the learning system as an embodiment, the learning processing unit uses a three-dimensional reconstruction model that uses event images for error calculation. This model calculates the error between a training grayscale image and the sum of rendering images at multiple timings within the exposure period of the training grayscale image as error information for learning. This makes it possible to ensure that the three-dimensional reconstruction model is properly trained even when using grayscale images that have not undergone deblurring. Therefore, it is possible to improve the learning accuracy of the three-dimensional reconstruction model when the subject is moving.

[0192] Furthermore, in the learning system as an embodiment, a first frequency adjustment unit (frequency adjustment unit F6) is provided that adjusts the imaging frequency for grayscale images that can be acquired by the imaging unit based on the magnitude of motion of the subject estimated based on the image captured by the imaging unit. This makes it possible to increase the frequency of capturing grayscale images when the subject is moving significantly and requires more training samples for training the three-dimensional reconstruction model, and conversely, decrease the frequency of capturing grayscale images when the subject is moving little and fewer training samples can be used. In this way, it becomes possible to adjust the frequency of capturing grayscale images according to the required number of training samples, thereby achieving both improved accuracy in generating free-viewpoint images and increased efficiency in the training process.

[0193] Furthermore, in the learning system as an embodiment, a second frequency adjustment unit (frequency adjustment unit F6) is provided that adjusts the imaging frequency for grayscale images that can be acquired by the imaging unit based on the degree of complexity of the shape of the subject estimated based on the image captured by the imaging unit. This makes it possible to increase the frequency of capturing grayscale images when the complexity of the subject shape is high and more training samples are required for training the three-dimensional reconstruction model, and conversely, to decrease the frequency of capturing grayscale images when the complexity of the subject shape is low and fewer training samples are needed. In this way, it becomes possible to adjust the frequency of capturing grayscale images according to the required number of training samples, thereby achieving both improved accuracy in generating free-viewpoint images and increased efficiency in the training process.

[0194] Furthermore, in the learning system as an embodiment, while the imaging unit is capturing images to be used for learning, a guide presentation processing unit (F7) is included that performs processing to present guide information related to the movement speed of the imaging viewpoint to the user based on the error between the target movement speed of the imaging viewpoint determined based on the images captured by the imaging unit and the measured movement speed of the imaging viewpoint. This makes it possible to guide the user, for example, to slow down the movement speed of the imaging viewpoint when the subject is moving or has a complex shape and the target movement speed is slow, and to speed up the movement speed of the imaging viewpoint when the subject is a stationary object with a simple shape and the target movement speed is fast. Therefore, it is possible to ensure that an appropriate number of training images are acquired according to the imaging scene, thereby improving the training accuracy of the three-dimensional reconstruction model, and in other words, improving the accuracy of generating free-viewpoint images.

[0195] The learning method as an embodiment is an information processing system equipped with an information processing device, an imaging unit that captures images for training a three-dimensional reconstruction model, the imaging unit configured to acquire multiple types of captured images, acquires scene information indicating the type of scene to be captured, selects the type of three-dimensional reconstruction model to be used for training and the type of captured image based on the acquired scene information, and performs training of the three-dimensional reconstruction model using the selected type of captured image. This learning method can also provide the same functions and effects as the learning system described in the above-described embodiment.

[0196] Furthermore, the imaging device as an embodiment (2,2B) is an imaging unit that captures images for training a three-dimensional reconstruction model, and comprises an imaging unit configured to acquire multiple types of captured images, a scene information acquisition unit that acquires scene information indicating the type of scene to be captured by the imaging unit, and a selection processing unit that selects the type of three-dimensional reconstruction model to be used for training and the type of captured image based on the scene information acquired by the scene information acquisition unit. With such an imaging device, as with the learning system in the embodiment described above, it is possible to ensure that the learning of the three-dimensional reconstruction model is carried out appropriately according to the imaging scene of the images used for learning. Therefore, it is possible to achieve both improved accuracy in generating free-viewpoint images and increased efficiency in the learning process.

[0197] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0198] <5. This Technology> Furthermore, this technology can also be configured as follows. (1) An imaging unit for capturing images for training a three-dimensional reconstruction model, the imaging unit being configured to acquire multiple types of captured images as said images, The aforementioned imaging unit acquires scene information indicating the type of scene to be imaged, A selection processing unit selects the type of the three-dimensional reconstruction model to be used for learning and the type of the captured image based on the scene information acquired by the scene information acquisition unit. The system comprises a learning processing unit that performs learning of a three-dimensional reconstruction model based on the model type selected by the selection processing unit, using captured images of the type selected by the selection processing unit. Learning system. (2) The imaging control unit controls the imaging unit so that only images of the type selected by the selection processing unit are acquired from among the types of images that the imaging unit can acquire. The learning system described in (1) above. (3) The selection processing unit, As part of selecting the type of three-dimensional reconstruction model, a selection is made from among multiple three-dimensional reconstruction models that use different types of images for error calculation during learning. The learning system described in (1) or (2) above. (4) The aforementioned scene information acquisition unit, The scene is determined and the scene information is acquired based on the image captured by the imaging unit. A learning system as described in any of (1) to (3) above. (5) The aforementioned scene information acquisition unit, As part of the scene determination, it is determined whether or not the subject is a moving subject. The learning system according to claim 4. The learning system described in (4) above. (6) The selection processing unit, If the scene information indicates the moving subject, the type of three-dimensional reconstruction model selected is a three-dimensional reconstruction model with a time-deformed field. The learning system described in (5) above. (7) The imaging unit is configured to acquire an event image as one of the multiple types of captured images. The selection processing unit, If the scene information indicates the moving subject, the type of three-dimensional reconstruction model selected is the type of three-dimensional reconstruction model that uses the event image for error calculation. The learning system described in (5) or (6) above. (8) The aforementioned scene information acquisition unit, As part of the scene determination, a determination is made as to whether or not the subject has a complex shape. A learning system as described in any of (4) to (7) above. (9) The imaging unit is configured to acquire a depth image as one of the multiple types of captured images, The selection processing unit, If the scene information indicates that the shape is complex, the type of three-dimensional reconstruction model selected is one in which the depth image is used for error calculation. The learning system described in (8) above. (10) The selection processing unit, Based on the confidence estimation result of the depth information shown in the depth image, the type of three-dimensional reconstruction model is selected from a three-dimensional reconstruction model that uses the relative error of the depth information for error calculation and a three-dimensional reconstruction model that uses the absolute error of the depth information for error calculation. The learning system described in (9) above. (11) The imaging unit is configured to acquire a grayscale image and an event image. Based on the learning grayscale images, which are multiple grayscale images captured while changing the viewpoint for the purpose of learning, a position and orientation estimation unit estimates position and orientation information, which is information indicating the viewpoint position and direction of gaze at the time of capture, for each of the learning grayscale images. The system includes a deblurring processing unit that performs deblurring on the learning grayscale image based on the event image, The position and orientation estimation unit, Based on the deblurred image, which is the learning grayscale image deblurred by the deblurring processing unit, the position and orientation information is estimated. The learning system described in (7) above. (12) The aforementioned deblurring processing unit is As the deblurred image, a deblurred image is generated with a predetermined timing within the exposure period of the learning grayscale image to be subjected to the deblurring process as the image generation reference time. The position and orientation estimation unit, A timestamp indicating the image generation reference time for the deblurred image is added to the position and orientation information estimated from the aforementioned deblurred image. The learning system described in (11) above. (13) The deblurring processing unit performs the deblurring process on the condition that the magnitude of motion of the subject estimated based on the image captured by the imaging unit is greater than a reference value. The learning system described in (11) or (12) above. (14) The aforementioned learning processing unit, When the selection processing unit selects the type of three-dimensional reconstruction model to be used for error calculation of the event image, the three-dimensional reconstruction model is trained using a grayscale image that has not been deblurred by the deblur processing unit as the grayscale image for training. A learning system as described in any of (11) to (13) above. (15) The aforementioned learning processing unit, As a three-dimensional reconstruction model that uses the aforementioned event image for error calculation, the learning process uses a three-dimensional reconstruction model that calculates the error between the training grayscale image and the sum of rendering images at multiple timings within the exposure period of the training grayscale image as error information for learning. The learning system described in (14) above. (16) The system includes a first frequency adjustment unit that adjusts the imaging frequency for the grayscale images that the imaging unit can acquire, based on the magnitude of the subject's movement estimated from the image captured by the imaging unit. The learning system according to any one of (1) to (15) above. (17) A second frequency adjustment unit that adjusts the imaging frequency of the gradation image that can be acquired by the imaging unit based on the complexity of the shape of the subject estimated based on the captured image by the imaging unit. The learning system according to any one of (1) to (16) above. (18) A guide presentation processing unit that performs a process of presenting guide information related to the moving speed of the imaging viewpoint to the user based on the error between the target moving speed of the imaging viewpoint determined based on the captured image by the imaging unit and the measured value of the moving speed of the imaging viewpoint in a state where the imaging unit is capturing an image used for the learning. The learning system according to any one of (1) to (17) above. (19) An information processing system including an information processing device, An imaging unit that captures images for learning a three-dimensional reconstruction model, the imaging unit being configured to be able to acquire a plurality of types of captured images as the images, and acquiring scene information indicating a difference in the scene to be imaged by the imaging unit, Based on the acquired scene information, selecting the type of the three-dimensional reconstruction model used for the learning and the type of the captured image, Performing learning of a three-dimensional reconstruction model according to the selected model type using the captured images of the selected type. Learning method. (20) An imaging unit that captures images for learning a three-dimensional reconstruction model, the imaging unit being configured to be able to acquire a plurality of types of captured images as the images, A scene information acquisition unit that acquires scene information indicating a difference in the scene to be imaged by the imaging unit, A selection processing unit that selects the type of the three-dimensional reconstruction model used for the learning and the type of the captured image based on the scene information acquired by the scene information acquisition unit. Imaging device.

Explanation of symbols

[0199] 1.1B Learning System 2,2B Imaging device 3 Subject 4,4B Server Device Microsoft Network F41 Learning Processing Unit 21 Imaging Department 211-level grayscale image acquisition unit 211a Grayscale sensor 212 Event Image Acquisition Section 212a Event Sensor 213 Depth Image Acquisition Unit 213a Depth Sensor 213b Light-emitting part 22, 22A, 22B, 22C Control Unit 23 Memory section 24 Input section 25 Output section 26 IMU 27 Communications Department 28 buses 41,41B Processor 42 ROM 43 RAM 44 bus 45 Input / Output Interface (I / F) 46 Input section 47 Display section 48 Audio output section 49 Memory section 50 Communications Department 51 Drive 52 Removable recording media F1 Scene Information Acquisition Unit F2 Selection Processing Unit F3 Imaging Control Unit F4 Position and orientation estimation unit F5 Deblurring Section F6 Frequency adjustment unit F7 Guide Presentation Processing Unit

Claims

1. An imaging unit for capturing images for training a three-dimensional reconstruction model, the imaging unit being configured to acquire multiple types of captured images as said images, The aforementioned imaging unit acquires scene information indicating the type of scene to be imaged, A selection processing unit selects the type of the three-dimensional reconstruction model to be used for learning and the type of the captured image based on the scene information acquired by the scene information acquisition unit. The system comprises a learning processing unit that performs learning of a three-dimensional reconstruction model based on the model type selected by the selection processing unit, using captured images of the type selected by the selection processing unit. Learning system.

2. The imaging control unit controls the imaging unit so that only images of the type selected by the selection processing unit are acquired from among the types of images that the imaging unit can acquire. The learning system according to claim 1.

3. The selection processing unit, As part of selecting the type of three-dimensional reconstruction model, a selection is made from among multiple three-dimensional reconstruction models that use different types of images for error calculation during learning. The learning system according to claim 1.

4. The aforementioned scene information acquisition unit, The scene is determined and the scene information is acquired based on the image captured by the imaging unit. The learning system according to claim 1.

5. The aforementioned scene information acquisition unit, As part of the scene determination, it is determined whether or not the subject is a moving subject. The learning system according to claim 4.

6. The selection processing unit, If the scene information indicates the moving subject, the type of three-dimensional reconstruction model selected is a three-dimensional reconstruction model with a time-deformed field. The learning system according to claim 5.

7. The imaging unit is configured to acquire an event image as one of the multiple types of captured images. The selection processing unit, If the scene information indicates the moving subject, the type of three-dimensional reconstruction model selected is the type of three-dimensional reconstruction model that uses the event image for error calculation. The learning system according to claim 5.

8. The aforementioned scene information acquisition unit, As part of the scene determination, a determination is made as to whether or not the subject has a complex shape. The learning system according to claim 4.

9. The imaging unit is configured to acquire a depth image as one of the multiple types of captured images, The selection processing unit, If the scene information indicates that the shape is complex, the type of three-dimensional reconstruction model selected is one in which the depth image is used for error calculation. The learning system according to claim 8.

10. The selection processing unit, Based on the confidence estimation result of the depth information shown in the depth image, the type of three-dimensional reconstruction model is selected from between a three-dimensional reconstruction model that uses the relative error of the depth information for error calculation and a three-dimensional reconstruction model that uses the absolute error of the depth information for error calculation. The learning system according to claim 9.

11. The imaging unit is configured to acquire a grayscale image and an event image. Based on the learning grayscale images, which are multiple grayscale images captured while changing the viewpoint for the purpose of learning, a position and orientation estimation unit estimates position and orientation information, which is information indicating the viewpoint position and direction of the gaze at the time of capture, for each of the learning grayscale images. The system includes a deblurring processing unit that performs deblurring on the learning grayscale image based on the event image, The position and orientation estimation unit, Based on the deblurred image, which is the learning grayscale image deblurred by the deblurring processing unit, the position and orientation information is estimated. The learning system according to claim 7.

12. The aforementioned deblurring processing unit is As the deblurred image, a deblurred image is generated with a predetermined timing within the exposure period of the learning grayscale image to be subjected to the deblurring process as the image generation reference time. The position and orientation estimation unit, A timestamp indicating the image generation reference time for the deblurred image is added to the position and orientation information estimated from the aforementioned deblurred image. The learning system according to claim 11.

13. The deblurring processing unit performs the deblurring process on the condition that the magnitude of motion of the subject estimated based on the image captured by the imaging unit is greater than a reference value. The learning system according to claim 11.

14. The aforementioned learning processing unit, When the selection processing unit selects the type of three-dimensional reconstruction model to be used for error calculation of the event image, the three-dimensional reconstruction model is trained using a grayscale image that has not been deblurred by the deblur processing unit as the grayscale image for training. The learning system according to claim 11.

15. The aforementioned learning processing unit, As a three-dimensional reconstruction model that uses the aforementioned event image for error calculation, the learning process uses a three-dimensional reconstruction model that calculates the error between the training grayscale image and the sum of rendering images at multiple timings within the exposure period of the training grayscale image as error information for learning. The learning system according to claim 14.

16. The system includes a first frequency adjustment unit that adjusts the imaging frequency for the grayscale images that the imaging unit can acquire, based on the magnitude of the subject's movement estimated from the image captured by the imaging unit. The learning system according to claim 1.

17. The system includes a second frequency adjustment unit that adjusts the imaging frequency for the grayscale images that the imaging unit can acquire, based on the degree of complexity of the subject's shape estimated from the image captured by the imaging unit. The learning system according to claim 1.

18. While the imaging unit is capturing images to be used for learning, the system includes a guide presentation processing unit that presents guide information related to the movement speed of the imaging viewpoint to the user based on the error between the target movement speed of the imaging viewpoint determined based on the images captured by the imaging unit and the measured movement speed of the imaging viewpoint. The learning system according to claim 1.

19. Information processing system equipped with information processing device, An imaging unit for capturing images for training a three-dimensional reconstruction model, wherein the imaging unit is configured to acquire multiple types of captured images, and it acquires scene information indicating the type of scene to be captured. Based on the acquired scene information, the type of the three-dimensional reconstruction model to be used for learning and the type of the captured image are selected. The three-dimensional reconstruction model is trained using the selected model type, with the selected type of captured images. Learning methods.

20. An imaging unit for capturing images for training a three-dimensional reconstruction model, the imaging unit being configured to acquire multiple types of captured images as said images, The aforementioned imaging unit acquires scene information indicating the type of scene to be imaged, The system includes a selection processing unit that selects the type of the three-dimensional reconstruction model to be used for learning and the type of the captured image based on the scene information acquired by the scene information acquisition unit. Imaging device.

Images