Shape estimation system, shape estimation method, and program
The shape estimation system improves accuracy by using high-resolution time-series data and spatial smoothness constraints to optimize shape estimation in harsh environments.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NIPPON TELEGRAPH & TELEPHONE CORP
- Filing Date
- 2023-01-23
- Publication Date
- 2026-07-23
Smart Images
Figure 0007894105000022 
Figure 0007894105000023 
Figure 0007894105000024
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to a shape estimation system, a shape estimation method, and a program.
Background Art
[0002] Estimation of the three-dimensional shape of an object or environment in a harsh environment remains an important issue in machine vision and its applications. For example, by estimating the three-dimensional shape, it becomes possible to identify the type and attributes of an object in an image and understand the spatial structure with higher accuracy. For example, methods for estimating the shape from a plurality of visible images have been disclosed.
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, for example, when the lighting conditions are poor, visibility is poor due to fog, etc., or there are obstacles, the shape estimation accuracy significantly decreases in situations where it is impossible to obtain a high-definition image with a visible image. In particular, the estimation of the shape in a situation where the object to be measured, such as a translucent object or a wall surface, is shielded from the sensor is very difficult when using existing visible images.
[0005] In recent years, Non-Patent Document 2 has disclosed a method for estimating the shape of an object on the other side of a wall from reflected and scattered light on a relay wall surface, using active measurement with pulsed laser light and a SPAD (Single Photon Avalanche Diode) sensor (hereinafter sometimes referred to as single-photon measurement), known as NLOS (Non-Line-of-Sight Launch System) imaging, and a shape estimation method using the measurement results.
[0006] However, these methods cannot be applied to shape estimation in more common imaging environments, such as objects seen through translucent objects.
[0007] In view of the above circumstances, the present invention aims to provide a technique for improving the accuracy of shape estimation. [Means for solving the problem]
[0008] One aspect of the present invention includes a data input unit that acquires observational data measured using an optical system in the presence of one or more objects in the environment as time-series data having a time resolution greater than or equal to a predetermined time resolution; an observation model input unit that acquires the parameters of the observation system that obtains the observational data obtained by the data input unit, the integral kernel of the spatiotemporal convolution function, and a transformation function model as an observation model; a preprocessing unit that transforms the time-series data obtained by the data input unit by normalization processing; and a term obtained based on the time-series data transformed by the preprocessing unit, which expresses the condition that the difference between the observation model and the time-series data is less than a predetermined value. The shape estimation system comprises: an optimization unit that optimizes the density field of an object so as to minimize an optimization function, which is a predetermined function consisting of an observation data constraint term that represents the observed data and a shape constraint term that expresses the condition that the estimated object shape is spatially smooth; an observation model estimation unit that estimates the parameters of the observation system, the integral kernel of the spatiotemporal convolution function and the transformation function model by minimizing the optimization function using the parameters of the observation system obtained from the observation model input unit, the integral kernel of the spatiotemporal convolution function and the transformation function model as initial values; and a model output unit that outputs the optimization result of the optimization unit.
[0009] One aspect of the present invention includes a data input step of acquiring observational data, measured using an optical system in the presence of one or more objects in the environment, as time-series data having a time resolution greater than or equal to a predetermined time resolution; an observation model input step of acquiring the parameters of the observation system that obtains the observational data obtained in the data input step, the integral kernel of the spatiotemporal convolution function, and a transformation function model as an observation model; a preprocessing step of transforming the time-series data obtained in the data input step by normalization processing; and a term obtained based on the time-series data transformed in the preprocessing step, wherein the difference between the observation model and the time-series data is less than a predetermined value. The shape estimation method comprises: an optimization step of optimizing the density field of an object so that a predetermined optimization function, which consists of an observation data constraint term that represents the object and a shape constraint term that represents the condition that the estimated object shape is spatially smooth, is minimized; an observation model estimation step of estimating the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model obtained in the observation model input step, by minimizing the optimization function; and a model output step of outputting the optimization result of the optimization step.
[0010] One aspect of the present invention is a program for causing a computer to function as the shape estimation system described above. [Effects of the Invention]
[0011] According to the present invention, the accuracy of shape estimation can be improved. [Brief explanation of the drawing]
[0012] [Figure 1] An explanatory diagram illustrating the outline of the shape estimation system of the first embodiment. [Figure 2] A diagram showing an example of the configuration of the control unit included in the learning device in the first embodiment. [Figure 3] A figure showing a first example of modeling in the first embodiment. [Figure 4] A diagram showing a second example of modeling in the first embodiment. [Figure 5] It is a diagram showing a first example of an optical path in the first embodiment. [Figure 6] It is a diagram showing a second example of an optical path in the first embodiment. [Figure 7] It is a diagram showing a third example of an optical path in the first embodiment. [Figure 8] A diagram showing an example of the configuration of a control unit included in the shape estimation device in the first embodiment. [Figure 9] A flowchart showing an example of the flow of processing executed by the control unit included in the learning device of the first embodiment. [Figure 10] A flowchart showing an example of the flow of processing executed by the control unit included in the shape estimation device of the first embodiment. [Figure 11] A diagram showing an example of the hardware configuration of the learning device of the first embodiment. [Figure 12] A diagram showing an example of the hardware configuration of the shape estimation device of the first embodiment. [Figure 13] An explanatory diagram explaining the outline of the shape estimation system of the second embodiment. [Figure 14] A diagram showing an example of the configuration of a control unit included in the shape estimation device in the second embodiment. [Figure 15] A flowchart showing an example of the flow of processing executed by the control unit included in the shape estimation device in the second embodiment. [Figure 16] A diagram showing an example of the hardware configuration of the shape estimation device in the second embodiment. [Figure 17] A diagram showing an example of an optical system for obtaining observation data obtained by measuring the environment in the first and second embodiments.
Embodiments for Carrying Out the Invention
[0013] (First Embodiment) FIG. 1 is an explanatory diagram explaining the outline of the shape estimation system 100 of the first embodiment. The shape estimation system 100 includes a learning device 1 and a shape estimation device 2.
[0014] The learning device 1 executes learning processing. The learning processing is processing for learning a shape estimation model by a machine learning method such as deep learning. The shape estimation model is a mathematical model that estimates the shape of an estimation target based on the input data. The shape of the estimation target is, for example, the density field of the estimation target.
[0015] The data input to the shape estimation model is time-series data (hereinafter referred to as "input time-series data") obtained using an optical system in the environment where the estimation target exists. The time-series data obtained in the environment where the estimation target exists is, for example, time-series data obtained by single-photon measurement executed in the environment where the estimation target exists.
[0016] In the field of machine learning, there is an expression "trained". Therefore, using this expression, the shape estimation model at the time when a learning end condition, which is a predetermined condition regarding the end of learning, is satisfied is a trained shape estimation model. The learning end condition may be, for example, a condition that the learning of the shape estimation model has been performed a predetermined number of times, or a condition that the change in the shape estimation model due to learning is smaller than a predetermined change.
[0017] The trained shape estimation model is used by the shape estimation device 2. The shape estimation device 2 estimates the shape of the estimation target using the trained shape estimation model.
[0018] <Regarding the learning device 1> The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) connected by a bus and a memory 92, and executes a program. FIG. 2 is a diagram showing an example of the configuration of the control unit 11 included in the learning device 1 in the first embodiment.
[0019] The control unit 11 includes a data input unit 101, an observation model input unit 102, a preprocessing unit 201, an observation model estimation unit 202, an observation data constraint unit 203, a shape constraint unit 204, an optimization unit 205, and a model output unit 300. Hereafter, subscripts may be represented using underscores. When using underscores, for example, A_B would be represented as A B It means...
[0020] The data input unit 101 acquires input time series data. Specifically, the data input unit 101 is characterized by acquiring observational data measured in an environment where one or more objects are present, as time series data with high temporal resolution. Therefore, the input time series data is two-dimensional time series data.
[0021] More specifically, the data input unit 101 is characterized by inputting observation data (time-series data of photon counts at each spatial coordinate) obtained by measuring the environment when one or more objects are present in the observation environment, for example, by single-photon measurement. Alternatively, the time-series data referred to here may be time-series data obtained using a sensor such as a ToF (Time of Flight) sensor. Hereinafter, for example, the time-series data will be represented as I_raw. For example, I_raw may include three arguments: coordinates x, y, and t. That is, I_raw (x,y,t) is the signal strength at observation time t at coordinate (x,y), and in the case of single-photon measurement, this may be the number of photons.
[0022] The observation model input unit 102 acquires the parameters of the optical system and the convolution kernels in the time and space directions. The terms "convolution kernels in the time and space directions" are well known to those skilled in the art. Specifically, the observation model input unit 102 is characterized by acquiring the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model as the observation model. More specifically, the observation model input unit 102 is characterized by acquiring the parameters of the optical system and the convolution kernels in the time and space directions (hereinafter sometimes referred to as the integral kernel).
[0023] The parameters of the optical system refer to the parameters that describe the optical system used to acquire the input time-series data. These parameters could be, for example, the resolution of the XY coordinate axes and time axis of the optical system, or they could be the rough scale to the object. The extent of this rough scale to the object is well known. The degree of "roughness" is also well known.
[0024] For example, the integral kernel can be a rectangular function, an exponential function, a Gaussian function, etc., which will be denoted as W(tt') below. For example, in the case of a rectangular function, it can be expressed as follows.
[0025]
number
[0026]
number
[0027]
number
[0028] Here, Δt is a parameter for adjusting the magnitude of the time scale, and τ or η is a value predetermined by the user. Furthermore, the observation model input unit 102 may acquire parameters of the observation system. More specifically, it may acquire the position of the optical center, the origin of the coordinate system, the speed of light, and the position of the focal point (whether it is confocal, etc.).
[0029] Furthermore, the observation model input unit 102 may also accept a function that outputs the density field σ(x, y, z) and reflectance or albedo ρ(x, y, z) of an object from three-dimensional coordinates (x, y, z). Here, a function (hereinafter also referred to as a model) is a function that, when given a certain coordinate value as input, outputs the density field and the reflectance of the object at that coordinate value. For example, this function may be given as a function that includes optimized fitting parameters, or as a function with a more complex nested structure.
[0030] Alternatively, this function may be represented by a neural network, or more specifically, by a method such as a neural radiance field. More specifically, as shown in Figure 3 or Figure 4, it can be modeled with a positional encoding PE that encodes the space with a periodic function and a function that has a nested structure of multiple functions. Figure 3 is a diagram showing a first example of the modeling in the first embodiment. Figure 4 is a diagram showing a second example of the modeling in the first embodiment. More specifically, a function like the following can be input. Then, for example, the parameters (w_F, w_G, w_H) included in functions such as F, H, G can be optimized by the optimization unit 205 described later.
[0031]
number
[0032]
number
[0033] Thus, the observation model input unit 102 acquires, for example, the spatial and temporal resolution as parameters of the optical system, the approximate scale to the object whose shape is to be estimated, and the general shape of the integral kernel in the temporal and spatial directions. The rectangular function, exponential function, and Gaussian function expressed as W(tt') above are all examples of the general shape of the integral kernel in the temporal and spatial directions. The acquisition of the observation model by the observation model input unit 102 is, for example, by reading from a storage unit that stores information on the target to be acquired in advance.
[0034] The preprocessing unit 201 is characterized by transforming the two-dimensional time series data acquired by the data input unit 101. Specifically, it is characterized by transforming the values by normalization processing, etc. For example, if the time series data acquired by the data input unit 101 is represented as I_raw (x,y,t), and I_raw (x,y,t) includes three arguments as coordinates x,y and t, the preprocessing unit 201 calculates the average in the time direction.<I_raw (x,y,t)> _t, average in the spatial direction<I_raw (x,y,t)> _(x,y), or the average of all arguments<I_raw (x,y,t)> The data can be normalized using _(x,y,t). More specifically, it can be averaged using the following formula.
[0035]
number
[0036]
number
[0037]
number
[0038] The observation model estimation unit 202 is characterized by using the convolution kernels in the time and space directions obtained by the observation model input unit 102 as initial values, and reestimating the parameters of the observation model (convolution kernels in the time and space directions and parameters of the optical system) so that the optimization function defined in the optimization unit 205 is minimized. In other words, it is sufficient to reestimate the parameters such as τ or η included in the time scale Δt or W in equations (1) to (3). The optimization function defined in the optimization unit 205 is a well-known predetermined function consisting of an observation data constraint term and a shape constraint term, which will be described later.
[0039] The observation data constraint unit 203 is characterized by constructing observation data constraint terms defined in the optimization unit 205 such that the difference between the observation model and the time series data is minimized, using the time series data preprocessed in the preprocessing unit 201 and the parameters of the observation model estimated in the observation model estimation unit 202 (convolution kernels in the time and spatial directions and parameters of the optical system). More specifically, if we first assume that all optical paths are linear as shown in Figures 5 to 7, we can model the light reflected from the object, or the light reflected from the object via the shielding plate and the shielding plate, and express the intensity of the light reaching the sensor surface (number of single photons) as I_1 (x,y,t) or I_2 (x,y,t) as follows.
[0040] Figure 5 shows a first example of the optical path in the first embodiment. Figure 6 shows a second example of the optical path in the first embodiment. Figure 7 shows a third example of the optical path in the first embodiment.
[0041]
number
[0042]
number
[0043] Note that Γ_0 is a parameter, I_0 is the intensity of light reflected from the shielding object, and δ(·) is the Dirac delta function. Alternatively, if we assume that the light passing through the shielding plate diffuses radially (the shielding plate diffusely reflects light), the intensity of light reaching the sensor surface (number of single photons) can be calculated using the following formula.
[0044]
number
[0045] Alternatively, if we assume that the light passing through the shielding plate is a mixture of light that is diffusely reflected radially (reflected by the shielding plate) and light that passes through directly, the intensity of the light reaching the sensor surface (number of single photons) can be calculated using the following formula.
[0046]
number
[0047] The observation data constraint term given by the observation data constraint unit 203 can be defined as a function whose value increases as the difference between the light intensity (number of single photons) reaching the sensor surface calculated in this way and the normalized observed value I(x,y,t) obtained by the preprocessing unit 201 increases. More specifically, it can be given as the L2 norm as shown in equations (13) to (16) below.
[0048]
number
[0049]
number
[0050]
number
[0051]
number
[0052] The shape constraint unit 204 is characterized by constraining the density field (or the density field plus reflectance) by estimating that it be spatially smooth, adding this as a constraint term to the optimization function defined in the optimization unit 205, and minimizing this optimization function. More specifically, for example, the constraint can be set that the density σ(x,y,z), reflectance, or albedo ρ(x,y,z) is spatially smooth. More specifically, the constraint can be set that the Lp norm of the magnitude of the spatial derivative of the density, reflectance, or albedo is small, as shown in the following equation.
[0053]
number
[0054]
number
[0055] Alternatively, for example, a constraint can be imposed that the density σ(x,y,z), reflectance, or albedo ρ(x,y,z) is spatially sparse. More specifically, a constraint can be imposed that the Lp norm of the magnitude (absolute value) of the density, reflectance, or albedo is small, as shown in the following equation.
[0056]
number
[0057]
number
[0058] The optimization unit 205 optimizes the density field (or reflectance in addition to the density field) of the object to be estimated so as to minimize an optimization function consisting of an observation data constraint term, which is characterized by reducing the difference between the observation model defined in the observation data constraint unit 203 and the two-dimensional time series data, and a shape constraint term, which is defined in the shape constraint unit 204. Specifically, for example, the parameters (w_F, w_G, w_H) included in functions such as F, G, and H expressed in equations (4) and (5) can be minimized so as to minimize the function so as to minimize the energy function defined by the following equation.
[0059]
number
[0060] Here, a or b are parameters used to adjust the magnitude of each term in equation (21), and can be provided in advance by the user. Furthermore, methods such as gradient descent can be used to minimize the optimization function of equation (21).
[0061] The model output unit 300 is characterized by outputting the model optimized by the optimization unit 205. The target of optimization by the optimization unit 205 is the shape estimation model, and the model optimized by the optimization unit 205 is the trained shape estimation model.
[0062] The output format of the model output unit 300 may, for example, be output to a display unit in real time, or it may be saved as image data to a storage unit such as semiconductor memory or an HDD (hard disk drive). This storage unit may be provided within the shape restoration device, or an external storage device may be used.
[0063] <About Shape Estimation Device 2> The shape estimation device 2 receives multiple input data sets and outputs the calculated identification results as required. The output format may be, for example, output in real time to a display unit, or it may be saved as image data to a storage unit such as a semiconductor memory or an HDD (Hard Disk Drive). This storage unit may be provided within the shape estimation device 2, or an external storage device may be used.
[0064] The shape estimation device 2 includes a control unit 21 which has a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. Figure 8 is a diagram showing an example of the configuration of the control unit 21 of the shape estimation device 2 in the first embodiment.
[0065] The control unit 21 includes a model input unit 103, a restored coordinate input unit 104, an inference unit 206, and an output unit 301.
[0066] The model input unit 103 is characterized by acquiring the model output by the model output unit 300 for inference. More specifically, it is sufficient to acquire functions such as F, G, and H represented by equations (4) and (5), and the parameters (w_F, w_G, w_H) optimized by the optimization unit 205.
[0067] The reconstructed coordinate input unit 104 is characterized by acquiring the coordinates of the location of the target to be estimated from the model acquired by the model input unit 103 as input values necessary for inference. More specifically, the reconstructed coordinate input unit 104 only needs to acquire the spatial coordinates (x, y, z) and incident angle (θ, Φ) of the density field and albedo (or reflectance) to be calculated.
[0068] The inference unit 206 is characterized by calculating the shape of an object (for example, the density field or the reflectance and albedo of the object) from the model obtained by the model input unit 103 and the coordinates obtained by the restored coordinate input unit 104. More specifically, the density field σ(x, y, z) and albedo ρ(x, y, z, θ, Φ) can be calculated from equations (4) and (5) and the parameters (w_F, w_G, w_H) optimized by the optimization unit 205, and the spatial coordinates (x, y, z) and incident angle (θ, Φ) input by the restored coordinate input unit 104.
[0069] The output unit 301 is characterized by outputting the shape of the object calculated by the inference unit 206. More specifically, it should output the density field σ(x, y, z) and albedo ρ(x, y, z, θ, Φ) calculated by the inference unit 206.
[0070] Figure 9 is a flowchart showing an example of the processing flow performed by the control unit 11 of the learning device 1 of the first embodiment. The data input unit 101 acquires observation data (step S101). Next, the observation model input unit 102 acquires an observation model (step S102). Next, the preprocessing unit 201 preprocesses the data acquired by the data input unit (step S103). Next, the observation model estimation unit 202 estimates the parameters of the observation model from the observation model acquired by the observation model input unit 102 (step S104). Next, the observation data constraint unit 203 constructs the observation data constraint terms (step S105).
[0071] Next, the shape constraint unit 204 constitutes the shape constraint term (step S106). Then, the optimization unit 205 optimizes the density field (or density field plus reflectivity) of each object to be estimated, based on the two-dimensional time series data preprocessed by the preprocessing unit 201 and the parameters of the observation model estimated by the observation model estimation unit 202 (convolution kernels in the time and spatial directions and parameters of the optical system), so that the difference between the observation model and the two-dimensional time series data becomes small (step S107). Note that minimizing the difference between the observation model and the two-dimensional time series data means that the optimization function consisting of the observation data constraint term and the shape constraint term is minimized.
[0072] Next, the control unit 11 performs a convergence check, and if the optimal value of the optimization function is below a certain level, or if the number of iterations is above a certain level, the optimization is terminated. More specifically, the control unit 11 performs a convergence check (step S108). If convergence is achieved (step S108: YES), the optimization is terminated. On the other hand, if convergence is not achieved (step S108: NO), the process returns to step S104. Note that convergence means, as described above, that the optimal value of the optimization function is below a certain level, or that the number of iterations is above a certain level. If convergence is achieved, the model output unit 300 then outputs the density field of each object obtained by optimization (step S109).
[0073] Figure 10 is a flowchart showing an example of the processing flow performed by the control unit 21 of the shape estimation device 2 of the first embodiment. The model input unit 103 acquires the model output by the model output unit 300 for inference (step S201). Next, the reconstruction coordinate input unit 104 acquires the coordinates to be estimated from the model acquired by the model input unit 103 as input values necessary for inference (step S202). Next, the inference unit 206 calculates the shape of the object (e.g., density field) from the model acquired by the model input unit 103 and the coordinates acquired by the reconstruction coordinate input unit 104 (step S203). Next, the output unit 301 outputs the shape of the object calculated by the inference unit 206 (step S204).
[0074] Figure 11 shows an example of the hardware configuration of the learning device 1 according to the first embodiment. As described above, the learning device 1 includes a control unit 11 which has a processor 91 such as a CPU and a memory 92 connected by a bus, and executes a program. The learning device 1 functions as a device comprising a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15 by executing the program.
[0075] More specifically, the processor 91 reads the program stored in the storage unit 14 and stores the read program in the memory 92. By executing the program stored in the memory 92, the learning device 1 functions as a device comprising a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.
[0076] The control unit 11 controls the operation of various functional units provided by the learning device 1.
[0077] The input unit 12 includes input devices such as a mouse, keyboard, or touch panel. The input unit 12 may also be configured as an interface for connecting these input devices to the learning device 1. The input unit 12 receives various types of information for input to the learning device 1.
[0078] The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless connection. The external device is, for example, the shape estimation device 2. Through communication with the shape estimation device 2, the communication unit 13 transmits, for example, the model output by the model output unit 300 to the shape estimation device 2.
[0079] The observation data obtained by the data input unit 101 is, for example, information input to the input unit 12 or the communication unit 13.
[0080] The memory unit 14 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The memory unit 14 stores various information related to the learning device 1. The memory unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The memory unit 14 stores various information generated by processing performed by, for example, the control unit 11. The memory unit 14 stores, for example, parameters, the integral kernel of the spatiotemporal convolution function, and the transformation function model in advance.
[0081] The output unit 15 outputs various types of information. The output unit 15 is comprised of a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may also be configured as an interface for connecting these display devices to the learning device 1. The output unit 15 outputs information input to, for example, the input unit 12 or the communication unit 13. The output unit 15 may also output information output by, for example, the model output unit 300.
[0082] Figure 12 shows an example of the hardware configuration of the shape estimation device 2 according to the first embodiment. As described above, the shape estimation device 2 includes a control unit 21 which has a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. The shape estimation device 2 functions as a device comprising the control unit 21, input unit 22, communication unit 23, storage unit 24 and output unit 25 by the execution of the program.
[0083] More specifically, the processor 93 reads the program stored in the storage unit 24 and stores the read program in the memory 94. By executing the program stored in the memory 94, the shape estimation device 2 functions as a device comprising a control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.
[0084] The control unit 21 controls the operation of various functional parts of the shape estimation device 2.
[0085] The input unit 22 includes input devices such as a mouse, keyboard, or touch panel. The input unit 22 may also be configured as an interface for connecting these input devices to the shape estimation device 2. The input unit 22 receives various types of information for input to the shape estimation device 2.
[0086] The communication unit 23 is configured to include a communication interface for connecting the shape estimation device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless connection. The external device is, for example, the learning device 1. Through communication with the learning device 1, the communication unit 23 acquires, for example, the model output by the model output unit 300.
[0087] The model obtained by the model input unit 103 and the input values obtained by the restored coordinate input unit 104 are, for example, information input to the input unit 22 or the communication unit 23.
[0088] The storage unit 24 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 24 stores various information related to the shape estimation device 2. The storage unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The storage unit 24 also stores various information generated by processing performed by, for example, the control unit 21.
[0089] The output unit 25 outputs various types of information. The output unit 25 is comprised of a display device such as a CRT display, liquid crystal display, or organic EL display. The output unit 25 may be configured as an interface for connecting these display devices to the shape estimation device 2. The output unit 25 outputs information input to, for example, the input unit 22 or the communication unit 23. The output unit 25 may also output information output by, for example, the output unit 301.
[0090] Thus, the shape estimation system 100 is A data input unit 101 acquires observational data, measured using an optical system, of the environment when one or more objects are present in that environment, as time-series data with a time resolution greater than or equal to a predetermined time resolution. The observation model input unit 102 acquires the parameters of the observation system that obtains the observation data obtained by the data input unit 101, the integral kernel of the spatiotemporal convolution function, and the transformation function model as an observation model. A preprocessing unit 201 transforms the time-series data obtained from the data input unit 101 by normalization processing, An optimization unit 205 optimizes the density field of an object so that an optimization function, which is a predetermined function consisting of an observation data constraint term (a term obtained based on the time series data converted by the preprocessing unit 201, and which expresses the condition that the difference between the observation model and the time series data is smaller than a predetermined value) and a shape constraint term (which expresses the condition that the estimated object shape is spatially smooth), is minimized. The observation model estimation unit 202 estimates the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model by minimizing an optimization function, using the parameters of the observation system obtained from the observation model input unit 102, the integral kernel of the spatiotemporal convolution function, and the transformation function model as initial values. A model output unit 300 outputs the optimization results of the optimization unit 205, It is equipped with.
[0091] Therefore, the shape estimation system 100 configured in this way can optimize the mathematical model used to estimate the shape. Consequently, the shape estimation system 100 can improve the accuracy of shape estimation.
[0092] (Second Embodiment) Figure 13 is an explanatory diagram illustrating the outline of the shape estimation system 100a of the second embodiment. The shape estimation system 100a includes a shape estimation device 2a.
[0093] The shape estimation device 2a receives multiple input data sets and outputs the calculated identification results as required. The output format may be, for example, output in real time to a display unit, or it may be stored as image data in a storage unit such as a semiconductor memory or HDD. This storage unit may be provided within the shape estimation system 100, or an external storage device may be used.
[0094] The shape estimation device 2a includes a control unit 21a which has a processor 93a such as a CPU and a memory 94a connected by a bus, and executes a program. Figure 14 is a diagram showing an example of the configuration of the control unit 21a of the shape estimation device 2a in the second embodiment.
[0095] Hereafter, components having functions similar to those of the shape estimation system 100 will be denoted by the same reference numerals as in Figures 1 to 12, and their explanations will be omitted.
[0096] The control unit 21a includes a data input unit 101, an observation model input unit 102, a background data input unit 105, a preprocessing unit 201, an observation model estimation unit 202, an observation data constraint unit 203a, a shape constraint unit 204, an optimization unit 205a, and an observation data separation unit 207.
[0097] The background data input unit 105 is characterized by acquiring two-dimensional time-series data (time-series data of the number of photons at each x and y coordinate) as background data by measuring the observation environment (e.g., a translucent object in the foreground) using single-photon measurement. More specifically, it is sufficient to acquire I_0(x, y, t) given in equations (10) to (12) as background data. The definition of the observation environment is well known. The definition of background data is also well known.
[0098] The observation data constraint unit 203a is characterized by configuring an observation data constraint term in the observation model such that the difference between the observation data separated for each object obtained by the observation data separation unit 207 and the two-dimensional time series data becomes small. Here, the observation model refers to the observation model defined in the observation data constraint unit 203 of the first embodiment. Thus, the observation data constraint term configured in the observation data constraint unit 203a is a more specific form of the observation data constraint term configured in the observation data constraint unit 203.
[0099] The optimization unit 205a optimizes the density field (or reflectance in addition to the density field) of each object to be estimated, based on the two-dimensional time series data preprocessed by the preprocessing unit 201 and the parameters of the observation model estimated by the observation model estimation unit 202 (convolution kernels in the time and spatial directions and parameters of the optical system), so as to minimize the difference between the observation model and the two-dimensional time series data. Minimizing the difference between the observation model and the two-dimensional time series data means minimizing the optimization function consisting of the observation data constraint term and the shape constraint term.
[0100] The observation data separation unit 207 is characterized by separating the two-dimensional time series data, which has been preprocessed by the preprocessing unit 201, into background data acquired by the background data input unit 105 and observed values corresponding to each of the objects to be estimated. More specifically, this can be done by dividing I_0(x,y,t) by I_2(x,y,t), I_3(x,y,t), and I_4(x,y,t) given in equations (10) to (12).
[0101] Figure 15 is a flowchart showing an example of the processing flow performed by the control unit 21a of the shape estimation device 2a in the second embodiment. Note that the processing flow shown in Figure 15 is an example of the processing flow during learning. The processing flow during inference is the same as the processing flow performed by the shape estimation device 2 in the first embodiment, so its explanation is omitted.
[0102] The data input unit 101 acquires observation data (step S301). Next, the observation model input unit 102 acquires the observation model (step S302). Next, the background data input unit 105 acquires background data (step S303). Next, the preprocessing unit 201 preprocesses the data acquired by the data input unit 101 (step S304). Next, the observation model estimation unit 202 estimates the parameters of the observation model from the observation model acquired by the observation model input unit 102 (step S305). Next, the observation data separation unit 207 separates the two-dimensional time series data that has been preprocessed by the preprocessing unit 201 into background data acquired by the background data input unit 105 and the observation values corresponding to each object (step S306).
[0103] Next, the observation data constraint unit 203a constitutes the observation data constraint term (step S307). Next, the shape constraint unit 204 constitutes the shape constraint term (step S308).
[0104] Next, the optimization unit 205a optimizes the density field (or reflectance in addition to the density field) of each object so as to minimize the difference between the observation model and the two-dimensional time series data, based on the two-dimensional time series data preprocessed by the preprocessing unit 201 and the parameters of the observation model estimated by the observation model estimation unit 202a (convolution kernels in the time and spatial directions and parameters of the optical system) (step S309).
[0105] Next, the control unit 21a performs a convergence check, and if the optimal value of the optimization function is below a certain level, or if the number of iterations is above a certain level, the optimization is terminated. More specifically, the control unit 21a performs a convergence check (step S310). If convergence is achieved (step S310: YES), the optimization is terminated. On the other hand, if convergence is not achieved (step S310: NO), the process returns to step S305. Note that convergence means, as described above, that the optimal value of the optimization function is below a certain level, or that the number of iterations is above a certain level. Next, the model output unit 300 outputs the density field of each object obtained by optimization (step S311).
[0106] Figure 16 shows an example of the hardware configuration of the shape estimation device 2a in the second embodiment. As described above, the shape estimation device 2a includes a control unit 21a which has a processor 93a such as a CPU and a memory 94a connected by a bus, and executes a program. The shape estimation device 2a functions as a device comprising the control unit 21a, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25 through the execution of the program.
[0107] More specifically, the processor 93a reads the program stored in the storage unit 24 and stores the read program in the memory 94a. By executing the program stored in the memory 94a, the shape estimation device 2a functions as a device comprising a control unit 21a, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25. The control unit 21a controls the operation of the various functional units of the shape estimation device 2a.
[0108] Thus, the shape estimation system 100a is A data input unit 101 acquires observational data, measured using an optical system, of the environment when one or more objects are present in the environment, as time-series data with a time resolution greater than or equal to a predetermined time resolution. The observation model input unit 102 acquires the parameters of the observation system that obtains the observation data obtained by the data input unit 101, the integral kernel of the spatiotemporal convolution function, and the transformation function model as an observation model. A preprocessing unit 201 that converts the time-series data obtained by the data input unit 101 through normalization processing, An optimization unit 205a optimizes the density field of the object so that an optimization function, which is a predetermined function consisting of an observation data constraint term, which is a term obtained based on the time series data converted by the preprocessing unit 201 and expresses the condition that the difference between the observation model and the time series data is smaller than a predetermined value, and a shape constraint term, which expresses the condition that the estimated object shape is spatially smooth, is minimized. An observation model estimation unit 202 estimates the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model by minimizing the optimization function, using the parameters of the observation system obtained from the observation model input unit 102, the integral kernel of the spatiotemporal convolution function, and the transformation function model as initial values. The model output unit 300 outputs the optimization results of the optimization unit 205a, A background data input unit 105 acquires time-series data measuring only the observation environment as background data, The observation data separation unit 207 separates the two-dimensional time-series data, which has been pre-processed by the pre-processing unit 201, into background data and observed values corresponding to each of the objects. Equipped with, The aforementioned observation data constraint term expresses the condition that the difference between the observation data separated for each object acquired by the observation data separation unit and the time series data is smaller than a predetermined value.
[0109] The shape estimation system 100a of the second embodiment configured in this way can optimize the mathematical model used to estimate the shape. Therefore, the shape estimation system 100a can improve the accuracy of shape estimation.
[0110] (An example of an optical system for obtaining observational data measuring the environment in the first and second embodiments) Figure 17 shows an example of an optical system for obtaining observational data measuring the environment in the first and second embodiments. In Figure 17, "250" means a distance of 250 times a predetermined unit length, such as 1 centimeter. In Figure 17, "75" means a distance of 75 times a unit length in the same units as "250".
[0111] (modified version) Furthermore, each of the learning device 1, shape estimation device 2, and shape estimation device 2a may be implemented using multiple information processing devices connected to each other via a network. In this case, each of the functional units of the learning device 1, shape estimation device 2, and shape estimation device 2a may be distributed and implemented across multiple information processing devices.
[0112] All or part of the functions of learning device 1, shape estimation device 2, and shape estimation device 2a may be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.
[0113] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]
[0114] 100, 100a...Shape estimation system, 1...Learning device, 2, 2a...Shape estimation device, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Storage unit, 15...Output unit, 21, 21a...Control unit, 22...Input unit, 23...Communication unit, 24...Storage unit, 25...Output unit, 91, 93, 93a...Processor, 92, 94, 94a...Memory, 101...Data input unit, 102...Observation model input unit, 201...Preprocessing unit, 202...Observation model estimation unit, 203, 203a...Observation data constraint unit, 204...Shape constraint unit, 205, 205a...Optimization unit, 300...Model output unit, 103...Model input unit, 104...Restored coordinate input unit, 206...Inference unit, 301...Output unit, 105...Background data input unit, 207...Observation data separation unit
Claims
1. A data input unit that acquires observational data, measured using an optical system, of the environment when one or more objects are present in the environment, as time-series data with a time resolution of a predetermined or higher time resolution, An observation model input unit obtains the parameters of the observation system that obtains the observation data obtained by the data input unit, the integral kernel of the spatiotemporal convolution function, and the transformation function model as an observation model. A preprocessing unit that transforms the time-series data obtained from the data input unit by normalization processing, An optimization unit optimizes the density field of an object so that an optimization function, which is a predetermined function consisting of an observation data constraint term, which is a term obtained based on the time series data converted by the preprocessing unit and expresses the condition that the difference between the observation model and the time series data is smaller than a predetermined value, and a shape constraint term, which expresses the condition that the estimated object shape is spatially smooth, is minimized. An observation model estimation unit estimates the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model by minimizing the optimization function, using the parameters of the observation system obtained from the observation model input unit, the integral kernel of the spatiotemporal convolution function, and the transformation function model as initial values. A model output unit that outputs the optimization results of the optimization unit, A shape estimation system equipped with the following features.
2. A background data input unit that acquires time-series data measuring only the observation environment as background data, An observation data separation unit separates the two-dimensional time-series data, which has been preprocessed in the aforementioned preprocessing unit, into background data and observed values corresponding to each of the aforementioned objects. Equipped with, The aforementioned observation data constraint term expresses the condition that the difference between the observation data separated for each object acquired by the observation data separation unit and the time series data is smaller than a predetermined value. The shape estimation system according to claim 1.
3. The time-series data acquired by the aforementioned data input unit is data obtained by single-photon measurement. The shape estimation system according to claim 1.
4. The observation model input unit acquires the spatial and temporal resolution, the approximate scale to the object whose shape is to be estimated, and the approximate shape of the integral kernel in the temporal and spatial directions as parameters of the optical system. The shape estimation system according to claim 1.
5. A data input step involves acquiring observational data obtained by measuring the environment using an optical system when one or more objects are present in the environment, as time-series data with a time resolution greater than or equal to a predetermined time resolution, An observation model input step is performed to obtain the parameters of the observation system that obtains the observation data obtained in the aforementioned data input step, the integral kernel of the spatiotemporal convolution function, and the transformation function model as an observation model. A preprocessing step in which the time-series data obtained in the aforementioned data input step is transformed by normalization processing, An optimization step to optimize the density field of an object such that an optimization function, which is a predetermined function consisting of an observation data constraint term, which is a term obtained based on the transformed time series data of the preprocessing step and expresses the condition that the difference between the observation model and the time series data is smaller than a predetermined value, and a shape constraint term, which expresses the condition that the estimated object shape is spatially smooth, is minimized. An observation model estimation step in which the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model obtained in the observation model input step are used as initial values, and the optimization function is minimized to estimate the parameters of the observation system, the integral kernel of the spatiotemporal convolution function, and the transformation function model; A model output step that outputs the optimization results of the optimization step, A shape estimation method having the following characteristics.
6. A program for causing a computer to function as a shape estimation system according to any one of claims 1 to 4.