Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device, an information processing method, and a program that convert relative depth estimated from an input image into absolute depth by more appropriately estimating the length of an object contained in the input image. [Solution] The information processing device 10 includes an acquisition unit 11 that acquires an input image, a scale factor estimation unit 12 that estimates a scale factor for converting the relative depth estimated based on the input image into an absolute depth, and an absolute depth estimation unit 13 that converts the relative depth into an absolute depth based on the scale factor.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for estimating the height of a person contained in an acquired depth image. More specifically, Patent Document 1 extracts an image of a person contained in a depth image and segments it into parts: head, upper body, thighs, and lower legs. The actual length of each part is then estimated using a trained neural network model, and the person's posture is estimated from the estimated lengths of each part and the topological relationships analyzed from each part. The actual height of the person is then estimated by further applying this information to the trained neural network model. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Chinese Patent Application Publication No. 112861699 Specification [Overview of the project] [Problems that the invention aims to solve]

[0004] However, conventional technologies require, for example, the preparation of a pre-trained neural network model for each part of a person, leaving room for improvement in terms of computational cost and other factors.

[0005] The purpose of this disclosure is to provide a technique that can more accurately estimate the length of an object included in a depth image. [Means for solving the problem]

[0006] A first aspect of the present disclosure provides an information processing device having: an acquisition unit for acquiring an input image; a scale factor estimation unit for estimating a scale factor for converting the relative depth estimated based on the input image into an absolute depth; and an absolute depth estimation unit for converting the relative depth into an absolute depth based on the scale factor.

[0007] Furthermore, a second aspect of the present disclosure provides an information processing method that acquires an input image, estimates a scale factor for converting the relative depth estimated based on the input image into an absolute depth, and converts the relative depth into an absolute depth based on the scale factor.

[0008] Furthermore, a third aspect of the present disclosure provides a program that causes a computer to perform the following processes: acquire an input image, estimate a scale factor for converting the relative depth estimated based on the input image into absolute depth, and convert the relative depth into absolute depth based on the scale factor. [Effects of the Invention]

[0009] From one perspective, it is possible to more accurately estimate the length of objects contained in depth images. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of an information processing device according to the embodiment. [Figure 2] A flowchart showing an example of processing performed by the information processing apparatus according to the present invention. [Figure 3] A flowchart showing an example of processing performed by the information processing apparatus according to the present invention. [Figure 4] This figure shows an example of the hardware configuration of the information processing device according to the embodiment. [Modes for carrying out the invention]

[0011] The principles of the present disclosure will be described with reference to some exemplary embodiments. These embodiments are described for illustrative purposes only and are intended to assist those skilled in the art in understanding and implementing the present disclosure without suggesting any limitations on the scope of the present disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0012] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Embodiments of the present disclosure will be described below with reference to the drawings.

[0013] <Configuration> Referring to FIG. 1, the configuration of the information processing apparatus 10 according to an embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing apparatus 10 according to an embodiment. In the example of FIG. 1, the information processing apparatus 10 includes an acquisition unit 11, a scale factor estimation unit 12, and an absolute depth estimation unit 13. Each of these units may be realized by the cooperation of one or more programs installed in the information processing apparatus 10 and the hardware such as the processor and memory of the information processing apparatus 10.

[0014] The acquisition unit 11 acquires an input image (two-dimensional image) captured by a photographing device. The scale factor estimation unit 12 estimates (calculates, specifies) a scale factor for converting relative depth to absolute depth. The absolute depth estimation unit 13 converts relative depth to absolute depth based on the scale factor estimated by the scale factor estimation unit 12.

[0015] <Processing> Next, referring to FIG. 2, an example of the processing of the information processing apparatus 10 according to an embodiment will be described. FIG. 2 is a flowchart showing an example of the processing of the information processing apparatus 10 according to an embodiment. The processing in FIG. 2 may be executed, for example, in response to a specific operation by an administrator of the information processing apparatus 10.

[0016] In step S101, the acquisition unit 11 acquires N (where N is an integer greater than or equal to 1) input images. The input images may be, for example, two-dimensional images taken with a regular (common) camera (monocular camera).

[0017] Next, the scale factor estimation unit 12 estimates (calculates, identifies) a scale factor for converting the relative depth estimated based on the input image into absolute depth (step S102).

[0018] Next, the absolute depth estimation unit 13 converts the relative depth to absolute depth based on the scale factor estimated by the scale factor estimation unit 12 (step S103). This makes it possible to estimate the distance between each object in the subject, as well as the distance from the imaging device to the subject, using a relatively inexpensive imaging device (camera), without having to use relatively expensive stereo cameras or LiDAR, by performing monocular depth estimation.

[0019] Next, referring to Figure 3, the processing by the scale factor estimation unit 12 in step S102 and the processing by the absolute depth estimation unit 13 in step S103 of Figure 2 will be described in more detail. Figure 3 is a diagram showing an example of the processing of the information processing device 10 according to the embodiment. Note that each process in Figure 3 may be executed in a different order, as long as it is not contradictory.

[0020] In step S201, the scale factor estimation unit 12 generates (estimates) a relative depth image 302 based on the input image 301. The relative depth image is a depth image that does not contain information of an absolute length dimension such as meters, but contains information of a relative length dimension.

[0021] The dataset D used for estimating the scale factor is represented by the following equation (1). Dataset D may be, for example, a collection of images that include people as subjects. K is an intrinsic parameter of the camera that took the image.

number

[0022] Here, the scale factor estimator 12, for example, as in the following formula (2), inputs the image I into the learned relative depth estimator f d (I) to calculate the relative depth image d (n) . Note that the learned relative depth estimator may be, for example, a Convolutional Neural Network (CNN) or the like. Also, as shown in formula (1), n is an integer from 1 to N (n) .

Equation

[0023] Subsequently, the scale factor estimator 12 generates (estimates) the two-dimensional skeleton information 303 of the subject person based on the input image 301 (step S202). Here, the scale factor estimator 12, for example, as in the following formula (3), inputs the image I into the learned human skeleton estimator f HP (I) to calculate the two-dimensional skeleton set S (n) . Note that the learned human skeleton estimator may be, for example, a convolutional neural network or the like uv (n) .

Equation

[0024] Here, p uv (n,k) represents the two-dimensional simultaneous coordinate vector (u, v, 1) T of the k-th joint. Also, let the set P corr ={(k m1 , k m2 )} of the set of indices of joints corresponding to the statistical data of body dimensions

[0025] Next, the scale factor estimation unit 12 generates three-dimensional skeletal information 304 by projecting the two-dimensional skeletal information 303 onto three-dimensional space using the relative depth image 302 (step S203). By using the three-dimensional skeletal information, the scale factor can be calculated more appropriately, for example, regardless of the person's posture. Here, the scale factor estimation unit 12 may generate the three-dimensional skeletal information as shown in equation (4) below, for example.

number

[0026] Next, the scale factor estimation unit 12 calculates the scale factor 306 by performing statistical processing on the 3D skeletal information 304 based on statistical data 305 of human body dimensions (step S204). This makes it possible to calculate the scale factor while mitigating the influence of the magnitude of the human body's variance, even in training data where the dimensions of the subject are unknown.

[0027] Here, the scale factor estimation unit 12 uses the joint length l obtained from 3D skeletal information and the joint length l obtained from statistical data of human body dimensions. p The scale factor s^ may also be calculated by statistical processing of the ratio, as shown in equation (5) below.

number

[0028] Note: Statistical data on human body dimensions L p The following equation (6) is given, where m is the index corresponding to the body part (e.g., femur, etc.), l (m) This represents the length of the part in question. The length of the part may also be a representative value of the part's length obtained through measurement surveys (e.g., mean, most frequent value, or median).

number

[0029] Also, L(I(n) ) is shown by the following equation (7).

number

[0030] In the example of equation (5), the scale factor estimation unit 12 calculates the scale factor using the mean of the ratio, but the statistical calculation method is not limited to this example. The scale factor estimation unit 12 may also calculate the scale factor using the median of the ratio.

[0031] Furthermore, the scale factor estimation unit 12 may estimate not only the relative scale quantity s, but also the relative scale quantity s and the relative shift quantity t as the scale factor. In this case, the scale factor estimation unit 12 may, for example, use s^ and t^ estimated by the minimization problem of equation (8) below to convert the relative depth to absolute depth as s^·d+t^. Here, d * The true depth is N p This is the number of valid true depth values.

[0032] Next, the absolute depth estimation unit 13 generates (estimates) a relative depth image 302 based on the input image 301 (step S205). The process of generating the relative depth image 302 by the absolute depth estimation unit 13 may be the same as the process of generating the relative depth image 302 by the scale factor estimation unit 12. Alternatively, the absolute depth estimation unit 13 may use the relative depth image 302 generated by the scale factor estimation unit 12.

[0033] Next, the absolute depth estimation unit 13 calculates an absolute depth image 307 by multiplying the relative depth image 302 by a scale factor 306 (step S206). This allows for the determination of absolute depth, which has dimensions such as meters, and enables a quantitative understanding of the distance and positional relationship to people and objects in space.

[0034] Here, the absolute depth estimation unit 13 may convert the relative depth d to the absolute depth d^ using the calculated scale quantity s^. In this case, the absolute depth estimation unit 13 may calculate the absolute depth d^ by multiplying the scale quantity s^, which is the scale factor, by the relative depth d, as shown in equation (8) below.

number

[0035] The absolute depth estimation unit 13 may also convert the relative depth d to the absolute depth d^ using the relative scale amount s and relative shift amount t estimated by the scale factor estimation unit 12, according to the following equation (9).

number

[0036] <Hardware Configuration> Figure 4 shows an example of the hardware configuration of each information processing device 10 according to the embodiment. In the example in Figure 4, the information processing device 10 (computer 100) includes a processor 101, memory 102, and a communication interface 103. These parts may be connected by a bus or the like. The memory 102 stores at least a portion of the program 104. The communication interface 103 includes an interface necessary for communication with other network elements.

[0037] When program 104 is executed in cooperation with the processor 101 and memory 102, etc., the computer 100 performs at least some of the processing of embodiments of this disclosure. Memory 102 may be of any type. Memory 102 may, in non-limiting examples, be a non-temporary computer-readable storage medium. Memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown for computer 100, computer 100 may have several physically different memory modules. Processor 101 may be of any type. Processor 101 may include one or more general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and, in non-limiting examples, processors based on multicore processor architectures. Computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-dependent to a clock that synchronizes the main processor.

[0038] Embodiments of the present disclosure may be implemented in hardware or in dedicated circuitry, software, logic, or any combination thereof. Some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device.

[0039] This disclosure also provides at least one computer program product tangibly stored on a non-temporary computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in a program module, and is executed on a device on a target real or virtual processor to perform the processes or methods of this disclosure. The program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of the program module may be combined or divided among the program module as desired in various embodiments. The machine-executable instructions of the program module can be executed on a local or distributed device. On a distributed device, the program module can reside on both local and remote storage media.

[0040] Programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media, magneto-optical recording media, optical disc media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disc media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (ReWritable). Semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0041] <Other> This study examines the estimation of a person's height based on a regular 2D image, not a depth image. In this case, if the person's arms are bent or they are sitting in a chair, the accuracy of estimating the length of each joint decreases because not all joints are on the same plane. Furthermore, if the estimation of joint lengths is based on the assumption that the person's actual height is known, it is not possible to estimate the joint lengths for a person being photographed for the first time. Additionally, since 2D images do not provide information such as the distance from the camera to the person, this method cannot be applied to processes that require information such as the distance to a person or the relative positions of people.

[0042] Furthermore, when using stereo cameras or LiDAR to determine the distance to an object, more expensive equipment is required compared to general-purpose cameras.

[0043] On the other hand, the technology disclosed herein enables distance estimation using a conventional camera. Furthermore, the technology disclosed herein can be used in a variety of situations where it is necessary to obtain the distance to an object. For example, the technology disclosed herein can be applied to object grasping tasks using robots. When a robot grasps an object, it is necessary to recognize the position of the object and plan the actions of the end-user. In this process, distance information to the object may be used. By applying the technology disclosed herein, object grasping tasks can be realized using an inexpensive camera.

[0044] Furthermore, by using the technology disclosed herein, it is possible to reconstruct a three-dimensional space using, for example, images or depth information calculated based on those images. For example, in order to reflect the size of the real world (for example, information such as "the entrance is 2 meters high" or "the road is 3 meters wide") in three-dimensional spatial information generated from images using SfM (Structure from Motion), NeRF (Neural Radiance Fields), or Gaussian Splatting, it is necessary to use information that has the dimension of meters. By applying the technology disclosed herein, it becomes possible to give depth information the dimension of meters.

[0045] <Variation> The information processing device 10 may be a device contained in a single enclosure, but the information processing device 10 of this disclosure is not limited to this. Each part of the information processing device 10 may be implemented by cloud computing, which is composed of, for example, one or more computers.

[0046] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. [Explanation of Symbols]

[0047] 10 Information Processing Devices 11 Acquisition Department 12. Scale Factor Estimation Unit 13 Absolute Depth Estimation Unit

Claims

1. An acquisition unit that acquires the input image, A scale factor estimation unit estimates a scale factor for converting the relative depth estimated based on the input image into absolute depth. An absolute depth estimation unit that converts the relative depth to absolute depth based on the scale factor, An information processing device having

2. The scale factor estimation unit, Based on the aforementioned input image, the two-dimensional skeletal information of the person in the subject is estimated. Based on the relative depth and the two-dimensional skeleton information, three-dimensional skeleton information is generated. The scale factor is estimated based on the three-dimensional skeletal information and statistical data of human body dimensions. The information processing apparatus according to claim 1.

3. The absolute depth estimation unit calculates the absolute depth by multiplying the scale factor by the relative depth. The information processing apparatus according to claim 1.

4. Obtain the input image, A scale factor is estimated to convert the relative depth estimated based on the input image into absolute depth. Based on the aforementioned scale factor, the relative depth is converted to absolute depth. Information processing methods.

5. Obtain the input image, A scale factor is estimated to convert the relative depth estimated based on the input image into absolute depth. Based on the aforementioned scale factor, the relative depth is converted to absolute depth. A program that instructs a computer to perform a process.

Images