3D spatial estimation system

Abstract

This three-dimensional space estimation system comprises: a relative depth acquisition unit that acquires relative depth information pertaining to a two-dimensional image; a reference line setting unit that sets a straight line detected from the two-dimensional image as a reference line; a reference point setting unit that sets a plurality of reference points on the reference line and sets the three-dimensional coordinates of each reference point by using a first coefficient for converting the relative depth to an absolute depth; a two-dimensional coordinate acquisition unit that acquires two-dimensional coordinates of each reference point; a parameter calculation unit that substitutes the three-dimensional coordinates and the two-dimensional coordinates of the reference points into a conversion formula representing the relationship between the three-dimensional coordinates and the two-dimensional coordinates, thereby calculating a conversion formula parameter and the first coefficient; and a three-dimensional space model generation unit that arranges each pixel of the two-dimensional image in the three-dimensional space using the parameter, the first coefficient, and the relative depth of each pixel and generates a three-dimensional space model.

Description

[Technical Field]

[0001] This invention relates to a three-dimensional spatial estimation system. [Background technology]

[0002] There is a need for a technique to estimate three-dimensional space based on two-dimensional images (monocular images). For example, Patent Document 1 discloses a technique for formulating the correspondence between image coordinates and global coordinates and determining the calibration parameters of an imaging device using a target image containing one or more reference markers captured by an imaging device. Based on the correspondence between image coordinates and global coordinates, it becomes possible to estimate three-dimensional space based on a two-dimensional image. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 2017-513079 [Overview of the project] [Problems that the invention aims to solve]

[0004] As mentioned above, in order to estimate a 3D space from a 2D image, it is necessary to determine various parameters related to the projection of the 3D space onto the 2D image, as well as the correspondence between coordinates in the 2D image and coordinates in the 3D space. However, it is necessary to pre-place markers or other objects whose coordinates in the 3D space are known, and pre-placing markers in the target 3D space is time-consuming. Furthermore, it was impossible to determine various parameters and the correspondence between coordinates in the 2D image and coordinates in the 3D space from past images where markers were not placed.

[0005] Therefore, the present invention has been made in view of the above problems, and aims to estimate a three-dimensional space from a two-dimensional image without using any installed objects such as markers. [Means for solving the problem]

[0006] To solve the above problems, a three-dimensional spatial estimation system relating to one aspect of this disclosure is a three-dimensional spatial estimation system that estimates a three-dimensional space represented in a two-dimensional image, comprising: a relative depth acquisition unit that acquires relative depth information representing the relative depth of each pixel in a two-dimensional image which is an image onto which a three-dimensional space is projected; a reference line setting unit that detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines; a reference point setting unit that sets multiple points on each reference line as reference points, and sets the three-dimensional coordinates of each reference point in three-dimensional space using a first coefficient that converts relative depth to absolute depth in common at least at each reference point; and each reference point The system comprises: a 2D coordinate acquisition unit that acquires 2D coordinates in a 2D image; a parameter calculation unit that calculates parameters and a first coefficient of a predetermined transformation formula by substituting the 3D coordinates and 2D coordinates of a reference point into a predetermined transformation formula that expresses the relationship between 3D coordinates in 3D space and 2D coordinates in a 2D image using predetermined parameters; and a 3D space model generation unit that generates a 3D space model representing 3D space by arranging the pixel values ​​of each pixel in the 2D image at corresponding positions in 3D space using at least some of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel.

[0007] According to the above aspects, relative depth information of a 2D image is obtained. A straight line detected from the 2D image is set as a reference line, and multiple reference points are set on the reference line. Since each reference point is linearly arranged on the straight line, the 3D coordinates of each reference point can be set using a common first coefficient. Then, by applying the 2D and 3D coordinates of the reference points to a transformation formula, the parameters of the transformation formula and the first coefficient are calculated. Therefore, based on the calculated parameters, the transformation formula and the first coefficient, and the relative depth of each pixel, the pixel value of each pixel can be placed at the corresponding position in 3D space, thus enabling estimation of 3D space. [Effects of the Invention]

[0008] This makes it possible to estimate a 3D space from a 2D image without using markers or other external objects. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the functional configuration of the 3D spatial estimation system and 3D spatial estimation device of this embodiment. [Figure 2] This figure shows an example of a 2D image to be processed. [Figure 3] This figure shows an example of a relative depth map corresponding to a 2D image. [Figure 4] This figure shows an example of the process of setting a reference line on a 2D image. [Figure 5] This figure shows an example of setting a reference point. [Figure 6] This is a diagram illustrating the first and second coefficients. [Figure 7] This figure shows an example of an estimated 3D spatial model. [Figure 8] This is a flowchart showing the processing steps of the 3D spatial estimation method in a 3D spatial estimation device. [Figure 9] This diagram shows the configuration of a 3D spatial estimation program. [Figure 10] This is a hard block diagram of a 3D spatial estimation device. [Modes for carrying out the invention]

[0010] Embodiments of the three-dimensional spatial estimation system according to the present invention will be described with reference to the drawings. Where possible, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] Figure 1 shows the functional configuration of the 3D spatial estimation system and 3D spatial estimation device according to this embodiment. The 3D spatial estimation system 1 of this embodiment is a system that estimates a 3D space represented in a 2D image, and is configured, for example, by a 3D spatial estimation device 10. The 2D image is, for example, a monocular camera image.

[0012] As shown in FIG. 1, the three-dimensional space estimation apparatus 10 functionally comprises an image acquisition unit 11, a relative depth acquisition unit 12, a reference line setting unit 13, a reference point setting unit 14, a two-dimensional coordinate acquisition unit 15, a parameter calculation unit 16, an output unit 17, and a three-dimensional space model generation unit 18. Each of these functional units 11 to 18 may be configured in one apparatus as illustrated in FIG. 1, or may be dispersedly configured in a plurality of apparatuses.

[0013] Each of the functional units 11 to 18 of the three-dimensional space estimation apparatus 10 is configured to be accessible to storage means such as a two-dimensional image storage unit 21, a conversion formula storage unit 22, and a three-dimensional space model storage unit 23. Each of the storage units 21 to 23 may be provided in the three-dimensional space estimation apparatus 10, or may be configured in another apparatus that is configured to be accessible from the three-dimensional space estimation apparatus 10 as illustrated in FIG. 1.

[0014] Next, each functional unit of the three-dimensional space estimation apparatus 10 will be described. The image acquisition unit 11 acquires a two-dimensional image obtained by projecting a three-dimensional space. FIG. 2 is a diagram illustrating an example of a two-dimensional image A acquired by the image acquisition unit. The two-dimensional image A is, for example, an image obtained by capturing a three-dimensional space with a monocular camera. The two-dimensional image A may be an image extracted from a moving image obtained by capturing a three-dimensional space.

[0015] The image acquisition unit 11 may acquire the two-dimensional image A from a camera. Further, the image acquisition unit 11 may acquire the two-dimensional image A from the two-dimensional image storage unit 21. The two-dimensional image storage unit 21 is a storage means that stores two-dimensional images in advance.

[0016] The relative depth acquisition unit 12 acquires relative depth information representing the relative depth of each pixel of a two-dimensional image. The relative depth acquisition unit 12 may acquire a relative depth map corresponding to the two-dimensional image as the relative depth information. The relative depth map is information indicating the relative depth information for each pixel by numerical values.

[0017] Figure 3 shows an example of a relative depth map corresponding to the 2D image shown in Figure 2. In relative depth map B illustrated in Figure 3, the closer the object represented by each pixel is to the camera, the smaller the relative depth value (closer to black), and the farther the object is from the camera, the larger the relative depth value (closer to white). Since relative depth values ​​are relative, it is not possible to project each pixel into 3D space at the same ratio as the real object. There is no upper limit to the value of each pixel, but as illustrated in Figure 3, representations like black and white photographs are possible.

[0018] Relative depth information can be obtained using well-known machine learning-based technologies such as MiDaS, and the method used by the relative depth acquisition unit 12 to acquire relative depth information is not limited. The relative depth acquisition unit 12 may also acquire a relative depth map B corresponding to a two-dimensional image A, which has been generated in advance and stored in the two-dimensional image storage unit 21.

[0019] The reference line setting unit 13 detects one or more straight lines from the 2D image and sets the detected straight lines as reference lines. Figure 4 shows an example of the process of setting reference lines in a 2D image. As shown in Figure 4, the reference line setting unit 13 detects straight lines from the 2D image.

[0020] The reference line setting unit 13 may detect straight lines from the 2D image A using the Hough transform. The method used by the reference line setting unit 13 to detect straight lines from the 2D image is not limited; for example, well-known techniques for edge detection from images may be applied.

[0021] The reference line setting unit 13 sets a straight line detected from the 2D image A as the reference line rl. The reference line setting unit 13 may detect multiple straight lines from the 2D image A. The reference line setting unit 13 may then set multiple straight lines from among the multiple straight lines detected from the 2D image A as the reference line rl. Setting multiple reference lines makes it possible to improve the accuracy of calculating the parameters of the transformation formula described later.

[0022] Furthermore, the reference line setting unit 13 may set one or more of the longest lines among the multiple lines detected from the 2D image A as the reference line rl. By setting a longer line as the reference line rl, it is possible to improve the accuracy in calculating the parameters of the conversion formula described later.

[0023] The reference point setting unit 14 sets multiple points as reference points on the reference line rl for each reference line. The reference point setting unit 14 then sets the three-dimensional coordinates of each reference point in three-dimensional space by using a first coefficient that converts relative depth to absolute depth, which is common to at least each reference point.

[0024] Figure 5 shows an example of setting reference points. As shown in Figure 5, the reference point setting unit 14 sets multiple reference points rp1 to rp4 (rp) on the reference line rl set by the reference line setting unit 13. The number of reference points rp to be set is not limited as long as there are multiple, but a minimum number may be specified due to constraints arising from the conversion formula which will be explained later.

[0025] The reference point setting unit 14 may set the intervals between multiple reference points rp to be equal. By setting the intervals between reference points rp to be equal, the ratio of the distances between the three-dimensional coordinates set for each reference point is simplified, making it easier to calculate the parameters and coefficients of the transformation formula, which will be explained later.

[0026] The reference point setting unit 14 may set the first of the first to third coordinate axes in three-dimensional space to be aligned with the reference line rl, and set the three-dimensional coordinates of each reference point rp in three-dimensional space using a first coefficient and a second coefficient that converts absolute depth to distance along the first coordinate axis, which are common to each reference point.

[0027] Specifically, the reference point setting unit 14 sets a coordinate system composed of the X, Y, and Z axes such that the Z axis aligns with the reference line. The reference point setting unit 14 may set the three-dimensional coordinates of equally spaced reference points rp1 to rp4 as follows, for example, by using α as a first coefficient for converting relative depth to absolute depth and β as a second coefficient for converting absolute depth to distance along the Z axis. rp1(0,0,1×α×β) rp2(0,0,2×α×β) rp3(0,0,3×α×β) rp4(0,0,4×α×β)

[0028] In other words, since the Z-axis is set to align with the reference line rl, the X and Y coordinates of each reference point rp can be set to 0.

[0029] Figure 6 illustrates the first and second coefficients. As shown in Figure 6, the absolute depths 1×α and 2×α along the camera's optical axis ca, expressed using the first coefficient α, are transformed into 1×α×β and 2×α×β, respectively, using the second coefficient β in a coordinate system where the reference line rl is the Z-axis za. Therefore, the second coefficient β differs depending on the angle of the reference line rl with respect to the camera's optical axis ca.

[0030] In this way, by setting a coordinate system in which the Z-axis aligns with the reference line, and setting the 3D coordinates of each reference point using a common second coefficient β in addition to the first coefficient α, the reference line rl can be set regardless of the angle of the 2D image with respect to the optical axis of the camera. Note that if the Z-axis aligns with the optical axis of the camera, β=1, so the 3D coordinates of the reference point rp may be set without using the second coefficient β.

[0031] The 2D coordinate acquisition unit 15 acquires the 2D coordinates of each reference point rp in the 2D image. Specifically, the 2D coordinate acquisition unit 15 acquires the coordinates of reference points rp1 to rp4 in the coordinate system set in the 2D image A. The coordinates of reference points rp1 to rp4 can be expressed, for example, as follows. rp1(u1,v1) rp2(u2,v2) rp3(u3,v3) rp4(u4,v4)

[0032] The parameter calculation unit 16 calculates the parameters of the transformation formula and the coefficients included in the 3D coordinates by substituting the 3D coordinates and 2D coordinates set for the reference point rp into a predetermined transformation formula. The predetermined transformation formula is an expression that represents the relationship between the 3D coordinates in 3D space and the 2D coordinates in a 2D image using predetermined parameters.

[0033] The parameter calculation unit 16 calculates at least a first coefficient α included in the three-dimensional coordinates of the reference point rp. Furthermore, if the three-dimensional coordinates of the reference point rp include both the first coefficient α and the second coefficient β, the parameter calculation unit 16 calculates the second coefficient β in addition to the first coefficient α.

[0034] The parameters of the transformation formula may include six external camera parameters, four internal camera parameters, and five parameters related to the projection of a three-dimensional space onto a two-dimensional image, as well as five parameters related to the camera's lens distortion.

[0035] If the three-dimensional coordinates of the reference point rp include the first coefficient α and the second coefficient β, the parameter calculation unit 16 calculates 17 variables, which are the sum of the 15 parameters of the transformation formula and the first and second coefficients.

[0036] In calculating the parameters of the transformation formula using the 3D and 2D coordinates of the reference point rp, the parameters that change with respect to the coordinate system determined by the setting of the reference line rl are six camera external parameters and a second coefficient β. Therefore, six variables need to be calculated for one reference line, and since two variables can be calculated for each reference point, the reference point setting unit 14 sets at least four reference points rp for one reference line rl. In this way, by setting an appropriate number of reference points according to the number of parameters constituting the transformation formula, it becomes possible to reliably calculate those parameters.

[0037] The parameter calculation unit 16 uses the following transformation formula to represent the relationship between three-dimensional coordinates (X, Y, Z) in three-dimensional space and two-dimensional coordinates (u, v) in a two-dimensional image.

number

[0038] The defined transformation formula makes it possible to appropriately represent the correspondence between 3D space and 2D images. Furthermore, by calculating the parameters of the transformation formula and the first and second coefficients, it becomes possible to correct the distortion of 2D image A and obtain the absolute depth of each pixel in 2D image A.

[0039] The absolute depth of each pixel in a 2D image can be obtained based on the relative depth of that pixel (obtained from a relative depth map) and a first coefficient α. This makes it possible to project each pixel of 2D image A to the position indicated by its absolute depth.

[0040] Note that the above conversion formula is just one example and is not limited to it. For example, by ignoring camera lens distortion and setting parameters k1, k2, k3, p1, and p2 to 0, the above conversion formula can be simplified.

[0041] Referring again to Figure 1, the output unit 17 may output a conversion formula. Specifically, the output unit 17 may store the parameters calculated by the parameter calculation unit 16 in the conversion formula storage unit 22. The output unit 17 may also store at least the calculated first coefficient in the conversion formula storage unit 22. Furthermore, the output unit 17 may store a second coefficient in the conversion formula storage unit 22. The conversion formula storage unit 22 is a storage means capable of storing the conversion formula defined by the parameters etc. calculated by the parameter calculation unit 16, as well as the first and second coefficients etc.

[0042] The 3D spatial model generation unit 18 generates a 3D spatial model representing the 3D space by placing the pixel values ​​of each pixel in the 2D image A at corresponding positions in the 3D space.

[0043] Figure 7 shows an example of a three-dimensional spatial model. Specifically, the three-dimensional spatial model generation unit 18 corrects the two-dimensional image A using various parameters calculated by the parameter calculation unit 16. More specifically, the three-dimensional spatial model generation unit 18 may correct the distortion of the two-dimensional image A using a well-known method based on camera lens distortion parameters k1, k2, k3, p1, p2.

[0044] Furthermore, the 3D spatial model generation unit 18 may obtain the absolute depth of each pixel of the 2D image A based on the relative depth map B and at least a first coefficient α. Specifically, the 3D spatial model generation unit 18 obtains the relative depth of each pixel of the 2D image A from the relative depth map B and obtains the absolute depth by applying the first coefficient α to the relative depth. The 3D spatial model generation unit 18 may further apply a second coefficient β to the relative depth to obtain the absolute depth of each pixel of the 2D image A. Then, the 3D spatial model generation unit 18 projects each pixel of the 2D image A to the position represented by the absolute depth, thereby arranging the pixel value of each pixel in 3D space and generating a 3D spatial model md that represents the 3D space corresponding to the 2D image A.

[0045] The 3D spatial model generation unit 18 may store data representing the generated 3D spatial model md in the 3D spatial model storage unit 23. The 3D spatial model storage unit 23 is a storage means for storing the generated 3D spatial model md.

[0046] Figure 8 is a flowchart showing the processing steps of the 3D spatial estimation method in 3D spatial estimation system 1.

[0047] In step S1, the image acquisition unit 11 acquires a 2D image A projected onto a 3D space. In step S2, the relative depth acquisition unit 12 acquires relative depth information (for example, a relative depth map B) representing the relative depth of each pixel in the 2D image A.

[0048] In step S3, the reference line setting unit 13 detects one or more straight lines from the 2D image A. In step S4, the reference line setting unit 13 sets the detected straight line as the reference line rl.

[0049] In step S5, the reference point setting unit 14 sets a predetermined number of reference points rp on the reference line rl. In step S6, the reference point setting unit 14 sets the 3D coordinates of each reference point rp in the coordinate system set in 3D space. The 3D coordinates set for each reference point rp are set using a first coefficient α, which converts relative depth to absolute depth, in common at least for each reference point.

[0050] In step S7, the 2D coordinate acquisition unit 15 acquires the 2D coordinates of each reference point rp in the 2D image. Specifically, the 2D coordinate acquisition unit 15 acquires the coordinates of the reference point rp in the coordinate system set in the 2D image A.

[0051] In step S8, the parameter calculation unit 16 calculates the parameters of the transformation formula and the coefficients included in the 3D coordinates by substituting the 3D coordinates and 2D coordinates set for the reference point rp into a predetermined transformation formula. The predetermined transformation formula is an expression that represents the relationship between the 3D coordinates in 3D space and the 2D coordinates in a 2D image using predetermined parameters.

[0052] In step S9, the 3D spatial model generation unit 18 generates a 3D spatial model representing the 3D space by placing the pixel values ​​of each pixel in the 2D image A at corresponding positions in the 3D space. Specifically, the 3D spatial model generation unit 18 corrects the 2D image A using various parameters calculated by the parameter calculation unit 16. Then, the 3D spatial model generation unit 18 places the pixel values ​​of each pixel in the 3D space by projecting each pixel of the 2D image A to a position represented by the absolute depth obtained based on the first coefficient and relative depth information.

[0053] Next, with reference to Figure 9, a 3D spatial estimation program for causing the computer to function as the 3D spatial estimation device 10 of this embodiment will be described. Figure 9 is a diagram showing the configuration of the 3D spatial estimation program. The 3D spatial estimation program P1 is composed of a main module m10 that comprehensively controls the 3D spatial estimation process in the 3D spatial estimation device 10, an image acquisition module m11, a relative depth acquisition module m12, a reference line setting module m13, a reference point setting module m14, a 2D coordinate acquisition module m15, a parameter calculation module m16, an output module m17, and a 3D spatial model generation module m18. Each of the modules m11 to m18 realizes the respective functions for each of the functional units 11 to 18.

[0054] The 3D spatial estimation program P1 may be transmitted via a transmission medium such as a communication line, or it may be stored on a recording medium M1, as shown in Figure 9.

[0055] According to the 3D spatial estimation system, 3D spatial estimation device 10, 3D spatial estimation method, and 3D spatial estimation program P1 of this embodiment described above, relative depth information B of a 2D image A is acquired. A straight line detected from the 2D image A is set as a reference line rl, and multiple reference points rp are set on the reference line rl. Since each reference point rp is linearly arranged on a straight line, the 3D coordinates of each reference point rp can be set using a common first coefficient α. Then, by applying the 2D and 3D coordinates of the reference points rp to a transformation formula, the parameters of the transformation formula and the first coefficient α are calculated. Therefore, based on the calculated parameters, the first coefficient α, and the relative depth of each pixel, the pixel value of each pixel can be placed at the corresponding position in 3D space, thus enabling 3D spatial estimation.

[0056] The three-dimensional spatial estimation system relating to this disclosure may have the following configurations. The operation and effects of each configuration are explained below.

[0057] A 3D spatial estimation system relating to one aspect of this disclosure is a 3D spatial estimation system for estimating a 3D space represented in a 2D image, comprising: a relative depth acquisition unit that acquires relative depth information representing the relative depth of each pixel in a 2D image which is an image onto which a 3D space is projected; a reference line setting unit that detects one or more straight lines from the 2D image and sets the detected straight lines as reference lines; a reference point setting unit that sets multiple points on each reference line as reference points, and sets the 3D coordinates of each reference point in 3D space using a first coefficient that converts relative depth to absolute depth in common at least for each reference point; and the 2D image of each reference point The system comprises: a 2D coordinate acquisition unit that acquires 2D coordinates in a 3D space; a parameter calculation unit that calculates parameters and a first coefficient of a predetermined transformation formula by substituting the 3D coordinates and 2D coordinates of a reference point into a predetermined transformation formula that expresses the relationship between 3D coordinates in 3D space and 2D coordinates in a 2D image using predetermined parameters; and a 3D space model generation unit that generates a 3D space model representing 3D space by arranging the pixel values ​​of each pixel in the 2D image at corresponding positions in 3D space using at least some of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel.

[0058] Based on the above aspects, relative depth information of a 2D image is obtained. A straight line detected from the 2D image is set as a reference line, and multiple reference points are set on the reference line. Since each reference point is linearly arranged on the straight line, the 3D coordinates of each reference point can be set using a common first coefficient. Then, by applying the 2D and 3D coordinates of the reference points to a transformation formula, the parameters of the transformation formula and the first coefficient are calculated. Therefore, based on the calculated parameters, the first coefficient, and the relative depth of each pixel, the pixel value of each pixel can be placed at the corresponding position in 3D space, thus enabling estimation of 3D space.

[0059] Furthermore, in a 3D spatial estimation system relating to other aspects, the reference point setting unit may set the intervals between multiple reference points to be equal.

[0060] Based on the above aspects, the ratio of the distances between the three-dimensional coordinates of each reference point is simplified, making it easier to calculate the parameters of the transformation formula and the first coefficient.

[0061] Furthermore, in a 3D spatial estimation system relating to other aspects, the reference point setting unit may set the first of the first to third coordinate axes in 3D space to be aligned with the reference line, and set the 3D coordinates of each reference point in 3D space using a first coefficient and a second coefficient that converts absolute depth to a distance along the first coordinate axis, which are common to each reference point.

[0062] Based on the above aspects, by setting a coordinate system in which the first coordinate axis is aligned with the reference line, and setting the three-dimensional coordinates of each reference point using a second coefficient in addition to the first coefficient, the reference line can be set regardless of the angle with respect to the optical axis.

[0063] Furthermore, in a 3D space estimation system relating to other aspects, the parameters of the transformation formula may include six predetermined camera external parameters, four camera internal parameters, and five parameters relating to the camera's lens distortion, and the reference point setting unit may set at least four reference points for a single reference line.

[0064] Based on the above aspects, the parameters can be reliably calculated by setting an appropriate number of reference points corresponding to the number of parameters that make up the transformation formula. This makes it possible to appropriately represent the correspondence between 3D space and 2D images using the transformation formula.

[0065] Furthermore, in the 3D spatial estimation system relating to other aspects, the parameter calculation unit uses the following transformation formula to represent the relationship between the 3D coordinates (X,Y,Z) in 3D space and the 2D coordinates (u,v) in the 2D image.

number

[0066] Based on the aspects described above, the defined transformation formula makes it possible to appropriately represent the correspondence between 3D space and 2D images using the transformation formula.

[0067] Furthermore, in a 3D spatial estimation system relating to other aspects, the reference line setting unit may set one or more straight lines with the highest length among the one or more detected straight lines as the reference line.

[0068] Based on the above aspects, by setting the longer of the lines detected from the 2D image as the reference line, it becomes possible to improve the accuracy of the calculated parameters.

[0069] Furthermore, in a 3D spatial estimation system relating to other aspects, the reference line setting unit may set multiple lines from the one or more detected lines as reference lines.

[0070] Based on the above aspects, by setting multiple lines detected from a 2D image as reference lines, it becomes possible to improve the accuracy of the calculated parameters.

[0071] The block diagram shown in Figure 1 represents functional units. These functional blocks (components) are implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each functional block is not particularly limited. That is, each functional block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be implemented by combining the above one device or the above multiple devices with software.

[0072] Functions include, but are not limited to, judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. As mentioned above, the method of implementation is not particularly limited.

[0073] For example, the 3D spatial estimation device 10 in one embodiment of the present invention may function as a computer. Figure 10 shows an example of the hardware configuration of the 3D spatial estimation device 10 according to this embodiment. Physically, the 3D spatial estimation device 10 may be configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.

[0074] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of the 3D spatial estimation device 10 may include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.

[0075] Each function in the 3D space estimation device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, allowing the processor 1001 to perform calculations and control communication by the communication device 1004, as well as the reading and / or writing of data to the memory 1002 and storage 1003.

[0076] The processor 1001 controls the entire computer, for example, by running the operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, etc. For example, the various functional units 11 to 18 shown in Figure 1 may be implemented by the processor 1001.

[0077] Furthermore, the processor 1001 reads programs (program code), software modules, and data from the storage 1003 and / or communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment. For example, each functional unit 11 to 18 of the 3D space estimation device 10 may be stored in the memory 1002 and implemented by a control program that runs on the processor 1001. Although the above processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The program may also be transmitted from a network via a telecommunications line.

[0078] Memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory, etc. Memory 1002 can store executable programs (program code), software modules, etc., for carrying out the three-dimensional space estimation method according to one embodiment of the present invention.

[0079] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including memory 1002 and / or storage 1003.

[0080] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via a wired and / or wireless network, and is also referred to as a network device, network controller, network card, communication module, etc.

[0081] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0082] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may consist of a single bus or different buses may be used for communication between devices.

[0083] Furthermore, the 3D spatial estimation device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0084] Information notification is not limited to the embodiments described herein and may be carried out by other means. For example, information notification may be carried out by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block))), other signals, or combinations thereof. RRC signaling may also be called RRC messages, and may be, for example, RRC Connection Setup messages, RRC Connection Reconfiguration messages, etc.

[0085] Each aspect / embodiment described in this disclosure may be applied to at least one of the following systems: LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, as well as next-generation systems extended based thereon. Furthermore, multiple systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A with 5G).

[0086] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be reordered, provided they are consistent with each other. For example, the methods described herein present various step elements in an exemplary order and are not limited to that specific order.

[0087] The specific operations described in this disclosure as being performed by a base station may, in some cases, be performed by its upper node. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal can be performed by the base station and at least one other network node (for example, an MME or S-GW, but not limited to these). Although the above example illustrates a case where there is one other network node besides the base station, it may also be a combination of multiple other network nodes (for example, an MME and an S-GW).

[0088] Information can be output from a higher layer (or lower layer) to a lower layer (or higher layer). Input and output may also occur via multiple network nodes.

[0089] Input and output information may be stored in a specific location (e.g., memory) or managed in a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be sent to other devices.

[0090] The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0091] Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0092] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0093] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.

[0094] Furthermore, software, instructions, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber lines (DSL) and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included in the definition of a transmission medium.

[0095] The information, signals, etc. described in this disclosure may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0096] In addition, terms described in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meaning.

[0097] The terms “system” and “network” as used in this disclosure are interchangeable.

[0098] Furthermore, the information, parameters, etc., described in this disclosure may be expressed as absolute values, relative values ​​from a given value, or by corresponding other information. For example, wireless resources may be indicated by an index.

[0099] The names used for the parameters described above are not restrictive in any way. Furthermore, the formulas and other expressions using these parameters may differ from those expressly disclosed in this disclosure. Various channels (e.g., PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, and therefore, the various names assigned to these various channels and information elements are not restrictive in any way.

[0100] As used in this disclosure, the terms “determining” and “determining” may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database, or other data structure), and ascertaining. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0101] As used in this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based on at least."

[0102] Where the terms “first,” “second,” etc., are used in this disclosure, no reference to those elements shall generally limit the quantity or order of those elements. These terms may be used herein as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements shall not imply that only two elements may be employed therein, or that the first element must precede the second element in any way.

[0103] In the configuration of each of the above devices, "means" may be replaced with "part," "circuit," "device," etc.

[0104] To the extent that “include,” “including,” and their variations are used herein or in the claims, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used herein or in the claims is not intended to be exclusive OR.

[0105] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0106] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0107] The three-dimensional spatial estimation system 1 of this disclosure may have the following configuration. [1] A 3D space estimation system that estimates a 3D space represented in a 2D image, A relative depth acquisition unit that acquires relative depth information representing the relative depth of each pixel in the two-dimensional image, which is an image projected onto the three-dimensional space, A reference line setting unit that detects one or more straight lines from the aforementioned two-dimensional image and sets the detected straight lines as reference lines, A reference point setting unit sets multiple points as reference points on the aforementioned reference line, and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts the relative depth to absolute depth in common at least for each reference point. A two-dimensional coordinate acquisition unit that acquires the two-dimensional coordinates of each reference point in the two-dimensional image, A parameter calculation unit calculates the parameters of a predetermined transformation formula and the first coefficient by substituting the three-dimensional coordinates and the two-dimensional coordinates of the reference point into a predetermined transformation formula that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters, A 3D spatial model generation unit generates a 3D spatial model representing the 3D space by arranging the pixel values ​​of each pixel in the 2D image at corresponding positions in the 3D space using at least a portion of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel. A 3D spatial estimation system equipped with the following features. [2] The reference point setting unit sets the intervals between the plurality of reference points to be equal. The three-dimensional spatial estimation system described in [1]. [3] The aforementioned reference point setting unit is The first coordinate axis among the first to third coordinate axes in the aforementioned three-dimensional space is set to align with the reference line, The three-dimensional coordinates of each reference point in the three-dimensional space are set using a first coefficient and a second coefficient that converts the absolute depth to a distance along the first coordinate axis, which are common to each reference point. A three-dimensional spatial estimation system as described in [1] or [2]. [4] The parameters of the conversion formula include six predetermined camera external parameters, four camera internal parameters, and five parameters relating to the projection of the three-dimensional space onto the two-dimensional image, The reference point setting unit sets at least four reference points for one reference line, The three-dimensional space estimation system according to [3]. [5] The parameter calculation unit: the following conversion expression representing the relationship between three-dimensional coordinates (X, Y, Z) in a three-dimensional space and two-dimensional coordinates (u, v) in a two-dimensional image [Math.]] calculates camera extrinsic parameters R and t each having 3 degrees of freedom, camera intrinsic parameters f x , f y , c x , c y , parameters k1, k2, k3, p1, p2 related to camera lens distortion, as well as the first coefficient and the second coefficient, The three-dimensional space estimation system according to [4]. [6] The reference line setting unit sets one or more straight lines with higher lengths among the detected one or more straight lines as the reference lines, The three-dimensional space estimation system according to any one of [1] to [5]. [7] The reference line setting unit sets a plurality of straight lines among the detected one or more straight lines as the reference lines, The three-dimensional space estimation system according to any one of [1] to [6]. Description of Reference Numerals

[0108] 1...3D spatial estimation system, 10...3D spatial estimation device, 11...Image acquisition unit, 12...Relative depth acquisition unit, 13...Reference line setting unit, 14...Reference point setting unit, 15...2D coordinate acquisition unit, 16...Parameter calculation unit, 17...Output unit, 18...3D spatial model generation unit, 21...2D image storage unit, 22...Transformation formula storage unit, 23...3D spatial model storage unit, M1...Recording medium, m10...Main module, m11...Image acquisition module, m12...Relative depth acquisition module, m13...Reference line setting module, m14...Reference point setting module, m15...2D coordinate acquisition module, m16...Parameter calculation module, m17...Output module, m18...3D spatial model generation module, P1...3D spatial estimation program, md...3D spatial model, rl...Reference line, rp...Reference point.

Claims

1. A 3D space estimation system that estimates a 3D space represented in a 2D image, A relative depth acquisition unit acquires relative depth information representing the relative depth of each pixel in the two-dimensional image, which is an image projected onto the three-dimensional space. A reference line setting unit that detects one or more straight lines from the aforementioned two-dimensional image and sets the detected straight lines as reference lines, A reference point setting unit sets multiple points as reference points on the aforementioned reference line, and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts the relative depth to absolute depth in common at least for each reference point. A two-dimensional coordinate acquisition unit that acquires the two-dimensional coordinates of each reference point in the two-dimensional image, A parameter calculation unit calculates the parameters of a predetermined transformation formula and the first coefficient by substituting the three-dimensional coordinates and the two-dimensional coordinates of the reference point into a predetermined transformation formula that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters, A three-dimensional spatial model generation unit generates a three-dimensional spatial model representing the three-dimensional space by arranging the pixel values ​​of each pixel in the two-dimensional image at corresponding positions in the three-dimensional space using at least a portion of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel. A three-dimensional spatial estimation system equipped with the following features.

2. The reference point setting unit sets the intervals between the plurality of reference points to be equal. The three-dimensional spatial estimation system according to claim 1.

3. The aforementioned reference point setting unit is The first coordinate axis among the first to third coordinate axes in the three-dimensional space is set to align with the reference line. The three-dimensional coordinates of each reference point in the three-dimensional space are set using a first coefficient and a second coefficient that converts the absolute depth into a distance along the first coordinate axis, which are common to each reference point. The three-dimensional spatial estimation system according to claim 1.

4. The parameters of the conversion formula include six predetermined external camera parameters, four internal camera parameters, and five parameters relating to the projection of the three-dimensional space onto the two-dimensional image. The reference point setting unit sets at least four reference points for one reference line. The three-dimensional spatial estimation system according to claim 3.

5. The parameter calculation unit, The following transformation formula represents the relationship between three-dimensional coordinates (X, Y, Z) in three-dimensional space and two-dimensional coordinates (u, v) in a two-dimensional image. [Math 1] in which, camera extrinsic parameters R and t each having 3 degrees of freedom, camera intrinsic parameter f ｘ , f ｙ , c ｘ , c ｙ , and a parameter k related to camera lens distortion １ , k ２ , k ３ , p １ , p ２ , and calculates the first coefficient and the second coefficient, The three-dimensional spatial estimation system according to claim 4.

6. The reference line setting unit sets one or more straight lines with the longer length among the one or more detected straight lines as the reference line. The three-dimensional spatial estimation system according to claim 1.

7. The reference line setting unit sets multiple straight lines from among the one or more straight lines detected as reference lines. The three-dimensional spatial estimation system according to claim 1.

Images