Three-dimensional space estimation system
The three-dimensional space estimation system addresses the challenge of estimating three-dimensional spaces from two-dimensional images without markers by using relative depth information and calculated coefficients to generate accurate three-dimensional models, eliminating the need for installed markers and streamlining the estimation process.
Patent Information
- Application Number
- PCT/JP2024/017978
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-22
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-28
AI Technical Summary
Existing methods for estimating a three-dimensional space from a two-dimensional image require installed markers with known coordinates, making it time-consuming and impractical to obtain parameters and coordinate correspondences, especially when no markers are present.
A three-dimensional space estimation system that acquires relative depth information, detects straight lines, sets reference points, and uses conversion formulas with calculated coefficients to estimate three-dimensional coordinates without relying on installed markers, allowing for the generation of a three-dimensional space model from a two-dimensional image.
Enables efficient estimation of three-dimensional spaces from two-dimensional images without markers, improving accuracy and reducing the time required for setup by using relative depth information and calculated coefficients to project pixel values into corresponding positions in three-dimensional space.
Smart Images

Figure JP2024017978_28112024_PF_FP_ABST
Abstract
Description
3D space estimation system
[0001] The present invention relates to a three-dimensional space estimation system.
[0002] There is a demand for a technology for estimating a three-dimensional space based on a two-dimensional image (monocular image). For example, Patent Literature 1 discloses a technology for formulating a correspondence between image coordinates and global coordinates using a target image including one or more reference markers captured by an imaging device, and determining calibration parameters for the imaging device. Based on the correspondence between the image coordinates and the global coordinates, it becomes possible to estimate a three-dimensional space based on the two-dimensional image.
[0003] Special table 2017-513079 publication
[0004] As described above, in order to estimate a three-dimensional space from a two-dimensional image, it is necessary to calculate various parameters related to the projection of the three-dimensional space onto the two-dimensional image and the correspondence between coordinates in the two-dimensional image and coordinates in the three-dimensional space. However, markers or the like whose coordinates in the three-dimensional space are known must be installed in advance, and installing the markers in the target three-dimensional space in advance is time-consuming. Furthermore, it is impossible to calculate various parameters and the correspondence between coordinates in the two-dimensional image and coordinates in the three-dimensional space from past images or the like in which markers are not installed.
[0005] The present invention has been made in consideration of the above problems, and has as its object to estimate a three-dimensional space from a two-dimensional image without using any installed objects such as markers.
[0006] In order to solve the above problem, a three-dimensional space estimation system according to one aspect of the present disclosure is a three-dimensional space estimation system that estimates a three-dimensional space represented in a two-dimensional image, and includes: a relative depth acquisition unit that acquires relative depth information that indicates a relative depth of each pixel in the two-dimensional image that is a projected image of the three-dimensional space; a reference line setting unit that detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines; a reference point setting unit that sets a plurality of points per reference line as reference points on the reference lines and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts a relative depth into an absolute depth, which is common to at least each reference point; a parameter calculation unit that calculates parameters and a first coefficient of a conversion equation by substituting the three-dimensional coordinates and two-dimensional coordinates of the reference point into a predetermined conversion equation that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters; and a three-dimensional space model generation unit that generates a three-dimensional space model that represents the three-dimensional space by arranging the pixel values of each pixel of the two-dimensional image at corresponding positions in the three-dimensional space using at least some of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel.
[0007] According to the above aspect, relative depth information of a two-dimensional image is acquired. A straight line detected from the two-dimensional image is set as a reference line, and multiple reference points are set on the reference line. Because the reference points are linearly arranged on the line, the three-dimensional coordinates of each reference point can be set using a common first coefficient. Then, by applying the two-dimensional coordinates and three-dimensional coordinates of the reference points to a transformation equation, parameters of the transformation equation and a first coefficient are calculated. Therefore, based on the transformation equation to which the calculated parameters have been applied, the first coefficient, and the relative depth of each pixel, the pixel value of each pixel can be positioned at a corresponding position in three-dimensional space, thereby enabling estimation of the three-dimensional space.
[0008] It is possible to estimate a three-dimensional space from a two-dimensional image without using any installed objects such as markers.
[0009] FIG. 1 is a block diagram showing the functional configuration of a three-dimensional space estimation system and a three-dimensional space estimation device according to this embodiment; FIG. 2 is a diagram showing an example of a two-dimensional image to be processed; FIG. 3 is a diagram showing an example of a relative depth map corresponding to the two-dimensional image; FIG. 4 is a diagram showing an example of processing for setting a reference line in a two-dimensional image; FIG. 5 is a diagram showing an example of setting a reference point; FIG. 6 is a diagram explaining first and second coefficients; and FIG. 7 is a diagram showing an example of an estimated three-dimensional space model. It is a flowchart showing the processing content of a three-dimensional space estimation method in the three-dimensional space estimation device; It is a diagram showing the configuration of a three-dimensional space estimation program; It is a hardware block diagram of the three-dimensional space estimation device.
[0010] An embodiment of a three-dimensional space estimation system according to the present invention will be described with reference to the drawings. When possible, the same parts are designated by the same reference numerals and redundant description will be omitted.
[0011] 1 is a diagram showing the functional configuration of a three-dimensional space estimation system and a three-dimensional space estimation device according to this embodiment. The three-dimensional space estimation system 1 of this embodiment is a system that estimates a three-dimensional space represented in a two-dimensional image, and is, for example, configured by a three-dimensional space estimation device 10. The two-dimensional image is, for example, an image taken with a monocular camera.
[0012] 1, the three-dimensional space estimation device 10 functionally comprises an image acquisition unit 11, a relative depth acquisition unit 12, a reference line setting unit 13, a reference point setting unit 14, a two-dimensional coordinate acquisition unit 15, a parameter calculation unit 16, an output unit 17, and a three-dimensional space model generation unit 18. These functional units 11 to 18 may be configured in a single device as exemplified in FIG. 1, or may be configured separately in multiple devices.
[0013] Each of the functional units 11 to 18 of the three-dimensional space estimation device 10 is configured to be able to access storage means (storage) such as a two-dimensional image storage unit 21, a transformation formula storage unit 22, and a three-dimensional space model storage unit 23. Each of the storage units 21 to 23 may be provided in the three-dimensional space estimation device 10, or may be configured in another device configured to be accessible from the three-dimensional space estimation device 10, as exemplified in FIG.
[0014] Next, each functional unit of the three-dimensional space estimation device 10 will be described. The image acquisition unit 11 acquires a two-dimensional image projected from the three-dimensional space. Fig. 2 is a diagram showing an example of a two-dimensional image A acquired by the image acquisition unit. The two-dimensional image A is, for example, an image of the three-dimensional space captured by a monocular camera. The two-dimensional image A may also be an image extracted from a video capturing the three-dimensional space.
[0015] The image acquisition unit 11 may acquire the two-dimensional image A from a camera. Alternatively, the image acquisition unit 11 may acquire the two-dimensional image A from the two-dimensional image storage unit 21. The two-dimensional image storage unit 21 is a storage unit that stores two-dimensional images in advance.
[0016] The relative depth acquisition unit 12 acquires relative depth information that indicates the relative depth of each pixel of the two-dimensional image. The relative depth acquisition unit 12 may acquire a relative depth map corresponding to the two-dimensional image as the relative depth information. The relative depth map is information that numerically indicates the relative depth information for each pixel.
[0017] 3 is a diagram showing an example of a relative depth map corresponding to the two-dimensional image shown in FIG. 2. In the relative depth map B shown in FIG. 3, the closer the object represented by each pixel is from the camera, the smaller the relative depth value (closer to black), and the farther the object is from the camera, the larger the relative depth value (closer to white). Because the relative depth value is a relative value, it is not possible to project each pixel into 3D space at the same ratio as the real object. Note that there is no upper limit to the value of each pixel, but it is possible to represent it like a black-and-white photograph, as shown in FIG. 3.
[0018] The relative depth information can be acquired by a well-known machine learning-based technology such as MiDaS, and there is no limitation on the method used by the relative depth acquisition unit 12 to acquire the relative depth information. The relative depth acquisition unit 12 may acquire a relative depth map B corresponding to the two-dimensional image A, which has been generated in advance and stored in the two-dimensional image storage unit 21.
[0019] The reference line setting unit 13 detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines. Fig. 4 is a diagram showing an example of a process for setting a reference line in a two-dimensional image. As shown in Fig. 4, the reference line setting unit 13 detects straight lines from the two-dimensional image.
[0020] The reference line setting unit 13 may use a Hough transform to detect straight lines from the two-dimensional image A. The method by which the reference line setting unit 13 detects straight lines from the two-dimensional image is not limited, and for example, a well-known technique for detecting edges from an image may be applied.
[0021] The reference line setting unit 13 sets a straight line detected from the two-dimensional image A as the reference line rl. The reference line setting unit 13 may detect a plurality of straight lines from the two-dimensional image A. Then, the reference line setting unit 13 may set a plurality of straight lines from the plurality of straight lines detected from the two-dimensional image A as the reference line rl. Setting a plurality of reference lines makes it possible to improve the accuracy in calculating the parameters of the conversion equation described below.
[0022] Furthermore, the reference line setting unit 13 may set as the reference line rl one or more straight lines having the longest length among the multiple straight lines detected from the two-dimensional image A. Setting a longer straight line as the reference line rl makes it possible to improve the accuracy in calculating the parameters of the conversion equation described below.
[0023] The reference point setting unit 14 sets a plurality of points per reference line rl as reference points. The reference point setting unit 14 then sets the three-dimensional coordinates of each reference point in three-dimensional space by using a first coefficient that converts a relative depth into an absolute depth, which is common to at least each reference point.
[0024] Fig. 5 is a diagram showing an example of setting of reference points. As shown in Fig. 5, the reference point setting unit 14 sets a plurality of reference points rp1 to rp4 (rp) on the reference line rl set by the reference line setting unit 13. The number of reference points rp to be set is not limited as long as it is plural, but a minimum number may be specified due to constraints arising from the conversion formula described later.
[0025] The reference point setting unit 14 may set the plurality of reference points rp at equal intervals. By setting the reference points rp at equal intervals, the ratio of the distances between the three-dimensional coordinates set for each reference point is simplified, which makes it easier to calculate the parameters and coefficients of the conversion equation described later.
[0026] The reference point setting unit 14 may set a first coordinate axis among the first to third coordinate axes in the three-dimensional space to be along the reference line rl, and may set the three-dimensional coordinates of each reference point rp in the three-dimensional space using a first coefficient and a second coefficient that converts the absolute depth into a distance along the first coordinate axis, which are common to each reference point.
[0027] Specifically, the reference point setting unit 14 sets a coordinate system consisting of the X-axis, Y-axis, and Z-axis so that the Z-axis is aligned with the reference line. The reference point setting unit 14 may set the three-dimensional coordinates of the reference points rp1 to rp4, which are set at equal intervals, as follows, for example: rp1 (0,0,1×α×β) rp2 (0,0,2×α×β) rp3 (0,0,3×α×β) rp4 (0,0,4×α×β)
[0028] That is, since the Z axis is set to be along the reference line rl, the X and Y coordinates of each reference point rp can be set to zero.
[0029] 6 is a diagram illustrating the first coefficient and the second coefficient. As shown in FIG. 6, the absolute depths 1×α and 2×α along the optical axis ca of the camera, which are expressed using the first coefficient α, are converted to 1×α×β and 2×α×β, respectively, using the second coefficient β in a coordinate system in which the reference line rl is the Z axis za. Therefore, the second coefficient β varies depending on the angle of the reference line rl relative to the optical axis ca of the camera.
[0030] In this way, by setting a coordinate system in which the Z axis is aligned with the reference line and setting the three-dimensional coordinates of each reference point using the second coefficient β in addition to the first coefficient α, the reference line rl can be set regardless of the angle with respect to the optical axis of the camera for the two-dimensional image. Note that when the Z axis is aligned with the optical axis of the camera, β = 1, so the three-dimensional coordinates of the reference point rp may be set without using the second coefficient β.
[0031] The two-dimensional coordinate acquisition unit 15 acquires the two-dimensional coordinates of each reference point rp in the two-dimensional image. Specifically, the two-dimensional coordinate acquisition unit 15 acquires the coordinates of the reference points rp1 to rp4 in the coordinate system set in the two-dimensional image A. The coordinates of the reference points rp1 to rp4 are expressed, for example, as follows: rp1 (u1, v1) rp2 (u2, v2) rp3 (u3, v3) rp4 (u4, v4)
[0032] The parameter calculation unit 16 calculates the parameters of the conversion formula and the coefficients included in the three-dimensional coordinates by substituting the three-dimensional coordinates and two-dimensional coordinates set for the reference point rp into a predetermined conversion formula. The predetermined conversion formula is a formula that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters.
[0033] The parameter calculation unit 16 calculates at least a first coefficient α included in the three-dimensional coordinates of the reference point rp. Furthermore, if the three-dimensional coordinates of the reference point rp include the first coefficient α and the second coefficient β, the parameter calculation unit 16 calculates the second coefficient β in addition to the first coefficient α.
[0034] The parameters of the transformation equation may include six camera extrinsic parameters related to the projection of the three-dimensional space onto the two-dimensional image, four camera intrinsic parameters, and five parameters related to the camera's lens distortion.
[0035] When the three-dimensional coordinates of the reference point rp include the first coefficient α and the second coefficient β, the parameter calculation unit 16 calculates a total of 17 variables, including the 15 parameters of the conversion formula and the first and second coefficients.
[0036] In calculating the parameters of the transformation equation using the three-dimensional coordinates and two-dimensional coordinates of the reference point rp, the parameters that change with respect to the coordinate system determined by the setting of the reference line rl are six external camera parameters and the second coefficient β, so six variables must be calculated for one reference line, and two variables can be calculated for each reference point, so the reference point setting unit 14 sets at least four reference points rp for one reference line rl. In this way, by setting an appropriate number of reference points according to the number of parameters that make up the transformation equation, it becomes possible to reliably calculate the parameters.
[0037] The parameter calculation unit 16 calculates the following transformation formula, which represents the relationship between three-dimensional coordinates (X, Y, Z) in the three-dimensional space and two-dimensional coordinates (u, v) in the two-dimensional image: In this case, the camera external parameters R (3 degrees of freedom), t (3 degrees of freedom), and the camera internal parameters f x , f y , c x , c y , and a parameter k related to the lens distortion of the camera 1 , k 2 , k 3 , p 1 , p 2 , and the first coefficient and the second coefficient may be calculated.
[0038] The defined transformation formula makes it possible to appropriately represent the correspondence between the three-dimensional space and the two-dimensional image. Then, by calculating the parameters of the transformation formula and the first and second coefficients, it becomes possible to correct the distortion of the two-dimensional image A and obtain the absolute depth of each pixel of the two-dimensional image A.
[0039] The absolute depth of each pixel of the two-dimensional image can be obtained based on the relative depth of the pixel (obtained from the relative depth map) and the first coefficient α, which allows each pixel of the two-dimensional image A to be projected to a position indicated by the absolute depth.
[0040] The above conversion formula is an example and is not limiting. For example, ignoring the lens distortion of the camera, the parameter k 1 , k 2 , k 3 , p1 , p 2 By setting to 0, the above conversion formula can be simplified.
[0041] 1 , the output unit 17 may output a transformation formula. Specifically, the output unit 17 may store the parameters calculated by the parameter calculation unit 16 in the transformation formula storage unit 22. The output unit 17 may also store at least the calculated first coefficient in the transformation formula storage unit 22. The output unit 17 may also store the second coefficient in the transformation formula storage unit 22. The transformation formula storage unit 22 is storage means capable of storing the transformation formula defined by the parameters calculated by the parameter calculation unit 16, the first and second coefficients, etc.
[0042] The three-dimensional space model generating unit 18 arranges the pixel values of each pixel of the two-dimensional image A at corresponding positions in three-dimensional space to generate a three-dimensional space model representing the three-dimensional space.
[0043] 7 is a diagram showing an example of a three-dimensional space model. Specifically, the three-dimensional space model generation unit 18 corrects the two-dimensional image A using various parameters calculated by the parameter calculation unit 16. Specifically, the three-dimensional space model generation unit 18 corrects the two-dimensional image A using a parameter k related to the lens distortion of the camera. 1 , k 2 , k 3 , p 1 , p 2 The distortion of the two-dimensional image A may be corrected by a well-known method based on the above.
[0044] Furthermore, the three-dimensional space model generation unit 18 may obtain the absolute depth of each pixel of the two-dimensional image A based on the relative depth map B and at least the first coefficient α. Specifically, the three-dimensional space model generation unit 18 obtains the relative depth of each pixel of the two-dimensional image A from the relative depth map B and obtains the absolute depth by applying the first coefficient α to the relative depth. The three-dimensional space model generation unit 18 may further apply the second coefficient β to the relative depth to obtain the absolute depth of each pixel of the two-dimensional image A. Then, the three-dimensional space model generation unit 18 projects each pixel of the two-dimensional image A to a position represented by the absolute depth, thereby arranging the pixel value of each pixel in three-dimensional space and generating a three-dimensional space model md representing the three-dimensional space corresponding to the two-dimensional image A.
[0045] The three-dimensional space model generation unit 18 may store data representing the generated three-dimensional space model md in the three-dimensional space model storage unit 23. The three-dimensional space model storage unit 23 is a storage means for storing the generated three-dimensional space model md.
[0046] FIG. 8 is a flowchart showing the processing steps of the three-dimensional space estimation method in the three-dimensional space estimation system 1.
[0047] In step S1, the image acquisition unit 11 acquires a two-dimensional image A projected from a three-dimensional space. In step S2, the relative depth acquisition unit 12 acquires relative depth information (e.g., a relative depth map B) representing the relative depth of each pixel in the two-dimensional image A.
[0048] In step S3, the reference line setting unit 13 detects one or more straight lines from the two-dimensional image A. In step S4, the reference line setting unit 13 sets the detected straight lines as the reference line rl.
[0049] In step S5, the reference point setting unit 14 sets a predetermined number of reference points rp on the reference line rl. In step S6, the reference point setting unit 14 sets three-dimensional coordinates of each reference point rp in a coordinate system set in three-dimensional space. The three-dimensional coordinates set for each reference point rp are set using a first coefficient α that converts relative depth to absolute depth, which is common to at least each reference point.
[0050] In step S7, the two-dimensional coordinate acquisition unit 15 acquires the two-dimensional coordinates of each reference point rp in the two-dimensional image. Specifically, the two-dimensional coordinate acquisition unit 15 acquires the coordinates of the reference point rp in the coordinate system set in the two-dimensional image A.
[0051] In step S8, the parameter calculation unit 16 calculates the parameters of the conversion formula and the coefficients included in the three-dimensional coordinates by substituting the three-dimensional coordinates and two-dimensional coordinates set for the reference point rp into a predetermined conversion formula. The predetermined conversion formula is a formula that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters.
[0052] In step S9, the three-dimensional space model generation unit 18 generates a three-dimensional space model representing the three-dimensional space by arranging the pixel values of each pixel of the two-dimensional image A at corresponding positions in the three-dimensional space. Specifically, the three-dimensional space model generation unit 18 corrects the two-dimensional image A using the various parameters calculated by the parameter calculation unit 16. Then, the three-dimensional space model generation unit 18 projects each pixel of the two-dimensional image A to a position represented by the absolute depth acquired based on the first coefficient and the relative depth information, thereby arranging the pixel values of each pixel in the three-dimensional space.
[0053] Next, a three-dimensional space estimation program for causing a computer to function as the three-dimensional space estimation device 10 of this embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing the configuration of the three-dimensional space estimation program. The three-dimensional space estimation program P1 is configured to include a main module m10 that comprehensively controls the three-dimensional space estimation process in the three-dimensional space estimation device 10, an image acquisition module m11, a relative depth acquisition module m12, a reference line setting module m13, a reference point setting module m14, a two-dimensional coordinate acquisition module m15, a parameter calculation module m16, an output module m17, and a three-dimensional space model generation module m18. Each of the modules m11 to m18 realizes a function for each of the functional units 11 to 18.
[0054] The three-dimensional space estimation program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG.
[0055] According to the three-dimensional space estimation system, three-dimensional space estimation device 10, three-dimensional space estimation method, and three-dimensional space estimation program P1 of the present embodiment described above, relative depth information B of a two-dimensional image A is acquired. Furthermore, a straight line detected from the two-dimensional image A is set as a reference line rl, and multiple reference points rp are set on the reference line rl. Because the reference points rp are linearly arranged on a straight line, the three-dimensional coordinates of each reference point rp can be set using a common first coefficient α. Then, by applying the two-dimensional coordinates and three-dimensional coordinates of the reference points rp to a transformation equation, the parameters of the transformation equation and the first coefficient α are calculated. Therefore, based on the calculated parameters, the first coefficient α, and the relative depth of each pixel, the pixel value of each pixel can be positioned at a corresponding position in three-dimensional space, thereby enabling estimation of three-dimensional space.
[0056] The three-dimensional space estimation system according to the present disclosure may have the following configurations: The actions and effects of each configuration will be described as follows.
[0057] A three-dimensional space estimation system according to one aspect of the present disclosure is a three-dimensional space estimation system that estimates a three-dimensional space represented in a two-dimensional image, and includes: a relative depth acquisition unit that acquires relative depth information that indicates a relative depth of each pixel in the two-dimensional image that is a projected image of the three-dimensional space; a reference line setting unit that detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines; a reference point setting unit that sets a plurality of points per reference line as reference points on the reference lines and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts a relative depth into an absolute depth, which is used in common for at least each reference point; a parameter calculation unit that calculates parameters and a first coefficient of the conversion equation by substituting the three-dimensional coordinates and two-dimensional coordinates of the reference point into a predetermined conversion equation that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters; and a three-dimensional space model generation unit that generates a three-dimensional space model that represents the three-dimensional space by arranging the pixel values of each pixel of the two-dimensional image at a corresponding position in the three-dimensional space using at least a part of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel.
[0058] According to the above aspect, relative depth information of a two-dimensional image is acquired. A straight line detected from the two-dimensional image is set as a reference line, and multiple reference points are set on the reference line. Because the reference points are linearly arranged on the line, the three-dimensional coordinates of each reference point can be set using a common first coefficient. Then, by applying the two-dimensional coordinates and three-dimensional coordinates of the reference points to a transformation equation, parameters and a first coefficient of the transformation equation are calculated. Therefore, based on the calculated parameters, the first coefficient, and the relative depth of each pixel, the pixel value of each pixel can be located at a corresponding position in three-dimensional space, thereby enabling estimation of the three-dimensional space.
[0059] In the three-dimensional space estimation system according to another aspect, the reference point setting unit may set the reference points at equal intervals.
[0060] According to the above aspect, the ratio of the distances between the three-dimensional coordinates of the respective reference points is simplified, which makes it easier to calculate the parameters of the conversion equation and the first coefficient.
[0061] In addition, in a three-dimensional space estimation system according to another aspect, the reference point setting unit may set a first coordinate axis among the first to third coordinate axes in the three-dimensional space to be along the reference line, and set the three-dimensional coordinates of each reference point in the three-dimensional space using a first coefficient and a second coefficient that converts absolute depth into a distance along the first coordinate axis, which are common to each reference point.
[0062] According to the above aspect, by setting a coordinate system in which the first coordinate axis is aligned with the reference line, and setting the three-dimensional coordinates of each reference point using the second coefficient in addition to the first coefficient, the reference line can be set regardless of the angle with respect to the optical axis.
[0063] In addition, in a three-dimensional space estimation system according to another aspect, the parameters of the transformation formula may include six predetermined external camera parameters related to the projection of the three-dimensional space onto a two-dimensional image, four internal camera parameters, and five parameters related to the camera's lens distortion, and the reference point setting unit may set at least four reference points for one reference line.
[0064] According to the above aspect, an appropriate number of reference points is set according to the number of parameters constituting the transformation formula, thereby allowing the parameters to be reliably calculated, and the correspondence between the three-dimensional space and the two-dimensional image to be appropriately expressed by the transformation formula.
[0065] In a three-dimensional space estimation system according to another aspect, the parameter calculation unit calculates a three-dimensional coordinate system using the following conversion formula, which represents a relationship between three-dimensional coordinates (X, Y, Z) in the three-dimensional space and two-dimensional coordinates (u, v) in the two-dimensional image: The external camera parameters R, t and the internal camera parameters f each have three degrees of freedom. x , f y , c x , c y , and a parameter k related to the lens distortion of the camera 1 , k 2 , k 3 , p1 , p 2 , as well as the first coefficient and the second coefficient.
[0066] According to the above aspect, the defined transformation formula makes it possible to appropriately represent the correspondence between the three-dimensional space and the two-dimensional image by the transformation formula.
[0067] In addition, in a three-dimensional space estimation system according to another aspect, the reference line setting unit may set one or more straight lines that are longest among the one or more detected straight lines as the reference line.
[0068] According to the above aspect, the accuracy of the calculated parameters can be improved by setting the longer straight line among the straight lines detected from the two-dimensional image as the reference line.
[0069] In a three-dimensional space estimation system according to another aspect, the reference line setting unit may set a plurality of straight lines from among the one or more detected straight lines as the reference lines.
[0070] According to the above aspect, by setting a plurality of straight lines out of the straight lines detected from the two-dimensional image as the reference lines, it is possible to improve the accuracy of the calculated parameters.
[0071] The block diagram shown in FIG. 1 shows functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.
[0072] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.
[0073] For example, the three-dimensional space estimation device 10 according to an embodiment of the present invention may function as a computer. Fig. 10 is a diagram showing an example of the hardware configuration of the three-dimensional space estimation device 10 according to this embodiment. The three-dimensional space estimation device 10 may be physically configured as a computer including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.
[0074] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the three-dimensional space estimation apparatus 10 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.
[0075] Each function of the three-dimensional space estimation device 10 is realized by loading specific software (programs) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations and control communication via a communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.
[0076] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the functional units 11 to 18 shown in FIG. 1 may be realized by the processor 1001.
[0077] The processor 1001 also reads programs (program codes), software modules, and data from the storage 1003 and / or the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the functional units 11 to 18 of the 3D space estimation device 10 may be implemented by a control program stored in the memory 1002 and executed by the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The programs may also be transmitted from a network via a telecommunications line.
[0078] The memory 1002 is a computer-readable recording medium and may be composed of at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 may store executable programs (program codes), software modules, etc. for implementing the 3D space estimation method according to one embodiment of the present invention.
[0079] Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including memory 1002 and / or storage 1003.
[0080] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.
[0081] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0082] Furthermore, each device such as the processor 1001 and the memory 1002 is connected to a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.
[0083] The three-dimensional space estimation device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.
[0084] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.
[0085] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (New Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and next-generation systems enhanced based on these. Furthermore, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.) may also be applied.
[0086] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0087] In the present disclosure, a specific operation described as being performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and another network node other than the base station (for example, an MME or an S-GW, etc., but are not limited to these). Although the above example illustrates a case where there is one other network node other than the base station, a combination of multiple other network nodes (for example, an MME and an S-GW) may also be used.
[0088] Information etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer), or may be input / output via multiple network nodes.
[0089] Input and output information may be stored in a specific location (for example, memory) or managed in a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.
[0090] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0091] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).
[0092] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
[0093] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0094] Software, instructions, etc. may also be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.
[0095] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0096] It should be noted that terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.
[0097] As used in this disclosure, the terms "system" and "network" are used interchangeably.
[0098] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed as absolute values, relative values from a predetermined value, or other corresponding information. For example, a radio resource may be indicated by an index.
[0099] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.
[0100] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.
[0101] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0102] When designations such as "first," "second," etc. are used in this disclosure, any reference to an element does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.
[0103] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.
[0104] To the extent that the terms "include," "including," and variations thereof are used herein or in the claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used herein or in the claims, is not intended to be an exclusive or.
[0105] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0106] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."
[0107] A three-dimensional space estimation system 1 according to the present disclosure may have the following configuration: [1] A three-dimensional space estimation system for estimating a three-dimensional space represented in a two-dimensional image, comprising: a relative depth acquisition unit that acquires relative depth information representing a relative depth of each pixel of the two-dimensional image, which is an image obtained by projecting the three-dimensional space; a reference line setting unit that detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines; a reference point setting unit that sets a plurality of points per reference line as reference points on the reference line and sets three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts the relative depth into an absolute depth, the first coefficient being common to at least each reference point; a two-dimensional coordinate acquisition unit that acquires two-dimensional coordinates of each reference point in the two-dimensional image; and a parameter calculation unit that calculates parameters and the first coefficients of a conversion equation that expresses a relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters, by substituting the three-dimensional coordinates and the two-dimensional coordinates of the reference points into a predetermined conversion equation. and a three-dimensional space model generation unit that generates a three-dimensional space model representing the three-dimensional space by arranging pixel values of each pixel of the two-dimensional image at corresponding positions in the three-dimensional space using at least some of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel. [2] The three-dimensional space estimation system described in [1], wherein the reference point setting unit sets the plurality of reference points at equal intervals. [3] The three-dimensional space estimation system described in [1] or [2], wherein the reference point setting unit sets a first coordinate axis of first to third coordinate axes in the three-dimensional space to be along the reference line, and sets the three-dimensional coordinates in the three-dimensional space of each reference point by using the first coefficient and a second coefficient that converts the absolute depth into a distance along the first coordinate axis, which are common to each reference point.[4] The three-dimensional space estimation system according to [3], wherein the parameters of the transformation equation include six predetermined camera external parameters, four camera internal parameters, and five parameters related to camera lens distortion, and the reference point setting unit sets at least four reference points for one reference line. [5] The parameter calculation unit calculates the following transformation equation that expresses the relationship between three-dimensional coordinates (X, Y, Z) in the three-dimensional space and two-dimensional coordinates (u, v) in the two-dimensional image: The external camera parameters R, t and the internal camera parameters f each have three degrees of freedom. x , f y , c x , c y , and a parameter k related to the lens distortion of the camera 1 , k 2 , k 3 , p 1 , p 2 [4] The three-dimensional space estimation system according to [4], wherein the reference line setting unit calculates, as the reference line, one or more straight lines that are longest among the one or more detected straight lines. [6] The three-dimensional space estimation system according to any one of [1] to [5], wherein the reference line setting unit sets, as the reference line, one or more straight lines that are longest among the one or more detected straight lines. [7] The three-dimensional space estimation system according to any one of [1] to [6], wherein the reference line setting unit sets, as the reference line, a plurality of straight lines among the one or more detected straight lines.
[0108] 1...3D space estimation system, 10...3D space estimation device, 11...image acquisition unit, 12...relative depth acquisition unit, 13...baseline setting unit, 14...reference point setting unit, 15...2D coordinate acquisition unit, 16...parameter calculation unit, 17...output unit, 18...3D space model generation unit, 21...2D image storage unit, 22...conversion formula storage unit, 23...3D space model storage unit, M1...recording medium, m10...main module, m11...image acquisition module, m12...relative depth acquisition module, m13...baseline setting module, m14...reference point setting module, m15...2D coordinate acquisition module, m16...parameter calculation module, m17...output module, m18...3D space model generation module, P1...3D space estimation program, md...3D space model, rl...baseline, rp...reference point.
Claims
1. A three-dimensional space estimation system for estimating a three-dimensional space represented in a two-dimensional image, comprising: a relative depth acquisition unit that acquires relative depth information representing the relative depth of each pixel of the two-dimensional image, which is an image obtained by projecting the three-dimensional space; a reference line setting unit that detects one or more straight lines from the two-dimensional image and sets the detected straight lines as reference lines; a reference point setting unit that sets a plurality of points per reference line as reference points on the reference line and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using a first coefficient that converts the relative depth into an absolute depth commonly used for at least each reference point; a two-dimensional coordinate acquisition unit that acquires the two-dimensional coordinates of each reference point in the two-dimensional image; and a parameter calculation unit that calculates the parameters of the conversion equation and the first coefficient by substituting the three-dimensional coordinates and the two-dimensional coordinates of the reference points into a predetermined conversion equation that expresses the relationship between the three-dimensional coordinates in the three-dimensional space and the two-dimensional coordinates in the two-dimensional image using predetermined parameters; and a three-dimensional space model generation unit that generates a three-dimensional space model representing the three-dimensional space by placing pixel values of each pixel of the two-dimensional image at corresponding positions in the three-dimensional space using at least a portion of the parameters calculated by the parameter calculation unit, the first coefficient, and the relative depth of each pixel.
2. The three-dimensional space estimation system according to claim 1, wherein the reference point setting unit sets the reference points at equal intervals.
3. The three-dimensional space estimation system of claim 1, wherein the reference point setting unit sets a first coordinate axis among first to third coordinate axes in the three-dimensional space to be aligned with the reference line, and sets the three-dimensional coordinates of each reference point in the three-dimensional space by using the first coefficient and a second coefficient that converts the absolute depth into a distance along the first coordinate axis in common for each reference point.
4. The three-dimensional space estimation system of claim 3, wherein the parameters of the transformation equation include six predetermined camera external parameters related to the projection of the three-dimensional space onto the two-dimensional image, four camera internal parameters, and five parameters related to camera lens distortion, and the reference point setting unit sets at least four reference points for one reference line.
5. The parameter calculation unit calculates the following conversion formula that represents the relationship between three-dimensional coordinates (X, Y, Z) in a three-dimensional space and two-dimensional coordinates (u, v) in a two-dimensional image: The external camera parameters R,t and the internal camera parameters f, each of which has three degrees of freedom, x , f y , c x , c y , and a parameter k related to the lens distortion of the camera 1 , k 2 , k 3 , p 1 , p 2 The three-dimensional space estimation system according to claim 4 , further comprising: a first coefficient and a second coefficient calculating unit for calculating the first coefficient and the second coefficient.
6. The three-dimensional space estimation system according to claim 1, wherein the reference line setting unit sets, as the reference line, one or more straight lines which are longest among the one or more detected straight lines.
7. The three-dimensional space estimation system according to claim 1, wherein the reference line setting unit sets a plurality of straight lines from among the one or more detected straight lines as the reference lines.
Citation Information
Patent Citations
Calibration device, calibration method, and program
JP2016100698A
Information processing apparatus, information processing method, and program
JP2017129942A
Camera calibration method, camera calibration program and camera calibration device
JP2018179981A
Detection apparatus, detection method, and detection program
JP2021189946A
