Generating a training dataset for depth measurement

Abstract

A system for generating a training dataset is provided. The system controls a depth sensor to capture a first image of a first object and a first depth value associated with the first object from a first viewpoint. The system receives tracking information from a handheld device associated with the depth sensor based on movement of the handheld device and the depth sensor in 3D space. The system generates graphical information corresponding to the first object based on the received tracking information. The graphical information includes the first object from a second viewpoint. The system calculates a second depth value associated with the first object based on the graphical information. The system generates a training dataset for a neural network model, the training dataset including a first combination of the first image and the first depth value and a second combination of the second image and the second depth value corresponding to the graphical information.

Description

[Technical field]

[0001] CROSS REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE none

[0002] Various embodiments of the present disclosure relate to data set generation for neural network models, and more particularly, to systems and methods for generating training data sets for neural network models for depth measurement. [Background technology]

[0003] Traditionally, range imaging applications are used for depth estimation. Range imaging applications can utilize various types of depth sensors (such as time-of-flight (ToF) sensors) to perform depth estimation of objects present in a real environment. In general, depth sensors employed by range imaging applications can be expensive and structurally difficult to implement. Therefore, it can be a complex and costly task to deploy a large number of depth sensors to collect a large number of depth values ​​of different objects present in a real environment (or capture depth values ​​of objects from numerous positions or viewpoints). Summary of the Invention [Problem to be solved by the invention]

[0004] The limitations and disadvantages of the conventional approach will become apparent to one skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. [Means for solving the problem]

[0005] Provided are systems and methods for generation of a depth measurement training data set substantially as illustrated and / or described in connection with at least one of the figures and as more fully set forth in the claims.

[0006] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example network environment for generating a training dataset for depth measurement, according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of a system for generating a training data set for depth measurement, according to an embodiment of the present disclosure. [Diagram 3] 1 is an exemplary scenario illustrating capture of a first image for generation of a training dataset for depth measurement, according to an embodiment of the present disclosure. [Figure 4A] FIG. 4B shows a sequence diagram of an example operation for generating a training dataset for depth measurement according to an embodiment of the present disclosure. [Figure 4B] FIG. 4B shows, in conjunction with FIG. 4A, a sequence diagram of an example operation for generating a training dataset for depth measurement according to an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart illustrating an example method for generating a training dataset for depth measurement, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The disclosed system and method for generating a training dataset for depth measurement may find implementation as described below. An exemplary aspect of the present disclosure provides a system configured to control a depth sensor (such as a time-of-flight (ToF) sensor) to capture a first image of one or more first objects from a first viewpoint. Examples of the one or more first objects may include, but are not limited to, animate objects (such as a human) or inanimate objects (such as non-animate entities such as furniture, electronic appliances, or infrastructure). The first image may be associated with a first object (such as a chair) of the one or more first objects. The first image may include color information (such as red, green, blue (RGB) information) of the one or more first objects. The first viewpoint may be, for example, a front view of the first object (such as a chair) relative to the depth sensor. The system may further control the depth sensor to capture a first depth value associated with the one or more first objects from the first viewpoint (such as a front view). The first depth value may indicate depth information or distance between the depth sensor and one or more points on the surface of one or more first objects (i.e., the surface visible from a first viewpoint).

[0009] The system may further receive tracking information from the handheld device (e.g., a virtual reality device) based on a movement of the handheld device and the depth sensor in three-dimensional (3D) space. The handheld device may be connected (or joined or integrated) with the depth sensor. For example, after an initial image capture, the position and orientation of the depth sensor may change in the 3D space based on the movement of the handheld device and the depth sensor. Thus, the tracking information may indicate positional or orientation information of the depth sensor in the 3D space. The system may further generate graphic information (e.g., but not limited to, point cloud data, images, videos, or 3D meshes) corresponding to the one or more first objects based on the received tracking information (i.e., the movement of the depth sensor). According to an embodiment, the system may include a 3D graphics engine (e.g., an animation engine or animation application) for generating the graphic information. The graphic information may include or correspond to one or more first objects (e.g., a chair) from one or more second viewpoints (e.g., but not limited to, a side view, a top view, a rear view, etc.) different from the first viewpoint. The second viewpoint of the one or more first objects can be based on the movement of the handheld device and the depth sensor in 3D space. Thus, generation of graphical information by the disclosed system can enable conversion of real-world information (such as one or more first objects captured in a first image by a depth sensor) to virtual information (such as graphical information related to the one or more first objects generated from a second viewpoint different from the first viewpoint from which the first image was captured).

[0010] The system may further calculate one or more second depth values ​​associated with the one or more first objects based on the generated graphical information from the one or more second viewpoints. The one or more second depth values ​​may include depth information (such as distance information) between the depth sensor and the one or more first objects from the one or more second viewpoints. The system may further enable generation and rendering of graphical information in a virtual environment (such as in a 3D graphics engine) to calculate one or more second depth values ​​associated with the one or more first objects based on the generated graphical information. Thus, the disclosed system may enable calculation of one or more second depth values ​​from the graphical information in a time-efficient manner compared to depth values ​​calculated by a conventional depth sensor.

[0011] In some embodiments, the system can generate one or more second images corresponding to the graphic information. The one or more second images of the one or more first objects from the one or more second viewpoints can be referred to as synthetic data generated by the system (or by the 3D graphics engine of the system). The generation of synthetic data related to the one or more first objects (i.e., real objects) can be inexpensive compared to capturing a large amount of real data (such as a first image) of the objects available in the real environment by a depth sensor. Thus, the system can enable the generation of synthetic data (i.e., graphic information or images of the object generated from different viewpoints) from limited real-world data (i.e., a first image of the real object captured from a first viewpoint by a depth sensor). For example, the system (i.e., a 3D graphics engine) can enable the generation of synthetic data (i.e., graphic information) related to a shipwreck (i.e., an object) present in the sea by 3D reconstruction of the shipwreck in the 3D graphics engine without repeatedly scanning the shipwreck from different viewpoints (i.e., using a depth sensor).

[0012] Furthermore, the system can generate a training data set for the neural network model, the training data set including a first combination of a first image (i.e., a real image of the object captured from a first viewpoint) and a first depth value (i.e., a real depth value), and further including a second combination of one or more second images (i.e., images generated by a 3D graphics engine from different viewpoints) corresponding to graphic information of the object and one or more second depth values ​​(i.e., depth values ​​calculated from graphic information about the object). In another embodiment, the disclosed system can generate a third image based on the first depth value, and generate one or more fourth images based on the one or more second depth values ​​to further generate a training data set including the first combination of the first image and the third image (i.e., real data), and may further include a second combination of the second image and the fourth image (i.e., virtual data). Thus, the training data set generated by the system can include inexpensively generated synthetic data that can be used to train the neural network model. The neural network model trained based on the generated training dataset can be utilized for automatic prediction of depth values ​​from images (such as images having two-dimensional (2D) information) provided as input to the neural network model.

[0013] FIG. 1 is a diagram of an exemplary network environment for generating a training dataset for depth measurement, according to an embodiment of the present disclosure. FIG. 1 shows a block diagram of a network environment 100. The network environment 100 can include a system 102. The system 102 can include a three-dimensional (3D) graphics engine 104. The network environment 100 can further include a depth sensor 106 and a handheld device 108. The network environment 100 can further include a first scene 110. The first scene 110 can include one or more first objects 112. The one or more first objects 112 can include a first object 112A, a second object 112B, and an Nth object 112N. The network environment 100 can further include a first input / output (I / O) device 114. The system 102 can further include a training dataset 116 generated by the system 102 for one or more first objects 112 present in the first scene 110 or in a different scene. Additionally, the network environment 100 may include a communications network 118 that allows the system 102, the depth sensor 106, and the handheld device 108 to communicate.

[0014] The system 102 may include suitable logic, circuits, and interfaces that may be configured to receive a first image of one or more first objects 112 and a first depth value associated with the one or more first objects 112 from the depth sensor 106. The system 102 may further generate graphical information and calculate one or more second depth values ​​associated with the one or more first objects 112. The system 102 may further be configured to generate a training data set 116 for the neural network model that may include a first combination of the first image and the first depth value, and a second combination of one or more second images and one or more second depth values ​​corresponding to the graphical information. Examples of the system 102 may include, but are not limited to, an animation system, a computing device, a laptop, a mainframe machine, a server, a computing workstation, a smart phone, a cellular phone, a mobile phone, a gaming device, and / or a consumer electronics (CE) device.

[0015] The 3D graphics engine 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to generate graphical information corresponding to the one or more first objects 112 from one or more second viewpoints based on the tracking information received by the system 102. The 3D graphics engine 104 may further control a display device (such as included in the first I / O device 114) to render the generated graphical information related to the one or more first objects 112. The 3D graphics engine 104 may utilize a script generator (such as a python script generator) to generate the graphical information. In some embodiments, the 3D graphics engine 104 may render a first image of the one or more first objects 112 captured by the depth sensor 106 from a first viewpoint, receive one or more user inputs related to the rendered first image via the first I / O device 114, and control the display device to generate graphical information of the one or more first objects 112 from different viewpoints. The graphical information of the one or more first objects 112 may include, but is not limited to, rotated objects, resized objects, repositioned objects, point cloud data, video information, voxel information, or a 3D mesh. Examples of the 3D graphics engine 104 may include, but are not limited to, a 3D computer graphics engine, a 3D creation application, a 3D modeling or simulation application, a 3D animation engine, or a video editing application.

[0016] The depth sensor 106 may include suitable logic, circuitry, and interfaces that may be configured to capture a first image of one or more first objects 112. The depth sensor 106 may be further configured to capture a first depth value associated with one or more first objects 112, such as the first object 112A. The first depth value may indicate a distance between the depth sensor 106 and one or more points on a surface of the first object 112A. Examples of the depth sensor 106 may include, but are not limited to, a time-of-flight (ToF) sensor, a light detection and ranging (LiDAR) sensor, a stereoscopic image sensor, or a structured light sensor.

[0017] The handheld device 108 may include suitable logic, circuits, and interfaces that may be configured to generate tracking information related to the depth sensor 106 based on the movement of the handheld device 108 and the depth sensor 106 in 3D space. The handheld device 108 may be held or worn by a user (as shown in FIG. 3) and may be integrated with the depth sensor 106. The handheld device 108 may include one or more sensors that generate tracking information (i.e., position or orientation in 3D space). Examples of the handheld device 108 may include, but are not limited to, a virtual reality device, an augmented reality device, or a mixed reality device. In some embodiments, the handheld device 108 may be any computing device integrated with a sensor that detects the position or orientation of the handheld device 108 in 3D space. The sensor may include, but is not limited to, an inertial measurement unit (IMU) sensor, an accelerometer, a gyroscope, or a motion sensing device.

[0018] The first scene 110 may include or relate to a real-world environment, such as a room in a building, an open space area such as a park, an underwater scene such as an ocean scene, etc. The first scene 110 may include one or more first objects 112, such as a first object 112A, a second object 112B, and an Nth object 112N. Examples of the one or more first objects 112 may include, but are not limited to, living objects (such as humans, animals, etc.) or inanimate objects (such as furniture, electronic gadgets, infrastructure, transportation objects, or decorative objects).

[0019] The first input / output (I / O) device 114 may include suitable logic, circuits, and interfaces that may be configured to receive input from a user and provide output based on the received input. The first I / O device 114 may be configured to receive input from a user (as shown in FIG. 3) and generate graphical information corresponding to one or more first objects 112. Additionally, the first I / O device 114 may display a first image captured by the depth sensor 106, or display one or more calculated second depth values ​​or a fourth image associated with the one or more first objects 112. The first I / O device 114 may include a variety of input and output devices and may be configured to communicate with the system 102 or the 3D graphics engine 104. Examples of the first I / O device 114 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, or a display device. Although FIG. 1 illustrates the first I / O device 114 separate from the system 102, the disclosure is not so limited. Thus, in some embodiments, the first I / O device 114 may be integrated into the 3D graphics engine 104 of the system 102 as shown in FIG. 2 without departing from the scope of this disclosure.

[0020] The training dataset 116 may be an image-based dataset that the neural network model can utilize to predict depth values ​​from 2D images. The training dataset 116 may include a first set of paired datasets (such as a first set of paired images). The first set of paired images may include a first image that may include color information of the first object 112A and a corresponding third image that may include or correspond to depth information (i.e., a first depth value) of the first object 112A. The training dataset 116 may further include a first set of unpaired datasets (such as a first set of unpaired images). The first set of unpaired images may include graphic information (i.e., a second image) of the first object 112A generated by the 3D graphics engine 104 from a different viewpoint and a fourth image that may include or correspond to depth information (i.e., a second depth value) of the first object 112A from a different viewpoint. Thus, the neural network model may be trained based on the generated training data set 116, which may include a first set of paired images and a first set of unpaired images.

[0021] The communication network 118 may include any communication medium that allows the system 102, the depth sensor 106, and the handheld device 108 to communicate with each other. The communication network 118 may be either a wired connection or a wireless connection. Examples of the communication network 118 include, but are not limited to, the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 can be configured to connect to the communication network 118 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols can include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0022] During operation, the system 102 may be configured to control the depth sensor 106 to capture a first image of one or more first objects 112 present in a first scene 110 from a first viewpoint and to capture first depth values ​​associated with the one or more first objects 112. The first scene 110 may depict one or more first objects 112 and a surrounding environment, such as a wall around the one or more first objects 112. The first image may include color information of the one or more first objects 112. The first viewpoint may be, for example, a front view of the first object 112A of the one or more first objects 112. According to an embodiment, the first depth values ​​associated with the one or more first objects 112 may indicate depth information (i.e., distance) between the depth sensor 106 and a first plurality of points on a surface of the one or more first objects 112 captured from the first viewpoint. Details of controlling the depth sensor 106 to capture the first image and the first depth value associated with the one or more first objects 112 are further illustrated in, for example, FIG. 4A.

[0023] The system 102 may further receive tracking information from the handheld device 108 associated with the depth sensor 106 based on the movement of the handheld device 108 and the depth sensor 106 in 3D space. The tracking information may indicate position information (X-axis, Y-axis, Z-axis information) or orientation information of the depth sensor 106 in 3D space. The position information or orientation information of the depth sensor 106 may change based on changes in the position and orientation of the depth sensor 106 in 3D space. Details of receiving the tracking information are further shown, for example, in FIG. 4A.

[0024] The system 102 may further generate graphic information corresponding to the one or more first objects 112 based on the received tracking information. In some embodiments, the system 102 may generate graphic information corresponding to the one or more first objects 112 based on the one or more user inputs received from the first I / O device 114. The graphic information may include, but is not limited to, one or more second images, point cloud data, voxel sets, three-dimensional (3D) meshes, video information, LAS (LASer) format data, or proprietary format data. According to an embodiment, the 3D graphics engine 104 included in the system 102 may be configured to generate graphic information corresponding to the one or more first objects 112. The graphic information may include or correspond to the one or more first objects 112 from one or more second perspectives different from the first perspective. The one or more second perspectives may include, for example, a front view, a side view, a rear view, a top view, a rotational view, or a translational view of the one or more first objects 112. The system may further generate one or more second images corresponding to the graphical information, details of which are further shown, for example, in FIG.

[0025] The system 102 may further be configured to calculate one or more second depth values ​​associated with the one or more first objects 112 based on the generated graphic information of the one or more first objects 112 from the one or more second viewpoints. According to an embodiment, the one or more second depth values ​​may indicate depth information (i.e., distance) between the depth sensor 106 and a second plurality of points on the surface of the one or more first objects 112 captured from the one or more second viewpoints. Details of the calculation of the one or more second depth values ​​are further illustrated, for example, in FIG. 4B. The system 102 may further generate a training data set 116 for the neural network model, which may include a first combination of the first image and the first depth value, and a second combination of the one or more second images and the one or more second depth values ​​corresponding to the graphic information. Details of the generation of the training data set 116 are further illustrated, for example, in FIG. 4B.

[0026] FIG 2 is a block diagram of a system for generating a training dataset for depth measurement according to an embodiment of the present disclosure. The description of FIG 2 is provided with reference to the elements of FIG 1. FIG 2 illustrates a block diagram 200 of a system 102. The system 102 may include a processor 202 and a memory 204. The memory 204 may further include a training dataset 116 and a neural network model 204A. The system 102 may further include a 3D graphics engine 104, a first I / O device 114, and a network interface 206.

[0027] The processor 202 may include suitable logic, circuits, and interfaces that may be configured to execute a set of instructions stored in the memory 204. The processor 202 may be configured to execute program instructions associated with different operations performed by the system 102. For example, some of the operations may include controlling the depth sensor 106 to capture a first image of one or more first objects 112 from a first viewpoint, receiving tracking information from the handheld device 108, generating graphical information corresponding to the one or more first objects 112 from one or more second viewpoints, and calculating one or more second depth values ​​associated with the one or more first objects 112. The processor 202 may further be configured to generate a training data set 116 for training the neural network model 204A. The processor 202 may be implemented based on multiple processor technologies known in the art. Examples of processor technologies include, but are not limited to, central processing units (CPUs), X86-based processors, reduced instruction set computing (RISC) processors, application specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, graphical processing units (GPUs), and other processors.

[0028] The memory 204 may include suitable logic, circuits, and interfaces that may be configured to store one or more instructions executed by the processor 202. The memory 204 may be configured to store the training data set 116 and the neural network model 204A. The memory 204 may be further configured to store, but is not limited to, a first image, a first depth value, a third image associated with the first depth value, tracking information, graphic information corresponding to one or more first objects 112, one or more second depth values, and one or more fourth images associated with the one or more second depth values. Example implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0029] The neural network model 204A may be a computational network or system in which nodes are artificial neurons arranged in multiple layers. The multiple layers of the neural network model 204A may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or artificial neurons). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of the neural network model 204A. The output of each hidden layer may be coupled to the input of at least one node in the other layers of the neural network model 204A. The node(s) in the final layer may receive inputs from at least one hidden layer and output results. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the neural network model 204A. Such hyperparameters may be set before or during training of the neural network model 204A based on a training dataset.

[0030] Each node of the neural network model 204A may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters may include, for example, weight parameters and regularization parameters. Each node may use a mathematical function to calculate an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network model 204A. All or some of the nodes of the neural network model 204A may correspond to the same or different mathematical functions.

[0031] In training the neural network model 204A, one or more parameters of each node of the neural network model 104 may be updated based on whether the output of the final layer for a given input (from the training dataset 116) matches the correct result based on the loss function of the neural network model 204A. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved to minimize the training error. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0032] The neural network model 204A may include electronic data, such as software programs, software program codes, libraries, applications, scripts, or other logic or instructions, executed by a processing device, such as the system 102. The neural network model 204A may include code and routines configured to enable a computing device, such as the processor 202, to perform one or more operations to predict depth information based on a provided input, such as a new image or computer generated image (CGI) input. Additionally or alternatively, the neural network model 204A may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network model 204A may be implemented using a combination of hardware and software.

[0033] Examples of the neural network model 204A may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-recurrent neural networks (CNN-RNNs), R-CNNs, Fast R-CNNs, Faster R-CNNs, artificial neural networks (ANNs), (You Only Look Once) YOLO networks, long short-term memory (LSTM) network-based RNNs, CNNs+ANNs, LSTMs+ANNs, gated recurrent units (GRUs)-based RNNs, fully connected neural networks, Connectionist Temporal Classification (CTC)-based RNNs, deep Bayesian neural networks, generative adversarial networks (GANs), and / or combinations of these networks. In some embodiments, the neural network model 204A may include a numerical computation method using a data flow graph. In some embodiments, the neural network model 204A may be based on a hybrid architecture of multiple deep neural networks (DNNs). In some embodiments, the neural network model 204A may be an artificial intelligence (AI) engine.

[0034] The network interface 206 may include suitable logic, circuits, and interfaces that may be configured to facilitate communication between the system 102, the depth sensor 106, and the handheld device 108 over the communications network 118. The network interface 206 may be implemented using various known technologies that support wired or wireless communication between the system 102 and the communications network 118. The network interface 206 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit. The network interface 206 may be configured to communicate via wireless communication with a network, such as the Internet, an intranet, or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN). The wireless communication may be configured to use one or more of a number of communication standards, protocols, and technologies, such as Global System for Mobile communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), or Worldwide Interoperability for Microwave Access (Wi-MAX), protocols for email, instant messaging, and short message service (SMS).

[0035] FIG. 3 is an exemplary scenario illustrating the capture of a first image for the generation of a training data set for depth measurement according to an embodiment of the present disclosure. FIG. 3 is described with reference to elements of FIG. 1 and FIG. 2. FIG. 3 illustrates an exemplary scenario 300. The exemplary scenario 300 may include a first object 302 (such as a chair), a user 304, a handheld device 306, a depth sensor 308, and a display device 310 associated with the 3D graphics engine 104 (or part of the first I / O device 114). The display device 310 may display a first image 312 captured by the depth sensor 308. The display device 310 and the 3D graphics engine 104 of the disclosed system 102 may be communicatively coupled to the depth sensor 308 and the handheld device 306 via a communication network 118.

[0036] As shown in Fig. 3, a user 304 can capture a first object 302 of one or more first objects 112 via a depth sensor 308. For example, a first image 312 (such as a front view) of the first object 302 from a first viewpoint can be captured by the depth sensor 308. As shown in Fig. 3, the depth sensor 308 and the handheld device 306 (which can be a virtual reality device) can be integrated. In some embodiments, the processor 202 of the system 102 can be configured to control the depth sensor 308 to capture a first image of the first object 302 and to capture one or more first depth values ​​that can indicate a distance (e.g., in a particular unit of centimeters, inches, feet, yards, or meters) between the depth sensor 308 and a first plurality of points on a surface of the first object 302 (i.e., a surface that is visible from a first viewpoint). The processor 202 can transmit instructions or commands to control the depth sensor 308 to capture a first image and one or more first depth values. In some embodiments, the depth sensor 308 or handheld device 306 can receive user input directly from the user 304 to control the depth sensor 308 to capture a first image and one or more first depth values ​​associated with the first object 302. In an embodiment, the processor 202 can be configured to receive from the depth sensor 308 a first image 312 of the first object 302 and a corresponding first depth value associated with the first image 312.

[0037] In an embodiment, the user 304 can move the depth sensor 308 and the handheld device 306 in a 3D space (i.e., real space). The handheld device 306 can capture position information (XYZ information) and orientation information of the depth sensor 308 based on, for example, the movement of the depth sensor 308 and the handheld device 306 in the 3D space. The handheld device 306 can be further configured to generate tracking information, which can include position information and orientation information of the handheld device 306 in the 3D space. The processor 202 can be further configured to receive the tracking information from the handheld device 306 and generate graphic information corresponding to the first object 302. Details of the generation of the graphic information are further illustrated, for example, in FIGS. 4A-4B.

[0038] The graphical information may include or correspond to the first object 302 from one or more second viewpoints, which may be different from the first viewpoint from which the first image 312 of the first object 302 was captured. For example, the one or more second viewpoints may include or correspond to the first object 302 from a side view, a plan view, or a top view. Details of the one or more second viewpoints and the graphical information are further illustrated, for example, in FIG. 4A. In some embodiments, the processor 202 may receive one or more user inputs from the user 304 (i.e., via the first I / O device 114) for modifying the first image 312 (i.e., the image corresponding to the first object 302) displayed on the display device 310. The processor 202 may further generate the graphical information based on the received one or more user inputs. For example, the user 304 may provide a user input via the first I / O device 114 to rotate the first image 312 to obtain a side view of the first object 302 in the 3D graphics engine 104. In another example, the user 304 may provide a user input to zoom in on the first image 312 to obtain a close-up view of the first object 302 in the 3D graphics engine 104. In some embodiments, the 3D graphics engine 104 may control the display device 310 and the first I / O device 114 to rotate or zoom in the first image 312 to further generate graphical information from different perspectives.

[0039] The processor 202 may be further configured to calculate one or more second depth values ​​associated with the first object 302 based on the generated graphic information. For example, the processor 202 may be configured to calculate depth values ​​associated with a side view of the first object 302 and a close-up view of the first object 302 generated by the 3D graphics engine 104. Thus, by receiving one image (such as the first image 312) associated with the first object 302 from the depth sensor 308, synthetic data (such as one or more second images corresponding to the graphic information and the one or more second depth values) may be generated that may include depth information of the first object 302 from different viewpoints. The first image 312 of the first object 302 captured by the depth sensor 308 from the first viewpoint may be referred to as real data, and the virtually generated synthetic data (i.e., synthetic data generated by the system 102 or the 3D graphics engine 104) of the first object 302 from different viewpoints may be referred to as virtual data for the first object 302 from various viewpoints. The virtual data may closely mimic the real data with certain variations (i.e., modifying the viewpoint generating the graphical information, etc.). The processor 202 may be further configured to generate a training data set 116, which may include such real and virtual data for different objects, for example, as described in Figures 4A-4B. It is noted that the capture of the first image 312 of the first object 302 (e.g., a chair), the depth sensor 308, and the handheld device 306 are shown in Figure 3 as an example only. The scenario 300 may also include different types of objects, depth sensors, and handheld devices without departing from the scope of the present disclosure.

[0040] 4A-B collectively illustrate sequence diagrams of example operations for generating a training dataset for depth measurement, according to an embodiment of the present disclosure. The description of Figures 4A-B is provided with reference to elements of Figures 1, 2, and 3. Figures 4A-B illustrate a sequence diagram 400 illustrating a series of operations 402-418. These series of operations may be performed by various components of system 102 of Figure 1 or processor 202 of Figure 2.

[0041] At 402, a first image 402A and a first depth value may be captured. According to an embodiment, the processor 202 may be configured to control the depth sensor 308 to capture a first image 402A of a first object 302 of the one or more first objects 112 from a first viewpoint. The processor 202 may further control the depth sensor 308 to capture a first depth value associated with the first object 302. According to an embodiment, the depth sensor 308 may be, but is not limited to, a Time of Flight (ToF) sensor, a Light Detection and Ranging (LiDAR) sensor, a stereoscopic image sensor, or a structured light sensor. The ToF sensor may utilize an infrared light source to capture the first depth value associated with the first object 302. The LiDAR sensor may utilize a laser light source to capture the first depth value associated with the first object 302. Additionally, the stereoscopic image sensor may utilize two imagers (not shown) to capture the first depth value. The structured light sensor can utilize a light pattern projected onto the first object 302 to capture a first depth value.

[0042] The first image 402A may include color information of the first object 302. For example, the first image 402A may include color information such as actual color (e.g., red-green-blue (RGB) information) of the first object 302. According to an embodiment, the first depth value may indicate depth information or distance between the depth sensor 308 and a first plurality of points on a surface of the first object 302 of the one or more first objects 112. For example, the first plurality of points on the surface of the first object 302 may include points on the surface of the first object 302 that may be within a field of view of the depth sensor 308 from a first viewpoint. In one example, the surface that may be located within a field of view of the depth sensor 308 may include a surface of the first object 302 as seen from the first viewpoint. The first depth value may relate to a distance between each of the first plurality of points on the surface of the first object 302 and the depth sensor 308. For example, the first object 302 may be a chair that may include a vertical backrest portion, armrest portions, and a horizontal seat portion. The depth sensor 308 may capture a first image 402A and a corresponding first depth value from a first perspective such that the horizontal seat portion is closer in distance to the depth sensor 308 than the vertical backrest portion. Thus, the first depth value may indicate a distance value from each of a first plurality of points on the vertical backrest portion, armrest portions, and horizontal seat portion of the chair to the depth sensor 308.

[0043] At 404, a third image 404A may be generated. According to an embodiment, the processor 202 may be configured to generate the third image 404A corresponding to the first depth value associated with the first object 302. The processor 202 may receive the first depth value in a first text format from the depth sensor 308. For example, the first text format may be, but is not limited to, a semicolon separated value (".SKV") file format. The processor 202 may further convert the first depth value in the first text format to a second text format. For example, the second text format may be, but is not limited to, a comma separated value (".CSV") file format. The processor 202 may further generate a third image 404A corresponding to the first depth value based on the second text format. In some embodiments, the third image 404A may be generated based on the first text format of the first depth value without conversion to the second text format. The third image 404A may be in a different format, such as (but not limited to) Portable Network Graphics (PNG) format, Joint Committee on Photographic Experts (jpeg) format, Graphics Interchange Format (GIF) format, bitmap image file format, or Tagged Image File Format (TIFF).

[0044] According to an embodiment, the generated third image 404A may include different colors (i.e., RGB) representing different first depth values ​​associated with the first plurality of points on the surface of the first object 302. In an exemplary embodiment, the points on the surface of the first object 302 that are considered to be closest to the depth sensor 308 may include the darkest shade of blue that represents the corresponding pixel in the third image 404A, and the points that are considered to be furthest away may include the lightest shade of blue or a different color (such as red) that represents the corresponding pixel in the third image 404A. This color, such as blue, may change in a gradient (e.g., decrease in intensity) in the generated third image 404A as the distance between the depth sensor 308 and the first plurality of points on the surface of the first object 302 increases. It is noted that the different shades of blue that represent different depth values ​​in the third image 404A are merely exemplary. The third image 404A may include different colors or other various color tones indicative of the first depth value of the first object captured by the depth sensor 308 without departing from the scope of this disclosure.

[0045] In an embodiment, a background (such as the first scene 110 of FIG. 1) associated with the first object 302 captured by the depth sensor 308 may include another color, such as red, in the third image 404A. Thus, the third image 404A may indicate a first depth information (or a first depth value) of the first object 302 captured by the depth sensor 308. In other words, the third image 404A may further indicate depth information associated with the first scene 110 in which the first object 302 may be present. Thus, the disclosed system 102 may represent the depth information of the object (i.e., an object existing in the real world) captured by the depth sensor 308 as an image (such as the third image).

[0046] At 406, the first image 402A may be filtered. According to an embodiment, the processor 202 may be configured to filter the first image 402A of the first object 302 of the one or more first objects 112 captured by the depth sensor 308. The filtering may be performed to clean the first image 402A, such as to remove any redundant or unnecessary information related to the first object 302 in the first image 402A. For example, the background of the first image 402A may be blurred. In another example, the first object 302 may be zoomed in to obtain a particular view or portion of the first object 302, and other portions of the first image 402A may be filtered out accordingly. The processor 202 or the 3D graphics engine 104 may utilize the filtered first image 402A to generate graphic information, for example, as described at 412.

[0047] At 408, the tracking information may be received. According to an embodiment, the processor 202 may be configured to receive tracking information from the handheld device 306 associated with the depth sensor 308 based on the movement of the handheld device 306 and the depth sensor 308 in 3D space. The user 304 may move the handheld device 306 and the depth sensor 308 in 3D space (i.e., around the first object 302 shown in FIG. 3) to capture a first image 402A and a first depth value associated with the first object 302. For example, the depth sensor 308 may be moved closer or farther in the 3D space to capture the first object 302. Additionally, the depth sensor 308 may be moved in an upward or downward direction relative to the first object 302 to capture a first set of images (not shown). The handheld device 306 connected to or integrated with the depth sensor 308 may track the movement of the depth sensor 308 in 3D space to generate the tracking information.

[0048] The tracking information may include position information (XYZ information) or orientation information (rotation degree in different XY directions) of the depth sensor 308 in 3D space. For example, the position information may include the exact location of the depth sensor 308 in 3D space described in a Cartesian coordinate system (such as XYZ coordinates). The orientation information may include the rotation angle of the depth sensor 308 relative to a reference axis in the 3D space. For example, the depth sensor 308 may be at an angle of 0 degrees relative to a reference axis in the 3D space while capturing the first image 402A. In another example, the depth sensor 308 may be at an angle of 30 degrees relative to a reference axis in the 3D space while capturing the first image 402A.

[0049] At 410, one or more user inputs may be received. According to an embodiment, the processor 202 may be further configured to receive one or more user inputs from the user 304 to modify the first image 402A captured from the first viewpoint. In an exemplary scenario, the generated first image 402A may be displayed on a display device 310 associated with the system 102 or the 3D graphics engine 104. The displayed first image 402A may be modified based on one or more user inputs received from the user 304. For example, the user 304 may rotate the first object 302 in the first image 402A by a certain angle via the first I / O device 114 to obtain an image of the first object 302 from a different viewpoint other than the first viewpoint. The one or more user inputs may be received from the user 304 via the first I / O device 114 of the system 102. In another example, the displayed first image 402A of the first object 302 can be moved from one position to another on the display device 310 to obtain another image of the first object 302 from a viewpoint different from the first viewpoint from which the first image 402A was originally captured by the depth sensor 308.

[0050] Graphical information may be generated at 412. According to an embodiment, the processor 202 may be configured to generate graphical information corresponding to a first image 402A of a first object 302 (of the one or more first objects 112) based on the received tracking information. In some embodiments, the graphical information may be generated (i.e., filtered at 406) from the first image 402A based on one or more user inputs received from a user 304 via the first I / O device 114. The graphical information may include or correspond to the first object 302 from one or more second perspectives, which may be different from the first perspective.

[0051] According to an embodiment, the system 102 may further include a 3D graphics engine 104 that may be configured to generate graphical information corresponding to a first object 302 of the one or more first objects 112 from one or more second viewpoints based on the received tracking information and / or one or more user inputs received from the first I / O device 114. The generated graphical information may be displayed on a display device 310 associated with the system 102. Thus, the 3D graphics engine 104 may enable generation of graphical information of the first object 302 from different viewpoints that are not directly captured by the depth sensor 308 in the actual environment (as shown in FIG. 3 ).

[0052] According to an embodiment, the graphical information may include, but is not limited to, one or more second images, point cloud data, voxel sets, three-dimensional (3D) meshes, video information, LAS (LASer) format data, or proprietary format data. The one or more second images may correspond to an image that may include color information of the first object 302 from one or more second viewpoints. For example, the graphical information may include first graphical information 412A, second graphical information 412B, third graphical information 412C, and fourth graphical information 412D. The first graphical information 412A may correspond to an image of one or more second images of the first object 302 from a second viewpoint that may be different from the first viewpoint. For example, the first graphical information 412A may show a horizontally flipped version of the first image 402A of the first object 302. The second graphics information 412B may correspond to an image of the one or more second images from a third viewpoint (i.e., a third viewpoint of the one or more second viewpoints) different from the first viewpoint and the second viewpoint. For example, as shown in FIG. 4A, the second graphics information 412B may show a rotated and resized version of the first image 402A of the first object 302. Thus, the 3D graphics engine 104 of the system 102 may generate multiple images (such as one or more second images) corresponding to the first object 302 based on the received tracking information and / or one or more user inputs received from the user 304 via the first I / O device 114. For example, the 3D graphics engine 104 may receive a user input on the display device 310 to change the viewpoint (e.g., rotate, resize, translate, or move) of the first image 402A (i.e., an image captured from one viewpoint by the depth sensor 308) to generate the one or more second images.In another example, the 3D graphics engine 104 may change the viewpoint based on tracking information (i.e., position information and / or orientation information) of the handheld device 306 based on movement of the handheld device 306 in 3D space around the first object 302 and generate one or more second images of the first object 302 (i.e., first graphic information 412A showing a side view, etc.).

[0053] In an embodiment, the third graphical information 412C may correspond to point cloud data related to the first object 302. Each point of the point cloud data may represent a point on the surface of the first object 302. The fourth graphical information 412D may correspond to a 3D mesh related to the first object 302. The 3D mesh of the first object 302 may represent a 3D model of the first object 302 generated by the 3D graphics engine 104 of the system 102. Thus, as shown by way of example in FIG. 4A, the generated graphical information may include a representation of the first object 302 from one or more second viewpoints different from the first viewpoint at which the first image 402A was originally captured by the depth sensor 308 in the real environment. The generated graphical information may represent synthetic data (or virtual data) related to the first object 302 generated by the system 102. In some embodiments, the processor 202 may further control the 3D graphics engine 104 to render the first scene 110 (such as a real-world surrounding environment) of the first object 302. The real-world surrounding environment of the first object 302 may be captured by a depth sensor 308 and a handheld device 306 (e.g., a virtual reality device). The 3D graphics engine 104 may further generate the graphic information based on a placement of the first image 402A of the first object 302 in a different scene (such as a virtual surrounding environment).

[0054] At 414, one or more second depth values ​​may be calculated. According to an embodiment, the processor 202 may be configured to calculate one or more second depth values ​​associated with a first object 302 of the one or more first objects 112 based on the generated (at 412) graphical information from the one or more second viewpoints. In some embodiments, each of the one or more second depth values ​​may indicate depth information between the depth sensor 308 and a second plurality of points on the surface of the first object 302 captured from the one or more second viewpoints in the generated one or more second images (or graphical information). The one or more second depth values ​​may relate to a distance between each of the second plurality of points on the surface of the first object 302 in the graphical information and the depth sensor 308.

[0055] According to an embodiment, the processor 202 may be configured to receive at least one of first position information or first orientation information of the depth sensor 308 in the tracking information from the handheld device 306 at a first time instance. The first position information or first orientation information of the depth sensor 308 may correspond to an initial or reference position and / or orientation (e.g., with respect to a Cartesian coordinate system) of the depth sensor 308 in 3D space at the first time instance. For example, the first time instance may relate to a time instance at which a first image 402A and a first depth value of the first object 302 are captured from the depth sensor 308 (i.e., the time instance as described in 402 and FIG. 3). Thus, the first position information and / or first orientation information may indicate a reference position and / or orientation of the handheld device 306 or the depth sensor 308 at the time of capture of the first image 402A of the first object 302.

[0056] The processor 202 may be further configured to receive at least one of second position information or second orientation information in the tracking information from the handheld device 306 at a second time instance. The second time instance may be a time instance of movement of the handheld device 306 and the depth sensor 308 in 3D space or a time instance of receiving a user input via the first I / O device 114 to modify (i.e., rotate, resize, translate or move) the first object 302 displayed on the display device 310. Thus, the processor 202 may identify a change in position / orientation of the handheld device 306 at the second time instance and / or identify a change in user input that modifies the displayed object.

[0057] According to an embodiment, the processor 202 or the 3D graphics engine 104 may be further configured to calculate one or more second depth values ​​associated with a first object 302 of the one or more first objects 112 based on the first position information or first orientation information at a first time instance (i.e., a reference position / orientation) and the second position information or second orientation information at a second time instance. Thus, the one or more second depth values ​​may be calculated based on a change in position and / or orientation of the depth sensor 308 in 3D space from a reference position / orientation of the depth sensor 308 (i.e., a change in the tracked position and / or orientation at the first time instance at which the first depth value of the first object 302 was captured (at 402)). Thus, the change in tracking information may be synchronized to generate graphics information and second depth values ​​from one or more second viewpoints different from the first viewpoint. The generation of the graphic information (i.e., a second image) and second depth values ​​from a different viewpoint (e.g., a side view or a rear view) by the 3D graphics engine 104 can be referred to as capturing the first object 302 from a virtual sensor different from the depth sensor 308. Detection by the handheld device 306 of changes in the positional and / or orientation information of the depth sensor 308 in 3D space can accurately provide distances (i.e., second depth values) between the virtual sensor and each of the second plurality of points on the surface of the first object 302 as viewed from one or more second viewpoints.

[0058] In another embodiment, the processor 202 can receive one or more user inputs related to the first image 402A displayed on the display device 310 to generate the graphic information (i.e., as described in 412). The processor 202 can further calculate one or more second depth values ​​based on the first depth values ​​calculated during the capture of the first image 402A and further based on modifications made to the first image 402A as one or more user inputs. For example, the processor or 3D graphics engine 104 can utilize a user input provided to rotate the displayed first image 402A of the first object 302 on the display device 310 to calculate a second depth value based on the original first depth value and the rotation input (i.e., degrees).

[0059] According to an embodiment, the processor 202 may be further configured to extract background information from the first image 402A captured by the depth sensor 308. The background information may include information related to an environment surrounding the first object 302 (such as the first scene 110). The background information of the first scene 110 may include, but is not limited to, a background object or a background scene (including multiple objects). The processor 202 may provide the background information along with the first image 402A to the 3D graphics engine 104 of the system 102 for calculating the second depth value.

[0060] The processor 202 or the 3D graphics engine 104 may calculate one or more second depth values ​​associated with the first object 302 based on the background information. In some situations where the depth sensor 308 moves (i.e., tracked by the tracking information) or the first object 302 moves (i.e., based on the user input), the background information may not change. If the background of such a first object 302 does not move, the processor 202 may calculate a second depth value based on the first depth value and the changes in the tracking information and / or the user input. On the other hand, if there is any change in the background information, the processor 202 may determine the distance between the first object 302 and the background object or scene. In some embodiments, the processor 202 or the 3D graphics engine 104 may further receive one or more user inputs from the user 304 that change the position of the first object 302 within the background displayed on the display device 310, which may cause a change in the background or a change in the distance between the first object 302 and the background object or scene. The processor 202 may further calculate one or more second depth values ​​associated with the first object 302 based on the received tracking information, user input to modify the displayed first object 302, and / or background information.

[0061] At 416, one or more fourth images may be generated. According to an embodiment, the processor 202 may be configured to generate one or more fourth images corresponding to the calculated one or more second depth values ​​associated with the first object 302 of the one or more first objects 112. The generation of the one or more fourth images from the one or more second depth values ​​may be similar to the generation of the third image 404A from the first depth values ​​as described in 404 of FIG. 4A, for example. In some embodiments, the generated one or more fourth images may include different colors (i.e., RGB) representing different depth values ​​of the one or more second depth values ​​associated with the second plurality of points on the surface of the first object 302. Like the third image 404A, the one or more fourth images may be in a different format, such as (but not limited to) Portable Network Graphics (PNG) format, Joint Committee on Photographic Experts (jpeg) format, Graphics Interchange Format (GIF) format, bitmap image file format, or Tagged Image File Format (TIFF).

[0062] As shown in FIG. 4B, the one or more fourth images may be a set of images 416A-416D (e.g., image 416A, image 416B, image 416C, and image 416D). Each of the set of images 416A-416D may include different colors representing different depth values ​​(i.e., the second depth values ​​calculated in 414). For example, within the generated one or more fourth images, a color, such as a shade of blue, may change (e.g., decrease) in a gradient manner as the distance between the virtual sensor associated with the 3D graphics engine 104 and the second plurality of points on the surface of the first object 302 increases. As shown in FIG. 4A-4B, the image 416A may indicate a depth value corresponding to the first graphic information 412A, and the image 416B may indicate a depth value corresponding to the second graphic information 412B. Similarly, image 416C can indicate depth values ​​corresponding to third graphical information 412C, and image 416D can indicate depth values ​​corresponding to fourth graphical information 412D. In one embodiment, the set of images 416A-416D can correspond to virtual images or virtual data generated by the disclosed system 102 (including the 3D graphics engine 104) from various viewpoints for a first image 402A (i.e., real data) captured by the depth sensor 308 from one viewpoint in a real environment.

[0063] At 418, a training dataset 420 may be generated. According to an embodiment, the processor 202 may be configured to generate the training dataset 116 for training the neural network model 204A. The training dataset 420 may include a first combination of a first image 402A (i.e., an image captured by the depth sensor 308 at 402) and a third image 404A (i.e., an image generated from the first depth value at 404). The training dataset 420 may further include a second combination of one or more second images (i.e., an image generated by the 3D graphics engine 104 at 412) corresponding to the graphic information and one or more fourth images (i.e., an image generated at 416) corresponding to the one or more second depth values. FIG. 4B illustrates the training dataset 420 generated by the disclosed system 102.

[0064] The training data set 420 may include a first image set 420A, which may be a first combination of a first image 402A and a third image 404A. The first image set 420A may be a paired image set on which the neural network model 204A may be trained. For example, the neural network model 204A may be trained based on a relationship or mapping between the first image 402A (i.e., an image captured by the depth sensor 308 in the real environment) and the third image 404A generated for the first depth value (i.e., a depth value captured by the depth sensor 308 during the capture of the first image 402A). In an embodiment, the neural network model 204A may be trained based on a relationship or mapping between the first image 402A (i.e., an image captured by the depth sensor 308 in the real environment) and the first depth value (i.e., a depth value captured by the depth sensor 308 during the capture of the first image 402A). The neural network model 204A can be trained to predict a first depth value when the first image 402A or a new image having similar characteristics to the first image 402A is provided as an input to the trained neural network model 204A.

[0065] The training data set 420 may further include a second image set 420B, a third image set 420C, a fourth image set 420D, and a fifth image set 420E (as shown in FIG. 4B) as a second combination. The second image set 420B may include the first graphic information 412A and an image 416A capable of indicating a depth value corresponding to the first graphic information 412A. The third image set 420C may include the second graphic information 412B and an image 416B capable of indicating a depth value corresponding to the second graphic information 412B. The fourth image set 420D may include the third graphic information 412C and an image 416C capable of indicating a depth value corresponding to the third graphic information 412C. The fifth image set 420E may include the fourth graphic information 412D and an image 416D capable of indicating a depth value corresponding to the fourth graphic information 412D. In some embodiments, the second image set 420B, the third image set 420C, the fourth image set 420D, and the fifth image set 420E may correspond to respective combinations of one or more second images corresponding to the graphic information generated by the 3D graphics engine 104 of the system 102 and one or more fourth images corresponding to the one or more second depth values. In some embodiments, the second image set 420B, the third image set 420C, the fourth image set 420D, and the fifth image set 420E may be referred to as unpaired image sets such that the system 102 may synthetically or virtually generate each of the one or more second images (or graphic information) based on real data (i.e., the first image 402A captured by the depth sensor 308 in the real environment). Such unpaired images (i.e., virtual data) generated by the disclosed system 102 may closely mimic the real data since they are generated of the same object viewed from different viewpoints. Thus, for each first image captured by the depth sensor 308 of a different object from one viewpoint, the disclosed system 102 can generate, for example, multiple second images (or graphical information) of the same object from different viewpoints.The system 102 may further include a first image and corresponding first depth value (i.e., third image 404A) of a different object, and may include a plurality of second images and corresponding second depth values ​​(i.e., fourth image) of the object from different viewpoints as a large-scale training data set for training the neural network model 204A. The trained neural network model 204A may be utilized to predict (without requiring a depth sensor) the depth value of an image (such as a computer-generated image (CGI)) input to the trained neural network model 204A. Thus, the trained neural network model 204A may be utilized to provide or predict depth information based on the input image, rather than relying on a depth sensor in real time. The numerous second images (i.e., graphic information) provided by the disclosed system 102 (including the 3D graphics engine 104) may provide a large-scale training data set that enhances the prediction accuracy of the trained neural network model 204A. Furthermore, since it may be unnecessary to place depth sensors 308 (or similar expensive depth sensors) at different locations or viewpoints to capture a large-scale depth value dataset in a real environment, it may be inexpensive to generate graphical information from various viewpoints (i.e., the various viewpoints described in 412) to increase the training dataset.

[0066] In an exemplary scenario, the neural network model 204A may be provided with a training data set 420. Each of the first image 402A, the first graphic information 412A, the second graphic information 412B, the third graphic information 412C, and the fourth graphic information 412D of the training data set 420 may be provided to the neural network model 204A to be considered as an input to the neural network model 204A. Each of the third image 404A, the image 416A, the image 416B, the image 416C, and the image 416D of the training data set 420 (i.e., all images show corresponding depth values ​​of a particular object) may be provided to be considered as a corresponding output (i.e., an output for each respective input image) for training the neural network model 204A. The neural network model 204A may be trained to establish a relationship or mapping between each input image and an output image (i.e., an image associated with a depth value). In some embodiments, the neural network model 204A can be trained based on a training data set that includes a first combination of the first image 312 and the first depth value, and can further include a second combination of one or more second images (corresponding to graphic information) and one or more second depth values ​​of a different object. The trained neural network model 204A can automatically predict a depth value (i.e., one of the first depth value or the second depth value) of an input image (i.e., an input image similar to one of the first image 312 or the second image from which the neural network model 204A was trained) provided to the trained neural network model 204A.

[0067] Although diagram 400 is shown as discrete operations such as 402, 404, 406, 408, 410, 412, 414, 416, and 418, in some embodiments such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated depending on the particular implementation without departing from the essence of the disclosed embodiments.

[0068] FIG 5 is a flow chart illustrating an example method for generating a training dataset for depth measurement, according to an embodiment of the present disclosure. FIG 5 is described with reference to elements of FIG 1, FIG 2, FIG 3, FIG 4A, and FIG 4B. FIG 5 illustrates a flow chart 500. The example method of flow chart 500 may be performed by any computer system, such as, for example, system 102 of FIG 1 or processor 202 of FIG 2. The example method of flow chart 500 may start at 502 and proceed to 504.

[0069] At 504, the depth sensor 106 can be controlled to capture a first image 312 of one or more first objects 112 and a first depth value associated with the one or more first objects 112 from a first perspective. According to an embodiment, the processor 202 can be configured to control the depth sensor 106 to capture a first image of one or more first objects 112 and a first depth value associated with the one or more first objects 112 from a first perspective. The first image 312 can include color information (i.e., RGB) of the one or more first objects 112. Controlling the depth sensor 106 to capture the first image 312 of one or more first objects 112 and a first depth value associated with the one or more first objects 112 (from a first perspective) is described, for example, in 402 of FIGS. 4A-4B.

[0070] At 506, a third image 404A may be generated corresponding to the first depth value associated with the one or more first objects 112. According to an embodiment, the processor 202 may be configured to generate the third image 404A corresponding to the first depth value, for example as described at 404 in FIGS.

[0071] At 508, tracking information may be received from the handheld device 108 associated with the depth sensor 106 based on movement of the handheld device 108 and the depth sensor 106 in 3D space. According to an embodiment, the processor 202 may be configured to receive tracking information from the handheld device 108 associated with the depth sensor 106 based on movement of the handheld device 108 and the depth sensor 106 in 3D space. The tracking information may indicate at least one of positional information or orientation information of the depth sensor 106 in 3D space. Receiving the tracking information is described, for example, at 408 in FIGS. 4A-4B. Control may proceed to 512.

[0072] At 510, one or more user inputs may be received to modify the first image captured from the first perspective. According to an embodiment, the processor 202 may be configured to receive, via the first I / O device 114, one or more user inputs to modify the first image 402A, for example as described at 410 in FIGS. 4A-4B.

[0073] At 512, graphical information corresponding to the one or more first objects 112 may be generated based on the received tracking information and / or the received one or more user inputs. According to an embodiment, the processor 202 may be configured to generate graphical information corresponding to the one or more first objects 112 based on the received tracking information and / or the received one or more user inputs, for example, as described in 412 of FIGS. 4A-4B.

[0074] At 514, one or more second depth values ​​associated with the one or more first objects 112 may be calculated based on the graphical information generated from the one or more second perspectives. According to an embodiment, the processor 202 may be configured to calculate one or more second depth values ​​associated with the one or more first objects 112 based on the graphical information generated from the one or more second perspectives. Calculation of the one or more second depth values ​​is described, for example, at 414 in FIGS. 4A-4B.

[0075] At 516, one or more fourth images may be generated corresponding to the calculated one or more second depth values ​​(i.e., second depth values ​​associated with the one or more first objects 112). According to an embodiment, the processor 202 may be configured to generate one or more fourth images corresponding to the calculated one or more second depth values ​​associated with the one or more first objects 112. Generation of the one or more fourth images is described, for example, at 416 in FIGS. 4A-4B.

[0076] At 518, a training data set 116 may be generated. According to an embodiment, the processor 202 may be configured to generate a training data set 116 for the neural network model 204A that may include a first combination of the first image 312 and the third image 404A corresponding to the first depth value, and may include a second combination of one or more second images (corresponding to the graphical information) and one or more fourth images corresponding to the one or more second depth values. Generation of the training data set 116 is described, for example, in FIG. 4A-4B at 418. Control may proceed to an end.

[0077] Although flowchart 500 depicts discrete operations, such as 504, 506, 508, 510, 512, 514, 516, and 518, the disclosure is not so limited and, thus, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0078] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions stored thereon executable by a machine and / or computer (e.g., system 102). The instructions may cause the machine and / or computer (e.g., system 102) to perform operations including controlling a depth sensor (e.g., depth sensor 106) to capture a first image (e.g., first image 312) of one or more first objects (e.g., one or more first objects 112) from a first perspective and a first depth value associated with the one or more first objects 112. The first image 312 may include color information of the one or more first objects 112. The operations may further include receiving tracking information from a handheld device (e.g., handheld device 108) associated with the depth sensor 106 based on a movement of the handheld device 108 and the depth sensor 106 in three-dimensional (3D) space. The tracking information may indicate at least one of position information or orientation information of the depth sensor 106 in 3D space. The operations may further include generating graphical information corresponding to the one or more first objects 112 based on the received tracking information. The graphical information may include the one or more first objects 112 from one or more second viewpoints different from the first viewpoint. The operations may further include calculating one or more second depth values ​​associated with the one or more first objects 112 based on the graphical information generated from the one or more second viewpoints. The operations may further include generating a training data set (e.g., training data set 116) for a neural network model (e.g., neural network model 204A) that may include a first combination of the first image 312 and the first depth value, and a second combination of the one or more second images and the one or more second depth values ​​corresponding to the graphical information.

[0079] An exemplary aspect of the disclosure may include a system (such as system 102) that may include a processor (such as processor 202). The processor 202 may be configured to control a depth sensor (such as depth sensor 106) to capture a first image (such as first image 312) of one or more first objects (such as one or more first objects 112) and a first depth value associated with the one or more first objects 112 from a first perspective. The first image 312 may include color information of the one or more first objects 112. The processor 202 may be further configured to receive tracking information from a handheld device (such as handheld device 108) associated with the depth sensor 106 based on a movement of the handheld device 108 and the depth sensor 106 in three-dimensional (3D) space. The tracking information may indicate at least one of position information or orientation information of the depth sensor 106 in the 3D space. The processor 202 may be further configured to generate, based on the received tracking information, graphical information corresponding to the one or more first objects 112. The graphical information may include the one or more first objects 112 from one or more second viewpoints different from the first viewpoint. The processor 202 may be further configured to calculate one or more second depth values ​​associated with the one or more first objects 112 based on the graphical information generated from the one or more second viewpoints. The processor 202 may be further configured to generate a training data set (e.g., training data set 116) for the neural network model (e.g., neural network model 204A) that may include a first combination of the first image 312 and the first depth value and may include a second combination of one or more second images and one or more second depth values ​​corresponding to the graphical information.

[0080] According to one embodiment, the graphical information may correspond to at least one of one or more second images, point cloud data, a voxel set, a three-dimensional (3D) mesh, video information, LAS (LASer) format data, or dedicated format data.

[0081] According to an embodiment, the processor 202 may be further configured to receive the first depth value in the first text format from the depth sensor 106. The processor 202 may further convert the first depth value in the first text format to a second text format. The processor 202 may generate a third image 404A corresponding to the first depth value based on the second text format.

[0082] According to an embodiment, the processor 202 may be further configured to generate a third image 404A corresponding to the first depth value associated with each of the one or more first objects 112. The processor 202 may further generate one or more fourth images corresponding to the calculated one or more second depth values ​​associated with the one or more first objects 112. The processor 202 may further generate a training data set 116 for the neural network model 204A that includes a first combination of the first image 312 and the third image 404A corresponding to the first depth value, and may include a second combination of one or more second images corresponding to the graphical information and one or more fourth images corresponding to the one or more second depth values.

[0083] According to one embodiment, each of the generated third image 404A and one or more fourth images may include different colors representing different depth values ​​of the respective first depth values ​​and one or more second depth values ​​associated with each of the one or more first objects 112.

[0084] According to an embodiment, the first depth value may indicate depth information between the depth sensor 106 and a first plurality of points on a surface of the one or more first objects 112 captured from a first viewpoint. Additionally, each of the one or more second depth values ​​may indicate depth information between the depth sensor 106 and a second plurality of points on a surface of the one or more first objects 112 captured from one or more second viewpoints in the generated one or more second images.

[0085] According to an embodiment, the depth sensor 106 may be one of a Time of Flight (ToF) sensor, a Light Detection and Ranging (LiDAR) sensor, a stereo image sensor, or a structured light sensor.

[0086] According to an embodiment, the processor 202 may be further configured to receive at least one of first position information or first orientation information of the depth sensor 106 in the tracking information from the handheld device 108 at a first time instance. The processor 202 may further receive at least one of second position information or second orientation information of the depth sensor 106 in the tracking information from the handheld device 108 at a second time instance. The processor 202 may further calculate one or more second depth values ​​associated with the one or more first objects 112 based on the at least one of the first position information or first orientation information included in the tracking information received at the first time instance and the at least one of the second position information or second orientation information included in the tracking information received at the second time instance.

[0087] According to an embodiment, the processor 202 may be further configured to extract background information from the first image 312 captured by the depth sensor 106. The processor 202 may further calculate one or more second depth values ​​associated with the one or more first objects 112 based on the received tracking information and the extracted background information.

[0088] According to an embodiment, the system 102 may further include a 3D graphics engine 104. The 3D graphics engine 104 may be configured to generate graphical information corresponding to the one or more first objects 112 from one or more second viewpoints based on the received tracking information.

[0089] According to an embodiment, the processor 202 may be configured to receive one or more user inputs to modify the first image 312 captured from the first viewpoint. The processor 202 may further generate graphical information corresponding to the one or more first objects 112 from one or more second viewpoints based on the received one or more user inputs.

[0090] According to an embodiment, the processor 202 may be further configured to filter the first images 312 of the one or more first objects 112 captured by the depth sensor 106. The processor 202 may further generate graphical information corresponding to the one or more first objects 112 based on the filtered first images 312 of the one or more first objects 112 and the received tracking information.

[0091] The present disclosure can be realized in hardware or in a combination of hardware and software. The present disclosure can be realized in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be realized in hardware, including parts of an integrated circuit that also perform other functions.

[0092] The present disclosure may also be embodied in a computer program product, which includes all features enabling the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to execute a particular function, either directly, or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0093] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims.

Claims

1. controlling a depth sensor to capture a first image of the one or more first objects from a first viewpoint, the first image including color information of the one or more first objects, and one or more first depth values ​​associated with the one or more first objects; receiving tracking information from a handheld device associated with the depth sensor based on movement of the handheld device and the depth sensor in three-dimensional (3D) space, the tracking information indicating at least one of positional information or orientation information of the depth sensor in the 3D space; generating, based on the received tracking information, graphical information corresponding to the one or more first objects, the one or more first objects being displayed from one or more second viewpoints different from the first viewpoint, where the one or more second viewpoints are included in the tracking information; calculating one or more second depth values ​​associated with the one or more first objects based on the graphical information generated from the one or more second viewpoints; generating a training data set for the neural network model, the training data set including a first combination of the first image and the one or more first depth values ​​and a second combination of one or more second images and the one or more second depth values ​​corresponding to the graphical information; a processor configured to: the first image in the first combination and the one or more second images in the second combination are inputs in the training data set, and the one or more first depth values ​​in the first combination and the one or more second depth values ​​in the second combination are ground truths in the training data set; each of the one or more first depth values ​​indicating depth information between the depth sensor and a first plurality of points on a surface of the one or more first objects captured from the first viewpoint; and each of the one or more second depth values ​​indicating depth information between the depth sensor and a second plurality of points on the surface of the one or more first objects captured from the one or more second viewpoints in the generated graphical information; The system further comprises a three-dimensional (3D) graphics engine configured to generate graphical information corresponding to the one or more first objects from the one or more second viewpoints based on the received tracking information.

2. the generated graphical information corresponds to at least one of point cloud data, a voxel set, a three-dimensional (3D) mesh, or LAS (LASer) format data; The system of claim 1 .

3. The processor, receiving the one or more first depth values ​​in a first text format from the depth sensor; converting the one or more first depth values ​​in the first text format to a second text format; generating a third image corresponding to the one or more first depth values ​​based on the second text format; The system of claim 1 further configured to:

4. The processor, generating a third image corresponding to the one or more first depth values ​​associated with each of the one or more first objects; generating one or more fourth images corresponding to the calculated one or more second depth values ​​associated with the one or more first objects; generating the training data set for the neural network model, the training data set including a first combination of the first image and the third image corresponding to the one or more first depth values, and a second combination of the one or more second images corresponding to the graphical information and the one or more fourth images corresponding to the one or more second depth values; The system of claim 1 further configured to:

5. the generated third image includes different colors representing different depth values ​​of the one or more first depth values ​​associated with each of the one or more first objects; each of the generated one or more fourth images includes a different color representing a different depth value of the one or more second depth values ​​associated with each of the one or more first objects; The system of claim 4.

6. the depth sensor is one of a time-of-flight (ToF) sensor, a light detection and ranging (LiDAR) sensor, a stereo image sensor, or a structured light sensor; The system of claim 1 .

7. The processor, receiving at least one of first position information or first orientation information of the depth sensor in the tracking information from the handheld device at a first time instance corresponding to the first viewpoint; receiving at least one of second position information or second orientation information of the depth sensor in the tracking information from the handheld device at a second time instance corresponding to the one or more second viewpoints; calculating the one or more second depth values ​​associated with the one or more first objects based on the at least one of the first position information or the first orientation information included in the tracking information received at the first time instance and the at least one of the second position information or the second orientation information included in the tracking information received at the second time instance. The system of claim 1 configured to:

8. The processor, Extracting background information from the first image captured by the depth sensor; calculating a distance between the one or more first objects and a background based on the received tracking information and the extracted background information; The system of claim 1 further configured to:

9. The processor, receiving one or more user inputs to modify the first image captured from the first viewpoint; generating the graphical information corresponding to the one or more first objects from the one or more second viewpoints based on the received one or more user inputs. The system of claim 1 further configured to:

10. The processor, filtering the first image of the one or more first objects captured by the depth sensor; generating the graphical information corresponding to the one or more first objects based on the filtered first images of the one or more first objects and the received tracking information; The system of claim 1 further configured to:

11. In the system, controlling a depth sensor to capture a first image of the one or more first objects from a first viewpoint, the first image including color information of the one or more first objects, and one or more first depth values ​​associated with the one or more first objects; receiving tracking information from a handheld device associated with the depth sensor, the tracking information being indicative of at least one of positional information or orientation information of the depth sensor in three-dimensional (3D) space based on movement of the handheld device and the depth sensor in the 3D space; generating, based on the received tracking information, graphic information corresponding to the one or more first objects, the one or more first objects being viewed from one or more second viewpoints different from the first viewpoint, where the one or more second viewpoints are included in the tracking information; calculating one or more second depth values ​​associated with the one or more first objects based on the graphical information generated from the one or more second viewpoints; generating a training data set for a neural network model, the training data set including a first combination of the first image and the one or more first depth values ​​and a second combination of one or more second images and the one or more second depth values ​​corresponding to the graphical information; Including, the first image in the first combination and the one or more second images in the second combination are inputs in the training data set, and the one or more first depth values ​​in the first combination and the one or more second depth values ​​in the second combination are ground truths in the training data set; each of the one or more first depth values ​​indicating depth information between the depth sensor and a first plurality of points on a surface of the one or more first objects captured from the first viewpoint; and each of the one or more second depth values ​​indicating depth information between the depth sensor and a second plurality of points on the surface of the one or more first objects captured from the one or more second viewpoints in the generated graphical information; The method further comprises controlling a three-dimensional (3D) graphics engine to generate graphics information corresponding to the one or more first objects from the one or more second viewpoints based on the received tracking information.

12. receiving the one or more first depth values ​​in a first text format from the depth sensor; converting the one or more first depth values ​​in the first text format to a second text format; generating a third image corresponding to the one or more first depth values ​​based on the second text format; The method of claim 11 further comprising:

13. generating a third image corresponding to the one or more first depth values ​​associated with each of the one or more first objects; generating one or more fourth images corresponding to the calculated one or more second depth values ​​associated with the one or more first objects; generating the training data set for the neural network model, the training data set including a first combination of the first image and the third image corresponding to the one or more first depth values, and a second combination of the one or more second images corresponding to the graphical information and the one or more fourth images corresponding to the one or more second depth values; The method of claim 11 , comprising:

14. Receiving at least one of first position information or first orientation information of the depth sensor in the tracking information from the handheld device at a first time instance corresponding to the first viewpoint; receiving at least one of second position information or second orientation information of the depth sensor in the tracking information from the handheld device at a second time instance corresponding to the one or more second viewpoints; calculating the one or more second depth values ​​associated with the one or more first objects based on the at least one of the first position information or the first orientation information included in the tracking information received at the first time instance and the at least one of the second position information or the second orientation information included in the tracking information received at the second time instance; The method of claim 11 further comprising:

15. extracting background information from the first image captured by the depth sensor; calculating a distance between the one or more first objects and a background based on the received tracking information and the extracted background information; The method of claim 11 further comprising:

16. receiving one or more user inputs to modify the first image captured from the first viewpoint; generating the graphical information corresponding to the one or more first objects from the one or more second viewpoints based on the received one or more user inputs; The method of claim 11 further comprising:

17. A non-transitory computer-readable medium having computer-implemented instructions stored thereon, the computer-implemented instructions, when executed by a system, controlling a depth sensor to capture a first image of one or more first objects from a first viewpoint, the first image including color information of the one or more first objects and a first depth value associated with the one or more first objects; receiving tracking information from a handheld device associated with the depth sensor, the tracking information being indicative of at least one of positional information or orientation information of the depth sensor in three-dimensional (3D) space based on movement of the handheld device and the depth sensor in the 3D space; generating, based on the received tracking information, graphic information corresponding to the one or more first objects, the one or more first objects being viewed from one or more second viewpoints different from the first viewpoint, where the one or more second viewpoints are included in the tracking information; calculating one or more second depth values ​​associated with the one or more first objects based on the graphical information generated from the one or more second viewpoints; generating a training data set for a neural network model, the training data set including a first combination of the first image and the one or more first depth values ​​and a second combination of one or more second images and the one or more second depth values ​​corresponding to the graphical information; Including, the first image in the first combination and the one or more second images in the second combination are inputs in the training data set, and the one or more first depth values ​​in the first combination and the one or more second depth values ​​in the second combination are ground truths in the training data set; each of the one or more first depth values ​​indicating depth information between the depth sensor and a first plurality of points on a surface of the one or more first objects captured from the first viewpoint; and each of the one or more second depth values ​​indicating depth information between the depth sensor and a second plurality of points on the surface of the one or more first objects captured from the one or more second viewpoints in the generated graphical information; 11. The computer-readable medium of claim 10, further comprising: controlling a three-dimensional (3D) graphics engine to generate graphics information corresponding to the one or more first objects from the one or more second viewpoints based on the received tracking information.

Images