System and method for three-dimensional modeling based on object tracking

Abstract

Imaging systems and techniques are described. An imaging system determines a movement path of an indicative object relative to a target object based on image data of a scene received from an image sensor. And representing the target object in the image data from a first perspective. The imaging system identifies a contour of at least a portion of the target object based on the movement path of the pointing object and the first viewing angle. The imaging system generates a three-dimensional grid based on the contour of at least a portion of the target object. The imaging system may generate the grid in an additive or subtraction manner by adding a volume based on the contour to a previous grid or by subtracting the volume from the previous grid, respectively. The imaging system may generate a texture of the grid, the texture based on the target object in the image data.

Description

Technical Field

[0001] The present application relates to image capture and processing. More specifically, the present application relates to systems and methods for using image data from an image sensor to track the movement of an indicative object (such as a user's hand) relative to a detected outline of a target object and to generate a three-dimensional mesh based on the movement and / or the detected outline. Background Art

[0002] Many devices include one or more cameras. For example, a smartphone or tablet includes a front-facing camera for capturing selfie images and a rear-facing camera for capturing scene images, such as landscapes or other scenes of interest to the device user. A camera may capture images using an image sensor of the camera, which may include an array of photodetectors. Some devices may analyze image data captured by an image sensor to detect objects within the image data. Summary of the invention

[0003] In some examples, systems and techniques for image processing are described. An imaging system determines a movement path of an indicating object (e.g., a hand, finger, pointer, etc.) relative to a target object based on image data of a scene received from an image sensor. The target object is represented in the image data from a first perspective. The imaging system identifies a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective. The imaging system generates a three-dimensional mesh based on the contour of at least a portion of the target object. The imaging system may generate the mesh in an additive manner by adding a volume based on the contour to a previous mesh. The imaging system may generate the mesh in a subtractive manner by removing a volume based on the contour from a previous instance mesh. The imaging system may generate a texture of the mesh, the texture being based on the target object as represented in the image data.

[0004] According to at least one example, a method for image-based modeling is provided. The method includes: determining a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identifying a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective; and generating a three-dimensional mesh based on the contour of at least a portion of the target object.

[0005] In another example, an apparatus for image-based modeling is provided, the apparatus comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory. The at least one processor is configured to: determine a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identify a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective; and generate a three-dimensional mesh based on the contour of at least a portion of the target object.

[0006] In another example, a non-transitory computer-readable medium is provided, on which instructions are stored, which when executed by one or more processors cause the one or more processors to perform the following operations: determine a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identify an outline of at least a portion of the target object based on the movement path of the indicating object and the first perspective; and generate a three-dimensional mesh based on the outline of at least a portion of the target object.

[0007] In another example, an apparatus for image-based modeling is provided. The apparatus includes: a component for determining a movement path of an indicative object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; a component for identifying a contour of at least a portion of the target object based on the movement path of the indicative object and the first perspective; and a component for generating a three-dimensional mesh based on the contour of at least a portion of the target object.

[0008] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0013] Figure 2 is a block diagram illustrating an example architecture of a sensor data processing system that performs object tracking and three-dimensional modeling processes according to some examples;

[0014] Figure 3A is a perspective view illustrating a head mounted display (HMD) for use as part of an imaging system according to some examples;

[0015] Figure 3B is an example based on some examples Figure 3A A perspective view of a head mounted display (HMD) being worn by a user;

[0016] Figure 4Ais a perspective view illustrating a front surface of a mobile phone including a front-facing camera and usable as part of an imaging system according to some examples;

[0017] Figure 4B is a perspective view illustrating a rear surface of a mobile phone including a rear camera and usable as part of an imaging system according to some examples;

[0018] Figure 5 is a conceptual diagram illustrating a modeling process for generating a three-dimensional model of a target object (a tank) based on tracking a movement path of an indicating object (a user's finger) relative to the target object according to some examples;

[0019] Figure 6 is a conceptual diagram illustrating a first modeling process for generating a three-dimensional model of a target object (a person) in a subtractive manner based on tracking a movement path of an indicating object (a user's finger) relative to the target object (a person) according to some examples;

[0020] Figure 7 is a conceptual diagram illustrating a second modeling process for generating a three-dimensional model of a target object (person) in a subtractive manner based on tracking a movement path of an indicating object (a user's finger) relative to the target object (person) according to some examples;

[0021] Figure 8 is a conceptual diagram illustrating a modeling process for generating a three-dimensional model of a target object based on tracking the movement of the target object (a ball) and / or the movement of an indicating object (a user's hand) relative to the target object according to some examples;

[0022] Fig. 9 is a block diagram illustrating an example of a neural network that may be used for object tracking and three-dimensional modeling operations according to some examples;

[0023] Fig.10 is a flowchart illustrating a process of object tracking-based 3D modeling according to some examples; and

[0024] Fig.11 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0025] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects may be implemented without these specific details. Each drawing and description is not intended to be restrictive.

[0026] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0027] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera may be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop (aperture number), shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor used to capture one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, levels, curves or colors. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0028] A device including a camera can analyze image data captured by an image sensor to detect, identify, classify and / or track objects within the image data. For example, by detecting and / or identifying an object in multiple video frames of a video, the device can track the movement of the object over time.

[0029] In some examples, systems and techniques for image processing are described. An imaging system determines a movement path of an indicating object (e.g., a hand, finger, pointer, etc.) relative to a target object based on image data of a scene received from an image sensor. The target object is represented in the image data from a first perspective. The imaging system identifies a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective. The imaging system generates a three-dimensional mesh based on the contour of at least a portion of the target object. The imaging system may generate the mesh in an additive manner by adding a volume based on the contour to a previous mesh. The imaging system may generate the mesh in a subtractive manner by removing a volume based on the contour from a previous instance mesh. The imaging system may generate a texture of the mesh, the texture being based on the target object as represented in the image data.

[0030] The sensor processing and 3D modeling systems and techniques described herein provide many technical improvements over existing sensor processing and 3D modeling systems. For example, the sensor processing and 3D modeling systems and techniques described herein provide a fast and efficient process by which a 3D mesh and / or 3D model can be generated based on at least a portion (e.g., at least one edge, curvature, or shape) of a real-world object, and the 3D mesh and / or 3D model does not require a specialized multi-camera 3D scanning frame system. The sensor processing and 3D modeling systems and techniques described herein are flexible, allowing 3D modeling to be performed based on, for example, a combination of a first portion of a real-world object, a second portion of a second real-world object, and the like. The sensor processing and 3D modeling systems and techniques described herein improve the accuracy of 3D modeling because the edges (e.g., curvature and / or shape) of the target object are used as a template to simplify and guide scanning and modeling.

[0031] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). The image capture and processing system 100 may capture individual images (or photographs) and / or may capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene in an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For example, the scene 110 may be a scene of one or both of the user's eyes and / or at least a portion of the user's face.

[0032] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0033] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0034] The exposure control mechanism 125A of the control mechanism 120 may obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or number of apertures), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0035] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, the focusing lens may be the lens 115) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0036] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0037] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or an analog signal amplified by an analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0038] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more processors of any other type of processor 1110 discussed with respect to the computing system 1100. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), a memory, a connectivity component (e.g., Bluetooth™, a global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0039] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store the image frames and / or processed images in a random access memory (RAM) 140 and / or 1120, a read-only memory (ROM) 145 and / or 1125, a cache, a memory unit, another storage device, or some combination thereof.

[0040] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1135, any other input device 1145, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0041] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0042] like Figure 1 As shown, the vertical dotted line will Figure 1 The image capture and processing system 100 is divided into two parts, representing an image capture device 105A and an image processing device 105B. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some components illustrated in the image capture device 105A (such as the ISP 154 and / or the host processor 152) may be included in the image capture device 105A.

[0043] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone or cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 1102.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing device.

[0044] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0045] Figure 21 is a block diagram illustrating an example architecture of a sensor data processing system 200 that performs object tracking and three-dimensional modeling processes. The sensor data processing system 200 may include at least one of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the HMD 310, the mobile phone 410, the neural network 900, the sensor data processing system that performs the process 1000, the computing system 1100, the processor 1110, or a combination thereof. In some examples, the sensor data processing system 200 may include, for example, one or more laptop computers, phones, tablet computers, mobile phones, video game consoles, vehicle computers, desktop computers, wearable devices, televisions, media centers, extended reality (XR) systems, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality (MR) systems, head mounted display (HMD) devices, other types of computing devices discussed herein, or a combination thereof.

[0046] The sensor data processing system 200 includes at least one sensor 205 that captures sensor data 210. Examples of sensor 205 include image capture and processing system 100, image capture device 105A, image processing device 105B, image sensor 130, an image sensor of any of cameras 330A-330D, an image sensor of any of cameras 430A-430D, an image sensor that captures an image used in input layer 910 of NN 900, an image sensor of imaging process 1000, an image sensor of input device 1145, or a combination thereof. In some examples, sensor data 210 includes raw image data, image data, pixel data, image frames, raw video data, video data, video frames, or a combination thereof.

[0047] In some examples, at least one sensor 205 may be or include an image sensor having an array of photodetectors. The photodetectors of the image sensor may be sensitive to one or more subsets of an electromagnetic (EM) frequency domain, such as a radio EM frequency domain, a microwave EM frequency domain, an infrared (IR) EM frequency domain, a visible light (VL) EM frequency domain, an ultraviolet (UV) EM frequency domain, an X-ray EM frequency domain, a gamma ray EM frequency domain, a subset of any of these items, or a combination thereof. In some examples, different photodetectors of the image sensor may be configured to be sensitive to different EM frequency domains and / or different color channels. In some examples, the sensor 205 captures a plurality of image frames configured to be arranged in a sequence to form a video, and the sensor data 210 includes at least a subset of the video (e.g., at least one video frame of the video).

[0048] In some examples, the sensor 205 may be oriented toward the user (e.g., may face the user), and thus may capture (e.g., depict or otherwise represent) at least a portion of sensor data (e.g., image data) of the user. In some examples, the sensor 205 may be oriented away from the user (e.g., may face away from the user) and / or toward the environment in which the user is located, and thus may capture (e.g., depict or otherwise represent) at least a portion of sensor data (e.g., image data) of the environment. In some examples, the sensor data 210 captured by the sensor 205 is oriented away from the user and / or toward the user. In some examples, the sensor data 210 captured by the sensor 205 may have a field of view (FoV) that includes, is included in, overlaps with, and / or otherwise corresponds to the FoV of the user's eyes.

[0049] In some examples, sensor 205 may be or may include other types of sensors besides image sensors. In some examples, in addition to sensor 205, sensor data processing system 200 may also include one or more other sensors, such as one or more other image sensors and / or one or more other types of sensors. For example, sensor types may include image sensors, cameras, microphones, heart rate monitors, blood oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, one or more sensors 205 include at least one input device 1145 of computing system 1100. In some implementations, one or more of these additional sensors can supplement or refine the sensor readings from the sensor 205. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor can be used to identify the pose (e.g., position and / or orientation) and / or motion and / or acceleration of the sensor data processing system 200 and / or the user in the environment, which the sensor data processing system 200 can use to reduce motion blur, rotation blur, or a combination thereof.

[0050] The graphic representation of sensor 205 is shown in Figure 2, and illustrates a sensor (e.g., an image sensor) capturing a representation (e.g., an image) of a scene having a target object and an indicating object. Specifically, the target object is a person wearing a tuxedo, and the indicating object is a hand (e.g., the hand of a user using the sensor data processing system 200 or the hand of another user). A graphic representing sensor data 210 is shown in Figure 2 , and illustrates a representation (eg, image) of a scene having a target object and an indicating object captured by sensor 205.

[0051] The sensor data processing system 200 includes a sensor data processor 215. The sensor data processor 215 may include an image processing device 105B, an image processor 150, a host processor 152, an ISP 154, a target object tracker 220, an indicator object tracker 225, a three-dimensional (3D) mesh generator 240, a texture generator 250, one or more output devices 260, one or more trained machine learning (ML) models 280, a feedback engine 285, an HMD 310, a mobile phone 410, a neural network 900, a sensor data processing system that performs the process 1000, a computing system 1100, a processor 1110, or a combination thereof. Both the target object tracker 220 and the indicator object tracker 225 of the sensor data processor 215 analyze the sensor data 210 to detect, identify, classify, and / or track one or more features, objects, and / or environments. As used herein, tracking may refer to tracking an observed path of an element (e.g., a feature, an object, and / or an aspect of an environment), predicting the path of the element (e.g., based on the observed and / or tracked path), or a combination thereof. For example, features may include edges, corners, spots, other points of interest, descriptors, or a combination thereof. For example, objects may include faces, people, hands, fingers, pointers, vehicles, animals, plants, structures, any target object discussed herein, any indicating object discussed herein, any object pointed to by an indicating object as discussed herein, or a combination thereof.

[0052] In some examples, the target object tracker 220 and the indicating object tracker 225 of the sensor data processor 215 may input the sensor data 210 into the trained ML model 280 in order to detect, identify, classify and / or track elements (e.g., features, objects, and / or aspects of the environment). The sensor data processing system 200 may use the training data to pre-train the trained ML model 280 to detect, identify, classify and / or track one or more features, objects and / or environments. The training data may include sensor data (e.g., sensor data of the same type or similar type as the sensor data 210) and a predetermined indication of an area of ​​the sensor data that includes certain features, objects and / or aspects of the environment. In some examples, the sensor data in the training data may include video data, and the training data may include a tracked path of an element (e.g., certain features, objects and / or aspects of the environment) and / or a predicted path of an element (e.g., certain features, objects and / or aspects of the environment). In some examples, sensor data processing system 200 may further iteratively train trained ML model 280 based on a level of success in detecting, identifying, classifying, and / or tracking elements based on sensor data 210 .

[0053] Specifically, the target object tracker 220 is configured to detect, identify, classify and / or track the target object in the sensor data 210. The indicating object tracker 225 is configured to detect, identify, classify and / or track the indicating object in the sensor data 210. The indicating object is an object in the environment that is used to indicate a position, area, volume and / or path within the environment (for example, if the indicating object moves along the path), such as by pointing, touching and / or grabbing (for example, holding to) the position, area, volume and / or path. For example, the indicating object can be a hand, a finger, a fingertip, a pointer, a rod, a point of light of a laser pointer, a leg, a foot, a toe, an appendage, another type of indicating object described herein, a part thereof, or a combination thereof. The target object can be any type of object, such as a face, a person, a vehicle, an animal, a plant, a statue, a toy, a structure, any of the object types discussed above with respect to the indicating object, any other object type described herein, a part thereof, or a combination thereof.

[0054] By detecting, identifying, classifying, and / or tracking a target object within sensor data 210, target object tracker 220 generates target object contour 230. Target object contour 230 may identify contours of at least a portion of one or more edges, sides, corners, and / or surfaces of a target object. The contours in target object contour 230 may depend on the viewing angle of sensor data 210 relative to the target object. For example, if the target object is a person's head and the viewing angle is a side profile, the contours in target object contour 230 may include the curvature and / or shape of the person's nose, lips, and / or eyes. On the other hand, if the viewing angle is facing the front of the person's head, the resulting contours in target object contour 230 may alternatively include the curvature and / or shape of the person's ears.

[0055] A diagram showing the target object tracker 220 is shown in FIG. Figure 2 , and illustrates a bounding box (with dashed lines) around a representation of a target object (a person in a tuxedo) in exemplary sensor data in a graphic representing sensor data 210. The graphic representing the outline of the target object is shown in FIG. Figure 2 , and illustrates the outer edges of a representation of a target object (a person in a tuxedo) in sensor data 210 , as represented from the perspective of the target object in the exemplary sensor data in the graph representing sensor data 210 .

[0056] By detecting, identifying, classifying, and / or tracking an indicator object within sensor data 210, indicator object tracker 225 generates indicator object movement path 235. Indicator object movement path 235 may identify a path (e.g., an observed and / or predicted path) of an indicator object tracked by indicator object tracker 225. Specifically, when an indicator object moves and / or stays within a threshold distance of at least a portion of one or more edges, sides, corners, and / or surfaces of a target object, indicator object tracker 225 may generate indicator object movement path 235 to identify the path of the indicator object. In some examples, the indicator object may outline a portion of one or more edges, sides, corners, and / or surfaces of a target object. As discussed above, the edges, sides, corners, and / or surfaces of a target object that the indicator object may outline may depend on the viewing angle of sensor data 210 relative to the target object.

[0057] The graphic representation of the object tracker 225 is shown in FIG. Figure 2 , and illustrates a bounding box (with dashed lines) around a representation of an indicative object (hand) in exemplary sensor data in a graphic representing sensor data 210. A graphic representing a movement path 235 of an indicative object is shown in FIG. Figure 2, and illustrates an indicating object movement path 235 (using a dotted line) of an indicating object (hand) that is outlined around a portion of the exterior of a target object (a person in a tuxedo) in a graph representing the sensor data 210 .

[0058] In some examples, to generate the target object contour 230, the sensor data processor 215 (e.g., the target object tracker 220) may input the sensor data 210 into the trained ML model 280. In some examples, the sensor data processor 215 (e.g., the target object tracker 220) may also input an indication of an object serving as a target object into the trained ML model 280. The indication may be based on the indicating object tracking performed by the indicating object tracker 225, because the indicating object points to and / or outlines the edge of the target object, thereby identifying which object in the environment is the target object. In some examples, the trained ML model 280 may determine the identity of the target object based on the sensor data 210. The sensor data processing system 200 may use the training data to pre-train the trained ML model 280 to generate the target object contour 230 based on the sensor data 210 and / or the indication of the identity of the target object. The training data may include sensor data (e.g., sensor data of the same type or similar type as the sensor data 210), the identity of the target object in the sensor data, and a predetermined contour of the target object. In some examples, sensor data processing system 200 may further iteratively train trained ML model 280 based on a degree of success in generating target object contour 230 based on sensor data 210 .

[0059] In some examples, to generate the indicating object movement path 235, the sensor data processor 215 (e.g., indicating object tracker 225) may input the sensor data 210 into the trained ML model 280. In some examples, the sensor data processor 215 (e.g., indicating object tracker 225) may also input an indication of an object serving as an indicating object into the trained ML model 280. The indication may be based on the classification of the object type in the sensor data 210 to identify an indicating object of any type among a hand, a finger, a fingertip, a pointer, an appendage, or other types of indicating objects discussed above. The indication may also be about identifying an object that moves in the scene more than a threshold amount (e.g., a threshold distance) as an indicating object. In some examples, the trained ML model 280 may determine the identity of the indicating object based on the sensor data 210. The sensor data processing system 200 may use the training data to pre-train the trained ML model 280 to generate the indicating object movement path 235 based on the sensor data 210 and / or the indication of the identity of the indicating object. The training data may include sensor data (e.g., sensor data of the same type or a similar type as sensor data 210), an identity of an indicator object in the sensor data, and a predetermined movement path of the indicator object. In some examples, sensor data processing system 200 may further iteratively train trained ML model 280 based on a degree of success in generating indicator object movement path 235 based on sensor data 210.

[0060] The sensor data processing system 200 includes a three-dimensional (3D) mesh generator 240. The 3D mesh generator 240 may include an image processing device 105B, an image processor 150, a host processor 152, an ISP 154, a sensor data processor 215, a target object tracker 220, an indicating object tracker 225, a texture generator 250, one or more output devices 260, one or more trained machine learning (ML) models 280, a feedback engine 285, an HMD 310, a mobile phone 410, a neural network 900, a sensor data processing system that performs the process 1000, a computing system 1100, a processor 1110, or a combination thereof. The 3D mesh generator 240 receives the target object contour 230 and the indicating object movement path 235 from the sensor data processor 215. The 3D mesh generator 240 generates a three-dimensional (3D) mesh 245 based on the target object contour 230 and / or the indicating object movement path 235. The 3D mesh generator 240 may generate the 3D mesh 245 by projecting a 3D volume based on the target object contour 230 and / or the indicating object movement path 235. For example, the contour of an edge of the 3D mesh 245 (and / or the 3D volume) (e.g., the curvature and / or the shape of the edge) may match the contour of at least a portion of an edge of the target object contour 230 and / or the indicating object movement path 235. For example, the 3D mesh generator 240 may generate the 3D mesh 245 based on a specified portion of the target object contour 230 outlined in the indicating object movement path 235. In some examples, 3D mesh generator 240 may generate a portion of 3D mesh 245 based only on target object outline 230 and not on indicated object movement path 235 (e.g., in the case where there is an error or discontinuity in indicated object movement path 235). In some examples, 3D mesh generator 240 may generate a portion of 3D mesh 245 based only on indicated object movement path 235 and not on target object outline 230 (e.g., in the case where there is an error or discontinuity in target object outline 230).

[0061] In some examples, 3D mesh generator 240 may generate 3D mesh 245 in an additive manner. For example, 3D mesh generator 240 may generate a 3D volume by projecting the 3D volume based on target object outline 230 and / or indicated object movement path 235. 3D mesh generator 240 may generate 3D mesh 245 by including the 3D volume in 3D mesh 245. For example, if 3D mesh generator 240 has not yet generated any 3D mesh 245, the 3D volume may be 3D mesh 245. If 3D mesh generator 240 already has a predetermined 3D mesh, 3D mesh generator 240 may add the 3D volume to the predetermined 3D mesh to generate 3D mesh 245, so that the 3D volume becomes part of 3D mesh 245. Examples of 3D mesh 245 being generated by 3D mesh generator 240 in an additive manner are shown in FIG. Figure 5 and Figure 8 Example in.

[0062] In some examples, 3D mesh generator 240 may generate 3D mesh 245 in a subtractive manner. For example, 3D mesh generator 240 may generate a 3D volume by projecting a 3D volume based on target object outline 230 and / or indicated object movement path 235. 3D mesh generator 240 may generate 3D mesh 245 by removing 3D volumes from a predetermined 3D mesh. After 3D mesh generator 240 removes the 3D volumes from the predetermined 3D mesh, the result may be 3D mesh 245. For example, in the illustrative example, the predetermined 3D mesh may be a 3D block. 3D mesh generator 240 may generate 3D mesh 245 in a subtractive manner by removing portions of the 3D block to generate 3D mesh 245. An example of 3D mesh 245 being generated by 3D mesh generator 240 in a subtractive manner is shown in FIG. Figure 6 to Figure 7 Example in.

[0063] In some examples, to generate a 3D mesh 245 (and / or the 3D volume discussed above) based on the target object contour 230 and / or the indicated object movement path 235, the 3D mesh generator 240 may input the target object contour 230 and / or the indicated object movement path 235 into the trained ML model 280. The sensor data processing system 200 may pre-train the trained ML model 280 using the training data to generate a 3D mesh 245 (and / or the 3D volume discussed above) based on the target object contour 230 and / or the indicated object movement path 235. The training data may include sensor data (e.g., sensor data of the same type or a similar type as sensor data 210), a target object outline of a target object represented in the sensor data (e.g., as in target object outline 230), an indicative object movement path of an indicative object represented in the sensor data (e.g., as in indicative object movement path 235), and / or a pre-generated 3D mesh based on the sensor data and / or the target outline and / or the indicative object movement path (e.g., as in 3D mesh 245 and / or 3D volume discussed above). In some examples, sensor data processing system 200 may further iteratively train trained ML model 280 based on a degree of success in generating 3D mesh 245 (and / or 3D volume discussed above) based on target object outline 230 and / or indicative object movement path 235.

[0064] A graphic representation of the 3D mesh generator 240 is shown in Figure 2 , and an arrow from a graphic representing a target object outline 230 and indicating an object movement path 235 to a graphic representing a 3D mesh 245 is illustrated. The graphic representing the 3D mesh 245 is a 3D shape whose outline is based on the target object outline 230 of the target object (e.g., a person wearing a tuxedo). The graphic representing the 3D mesh 245 is illustrated as being similar to the 3D meshes at stages 615 and 620 of the modeling process 600.

[0065] The sensor data processing system 200 includes a texture generator 250. The texture generator 250 may include the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, one or more output devices 260, one or more trained machine learning (ML) models 280, the feedback engine 285, the HMD 310, the mobile phone 410, the neural network 900, the sensor data processing system performing the process 1000, the computing system 1100, the processor 1110, or a combination thereof. The texture generator 250 generates a texture of the 3D mesh 245 based on the shape of the 3D mesh 245 and based on the appearance of the target object in the sensor data 210 (e.g., as tracked using the target object tracker 220). For example, if the target object is a person, the 3D mesh 245 may have a shape based on the edges of the person as represented in the sensor data 210, and the texture may apply colors (e.g., skin color, eye color, lip color, clothing color, accessory color, and / or colors of other parts of the target object) and / or patterns (e.g., skin texture, clothing texture, clothing pattern, hair texture) from the representation of the person to the 3D mesh 245. Once the texture generator 250 generates the texture for the 3D mesh 245, the texture generator 250 may also apply the texture to the 3D mesh 245 to generate a 3D model 255 with the texture applied to the 3D mesh 245.

[0066] In some examples, to generate a texture based on the 3D mesh 245 and the sensor data 210, the texture generator 250 may input the 3D mesh 245 and the sensor data 210 into the trained ML model 280. In some examples, the texture generator 250 may also input an indication of the location and / or area of ​​a representation of a target object in the sensor data 210 (e.g., as determined and / or tracked by the target object tracker 220) and / or an indication of the identity of the target object. The sensor data processing system 200 may use the training data to pre-train the trained ML model 280 to generate a texture based on the 3D mesh 245 and the sensor data 210. The training data may include a 3D mesh (e.g., as in the 3D mesh 245), sensor data (e.g., sensor data of the same type or a similar type as the sensor data 210), and a texture of the 3D mesh pre-generated based on the sensor data. In some examples, sensor data processing system 200 may further iteratively train trained ML model 280 based on a level of success in generating a texture based on 3D mesh 245 and sensor data 210 .

[0067] A graphic representation of the texture generator 250 is shown in Figure 2, and illustrates the texture of a tuxedo extracted from a graphic representing sensor data 210. A graphic representing a 3D model 255 is shown in Figure 2 , and illustrates a texture of a tuxedo applied to a graphic representing a 3D mesh 245 .

[0068] The sensor data processing system 200 includes one or more output devices 260. In some examples, the sensor data processing system 200 transmits the 3D mesh 245 and / or the 3D model 255 directly to the output device 260. In some examples, the sensor data processing system 200 transmits a rendered image 270 of the 3D model 255 to the one or more output devices 260 instead of or in addition to the 3D mesh 245 and / or the 3D model 255. In some examples, the sensor data processing system 200 includes an image renderer 265. The image renderer 265 may include the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, one or more output devices 260, one or more trained machine learning (ML) models 280, the feedback engine 285, the HMD 310, the mobile phone 410, the neural network 900, the sensor data processing system performing the process 1000, the computing system 1100, the processor 1110, or a combination thereof. The image renderer 265 generates the rendered image 270 by rendering the 3D mesh 245 and / or the 3D model 255. In some examples, the image renderer 265 generates the rendered image 270 by rendering the 3D mesh 245 and / or the 3D model 255 from a specified perspective.

[0069] A graphic representation of the image renderer 265 and the texture generator 250 is shown in Figure 2 , and illustrates the generation of a rendered image from a graphic representing a 3D model 255. A graphic representing a rendered image 270 is shown in FIG. Figure 2 , and illustrates a rendered image based on a graphic representing a 3D model 255 .

[0070] Sensor data processing system 200 includes output device 260. Output device 260 may include one or more visual output devices, such as a display or a connector therefor. Output device 260 may include one or more audio output devices, such as speakers, headphones, and / or connectors therefor. Output device 260 may include one or more of output device 1135 and / or communication interface 1140 of computing system 1100. In some examples, sensor data processing system 200 causes a display of output device 260 to display 3D mesh 245, 3D model 255, a texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof.

[0071] In some examples, the output device 260 includes one or more transceivers. The transceiver may include a wired transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include a wireless transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include one or more of the output device 1135 and / or the communication interface 1140 of the computing system 1100. In some examples, the sensor data processing system 200 causes the transceiver to transmit the 3D mesh 245, the 3D model 255, the texture of the 3D model 255, the rendered image 270, a portion or subset of any of these items, or a combination thereof to a recipient device. In some examples, the recipient device may include another sensor data processing system 200, an HMD 310, a mobile phone 410, a computing system 1100, or a combination thereof. In some examples, the recipient device may include a display, and the data transmitted from the transceiver of output device 260 to the recipient device may cause the display of the recipient device to display 3D mesh 245, 3D model 255, a texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof.

[0072] In some examples, the display of the output device 260 of the sensor data processing system 200 is used as an optical "see-through" display that allows light from the real-world environment (scene) around the sensor data processing system 200 to traverse (e.g., pass through) the display of the output device 260 to one or both eyes of the user. For example, the display of the output device 260 may be at least partially transparent, translucent, light-transmissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output device 260 includes a transparent, translucent and / or light-transmissive lens and a projector. The display of the output device 260 may include a projector that projects virtual content (e.g., 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof) onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector may project virtual content (e.g., 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof) onto the lens so that the virtual content appears overlaid on the user's view of the environment from the perspective of one or both of the user's eyes. In some examples, the projector may project the virtual content onto one or both retinas of one or both of the user's eyes, rather than onto the lens, which may be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display.

[0073] In some examples, the display of the output device 260 of the sensor data processing system 200 is a digital "transparent" display that allows a user of the sensor data processing system 200 and / or a recipient device to view a view of the environment by displaying the view of the environment on the display of the output device 260. For example, based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by the sensor 205 and / or other sensors described herein (e.g., sensor data 210), the view of the environment displayed on the digital transparent display can be a view of the real-world environment surrounding the sensor data processing system 200. The view of the environment displayed on the digital transparent display can be a virtual environment (e.g., as in VR), which in some cases can include elements based on the real-world environment (e.g., the boundaries of a room). The view of the environment displayed on the digital transparent display can be an augmented environment based on the real-world environment (e.g., as in AR). The view of the environment displayed on the digital transparent display can be a mixed environment based on the real-world environment (e.g., as in MR). The view of the environment displayed on the digital pass-through display may include virtual content (e.g., 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof) overlaid on or otherwise incorporated into the view of the environment.

[0074] exist Figure 2 Within, a graphic representation of output device 260 illustrates a display, speaker, and wireless transceiver that outputs a graphic representation of rendered image 270 .

[0075] The trained ML model 280 may include one or more neural networks (NNs) (e.g., neural network 900), one or more convolutional neural networks (CNNs), one or more trained time delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (cGANs), one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, one or more classifiers, one or more transformers, or a combination thereof. Figure 2, a graphic representing a trained ML model 280 illustrates a collection of circles connected to another collection of circles. Each of these circles may represent a node (e.g., node 916), a neuron, a perceptron, a layer, a portion thereof, or a combination thereof. The circles are arranged in columns. The white circles in the leftmost column represent input layers (e.g., input layer 910). The white circles in the rightmost column represent output layers (e.g., output layer 914). The two columns of shaded circles between the leftmost column of white circles and the rightmost column of white circles each represent a hidden layer (e.g., hidden layers 912A to 912N).

[0076] In some examples, the sensor data processing system 200 includes a feedback engine 285 of the sensor data processing system 200. The feedback engine 285 can detect feedback received from a user interface of the sensor data processing system 200. The feedback can include feedback about outputs of various subsystems of the sensor data processing system 200 (e.g., the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, the texture generator 250, the output device 260, the image renderer 265, and / or the trained ML model 280), such as target object detection / recognition / tracking, indicating object detection / recognition / tracking, target object contour 230, indicating object movement path 235, 3D mesh 245, 3D model 255, texture of the 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof. The feedback engine 285 may detect feedback received from another engine of the sensor data processing system 200 about one engine of the sensor data processing system 200, such as whether one engine decides to use data from other engines and / or whether the use of the data is successful. The feedback received by the feedback engine 285 may be positive feedback or negative feedback. For example, if one engine of the sensor data processing system 200 successfully uses data from another engine of the sensor data processing system 200, or if positive feedback from a user is received through a user interface, the feedback engine 285 may interpret this as positive feedback. If one engine of the sensor data processing system 200 rejects data from another engine of the sensor data processing system 200, or cannot successfully use data from the other engine, or if negative feedback from a user is received through a user interface, the feedback engine 285 may interpret this as negative feedback. In an illustrative example, the feedback engine 285 may detect whether the 3D mesh generator 240 is able to successfully generate the 3D mesh 245 based on the target object contour 230 and / or the indicated object movement path 235. If yes, then 3D mesh generator 240 effectively provides positive feedback to target object tracker 220 and / or indicating object tracker 225. If no, then 3D mesh generator 240 effectively provides negative feedback to target object tracker 220 and / or indicating object tracker 225.

[0077] Positive feedback may also be based on attributes of the user as detected in the sensor data 210 from the sensor 205, such as the user smiling, laughing, nodding, uttering an affirmative statement (e.g., "yes," "confirm," "ok," "next"), or otherwise reacting positively to the output of one of the engines described herein, or an indication thereof. Negative feedback may also be based on attributes of the user as detected in the sensor data from the sensor 205, such as the user frowning, crying, shaking his head (e.g., making a "no" gesture), uttering a negative statement (e.g., "no," "reject," "bad," "not this"), or otherwise reacting negatively to the output of one of the engines described herein, or an indication thereof.

[0078] In some examples, the feedback engine 285 provides the feedback as training data to the trained ML model 280 and / or one or more subsystems of the sensor data processing system 200 that can use the trained ML model 280 (e.g., the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, the texture generator 250, and / or the image renderer 265) to update the one or more trained ML models 280 of the sensor data processing system 200. For example, the feedback engine 285 may provide the feedback as training data to the ML system and / or the trained ML model 280 to update the training for the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, the texture generator 250, the image renderer 265, the trained ML model 280, or a combination thereof. Positive feedback may be used to strengthen and / or enhance weights associated with the output of the ML system and / or trained ML model 280, and / or weaken or remove weights other than the weights associated with the output of the ML system and / or trained ML model 280. Negative feedback may be used to weaken and / or remove weights associated with the output of the ML system and / or trained ML model 280, and / or strengthen and / or enhance weights other than the weights associated with the output of the ML system and / or trained ML model 280.

[0079] In some examples, certain elements of the sensor data processing system 200 (e.g., sensor 205, sensor data processor 215, target object tracker 220, indicating object tracker 225, 3D mesh generator 240, texture generator 250, output device 260, image renderer 265, trained ML model 280, feedback engine 285, or a combination thereof) include software elements, such as an instruction set corresponding to a program, which is executed on a processor (such as processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, sensor data processor 215, or a combination thereof). In some examples, one or more of these elements of the sensor data processing system 200 may include one or more hardware elements, such as a dedicated processor (e.g., processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, sensor data processor 215, or a combination thereof). In some examples, one or more of these elements of sensor data processing system 200 may include a combination of one or more software elements and one or more hardware elements.

[0080] Figure 3A300 is a perspective view illustrating a head mounted display (HMD) 310 used as part of a sensor data processing system 200. HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. HMD 310 may be an example of sensor data processing system 200. HMD 310 includes a first camera 330A and a second camera 330B along a front portion of HMD 310. First camera 330A and second camera 330B may be examples of sensors 205 of sensor data processing system 200. HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eyes when the user's eyes face display 340. Third camera 330C and fourth camera 330D may be examples of sensors 205 of sensor data processing system 200. In some examples, HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of sensors 205 of the sensor data processing system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0081] The HMD 310 may include one or more displays 340 visible to a user 320 (wearing the HMD 310 on the head of the user 320). The one or more displays 340 of the HMD 310 may be an example of one or more displays of the output device 260 of the sensor data processing system 200. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be a digital "pass-through" display or an optical "see-through" display.

[0082] The HMD 310 may include one or more earpieces 335, which may be used as speakers and / or headphones to output audio to one or both ears of a user of the HMD 310 and may be an example of an output device 260. Figure 3A and Figure 3B 310, one earpiece 335 is illustrated, but it should be understood that the HMD 310 may include two earpieces, one for each ear of the user (left ear and right ear). In some examples, the HMD 310 may also include one or more microphones (not shown). In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using the one or more microphones.

[0083] Figure 3B is an example Figure 3A310 is a perspective view 350 of a head mounted display (HMD) being worn by a user 320. The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320. The HMD 310 may capture images through a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images toward the eyes of the user 320 using a display 340. In some examples, the output image may include a 3D mesh 245, a 3D model 255, a texture of the 3D model 255, a rendered image 270, a portion or subset of any of these items, or a combination thereof. The output image may be based on the images captured by the first camera 330A and the second camera 330B (e.g., sensor data 210), for example, overlaid with virtual content (e.g., a 3D mesh 245, a 3D model 255, a texture of the 3D model 255, a rendered image 270, a portion or subset of any of these items, or a combination thereof). The output image may provide a stereoscopic view of the environment, in some cases with virtual content superimposed and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on an image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on an image captured by the second camera 330B. For example, the HMD 310 may provide superimposed virtual content in the display image superimposed on the images captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the display 340. In this way, the sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earpiece 335 of the HMD 310 is illustrated as being in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear of the user 320 (not shown).

[0084] Figure 4A 4 is a perspective view 400 illustrating a front surface of a mobile handset 410 that includes a front-facing camera and can be used as part of the sensor data processing system 200. The mobile handset 410 can be an example of the sensor data processing system 200. The mobile handset 410 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop computer, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0085] The front surface 420 of the mobile phone 410 includes a display 440. The front surface 420 of the mobile phone 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of sensors 205 of the sensor data processing system 200. The first camera 430A and the second camera 430B may face the user, including the user's eyes, while content (e.g., 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof) is displayed on the display 440. The display 440 may be an example of a display of the output device 260 of the sensor data processing system 200.

[0086] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be positioned in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras positioned between the display 440 and the rest of the mobile phone 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the planar surface of the front surface 420 of the mobile phone 410. The first camera 430A and the second camera 430B may be two cameras of one or more cameras of the mobile phone 410. In some examples, the front surface 420 of the mobile phone 410 may have only a single camera.

[0087] In some examples, the display 440 of the mobile phone 410 displays one or more output images to a user using the mobile phone 410. In some examples, the output images may include the 3D mesh 245, the 3D model 255, the texture of the 3D model 255, the rendered image 270, a portion or subset of any of these items, or a combination thereof. The output image may be based on images captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D (e.g., sensor data 210), for example, overlaid with virtual content (e.g., the 3D mesh 245, the 3D model 255, the texture of the 3D model 255, the rendered image 270, a portion or subset of any of these items, or a combination thereof).

[0088] In some examples, the front surface 420 of the mobile phone 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of sensors 205 of the sensor data processing system 200. In some examples, the front surface 420 of the mobile phone 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of sensors 205 of the sensor data processing system 200. In some cases, the front surface 420 of the mobile phone 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile phone 410 may be examples of displays of the output device 260 of the sensor data processing system 200. For example, the one or more displays 440 may include one or more touch screen displays.

[0089] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4A 4 shows one speaker 435A, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile phone 410, where these microphones are examples of sensors 205 of the sensor data processing system 200. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0090] Figure 4B 4 is a perspective view 450 illustrating a rear surface 460 of a mobile phone that includes a rear camera and can be used as part of a sensor data processing system 200. The mobile phone 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile phone 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D can be examples of a sensor 205 of the sensor data processing system 200. The third camera 430C and the fourth camera 430D face a direction perpendicular to the planar surface of the rear surface 460 of the mobile phone 410.

[0091] The third camera 430C and the fourth camera 430D may be two cameras of the one or more cameras of the mobile phone 410. In some examples, the rear surface 460 of the mobile phone 410 may have only a single camera. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the sensor 205 of the sensor data processing system 200. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the sensor 205 of the sensor data processing system 200. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0092] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4B 435B, it should be understood that the mobile phone 410 may include more than one speaker and / or other audio device. In some examples, the mobile phone 410 may also include one or more microphones (not shown). In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the rear surface 460 of the mobile phone 410, where these microphones are examples of sensors 205 of the sensor data processing system 200. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0093] The mobile phone 410 can use the display 440 on the front surface 420 as a transparent display. For example, the display 440 can display an output image, such as a 3D mesh 245, a 3D model 255, a texture of a 3D model 255, a rendered image 270, a portion or a subset of any of these items, or a combination thereof. The output image can be based on an image captured by the third camera 430C and / or the fourth camera 430D (e.g., sensor data 210), for example, overlaid with virtual content (e.g., a 3D mesh 245, a 3D model 255, a texture of a 3D model 255, a rendered image 270, a portion or a subset of any of these items, or a combination thereof). The first camera 430A and / or the second camera 430B can capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, the sensor data from the first camera 430A and / or the second camera 430B can capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0094] Figure 5 2 is a conceptual diagram illustrating a modeling process 500 for generating a three-dimensional model of a target object (a tank) based on tracking a movement path of an indicating object (a user's finger) relative to the target object. The modeling process 500 is an additive process and is illustrated in stages. At each stage, a real-world depiction of a scene (e.g., as captured in sensor data 210 from sensor 205) is illustrated under the heading "real world", and virtual data corresponding to what is happening in the real-world scene is additionally illustrated under the heading "virtual world". The virtual data may include, for example, at least a portion of a target object outline 230, at least a portion of an indicating object movement path 235, at least a portion of a 3D mesh 245, and / or at least a portion of a 3D model 255.

[0095] At all stages, the target object - the can - is illustrated in the real world environment. At stage 505, in the real world, the user points a finger at the edge of the target object (the can). The tip of the user's finger is the indicator of the modeling process 500. At stage 505, the fingertip is illustrated (using an arrow) as moving down the edge of the target object, thereby outlining the edge of the target object. At stage 505, in the virtual world, the path of the fingertip (e.g., the indicator object movement path 235 that outlines the target object outline 230) is illustrated as it begins to form. The perspective of the target object in the sensor data (real world) determines which edges are discernible for outlining, and can therefore affect the indicator object movement path 235 and / or target object outline 230 generated in the virtual world.

[0096] At stage 510 and stage 515, in the real world, the fingertip is illustrated (using arrows) as continuing to outline the edge of the target object. At stage 510 and stage 515, in the virtual world, the path of the fingertip is illustrated (e.g., indicating object movement path 235 outlining the target object outline 230). In some examples, indicating object movement path 235 can be aligned to the nearest portion of the target object outline 230 to compensate for shaking or undesired movement in the indicating object (e.g., in the fingertip). At stage 515, in the virtual world, all of the edges of the target object have been outlined, and a texture-free 3D mesh (e.g., 3D mesh 245) representing the target object has been generated (e.g., by 3D mesh generator 240).

[0097] At stage 520, in the real world, the fingertip is no longer instantiated, indicating that the fingertip is no longer in the scene and / or indicating that the generation of the indicated object movement path 235 is complete. At stage 520, in the virtual world, a texture is generated (e.g., by a texture generator 250) based on the depiction of the target object (tank) in the real world and applied to the 3D mesh to generate a 3D model 255 of the target object (tank).

[0098] Figure 5 The modeling process 500 illustrated in FIG. 5 is an additive process because the 3D volume generated based on the indicated object movement path 235 and / or the target object outline 230 (e.g., by the 3D mesh generator 240) becomes the 3D mesh 245 of the target object (tank), and ultimately becomes the 3D model 255 of the target object. In some examples, the user can generate additional 3D volumes to add to the 3D model. Figure 5 The 3D mesh 245 and / or 3D model 255 of the target object.

[0099] Figure 6 6 is a conceptual diagram illustrating a first modeling process 600 for generating a three-dimensional model of a target object (a person) in a subtractive manner based on tracking the movement path of an indicating object (a user's finger) relative to the target object. The modeling process 600 is a subtractive process and is illustrated in stages. At each stage, a real-world depiction of a scene (e.g., as captured in sensor data 210 from sensor 205) is illustrated under the heading "real world", and virtual data corresponding to what is happening in the real-world scene is additionally illustrated under the heading "virtual world". The virtual data may include, for example, at least a portion of a target object outline 230, at least a portion of an indicating object movement path 235, at least a portion of a 3D mesh 245, and / or at least a portion of a 3D model 255.

[0100] At all stages, a target object - a bust of a person - is instantiated in a real-world environment. At stage 605, in the real world, the target object (bust) is instantiated without any referents; and in the virtual world, a pre-generated 3D mesh of a 3D block (e.g., a cube) is instantiated. At stage 610, in the real world, a user points a finger at the edge of the target object (bust). The tip of the user's finger is the referent for modeling processes 600 and 700. At stage 610, the fingertip is illustrated (using a dotted line) as moving down the edge of the target object, outlining the edge of the target object. At stage 610, in the virtual world, a 3D volume whose shape is projected based on the path of the fingertip (e.g., the referent movement path 235 that outlines the target object outline 230) is instantiated as removed, subtracted, or cut out of the pre-generated 3D mesh (the 3D block from stage 605). The viewing angle of the target object in the sensor data (real world) determines the shape of the outlined edges and therefore affects the shape of the 3D volume removed from the pre-generated 3D mesh.

[0101] At stage 615, in the real world, the fingertip completes outlining the edge of the target object around the exterior of the target object. At stage 615, in the virtual world, a 3D volume whose shape is projected based on the path of the fingertip (e.g., the indicated object movement path 235 outlining the target object outline 230) is instantiated as being removed, subtracted, or cut out from the pre-generated 3D mesh (the 3D block from stage 605 and / or the intermediate 3D mesh of stage 610).

[0102] At stage 620, the perspective of the target object is rotated 90 degrees in the real world, for example, by rotating the target object itself, by moving the sensor 205 relative to the target object, or a combination thereof. At stage 620, in the virtual world, the 3D mesh remains in its state in stage 615. At stage 625, in the real world, the fingertip is illustrated (using a dashed line) as moving down the edge of the target object, thereby outlining the edge of the target object. Due to the change in perspective, the edge of the target object outlined at stage 625 is a different edge than the edge of the target object outlined at stages 610 and 615. At stage 625, in the virtual world, a 3D volume whose shape is projected based on the path of the fingertip (e.g., the indicated object movement path 235 outlining the target object outline 230) is illustrated as being removed, subtracted, or cut out from the pre-generated 3D mesh (the intermediate 3D mesh of stages 615 and 620).

[0103] At stage 630, in the real world, the fingertip completes outlining the edge of the target object around the exterior of the target object. At stage 630, in the virtual world, a 3D volume whose shape is projected based on the path of the fingertip (e.g., the indicated object movement path 235 outlining the target object outline 230) is instantiated as being removed, subtracted, or cut out from the pre-generated 3D mesh (the intermediate 3D mesh of stages 615, 620, and / or 625).

[0104] Figure 7 is a conceptual diagram illustrating a second modeling process 700 for generating a three-dimensional model of a target object (person) in a subtractive manner based on tracking a movement path of an indicating object (a user's finger) relative to the target object (person). Figure 7 The second modeling process 700 is Figure 6 The first modeling process 600 is continued.

[0105] At stage 705, the perspective of the target object is rotated 45 degrees in the real world, for example, by rotating the target object itself, by moving the sensor 205 relative to the target object, or a combination thereof. At stage 705, in the virtual world, the 3D mesh remains in its state in stage 630. At stage 710, in the real world, the fingertip is illustrated (using a dashed line) as moving down the edge of the target object, thereby outlining the edge of the target object. Due to the change in perspective, the edge of the target object outlined at stage 710 is a different edge than the edge of the target object outlined at stages 610, 615, 625, and 630. At stage 710, in the virtual world, a 3D volume whose shape is projected based on the path of the fingertip (e.g., the indicated object movement path 235 outlining the target object outline 230) is illustrated as being removed, subtracted, or cut out from the pre-generated 3D mesh (the intermediate 3D mesh of stages 615 and 620).

[0106] At stage 715, in the real world, the fingertip completes outlining the edge of the target object by connecting the end point of the fingertip's path to the starting point of the fingertip's path. At stage 715, in the virtual world, the 3D volume whose shape is projected based on the path of the fingertip (e.g., the indicated object movement path 235 that outlines the target object outline 230) is illustrated as being removed, subtracted, or cut out from the pre-generated 3D mesh (the intermediate 3D mesh of stages 630, 705, and / or 710). This refines the curvature and / or shape of the target object in the 3D mesh. At stage 720, in the real world, the perspective of the target object is slightly rotated, for example, by rotating the target object itself, by moving the sensor 205 relative to the target object, or a combination thereof. At stage 720, in the virtual world, the 3D mesh is completed and the refinement performed in stage 715 is completed.

[0107] In the real world, the perspective of the target object is again slightly rotated, e.g., by rotating the target object itself, by moving the sensor 205 relative to the target object, or a combination thereof, at stage 725. In the virtual world, a texture is generated (e.g., by a texture generator 250) based on the depiction of the target object (bust) in the real world, and applied to the 3D mesh to generate a 3D model 255 of the target object (bust) at stage 725.

[0108] Figure 6 to Figure 7 The modeling process 600 and 700 illustrated in the example is a subtractive process, because the 3D volume generated based on the indicated object movement path 235 and / or the target object contour 230 (e.g., by the 3D mesh generator 240) is removed from the previous 3D mesh, which starts with the block in the virtual world at stage 605 and further iterates. In some examples, Figure 6 to Figure 7 Some stages and / or portions of the modeling processes 600 and 700 illustrated in stages 710 and 715 may be additive stages and / or portions. For example, some of the refinements of the face of the target object (bust) in the 3D mesh illustrated in stages 715 and 720 may add some 3D volume to the face relative to the pre-generated 3D mesh (the intermediate 3D mesh of stages 710 and / or 715).

[0109] Figure 8 8 is a conceptual diagram illustrating a modeling process 800 for generating a three-dimensional model of a target object based on tracking the movement of a target object (a ball) and / or the movement of an indicating object (a user's hand) relative to the target object. The modeling process 800 is an additive process and is illustrated in stages. At each stage, a real-world depiction of a scene (e.g., as captured in sensor data 210 from sensor 205) is illustrated under the heading "real world", and virtual data corresponding to what is happening in the real-world scene is additionally illustrated under the heading "virtual world". The virtual data may include, for example, at least a portion of a target object outline 230, at least a portion of an indicating object movement path 235, at least a portion of a 3D mesh 245, and / or at least a portion of a 3D model 255.

[0110] At all stages, the target object - a ball - is illustrated in the real world environment. At stage 805, in the real world, the user holds the target object (ball) in the user's hand. The user's hand is the referent in the modeling process 800. At stage 805, the shape of the user's hand holds the shape of the ball, so that some edges of the ball are effectively outlined by the shape of the user's hand. At stage 805, in the virtual world, key points along the surface of the target object (ball) are illustrated as circles with solid outlines, and representations of some fingers of the user's hand are illustrated in dashed lines.

[0111] At stages 810, 815, 820, 825, and 830, in the real world, the perspective of the target object (ball) changes based on the movement of the user's hand as the user's hand holds and / or grasps the target object, based on the movement of the sensor 205, or a combination thereof. In stages 810, 815, 820, 825, and 830, in the virtual world, more and more key points along the surface of the ball are captured, and a 3D mesh of the ball (illustrated using a shaded area with a speckled pattern) gradually forms and gradually increases in size to eventually cover the entire ball at stage 830. The movement of the user's hand can be tracked by the pointing object tracker 225 as the pointing object movement path 235. The outline of the ball can be tracked by the target object tracker as the target object outline 230, and is slowly formed and evolved over the course of stages 805 to 830 based on the different perspectives of the ball and the user's hand.

[0112] At stages 835 and 840, in the real world, the hand is no longer instantiated, indicating that the hand is no longer in the scene and / or indicating that the generation of the indicated object movement path 235 is complete. At stage 835, in the virtual world, the 3D mesh of the target object (ball) is completed. At stage 840, in the virtual world, a texture is generated (e.g., by a texture generator 250) based on the depiction of the target object (ball) in the real world and applied to the 3D mesh to generate a 3D model 255 of the target object (ball).

[0113] Figure 8 The modeling process 800 illustrated in FIG. 8 is an additive process because the 3D volume generated based on the indicated object movement path 235 and / or the target object outline 230 (e.g., by the 3D mesh generator 240) becomes the 3D mesh 245 of the target object (the ball), and ultimately becomes the 3D model 255 of the target object. In some examples, the user can generate additional 3D volumes to add to the 3D model. Figure 8 The 3D mesh 245 and / or 3D model 255 of the target object.

[0114] Figure 8 The modeling process 800 illustrated in FIG. 8 is an additive process because the 3D volume generated based on the indicated object movement path 235 and / or the target object outline 230 (e.g., by the 3D mesh generator 240) becomes the 3D mesh 245 of the target object (the ball), and ultimately becomes the 3D model 255 of the target object. In some examples, the user can generate additional 3D volumes to add to the 3D model. Figure 8 The 3D mesh 245 and / or 3D model 255 of the target object.

[0115] In some examples, a particular trigger condition may be used to initiate a modeling process (such as modeling process 500, 600, 700, or 800). For example, in an illustrative example, the trigger condition may be a user's hands entering the field of view of sensor 205 (e.g., and thus appearing in sensor data 210). In a second illustrative example, the trigger condition may be an audio command spoken by the user, recorded by sensor 205, and recognized by sensor data processing system 200. In a third illustrative example, the trigger condition may be the pressing of a button, the flipping of a switch, or another change in a user interface detected at sensor data processing system 200 (e.g., via input device 1145).

[0116] In some examples, a modeling process (such as modeling process 500, 600, 700, or 800) may combine corresponding portions of different real-world objects into one 3D mesh 245 and / or 3D model 255 in the real world. For example, a 3D mesh 245 and / or 3D model 255 may be generated to include a first portion of a real-world object (e.g., Figure 5 a portion of a tank), a second portion of a second real-world object (e.g., Figure 6 to Figure 7 a bust of a bust), a third part of a third real-world object (e.g. Figure 8 all of which are combined according to an arrangement specified by a user, by the sensor data processing system 200 (e.g., based on how the shapes of the parts best fit together), by the NN 900, or a combination thereof.

[0117] Fig. 9 is a block diagram illustrating an example of a neural network (NN) 900 that can be used for object tracking and three-dimensional modeling operations. The neural network 900 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 900 may be an example of a trained ML model 280. The neural network 900 may be used by various subsystems of the sensor data processing system 200, such as the sensor data processor 215, the target object tracker 220, the indicating object tracker 225, the 3D mesh generator 240, the texture generator 250, the output device 260, and / or the image renderer 265.

[0118] The input layer 910 of the neural network 900 includes input data. The input data of the input layer 910 may include data representing pixels of one or more input image frames. In some examples, the input data of the input layer 910 includes data representing pixels of image data. Examples of the image data include images captured using the image capture and processing system 100, sensor data 210, images captured by one of the cameras 330A-330D, images captured by one of the cameras 430A-430D, Figures 5 to 8 1005 and / or an image captured using input device 1145, any other image data described herein, any other sensor data described herein, or a combination thereof. The input data in input layer 910 may also include other data, such as data corresponding to target object detection / recognition / tracking by target object tracker 220, indicated object detection / recognition / tracking by indicated object tracker 225, target object outline 230, indicated object movement path 235, 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof.

[0119] The image may include image data from an image sensor, including raw pixel data (including a single color per pixel based on, for example, a Bayer color filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). Neural network 900 includes multiple hidden layers 912, 912B to 912N. Hidden layers 912, 912B to 912N include "N" hidden layers, where "N" is an integer greater than or equal to 1. The multiple hidden layers can include as many layers as required for a given application. Neural network 900 also includes an output layer 914, which provides output resulting from the processing performed by hidden layers 912, 912B to 912N.

[0120] Output layer 914 may provide output data for operations performed using NN 900. For example, output layer 914 may provide output data such as data corresponding to target object detection / recognition / tracking by target object tracker 220, indicated object detection / recognition / tracking by indicated object tracker 225, target object outline 230, indicated object movement path 235, 3D mesh 245, 3D model 255, texture of 3D model 255, rendered image 270, a portion or subset of any of these items, or a combination thereof.

[0121] Neural network 900 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with these filters is shared between different layers, and each layer retains information as it processes it. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0122] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 910 may activate a set of nodes in the first hidden layer 912A. For example, as shown in the figure, each of the input nodes of the input layer 910 may be connected to each of the nodes of the first hidden layer 912A. The nodes of the hidden layer may transform the information by applying an activation function (e.g., a filter) to the information of each input node. Then, the information derived from the transformation may be passed to the nodes of the next hidden layer 912B and may activate these nodes, which may perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 912B may activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 912N may activate one or more nodes of the output layer 914, which provides a processed output image. In some cases, although a node in neural network 900 (eg, node 916 ) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0123] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 900. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that can be tuned (e.g., based on a training data set), thereby allowing neural network 900 to adapt to the input and be able to learn as more and more data is processed.

[0124] The neural network 900 is pre-trained to process features of data from the input layer 910 using different hidden layers 912 , 912B to 912N to provide an output through an output layer 914 .

[0125] Fig.101 is a flowchart illustrating a three-dimensional modeling process 1000 based on object tracking. Process 1000 may be performed by a sensor data processing system. In some examples, the sensor data processing system may include, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, a sensor data processing system 200, a neural network 900, a computing system 1100, a processor 1110, or a combination thereof. In some examples, the sensor data processing system includes a display. In some examples, the imaging system includes a transceiver.

[0126] At operation 1005, the sensor data processing system is configured to and may determine a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor. The target object is represented in the image data from a first perspective. In some aspects, the sensor data processing system may detect a change from a second perspective to the first perspective based on the image data and previous image data of the scene (e.g., as previously described with respect to FIG. 1 ). Figure 6 , Figure 7 and Figure 8 ). In such aspects, the target object may be represented in the previous image data from the second perspective. The previous image data may be received from the image sensor before the image data. In some cases, the sensor data processing system may track the hand in the image data to determine the movement path of the indicating object relative to the target object. In such cases, the indicating object may include at least a portion of the hand. In some examples, the target object is held by the hand. In some examples, the movement path of the indicating object is configured to rotate the target object.

[0127] Examples of the image sensor include image sensor 130, sensor 205, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor for capturing an image used as input data for input layer 910 of NN 900, input device 1145, another image sensor described herein, another sensor described herein, or a combination thereof. Examples of the raw image data include raw image data 210.

[0128] At operation 1010 , the sensor data processing system is configured to and may identify an outline of at least a portion of the target object based on the movement path of the pointing object and the first viewing angle.

[0129] At operation 1015, the sensor data processing system is configured to and may generate a three-dimensional mesh based on the outline of at least a portion of the target object. In some aspects, the sensor data processing system may remove a portion of a previous three-dimensional mesh to generate the three-dimensional mesh. In such aspects, the portion of the previous three-dimensional mesh may be based on the outline of at least a portion of the target object. Additionally or alternatively, in some examples, the sensor data processing system may combine a volume with a previous three-dimensional mesh (or a different previous three-dimensional mesh) to generate the three-dimensional mesh. In such examples, the volume combined with the previous three-dimensional mesh may be based on the outline of at least a portion of the target object. Additionally or alternatively, in some cases, the sensor data processing system may apply a texture to the three-dimensional mesh to generate a three-dimensional model. For example, the texture may be based on a representation of the target object in the image data. In some aspects, the texture may be generated by the texture generator 250 described above. In some cases, the sensor data processing system may generate the shape of the edge of the three-dimensional mesh to match the shape of the outline of at least a portion of the target object (e.g., as previously described with respect to Figure 6 described).

[0130] In some aspects, the sensor data processing system may determine the second movement path of the indicating object relative to the target object based on the second image data of the scene received from the image sensor. In some aspects, the target object is represented in the second image data from the first viewing angle. In such aspects, the sensor data processing system may identify the second contour of at least the second part of the target object based on the second movement path of the indicating object and the first viewing angle. In some cases, the target object is represented in the second image data from a second viewing angle. In such cases, the sensor data processing system may identify the second contour of at least the second part of the target object based on the second movement path of the indicating object and the second viewing angle. In any of these aspects or situations, the three-dimensional grid may also be generated based on the second contour of at least the second part of the target object. In some cases, the shape of the first edge of the three-dimensional grid is generated to match the shape of the contour of at least a part of the target object, and the shape of the second edge of the three-dimensional grid is generated to match the shape of the second contour of at least the second part of the target object.

[0131] In some examples, the processes described herein (e.g., Figure 1 , Figure 2 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 The corresponding process Fig.10Process 1000 and / or other processes described herein) may be performed by a computing device or apparatus. In some examples, the processes described herein may be performed by image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, sensor data processing system 200, neural network 900, a sensor data processing system performing process 1000, computing system 1100, processor 1110, or a combination thereof.

[0132] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR head-mounted device, an AR head-mounted device, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capability to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0133] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0134] The processes described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer executable instruction stored on one or more computer-readable storage media that performs the described operations when executed by one or more processors. Generally speaking, computer executable instructions include routines, programs, objects, components, and data structures that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0135] Additionally, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0136] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. Specifically, Fig.11 An example of a computing system 1100 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0137] In some aspects, computing system 1100 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality that the component is described for. In some aspects, each component may be a physical or virtual device.

[0138] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to the processor 1110, in close proximity to the processor, or integrated as part of the processor.

[0139] Processor 1110 may include any general purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, that are configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0140] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 may also include an output device 1135 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning connector, or a USB port. ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low energy (BLE) wireless signal transmission, IBEACON ®Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1102.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0141] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smart card chip, an EMV Chip, Subscriber Identity Module (SIM) card, Mini / Micro / Nano / Pico SIM card, another integrated circuit (IC) chip / card, Random Access Memory (RAM), Static RAM (SRAM), Dynamic RAM (DRAM), Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash EPROM (FLASHEPROM), Cache Memory (L1 / L2 / L3 / L4 / L5 / L#), Resistive Random Access Memory (RRAM / ReRAM), Phase Change Memory (PCM), Spin Transfer Torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0142] Storage device 1130 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1110, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 1110, connection 1105, output device 1135, etc.).

[0143] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, or network sending, etc.

[0144] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0145] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0146] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0147] The processes and methods according to the above examples can be implemented using stored computer executable instructions or computer executable instructions otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or a group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0148] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0149] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0150] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it should be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0151] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ”) and greater than or equal to (“ ”) symbol instead.

[0152] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0153] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0154] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0155] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0156] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, and a magnetic or optical data storage medium, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0157] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0158] Illustrative aspects of the present disclosure include:

[0159] Aspect 1. A device for image-based modeling, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: determine a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identify a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective; and generate a three-dimensional mesh based on the contour of at least the portion of the target object.

[0160] Aspect 2. An apparatus according to aspect 1, wherein the at least one processor is configured to: apply a texture to the three-dimensional mesh to generate a three-dimensional model, wherein the texture is based on a representation of the target object in the image data.

[0161] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the at least one processor is configured to: track a hand in the image data to determine the movement path of the indicating object relative to the target object, wherein the indicating object includes at least a portion of the hand.

[0162] Aspect 4. The apparatus according to aspect 3, wherein the target object is held by the hand, and wherein the movement path of the indicating object is configured to rotate the target object.

[0163] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the at least one processor is configured to: remove a portion of a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the portion of the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

[0164] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein the at least one processor is configured to: combine a volume with a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the volume combined with the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

[0165] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is configured to: detect a change from a second perspective to the first perspective based on the image data and previous image data of the scene, wherein the target object is represented in the previous image data from the second perspective, and wherein the previous image data is received from the image sensor before the image data.

[0166] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is configured to: generate a shape of an edge of the three-dimensional mesh to match a shape of the outline of at least the portion of the target object.

[0167] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to: determine a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from the first perspective; and identify a second contour of at least a second part of the target object based on the second movement path of the indicating object and the first perspective, wherein the three-dimensional mesh is also generated based on the second contour of at least the second part of the target object.

[0168] Aspect 10. An apparatus according to Aspect 9, wherein the shape of the first edge of the three-dimensional mesh is generated to match the shape of the contour of at least the portion of the target object, and wherein the shape of the second edge of the three-dimensional mesh is generated to match the shape of the second contour of at least the second portion of the target object.

[0169] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to: determine a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from a second perspective; and identify a second contour of at least a second part of the target object based on the second movement path of the indicating object and the second perspective, wherein the three-dimensional mesh is also generated based on the second contour of at least the second part of the target object.

[0170] Aspect 12. An apparatus according to Aspect 11, wherein the shape of a first edge of the three-dimensional mesh is generated to match the shape of the contour of at least the portion of the target object, and wherein the shape of a second edge of the three-dimensional mesh is generated to match the shape of the second contour of at least the second portion of the target object.

[0171] Aspect 13. The device according to any one of aspects 1 to 12, further comprising: the image sensor.

[0172] Aspect 14. The apparatus according to any one of aspects 1 to 13, further comprising: a display configured to display the three-dimensional grid.

[0173] Aspect 15. The apparatus according to any one of aspects 1 to 14, further comprising: a communication transceiver configured to send the three-dimensional mesh to a recipient device.

[0174] Aspect 16. The apparatus according to any one of aspects 1 to 15, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

[0175] Aspect 17. A method for image-based modeling, the method comprising: determining a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identifying a contour of at least a portion of the target object based on the movement path of the indicating object and the first perspective; and generating a three-dimensional mesh based on the contour of at least the portion of the target object.

[0176] Aspect 18. The method according to aspect 17, the method further comprising: applying a texture to the three-dimensional mesh to generate a three-dimensional model, wherein the texture is based on a representation of the target object in the image data.

[0177] Aspect 19. A method according to any one of Aspects 17 to 18, the method further comprising: tracking a hand in the image data to determine the movement path of the indicating object relative to the target object, wherein the indicating object includes at least a portion of the hand.

[0178] Aspect 20. The method according to aspect 19, wherein the target object is held by the hand, and wherein the movement path of the indicating object is configured to rotate the target object.

[0179] Aspect 21. The method according to any one of Aspects 17 to 20, further comprising: removing a portion of a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the portion of the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

[0180] Aspect 22. The method according to any one of Aspects 17 to 21, further comprising: combining a volume with a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the volume combined with the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

[0181] Aspect 23. A method according to any one of Aspects 17 to 22, the method further comprising: detecting a change from a second perspective to the first perspective based on the image data and previous image data of the scene, wherein the target object is represented in the previous image data from the second perspective, and wherein the previous image data is received from the image sensor before the image data.

[0182] Aspect 24. The method according to any one of Aspects 17 to 23, further comprising: generating a shape of an edge of the three-dimensional mesh to match a shape of the outline of at least the portion of the target object.

[0183] Aspect 25. A method according to any one of Aspects 17 to 24, the method further comprising: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from the first perspective; and identifying a second contour of at least a second part of the target object based on the second movement path of the indicating object and the first perspective, wherein the three-dimensional mesh is also generated based on the second contour of at least the second part of the target object.

[0184] Aspect 26. A method according to Aspect 25, wherein the shape of the first edge of the three-dimensional mesh is generated to match the shape of the contour of at least the portion of the target object, and wherein the shape of the second edge of the three-dimensional mesh is generated to match the shape of the second contour of at least the second portion of the target object.

[0185] Aspect 27. A method according to any one of Aspects 17 to 26, the method further comprising: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from a second perspective; and identifying a second contour of at least a second part of the target object based on the second movement path of the indicating object and the second perspective, wherein the three-dimensional mesh is also generated based on the second contour of at least the second part of the target object.

[0186] Aspect 28. A method according to Aspect 27, wherein the shape of the first edge of the three-dimensional mesh is generated to match the shape of the contour of at least the portion of the target object, and wherein the shape of the second edge of the three-dimensional mesh is generated to match the shape of the second contour of at least the second portion of the target object.

[0187] Aspect 29. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations according to any one of aspects 1 to 17.

[0188] Aspect 30. An apparatus for image processing, the apparatus comprising one or more components for performing the operations according to any one of aspects 1 to 17.

Claims

1. An apparatus for image-based modeling, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: determining a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identifying an outline of at least a portion of the target object based on the movement path of the indicating object and the first viewing angle; as well as A three-dimensional mesh is generated based on the outline of at least the portion of the target object.

2. The apparatus of claim 1 , wherein the at least one processor is configured to: A texture is applied to the three-dimensional mesh to generate a three-dimensional model, wherein the texture is based on a representation of the target object in the image data.

3. The apparatus of claim 1 , wherein the at least one processor is configured to: A hand is tracked in the image data to determine the path of movement of the indicating object relative to the target object, wherein the indicating object includes at least a portion of the hand. 4 . The apparatus of claim 3 , wherein the target object is held by the hand, and wherein the movement path of the indicating object is configured to rotate the target object.

5. The apparatus of claim 1 , wherein the at least one processor is configured to: A portion of a previous three-dimensional mesh is removed to generate the three-dimensional mesh, wherein the portion of the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

6. The apparatus of claim 1, wherein the at least one processor is configured to: A volume is combined with a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the volume combined with the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

7. The apparatus of claim 1 , wherein the at least one processor is configured to: A change from a second perspective to the first perspective is detected based on the image data and previous image data of the scene, wherein the target object is represented in the previous image data from the second perspective, wherein the previous image data was received from the image sensor prior to the image data. 8 . The apparatus of claim 1 , wherein the at least one processor is configured to generate a shape of an edge of the three-dimensional mesh to match a shape of the outline of at least the portion of the target object.

9. The apparatus of claim 1 , wherein the at least one processor is configured to: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from the first perspective; and A second outline of at least a second portion of the target object is identified based on the second movement path of the pointing object and the first viewing angle, wherein the three-dimensional mesh is further generated based on the second outline of at least the second portion of the target object.

10. The apparatus of claim 9, wherein a first edge of the three-dimensional mesh is shaped to match a shape of the contour of at least the portion of the target object, and wherein a second edge of the three-dimensional mesh is shaped to match a shape of the second contour of at least the second portion of the target object.

11. The apparatus of claim 1 , wherein the at least one processor is configured to: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from a second perspective; and A second outline of at least a second portion of the target object is identified based on the second movement path of the pointing object and the second viewing angle, wherein the three-dimensional mesh is further generated based on the second outline of at least the second portion of the target object.

12. The apparatus of claim 11, wherein a first edge of the three-dimensional mesh is shaped to match a shape of the contour of at least the portion of the target object, and wherein a second edge of the three-dimensional mesh is shaped to match a shape of the second contour of at least the second portion of the target object.

13. The apparatus according to claim 1, further comprising: The image sensor.

14. The apparatus according to claim 1, further comprising: A display is configured to display the three-dimensional grid.

15. The apparatus according to claim 1, further comprising: A communication transceiver is configured to transmit the three-dimensional mesh to a recipient device.

16. The apparatus of claim 1, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

17. A method for image-based modeling, the method comprising: determining a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identifying an outline of at least a portion of the target object based on the movement path of the indicating object and the first viewing angle; as well as A three-dimensional mesh is generated based on the outline of at least the portion of the target object.

18. The method according to claim 17, further comprising: A texture is applied to the three-dimensional mesh to generate a three-dimensional model, wherein the texture is based on a representation of the target object in the image data.

19. The method according to claim 17, further comprising: A hand is tracked in the image data to determine the path of movement of the indicating object relative to the target object, wherein the indicating object includes at least a portion of the hand.

20. The method of claim 19, wherein the target object is held by the hand, and wherein the movement path of the indicating object is configured to rotate the target object.

21. The method according to claim 17, further comprising: A portion of a previous three-dimensional mesh is removed to generate the three-dimensional mesh, wherein the portion of the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

22. The method according to claim 17, further comprising: A volume is combined with a previous three-dimensional mesh to generate the three-dimensional mesh, wherein the volume combined with the previous three-dimensional mesh is based on the contour of at least the portion of the target object.

23. The method according to claim 17, further comprising: A change from a second perspective to the first perspective is detected based on the image data and previous image data of the scene, wherein the target object is represented in the previous image data from the second perspective, wherein the previous image data was received from the image sensor prior to the image data.

24. The method according to claim 17, further comprising: The shape of the edge of the three-dimensional mesh is generated to match the shape of the outline of at least the portion of the target object.

25. The method of claim 17, further comprising: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from the first perspective; as well as A second outline of at least a second portion of the target object is identified based on the second movement path of the pointing object and the first viewing angle, wherein the three-dimensional mesh is further generated based on the second outline of at least the second portion of the target object.

26. The method of claim 25, wherein a first edge of the three-dimensional mesh is shaped to match a shape of the contour of at least the portion of the target object, and wherein a second edge of the three-dimensional mesh is shaped to match a shape of the second contour of at least the second portion of the target object.

27. The method of claim 17, further comprising: determining a second movement path of the indicating object relative to the target object based on second image data of the scene received from the image sensor, wherein the target object is represented in the second image data from a second perspective; as well as A second outline of at least a second portion of the target object is identified based on the second movement path of the pointing object and the second viewing angle, wherein the three-dimensional mesh is further generated based on the second outline of at least the second portion of the target object.

28. The method of claim 27, wherein a first edge of the three-dimensional mesh is shaped to match a shape of the contour of at least the portion of the target object, and wherein a second edge of the three-dimensional mesh is shaped to match a shape of the second contour of at least the second portion of the target object.

29. A non-transitory computer readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the following operations: determining a movement path of an indicating object relative to a target object based on image data of a scene received from an image sensor, wherein the target object is represented in the image data from a first perspective; identifying an outline of at least a portion of the target object based on the movement path of the indicating object and the first viewing angle; as well as A three-dimensional mesh is generated based on the outline of at least the portion of the target object.

30. The non-transitory computer readable medium of claim 29, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: A texture is applied to the three-dimensional mesh to generate a three-dimensional model, wherein the texture is based on a representation of the target object in the image data.

Images