Object detection and tracking in augmented reality devices

Abstract

Methods, systems, and apparatus are provided for detecting and tracking objects in an augmented reality (XR) environment. For example, a computing device may detect an object in a placement area of a hybrid environment. In response to the detection, the computing device may determine at least one parameter of the object, may register the object based on the at least one parameter, and may track movement of the object based on the registration. In additional embodiments, the computing device may capture at least one image of the object, may generate a plurality of data points based on the at least one image, may generate a multi-dimensional model based on the plurality of data points, and may generate a plurality of action points based on the multi-dimensional model. The computing device may track movement of the object based on the plurality of action points.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application No. 17 / 849,431 filed on June 24, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates generally to extended reality environments, and more particularly to object detection and tracking in extended reality devices. Background Art

[0003] Extended reality (XR) frameworks such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) frameworks can detect objects, such as a user's hand, in the field of view (FOV) of a camera of a corresponding reality system. The XR framework can track the object as it moves throughout its corresponding environment. For example, in a gaming application, the XR framework can attempt to track a player's hand as it moves throughout the environment. Summary of the invention

[0004] According to one aspect, a method includes receiving image data from a camera. The method also includes detecting an object in a placement area of ​​a hybrid environment based on the image data. The hybrid environment includes a real environment and a virtual environment. The method also includes: in response to the detection, determining a value of at least one parameter of the object. The method also includes generating profile data based on the at least one parameter value. The profile data registers the object with a user. In addition, the method includes tracking movement of the object within the hybrid environment based on the profile data.

[0005] According to another aspect, a method includes capturing at least one image of an object in a real environment of a hybrid environment. The method also includes generating a plurality of data points of the object based on the at least one image. The method also includes generating a multi-dimensional model of the object based on the plurality of data points. The method also includes generating a plurality of action points based on the multi-dimensional model of the object. Furthermore, the method includes tracking movement of the object in a virtual environment of the hybrid environment based on the plurality of action points.

[0006] According to another aspect, an apparatus includes a non-transitory machine-readable storage medium storing instructions, and at least one processor coupled to the non-transitory machine-readable storage medium. The at least one processor is configured to receive image data from a camera. The at least one processor is also configured to detect an object in a placement area of ​​a hybrid environment based on the image data. The hybrid environment includes a real environment and a virtual environment. In addition, the at least one processor is configured to determine a value of at least one parameter of the object in response to the detection. The at least one processor is also configured to generate profile data based on the at least one parameter value. The profile data registers the object with a user. The at least one processor is further configured to track the movement of the object within the hybrid environment based on the profile data.

[0007] According to another aspect, an apparatus includes a non-transitory machine-readable storage medium storing instructions, and at least one processor coupled to the non-transitory machine-readable storage medium. The at least one processor is configured to capture at least one image of an object in a real environment of a hybrid environment. The at least one processor is also configured to generate a plurality of data points of the object based on the at least one image. In addition, the at least one processor is configured to generate a multi-dimensional model of the object based on the plurality of data points. The at least one processor is also configured to generate a plurality of action points based on the multi-dimensional model of the object. The at least one processor is further configured to track the movement of the object in the virtual environment of the hybrid environment based on the plurality of action points.

[0008] According to another aspect, a non-transitory machine-readable storage medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising receiving image data from a camera. The operations also include detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment. In addition, the operations include determining a value of at least one parameter of the object in response to the detection. The operations also include generating profile data based on the at least one parameter value, the profile data registering the object with a user. The operations also include tracking movement of the object within the hybrid environment based on the profile data.

[0009] According to another aspect, a non-transitory machine-readable storage medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising capturing at least one image of an object in a real environment of a hybrid environment. The operations also include generating a plurality of data points for the object based on the at least one image. The operations also include generating a multi-dimensional model of the object based on the plurality of data points. The operations also include generating a plurality of action points based on the multi-dimensional model of the object. Furthermore, the operations also include tracking movement of the object in a virtual environment of the hybrid environment based on the plurality of action points.

[0010] According to another aspect, an object detection and tracking device includes means for receiving image data from a camera. The object detection and tracking device also includes means for detecting an object in a placement area of ​​a hybrid environment based on the image data. The hybrid environment includes a real environment and a virtual environment. The object detection and tracking device also includes means for determining a value of at least one parameter of the object in response to the detection. The object detection and tracking device also includes means for generating profile data based on the at least one parameter value, the profile data registering the object with a user. The object detection and tracking device also includes means for tracking movement of the object within the hybrid environment based on the profile data.

[0011] According to another aspect, an object detection and tracking device includes means for capturing at least one image of an object in a real environment of a hybrid environment. The object detection and tracking device also includes means for generating a plurality of data points of the object based on the at least one image. The object detection and tracking device also includes means for generating a multi-dimensional model of the object based on the plurality of data points. The object detection and tracking device also includes means for generating a plurality of action points based on the multi-dimensional model of the object. The object detection and tracking device also includes means for tracking movement of the object in a virtual environment of the hybrid environment based on the plurality of action points. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an exemplary object detection and tracking device according to some specific implementations;

[0013] Figure 2A is a diagram illustrating tracking ranges in an XR system according to some implementations;

[0014] Figure 2B is a diagram illustrating a field of view (FOV) of a user according to some implementations;

[0015] Figure 3 is a diagram illustrating insertion of an object into a tracking range of an XR system according to some implementations;

[0016] Figure 4 is a diagram illustrating a landmark technique for identifying a hand according to some implementations;

[0017] Figure 5A is a diagram illustrating palm lines of a hand that may be used to uniquely identify the hand according to some implementations;

[0018] Figure 5B is a diagram illustrating a palm contour map that may be used to uniquely identify a hand according to some implementations;

[0019] Figure 6is a diagram illustrating tracking techniques in an XR system according to some specific implementations;

[0020] Figure 7 is a diagram illustrating a hand tracking technique using 20 different points of the hand according to some implementations;

[0021] Fig. 8A is a diagram illustrating a hand with a covering according to some implementations;

[0022] Figure 8B is a diagram illustrating a hand having an irregular shape according to some implementations;

[0023] Fig. 9 is a diagram illustrating a hand tracking technique for tracking a hand having an unexpected shape according to some implementations;

[0024] Fig.10 is a flow chart of an exemplary process for detecting and tracking an object according to some specific implementations; and

[0025] Fig.11 is a flow chart of an exemplary process for tracking an object based on a machine learning process according to some specific implementations. DETAILED DESCRIPTION

[0026] Although the features, methods, devices, and systems described herein may be embodied in various forms, some exemplary and non-limiting embodiments are shown in the drawings and described below. Some of the components described in this disclosure are optional, and some implementations may include additional components, different components, or fewer components than those explicitly described in this disclosure.

[0027] Various systems, such as gaming, computer vision, extended reality (XR), augmented reality (AR), virtual reality (VR), medical, and robotics-based applications, rely on receiving input from a user through one or more techniques. These techniques may include tracking the movement of one or more body parts of a user (e.g., a hand, a fist, fingers of a hand, etc.). For example, an imaging device, such as a digital camera, a smartphone, a tablet, a laptop, a car, or an Internet of Things (IoT) device (e.g., a security camera, etc.), may capture an image of a user, and a 3D image may be reconstructed based on one or more body parts of the user. For example, an imaging device may capture an image of a user's hand, such as a game player's hand, and a 3D image of the user's hand may be reconstructed for use as part of an avatar within, for example, an XR, VR, or AR-based game.

[0028] Existing image processing techniques may allow input to be received from a user through the movement of one or more objects or body parts of the user. However, these conventional techniques may suffer from several disadvantages, such as properly determining whether a detected object or body part belongs to the user, and determining whether the user intends to use the object or body part to transmit input. In addition, these conventional systems may also be unable to use objects to correctly detect gestures of the user, such as gestures performed with the user's hand or an object that the user is holding. For example, if one or both of the user's hands suffer from deformities (such as an irregular shape of the hand, more than five fingers on the hand, less than five fingers on the hand, etc.), or if the user holds an object with less than five fingers when intending to provide input, or if the user's hand is completely or partially covered by a material (e.g., mittens), the XR, VR, or AR system may not successfully detect the user's hand, and may not be able to recognize one or more gestures that the user may intend for the XR, VR, or AR system to detect.

[0029] In some implementations, the object detection and tracking device may include one or more optical elements, such as a camera, one or more motion sensors, a thermal camera, and a sensitive microphone for detecting sound based on movement, and may detect one or more objects or body parts of a user within a virtual environment to identify (e.g., determine) input gestures made by the user. In some implementations, the object detection and tracking device may detect an object in the field of view (FOV) of a camera and determine that the object corresponds to a particular user. For example, the object detection and tracking device may determine that the object corresponds to the user and is being used to provide an input gesture. Additionally or alternatively, the object detection and tracking device may determine that the object does not correspond to the user and therefore will not be used to provide an input gesture.

[0030] In one specific implementation, the object detection and tracking device may include one or more processors that execute instructions stored in a memory of the object detection and tracking device. The one or more processors may include, for example, a camera processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a neural processing unit (NPU). The object detection and tracking device may execute instructions to detect an object, such as a user's hand, based on one or more parameter values. The parameter value may correspond to, for example, an angle at which the object is inserted into a predetermined window within the FOV of a user within the virtual environment. The object detection and tracking device may also execute instructions to detect that the object corresponds to the user, and may track the object to identify an input gesture from the user. For example, the object detection and tracking device may detect an object within a predetermined window, determine that the object corresponds to the user, and may track the object on the user's FOV. Based on the movement of the tracked object, the object detection and tracking device may determine one or more input gestures from the user.

[0031] In another specific implementation, the object detection and tracking device may include one or more processors that execute instructions stored in a memory of the object detection and tracking device to detect an object (such as a hand) of the user based on a unique profile of the user. For example, the unique profile of the user may include data characterizing one or more of the shape of the user's hand, palm lines on the user's hand, palm contours, the size of the user's fingernails, the shape of the user's fingernails, the color of the object, a multi-point contour of the user's hand, and one or more identification marks on the object. The object detection and tracking device may execute instructions to track the object based on the user's profile. For example, the object detection and tracking device may execute instructions to detect one or more input gestures from the user based on the user's profile.

[0032] In some implementations, the object detection and tracking device may include one or more processors that execute one or more trained machine learning processes to detect an object of a user (such as a user's hand) for tracking and receiving one or more gesture inputs. The trained machine learning process may include, for example, a trained neural network process (e.g., a trained convolutional neural network (CNN) process), a trained deep learning process, a trained decision tree process, a trained support vector machine process, or any other suitable trained machine learning process. For example, during initialization, the object detection and tracking device may prompt the user to select an object detected by a camera or sensor of the object detection and tracking device as an object to be used to detect the user's gesture input. The object detection and tracking device may apply the trained machine learning process to image data representing the selected object to generate a plurality of data points for the object and a multi-dimensional model of the selected object. In addition, the object detection and tracking device may apply the trained machine learning process to the multi-dimensional model of the object to estimate an action point. For example, an action point may be an expected point in the 3-D space of a virtual environment that may move when certain gestures are made. In some cases, the object detection and tracking device may implement a training mode for the machine learning process, during which the machine learning process may iteratively change the action point in the 3-D space for the corresponding gesture. For example, the object detection and tracking device may determine a gesture based on the generated action points, and may request and receive verification from the user to confirm whether the determined gesture is correct.

[0033] In some implementations, and based on execution of instructions stored in non-volatile memory, one or more processors may apply a machine learning process to the multi-dimensional model of the object to generate a lookup table. The lookup table may include a list of gestures and a sequence of tracking points in 3-D space that the object may be expected to cross during the gesture. The tracking points may include, for example, x, y, z coordinates for each of the tracking points in 3-D space.

[0034] When the training process is complete, the one or more processors may store the values ​​and sequences of tracking points and corresponding gestures as lookup tables (e.g., each lookup table corresponds to a unique object / hand of a user) in a memory device of the object detection and tracking device. The lookup tables corresponding to the objects may enable the one or more processors to detect and identify gestures made by the object while tracking the movement of the object (e.g., during gesture input).

[0035] Among other advantages, the embodiments described herein can provide more accurate object (e.g., hand) detection capabilities for XR, VR, or AR systems. For example, the embodiments can more accurately identify which object corresponds to the user, and which object in the user's FOV may be a foreign or unexpected object to be ignored. In addition, the embodiments as described herein can provide users with greater flexibility in receiving input using irregularly shaped objects. The embodiments may be particularly helpful and enable users to be in physically challenging situations, such as those with biological defects in one or both hands. The embodiments may further allow users to engage in multitasking (e.g., holding one or more objects in their hands, engaging one or more fingers with other gadgets such as fitness trackers, smart watches, etc.), while making gestures that object detection and tracking devices can successfully detect and track. In addition, the embodiments can be used in a variety of applications, such as games, computer vision, AR, VR, medicine, biometrics, and robotics applications. Those of ordinary skill in the art who benefit from these disclosures will also recognize these and other benefits.

[0036] Figure 1 is a block diagram of an exemplary object detection and tracking device 100. The functionality of the object detection and tracking device 100 may be implemented in one or more processors, one or more field programmable gate arrays (FPGAs), one or more application specific integrated circuits (ASICs), one or more state machines, digital circuits, any other suitable circuits, or any suitable hardware. The object detection and tracking device 100 may perform one or more of the exemplary functions and processes described in the present disclosure. Examples of the object detection and tracking device 100 include, but are not limited to, a computer (e.g., a game console, a personal computer, a desktop computer, or a laptop computer), a mobile device (such as a tablet computer), a wireless communication device (such as, for example, a mobile phone, a cellular phone, etc.), a digital camera, a digital video recorder, a handheld device (such as a portable video game device or a personal digital assistant (PDA)), a drone device, a virtual reality device (e.g., a virtual reality headset), an augmented reality device (e.g., augmented reality glasses), a virtual reality device (e.g., a virtual reality headset), an extended reality device, or any device that may include one or more cameras.

[0037] like Figure 1 As illustrated in the example of , the object detection and tracking device 100 may include one or more image sensors 112 (such as image sensor 112A), a lens 113A, and one or more camera processors (such as camera processor 114). In some instances, the camera processor 114 may be an image signal processor (ISP) that uses various image processing algorithms to process image data (e.g., captured by corresponding lenses and sensors of these lenses and sensors). For example, the camera processor 114 may include an image front end (IFE) and / or an image processing engine (IPE) as part of a processing pipeline. In addition, the camera 115 may refer to a collective device including one or more image sensors 112, one or more lenses 113, and one or more camera processors 114.

[0038] The object detection and tracking device 100 may also include a central processing unit (CPU) 116, an encoder / decoder 117, a graphics processing unit (GPU) 118, a local memory 120 of the GPU 118, a user interface 122, a memory controller 124 that provides access to a system memory 130 and an instruction memory 132, and a display interface 126 that outputs signals that cause graphical data to be displayed on a display 128.

[0039] In some examples, one of the image sensors 112 may be assigned to each of the lenses 113. In addition, in some examples, one or more of the image sensors 112 may be assigned to a corresponding lens of the lenses 113 of a corresponding and different lens type (e.g., a wide-angle lens, an ultra-wide-angle lens, a telephoto lens, and / or a periscope lens, etc.). For example, the lens 113 may include a wide-angle lens, and a corresponding image sensor of the image sensor 112 having a first size (e.g., 108 MP) may be assigned to the wide-angle lens. In other examples, the lens 113 may include an ultra-wide-angle lens, and a corresponding image sensor of the image sensor 112 having a second and different size (e.g., 16 MP) may be assigned to the ultra-wide-angle lens. In another example, the lens 113 may include a telephoto lens, and a corresponding image sensor of the image sensor 112 having a third size (e.g., 12 MP) may be assigned to the telephoto lens.

[0040] In the illustrative example, a single object detection and tracking device 100 may include two or more cameras (e.g., two or more of cameras 115). Furthermore, in some examples, a single image sensor (e.g., image sensor 112A) may be assigned to multiple lenses in lenses 113. Additionally or alternatively, each of image sensors 112 may be assigned to a different lens in lenses 113, for example, to provide multiple cameras to object detection and tracking device 100.

[0041] exist Figure 1 In some examples not illustrated in , the object detection and tracking device 100 may include multiple cameras (e.g., a mobile phone having one or more front cameras and one or more rear cameras). For example, the object detection and tracking device 100 may include a first camera (such as camera 115 including a 16MP image sensor), a second camera (which includes a 108MP image sensor), and a third camera (which includes a 12MP image sensor).

[0042] Each image sensor in image sensors 112, including image sensor 112A, may represent an image sensor including processing circuitry, an array of pixel sensors (e.g., pixels) for capturing representations of light, memory, an adjustable lens (such as lens 113), and an actuator to adjust the lens. By way of example, image sensor 112A may be associated with a corresponding lens in lenses 113, such as lens 113A, and may capture images through the corresponding lens. In other examples, additional or alternative image sensors in image sensors 112 may be associated with corresponding additional lenses in lenses 113, and may capture images through the corresponding additional lenses.

[0043] Image sensor 112 may also include subsets of two or more different image sensors that operate in conjunction with each other to detect motion of an object / hand. For example, image sensor 112 may include two different "color" pixel sensors that operate in conjunction with each other. The different color pixel sensors may support different binning types and / or binning levels, and while operating in conjunction with each other, the different color pixel sensors may each operate with respect to a specific range of zoom levels.

[0044] Additionally, in some instances, the object detection and tracking device 100 may receive user input via the user interface 122, and in response to the received user input, the CPU 116 and / or the camera processor 114 may activate a corresponding lens in the lenses 113 or a combination of lenses 113. For example, the received user input may correspond to a confirmation that the object / hand in the field of view of the lens 113A is the object / hand of the user whose input gesture should be tracked. In some instances, the user input via the user interface 122 may be a confirmation that the object detection and tracking device 100 has identified the correct gesture during the machine learning process (as described above).

[0045] Although the various components of object detection and tracking device 100 are illustrated as separate components, in some examples, these components may be combined to form a system on a chip (SoC). As an example, instruction memory 132, CPU 116, GPU 118, and display interface 126 may be implemented on a common integrated circuit (IC) chip. In some examples, one or more of instruction memory 132, CPU 116, GPU 118, and display interface 126 may be implemented in separate IC chips. Various other arrangements and combinations are possible, and the techniques of this disclosure should not be considered limited to the present invention. Figure 1 .

[0046] The system memory 130 may store program modules and / or instructions and / or data that can be accessed by the camera processor 114, the CPU 116, and the GPU 118. For example, the system memory 130 may store user applications (e.g., instructions for the camera application) and the resulting images from the camera processor 114. The system memory 130 may additionally store information for use and / or generation by other components of the object detection and tracking device 100. For example, the system memory 130 may serve as device memory for the camera processor 114. The system memory 130 may include one or more volatile or non-volatile memories or storage devices, such as, for example, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, cloud-based storage media, or optical storage media.

[0047] The camera processor 114 may store data to and read data from the system memory 130. For example, the camera processor 114 may store a set of working instructions to the system memory 130, such as instructions loaded from the instruction memory 132. The camera processor 114 may also use the system memory 130 to store dynamic data generated during operation of the object detection and tracking device 100.

[0048] Similarly, the GPU 118 may store data to and read data from the local memory 120. For example, the GPU 118 may store a set of working instructions to the local memory 120, such as instructions loaded from the instruction memory 132. The GPU 118 may also use the local memory 120 to store dynamic data generated during operation of the object detection and tracking device 100. Examples of the local memory 120 include one or more volatile or non-volatile memory or storage devices, such as RAM, SRAM, DRAM, EPROM, EEPROM, flash memory, magnetic data media, cloud-based storage media, or optical storage media.

[0049] The instruction memory 132 may store instructions that may be accessed (e.g., read) and executed by one or more of the camera processor 114, the CPU 116, and the GPU 118. For example, the instruction memory 132 may store instructions that, when executed by one or more of the camera processor 114, the CPU 116, and the GPU 118, cause the camera processor 114, the CPU 116, and the GPU 118 to perform one or more of the operations described herein. For example, the instruction memory 132 may include a detection unit 132A that may include instructions that, when executed by one or more of the camera processor 114, the CPU 116, and the GPU 118, cause the camera processor 114, the CPU 116, and the GPU 118 to detect an object / hand of a user, as described in various embodiments. The instruction memory 132 may also include a tracking unit 132B, which may include instructions that, when executed by one or more of the camera processor 114, the CPU 116, and the GPU 118, cause the camera processor 114, the CPU 116, and the GPU 118 to track the movement of the object / hand, as described in various embodiments. In an optional embodiment, the tracking unit 132B may include a lookup table 132C, which may store a sequence of tracking points and a gesture corresponding to the sequence of tracking points for a particular object / hand. As described herein, the lookup table 132C may allow the gesture of the object / hand to be identified based on the tracking points that the object / hand crosses in the gesture. In addition, the tracking unit 132B may include instructions that, when executed by one or more of the camera processor 114, the CPU 116, and the GPU 118, cause the camera processor 114, the CPU 116, and the GPU 118 to perform a machine learning process, as described herein.

[0050] The instruction memory 132 may also store instructions that, when executed by one or more of the camera processor 114, the CPU 116, and the GPU 118, cause the camera processor 114, the CPU 116, and the GPU 118 to perform image processing operations, generate multiple data points for the object / hand, generate a multi-dimensional model of the object / hand, and generate multiple action points based on the multi-dimensional model of the object / hand, as described herein.

[0051] like Figure 1 As illustrated, the various components of the object detection and tracking device 100 may be configured to communicate with each other across a bus 135. The bus 135 may include any of a variety of bus structures, such as a third generation bus (e.g., a HyperTransport bus or an InfiniBand bus), a second generation bus (e.g., an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXtensible Interface (AXI) bus), or another type of bus or device interconnect. It will be appreciated that Figure 1 The specific configuration of components and communication interfaces between the different components shown are merely exemplary, and other configurations of components and / or other image processing systems having the same or different components may be configured to implement the operations and processes of the present disclosure.

[0052] The camera processor 114 may be configured to receive image frames (e.g., pixel data, image data) from the image sensor 112 and process the image frames to generate images and / or video content. For example, the image sensor 112A may be configured to capture individual frames, frame bursts, a sequence of frames for generating video content, a still image of a photo captured while recording a video, an image preview, or a motion photo from before and / or after a user captures an input gesture. The CPU 116, GPU 118, camera processor 114, or some other circuit may be configured to process the image and / or video content captured by the image sensor 112A into an image or video for display on the display 128. In an illustrative example, the CPU 116 may cause the image sensor 112A to capture an image frame and may receive pixel data from the image sensor 112A. In the context of the present disclosure, an image frame may generally refer to a data frame of a still image or a video data frame or a combination thereof, such as in the case of a motion photo. The camera processor 114 may receive pixel data of an image frame in any suitable format from the image sensor 112. For example, the pixel data may be formatted according to a color format such as RGB, YCbCr, or YUV.

[0053] In some examples, the camera processor 114 may include an image signal processor (ISP). For example, the camera processor 114 may include an ISP that receives signals from the image sensor 112, converts the received signals into image pixels, and provides pixel values ​​to the camera processor 114. Additionally, the camera processor 114 may be configured to perform various operations on image data captured by the image sensor 112, including automatic gain, automatic white balance, color correction, or any other image processing operation.

[0054] The memory controller 124 may be communicatively coupled to the system memory 130 and the instruction memory 132. The memory controller 124 may facilitate data transfers to and from the system memory 130 and / or the instruction memory 132. For example, the memory controller 124 may receive memory read and write commands, such as from the camera processor 114, the CPU 116, or the GPU 118, and service such commands to provide memory services to the system memory 130 and / or the instruction memory 132. Although in Figure 1 1 illustrates memory controller 124 as being separate from both CPU 116 and system memory 130, but in other examples, some or all of the functionality of memory controller 124 with respect to servicing system memory 130 may be implemented on one or both of CPU 116 and system memory 130. Similarly, some or all of the functionality of memory controller 124 with respect to servicing instruction memory 132 may be implemented on one or both of CPU 116 and instruction memory 132.

[0055] The camera processor 114 may also be configured, through the executed instructions, to analyze the image pixel data and store the resulting image (e.g., the pixel value of each of the image pixels) to the system memory 130 via the memory controller 124. The GPU 118 or some other processing unit (including the camera processor 114 itself) may perform operations to detect an object in the placement area, register the object / hand, generate a plurality of data points for the object / hand based on the image of the object / hand, generate a multi-dimensional model of the object / hand based on the plurality of data points, generate a plurality of motion points based on the multi-dimensional model, and track the movement of the object / hand in the virtual environment based on the plurality of motion points.

[0056] In addition, the object detection and tracking device 100 may include a video encoder and / or a video decoder 117, either of which may be integrated as part of a combined video encoder / decoder (CODEC). The encoder / decoder 117 may include a video decoder that encodes video captured by one or more cameras 115 or a decoder that decodes compressed or encoded video data. In some examples, the CPU 116 may be configured to encode and / or decode video data using the encoder / decoder 117.

[0057] The CPU 116 may include a general or special purpose processor that controls the operation of the object detection and tracking device 100. A user may provide input to the object detection and tracking device 100 to cause the CPU 116 to execute one or more software applications. The software applications executed by the CPU 116 may include, for example, a camera application, a graphics editing application, a media player application, a video game application, a graphical user interface application, or other programs. For example, and when executed by the CPU 116, the camera application may allow various settings of the camera 115 to be controlled, for example, via input provided to the object detection and tracking device 100 through the user interface 122. Examples of the user interface 122 include, but are not limited to, a pressure-sensitive touch screen unit, a keyboard, a mouse, or an audio input device (such as a microphone). For example, the user interface 122 may receive input from a user to verify an object to be tracked for an input gesture, or to verify a gesture recognized during a machine learning process (as described above, and with reference to the following Fig. 9 describe).

[0058] By way of example, the executed camera application may cause the CPU 116 to generate content for display on the display 128 and / or in the virtual environment of the hybrid environment of the AR, VR, or XR system. For example, the display 128 or the projection in the virtual environment may display information such as an image insertion guide for indicating to the user the insertion direction and / or angle for the object / hand detected by the object detection and tracking device 100. The executed hand detection and tracking application (for simplicity, the hand detection and tracking application 100 is stored in the system memory 130 and / or the instruction memory 132) may be displayed in the virtual environment of the AR, VR, or XR system. Figure 1 12A) may also cause the CPU 116 to direct the camera processor 114 to process images captured by the sensors 112A in a predefined manner. For example, the CPU 116 may direct the camera processor 114 to perform a zoom operation, or to increase the white balance on images captured by one or more of the sensors 112, e.g., in response to identifying that the hand / object is small or the hand / object is in a low-light environment, etc.

[0059] As described herein, one or more of CPU 116 and GPU 118 may perform operations to generate or update lookup table 132C using a machine learning process that applies training.

[0060] In some examples, one or more of CPU 116 and GPU 118 causes the output data to be displayed on display 128. In some examples, object detection and tracking device 100 sends the output data to a computing device, such as a server or a user's handheld device (e.g., a cell phone), via transceiver 119. For example, object detection and tracking device 100 may send a message based on the output data to another computing device, such as a handheld device for authenticating a user, or a projection device for simulating a virtual environment.

[0061] Figure 2A is a diagram illustrating a tracking range in an XR system including, for example, the object detection and tracking device 100 . Figure 2A A user 202 having a FOV 204 is included. The FOV 204 of the user 202 may have an angular extent of 120 degrees, such as Figure 2A 204 may be an area in which the VR, AR, or XR system may track (e.g., using one or more cameras 115) input gestures of the user 202. For example, the VR, AR, or XR system may track objects within the FOV 204, such as the user's hands. The FOV 204 may extend from a first radius from the user to a second radius from the user. For example, and as Figure 2A As shown, FOV 204 may extend from a radius of approximately 10 cm from the eyes of user 202 to a radius of 60 cm-80 cm from the user. Figure 2A The hand of user 202 in FOV 204 is illustrated, but multiple hands of various users may be present in FOV 204 (e.g., foreign hands, which may be hands of people other than user 202). The VR, AR, or XR system may detect the hands inserted into FOV 204, may determine whether each hand is associated with a corresponding user, and may track the hands associated with the corresponding users. For example, the VR, AR, or XR system may detect input gestures made with detected hands from each of the users (such as user 202).

[0062] Figure 2B is a diagram illustrating a user's FOV 204 with a placement area 206 for initializing object detection and tracking according to some embodiments. Figure 2B including having FOV 204 (as referenced above Figure 2A204). In one specific implementation, during initialization of the object detection and tracking process, the object detection and tracking device 100 may generate and display a highlight of the drop area 206 to the user 202 within the virtual environment. For example, the CPU 116 may execute instructions stored in the detection unit 132A to generate a request for the user 202 to insert an object, such as the user's hand, into the drop area 206. The display unit 208 may cause the drop area 206 to be highlighted in the virtual environment and displayed to the user 202 (e.g., as seen by the user 202 through the VR goggles). Upon initialization, the object detection and tracking device 100 may detect an object present in the drop area 206 as an object of the user 202 to be tracked for use in recognizing input gestures from the user 202.

[0063] Figure 3 This is an example of using Figure 1 1. An object detection and tracking device 100 is shown inserting an object into the tracking range of the XR system. Figure 3 Includes placement area 308 (similar to the above reference Figure 2B The placement area 206 ) highlights to the user 202 the insertion angle and insertion direction of the user's 202 hand 310 into the placement area 308 . Figure 3 A projection device 302 (e.g., a projection device of the object detection and tracking device 100) is included that can project a placement area 308 bounded by boundaries 304 and 306 of the placement area 308 for detection of a hand 310. The projection device 302 can highlight to a user an insertion angle into the placement area 308 through which the user can insert the hand 310 for detection by the object detection and tracking device 100. In some examples, the projection device 302 can generate and display an image identifying the insertion direction into the placement area 308 for detecting the hand 310 within the virtual environment.

[0064] In some examples, the object detection and tracking device 100 may determine whether the insertion angle of the hand 310 is within a predetermined range, and may generate profile data identifying the hand 310 as the user's hand based on the determination. For example, the predetermined range may be a range of angle values ​​based on the field of view of the user 202. When the object detection and tracking device 100 determines that the detected insertion angle of the hand 310 is within the predetermined range of values, the object detection and tracking device 100 may register the hand 310 as an object to be tracked by the user. Similarly, the object detection and tracking device 100 may determine that the insertion direction into the placement area 308 is an appropriate direction (e.g., from bottom to top), and the object detection and tracking device 100 may register the hand 310 as an object to be tracked by the user.

[0065] As another example, the object detection and tracking device 100 may determine that the insertion angle of the hand 310 is not within a predetermined range of values, and may not associate the hand 310 with the user. Similarly, the object detection and tracking device 100 may determine that the insertion direction into the placement area 308 is not an appropriate direction (e.g., from top to bottom), and may not associate the hand 310 with the user. Therefore, the object detection and tracking device 100 may not register the hand 310 as an object to be tracked. In one specific implementation, the object detection and tracking device 100 may request the user 202 to re-enter the hand 310 at a suggested angle and / or direction. For example, the object detection and tracking device 100 may provide a visual cue (e.g., providing an insertion guide image identifying one or more insertion angles for inserting the hand / object 310 into the placement area 308) through a projection in or near the placement area 308, which indicates to the user 202 the insertion angle and / or insertion direction at which the user 202 can insert the hand 310 to successfully register the hand 310 as the hand of the user 202 to the XR system.

[0066] Figure 4 is a diagram illustrating a landmark technique for identifying a hand. Figure 4 The object detection and tracking device 100 includes hands 402 and 404, each of which includes a plurality of landmark points 406 and 408, respectively. The object detection and tracking device 100 may uniquely identify the hands of the user 202 based on the landmark points 406 and 408, as described herein. For example, each of the landmark points 406 and 408 may be a set of points that uniquely describe the geometry of the hands 402 and 404 of the user 202, respectively. The object detection and tracking device 100 may detect and identify the hands 402 and 404 based on a hand line drawing. For example, the object detection and tracking device 100 may compare the landmark points 406 and 408 to a set of points within a hand line drawing stored in a memory of the object detection and tracking device 100 (e.g., the system memory 130 and / or the instruction memory 132). Upon detecting a successful match, the object detection and tracking device 100 may find an insertion placement area (e.g., as described above with reference to FIG. 1 ). Figure 3 The hand in the placement area 308 is the hand of the user 202, and the detected object is registered as the object of the user 202 for tracking and receiving input gestures from the user 202. For example, the object detection and tracking device 100 can generate profile data that associates the detected hand with the user, and can store the profile data in the system memory 132.

[0067] Figure 5A is a diagram illustrating palm lines of a hand that can be used to uniquely identify the hand. Figure 5A Including Figure 5AThe object detection and tracking device 100 may be based on combining the data representing the palm lines 506 with the data representing the palm lines 506 and stored in the memory of the object detection and tracking device 100 (e.g., the system memory 130 and / or the instruction memory 132, as described above with reference to Figure 1 to uniquely identify and detect the Figure 5A 306 as shown. Upon determining a successful match, the object detection and tracking device 100 may determine the hand inserted into the placement area 308 as the hand of the user 202, and may track the movement of the hand to determine the gesture of the user 202. The object detection and tracking device 100 is not limited to using the palm lines 506 as described above to determine a successful match. In some specific implementations, the object detection and tracking device 100 may use other unique features of the hand of the user 202 (such as palm contour, hand shape (e.g., unique geometry), fingernail size, fingernail shape, hand color, multi-point hand contour geometry, and / or one or more identification marks on the hand of the user 202) to uniquely identify the hand as the hand of the user 202. Once detected, the object detection and tracking device 100 may generate profile data, wherein the profile data registers the hand as the hand of the user 202 to the XR system. The object detection and tracking device 100 may track the movement of the hand for receiving input gestures from the user 202 based on the profile data.

[0068] Figure 5B is a diagram illustrating a palm contour map of a hand that may be used to uniquely identify a hand, such as a hand inserted into placement area 308 . Figure 5B Including Figure 5B The palm contour image data 504 is shown. The palm contour image data 504 may be based on an image captured by the camera 115 of the object detection and tracking device 100. The object detection and tracking device 100 may uniquely identify and detect a hand having a palm contour represented by the palm contour image data 504. For example, the object detection and tracking device 100 may store the palm contour image data 504 in a memory of the object detection and tracking device 100 (e.g., the system memory 130 and / or the instruction memory 132, as described above with reference to FIG. 1 ). Figure 1The object detection and tracking device 100 may be compared with the palm contour data in the palm contour image data 504 to determine whether the palm contour matches. In some cases, the system memory 132 stores palm contour data for multiple users. The palm contour data may identify and characterize multiple pixel positions along the contour of the hand captured in the image. The object detection and tracking device 100 may perform operations to determine whether any palm contour data of the user matches the contour of the palm contour image data 504 to identify the user. After determining a successful match, the object detection and tracking device 100 may determine the hand inserted into the placement area 308 as the hand of the user 202, and may detect and track the gesture of the hand as the input gesture of the user 202.

[0069] Figure 6 is a diagram illustrating tracking technology in an XR system. Figure 6 A user 602 is included having a FOV 604. The FOV 604 of the user 602 may have an angular spread of a certain number of degrees, such as Figure 6 604 can be an area in the real environment where the VR, AR, or XR system can track the input gestures of the user 602. Figure 6 As shown, FOV 604 may extend from a radius of approximately 10 cm from the eyes of user 602 to a radius within a range of 60 cm-80 cm from the user. Figure 6 Only the hands of user 602 in FOV 604 are shown, however, for conventional VR, AR, or XR systems, it may be challenging to detect and track the hands of user 602 when the hands may have irregular or unexpected shapes or be covered with materials (such as clothing), making it difficult to identify the hands / objects that should be tracked by the VR, AR, or XR system for receiving input gestures from user 602. However, as described herein, object detection and tracking device 100 may employ hand tracking techniques to detect irregularly shaped hands or objects.

[0070] For example, Figure 7 is a diagram illustrating a hand tracking graph using 20 different points of the hand. Figure 7 A 20-point diagram 702 of a user's hand is included, where each of the 20 points is described for a specific location on a conventional hand. However, in instances where the user's hand is irregularly shaped (e.g., the user has four fingers instead of five), or the user has a covering on the hand, Figure 7 Each of the 20 points shown in A may be absent or identifiable. Such examples of hands are referred to below. Fig. 8A and Figure 8B to describe.

[0071] For example, Fig. 8A is a diagram illustrating a hand with a cover. Fig. 8AA hand 802 is included with a mitten covering the hand, and an outline 804 indicates the shape of the hand 802. Figure 7 In contrast, a 20-point model for identifying and detecting a user's hand may not be useful for detecting or tracking the movement of hand 802 for recognizing input gestures, at least because hand 802 may not map to all 20 points or a sufficient number of points for detecting and tracking hand 802 using the 20-point technique.

[0072] Figure 8B is a diagram illustrating a hand having an irregular shape. Figure 8B A hand 806 having an irregular shape (such as a missing middle finger) is included, and an outline 808 describes the shape of the hand 806. Figure 7 In contrast, a 20-point model for identifying and detecting a user's hand may not be useful for detecting or tracking the motion of hand 806 for identifying input gestures, at least because hand 802 may not map to all 20 points or a sufficient number of points for detecting and tracking hand 802 using the 20-point technique. Fig. 9 An initialization start technique is illustrated by which the object detection and tracking device 100 can detect unexpected shapes and sizes of objects by seeking confirmation from a user (eg, user 202 ) via ground truth initialization.

[0073] Specifically, Fig. 9 is a diagram illustrating a hand tracking technique for tracking a hand having an unexpected or irregular shape. Fig. 9 The hand 902 is comprised of a hand having a contour line 904. Upon detecting a hand having a covering (e.g., Fig. 8A ), or have hands with irregular shapes (e.g. Figure 8B After detecting a hand 902 (as shown) or an object (e.g., a prosthesis), the object detection and tracking device 100 can generate a plurality of data points 906 of the hand 902 based on the image of the hand 902. The object detection and tracking device 100 can also generate a multi-dimensional model (e.g., a 3-D model) of the hand 902 based on the plurality of data points 906. For example, the object detection and tracking device 100 can capture one or more images of the hand 902 in the real environment of the mixed environment of the XR system, and plot the data points 906 in a 3-D space to generate a multi-dimensional model of the hand 902. The multi-dimensional model can be, for example, a 3D model of the hand 902. The object detection and tracking device 100 can also generate a plurality of action points based on the multi-dimensional model of the hand 902 and the detected gesture. The object detection and tracking device 100 can further determine a plurality of tracking points, which can be points in the 3-D space that the hand 902 is expected to span when making a gesture, and can store the tracking points in a lookup table specific to the hand 902 (e.g., as described above with reference to FIG. 1 ). Figure 1The object detection and tracking device 100 may utilize the lookup table to determine the gesture of the hand 902 as the hand 902 moves in the 3-D space. The detailed steps of this initialization technique for hand / object detection, registration, and tracking are described below.

[0074] To begin the initialization process, the object detection and tracking device 100 may generate a first request for placing an object in a mixed environment including a real environment and a virtual environment. For example, the object detection and tracking device 100 may generate and display a highlighted placement area (e.g., placement area 308, as described above with reference to FIG. 1 ) in the virtual environment of the XR system. Figure 3 The object detection and tracking device 100 may indicate a first request to the user by using an image of the XR device (described) to request the user to place an object, such as a hand, in the placement area. Upon detecting that the user may have placed the object in the placement area, the object detection and tracking device 100 may generate a second request for confirmation from the user that the object is the user's hand. For example, the object detection and tracking device 100 may seek verification from the user that the object placed in the placement area is the object that the user intended to use to provide gesture input. The object detection and tracking device 100 may display the second request within the virtual environment of the XR device, for example, through a projection in front of the user. After receiving a response from the user confirming that the identified object is the object that the user intended to use to provide input gestures, the object detection and tracking device 100 may generate multiple image data points for the object. For example, the multiple image data points may be similar to the following: Fig. 9 Data points 906 are shown. The number of data points in the plurality of image data points may be any number, including less than 20, greater than 20, or equal to 20. Furthermore, the plurality of image data points may differ from user to user based on the shape of the object registered by each user.

[0075] Based on the multiple image data points, the object detection and tracking device 100 can determine multiple action points, and the multiple action points can be stored in a memory. The multiple action points can be significant points in an object (such as a hand) that move enough to produce a gesture. For example, the multiple action points can correspond to joints or other points where the object or hand can bend (for example, the action points can be points in a 3-D model of the hand that are expected to be active / move in a certain way for a particular gesture). The object detection and tracking device 100 can apply a trained machine learning process (such as a trained neural network) to the multiple data points to determine the multiple action points. For example, the neural network can be trained using supervised learning based on elements of the image data that characterize the hand and the corresponding action points. The object detection and tracking device 100 can execute the trained neural network to ingest multiple data points and output multiple action points.

[0076] Based on the multiple action points and the captured images, the object detection and tracking device 100 can determine the user's gesture. For example, the object detection and tracking device 100 can capture an image of an object in the FOV 604, and can adapt the multiple action points to the image of the object to generate tracking points for the object. For example, the tracking points can be points in a 3-D space (such as the FOV 604) that the hand 902 is expected to span when making a gesture. In some instances, the object detection and tracking device 100 can determine multiple action points for multiple captured images, and can determine the gesture based on the tracking points generated for each of the captured images. For example, the object detection and tracking device 100 can capture a second image of the object in the FOV 604, and can adapt the multiple action points to the object in the second image to generate additional tracking points. The object detection and tracking device 100 can determine the gesture based on the tracking points and the additional tracking points (for example, the gesture can be the movement of the object from one location to another in the virtual environment). The object detection and tracking device 100 can use such a sequence of tracking points to identify the gesture and receive user input corresponding to the gesture. In some implementations, the object detection and tracking device 100 can save the sequence of tracking points crossed during a gesture in a lookup table (eg, lookup table 132C), as described above with reference to FIG. Figure 1 The object detection and tracking device 100 may determine the gesture made by the user based on the sequence of tracking points stored in the lookup table 132C.

[0077] In some examples, the object detection and tracking device 100 trains a machine learning process based on the image and the corresponding action points of the object within the image. For example, the machine learning process can be trained using supervised learning, where the corresponding machine learning model ingests elements of the image and the corresponding action points and generates elements of the output data. In addition, and during training, the object detection and tracking device 100 can determine one or more losses based on the generated output data. When one or more losses are within a threshold (e.g., less than a threshold), the object detection and tracking device 100 can determine that the training is complete. For example, the object detection and tracking device 100 can continue to train the machine learning process until one or more losses are each below a corresponding threshold.

[0078] In some examples, the object detection and tracking device 100 may generate a third request for the user to perform one or more gestures to train and / or optimize the machine learning process to better detect and track the hand 902. The object detection and tracking device 100 may generate a third request to the user and display the third request within the virtual environment of the system. For example, the object detection and tracking device 100 may request the user to make a gesture (e.g., a gesture of rotating the hand 902) using the hand 902 to make one or more shapes, such as letters, numbers (e.g., a figure of 8), or symbols. The object detection and tracking device 100 may then detect the execution of the gesture from the user in response to displaying the third request. The object detection and tracking device 100 may identify one or more points of the hand 902 that participate in the motion during the requested gesture, and adjust multiple action points based on the identified one or more points of the hand 902 of the requested gesture. The object detection and tracking device 100 may also identify a sequence of points in the 3-D space spanned by the hand 902 as described herein, and use such a sequence of points to update the sequence of tracking points stored in the lookup table 132C.

[0079] In some examples, the object detection and tracking device 100 may obtain hand attribute data from the system memory 130, which may include multiple attributes, such as palm lines, palm contours, shapes, fingernail sizes, fingernail shapes, colors, multi-point hand contour geometry, and one or more identification markers. The object detection and tracking device 100 may also obtain object data (which may be insertion angle and direction data) and user field of view data (which may represent at least one view within an angular range based on the user's field of view). The object detection and tracking device 100 may generate characteristics of the user's hand based on the hand attribute data and the user field of view data, and provide the generated characteristics to train, for example, a convolutional neural network (CNN) process, a deep learning process, a decision tree process, a support vector machine process, or any other suitable machine learning process.

[0080] In one specific implementation, the object detection and tracking device 100 may apply one or more data processing techniques to the hand attribute data and the user field of view data to determine one or more parameters corresponding to the user's hand. For example, the object detection and tracking device 100 may use the user field of view data to identify the core area of ​​the user's hand in the placement area. The object detection and tracking device 100 may then finely characterize the size, shape, position, and orientation of the hand in the placement area based on the hand attribute data. For example, and as described herein, the hand attribute data may characterize one or more of palm lines, palm contours, shapes, fingernail sizes, fingernail shapes, colors, multi-point hand contour geometries, and one or more identification marks of the hand. Based on the hand attribute data, the object detection and tracking device 100 may determine hand characteristic data of the hand, which may include one or more of the shape of the user's hand, the position of the user's hand, and the orientation of the user's hand in the placement area. The object detection and tracking device 100 may output hand characteristic data (e.g., including characteristics of the size, shape, position, and orientation of the hand), and may apply one or more data processing techniques and / or related techniques to the hand characteristic data and the insertion angle and direction data to generate object data. For example, the object detection and tracking device 100 may associate the insertion angle and direction data with the position and orientation of the hand (e.g., include in the hand characteristic data) to generate the object data. In one example, the object data may characterize a plurality of values ​​representing the position, orientation, and direction of the object inserted into the placement area. The object data may further characterize whether the object is a user's hand, which may be subsequently tracked as described herein.

[0081] In addition, the object detection and tracking device 100 may apply one or more trained machine learning processes (such as a trained CNN process) to the object data and the hand attribute data to generate detection output data. The detection output data may identify an object (e.g., a hand) inserted into the placement area of ​​the hybrid environment (as described herein with reference to Figures 1 to 5B and Fig.10 For example, the object may be tracked based on the detection output data.

[0082] By way of example, the object detection and tracking device 100 may train the CNN process against feature values ​​generated or obtained from a training data set including historical object data and hand attribute data, and the object detection and tracking device 100 may calculate one or more losses to determine whether the CNN process has converged. In some instances, the object detection and tracking device 100 may determine one or more of a triplet loss, a regression loss, and a classification loss (e.g., a cross entropy loss), etc. based on one or more of the detection output data and the object data. For example, the object detection and tracking device 100 may execute a sigmoid function that operates on the detection output data. In addition, the object detection and tracking device 100 may provide the output generated by the executed sigmoid function as feedback to the training process, for example to encourage more zeros and ones from the generated output.

[0083] The object detection and tracking device 100 may also calculate the classification loss based on the detection output data and the object data. In addition, the object detection and tracking device 100 may provide the classification loss and the triplet loss as feedback to the training process. The object detection and tracking device 100 may further determine whether one or more of the calculated losses meet the corresponding threshold to determine whether the training process has converged and whether the trained CNN process is available for deployment. For example, the object detection and tracking device 100 may compare each calculated loss with its corresponding threshold to determine whether each calculated loss meets or exceeds its corresponding threshold. In some examples, when each of the calculated losses meets or exceeds their corresponding thresholds, the object detection and tracking device 100 determines the convergence of the training process, and the training process is completed. In addition, the object detection and tracking device 100 generates training loss data characterizing the calculated losses, and stores the training loss data in the system memory 130.

[0084] In some examples, the object detection and tracking device 100 may perform additional operations to determine whether the CNN process is sufficiently trained. For example, the object detection and tracking device 100 may input elements of feature values ​​generated or obtained from a validation data set including validation object data and hand attribute data into the CNN process to generate additional detection output data. Based on the detection output data, the object detection and tracking device 100 calculates one or more losses (as described above with reference to FIG. 1 ) that characterize errors in the detection of an object (e.g., a hand). Figures 1 to 5B and Fig.10 If the calculated losses indicate that the CNN is not sufficiently trained (e.g., one or more of the calculated losses do not satisfy their corresponding thresholds), the object detection and tracking device 100 continues to train the CNN (e.g., using interpolated angle and direction data, user field of view data, and hand attribute data).

[0085] Although the object detection and tracking device 100 trains the CNN process as described, one or more of any suitable processing devices associated with the object detection and tracking device 100 can train the CNN process as described herein. For example, one or more servers, such as one or more cloud-based servers, can train the CNN process. In some examples, one or more processors (e.g., CPU, GPU) of a distributed or cloud-based computing cluster can train the CNN process. In some specific implementations, the CNN process can be trained by another processing device associated with the object detection and tracking device 100, and the other processing device stores the configuration parameters, hyperparameters, and / or weights associated with the trained CNN process in a data repository via a network (e.g., the Internet). In addition, the object detection and tracking device 100 can obtain the stored configuration parameters, hyperparameters, and / or weights via the network and store them in the instruction memory 132 (e.g., in the detection unit 132A). The object detection and tracking device 100 can then establish the CNN process based on the configuration parameters, hyperparameters, and / or weights stored in the instruction memory 132A.

[0086] In one example, the object detection and tracking device 100 implements machine learning techniques for object detection and tracking the movement of an object (such as a user's hand) in a placement area of ​​a hybrid environment. For example, as described above, the object detection and tracking device 100 can generate hand attribute data, which can be further processed in conjunction with the insertion angle and direction data to generate object data that can indicate whether the detected hand corresponds to the user's hand that should be tracked. The object detection and tracking device 100 can utilize the object data in conjunction with the hand attribute data and / or the insertion angle and direction data to generate detection output data identifying the user's hand in the placement area. The object detection and tracking device 100 can track the user's hand in the placement area of ​​the hybrid environment based on the detection output data.

[0087] In some examples, the object detection and tracking device 100 can generate action points using data points and multi-dimensional model data. For example, the data points can represent points of an image of a tracked hand / object, and the multi-dimensional model data can represent a model that defines the shape of the object. The object detection and tracking device 100 can apply one or more data processing techniques to generate action points, which in some examples can represent expected points of an object / hand in 3-D space, where these points can move when certain gestures are made by the object / hand (as described above with reference to FIG. Figures 6 to 9 and Fig.11The object detection and tracking device 100 may also utilize the data points, the multi-dimensional model data, and the action points to generate object orientation data. For example, the object detection and tracking device 100 may apply one or more data processing and / or related techniques to the action points and the multi-dimensional model data to identify the orientation of the object / hand in 3-D space (as described herein with reference to Figures 6 to 9 and Fig.11 described above).

[0088] In one implementation, the object detection and tracking device 100 may apply one or more trained machine learning processes (such as a trained CNN process) to the action points, object orientation data, gesture data, multi-dimensional model data, and / or data points to generate tracking point data. For example, the gesture data may represent a gesture performed by a user in response to a request to perform a gesture, such as a rotation, a graphic of one or more letters, numbers, or symbols (as described above with reference to FIG. Figures 6 to 9 and Fig.11 The object detection and tracking device 100 may apply one or more data processing and / or related techniques to the action points, the object orientation data, the gesture data, and the multi-dimensional model data to generate tracking point data for the hand / object corresponding to the gesture performed by the user. For example, the tracking point data may include a sequence of points in a 3-D space spanned by the hand / object during the gesture of the object / hand (as described herein with reference to Figure 6-Figure 9 and Fig.11 described above).

[0089] In addition, the object detection and tracking device 100 may perform operations to generate a lookup table. For example, in some examples, the object detection and tracking device 100 may apply one or more data processing and / or correlation techniques to the multi-dimensional model data, the object orientation data, and the tracking point data to generate the lookup table 132C. The lookup table 132C may include data representing one or more gestures and a sequence of tracking points corresponding to each gesture (e.g., as described herein with reference to FIG. 1 ). Figures 6 to 9 and Fig.11 described above).

[0090] In some examples, the object detection and tracking device 100 may perform operations to train a CNN process to generate a lookup table 132C using multidimensional model data, object orientation data, and tracking point data. For example, the object detection and tracking device 100 may generate features based on the historical lookup table, multidimensional model data, object orientation data, and tracking point data, and the generated features may be input to the CNN process for training (e.g., supervised learning). The CNN may be trained to generate output data representing the historical lookup table based on features generated from the multidimensional model data, object orientation data, and tracking point data.

[0091] The object detection and tracking device 100 may calculate one or more losses to determine whether the CNN has converged. For example, the object detection and tracking device 100 may determine one or more of a triplet loss, a regression loss, and a classification loss (e.g., a cross entropy loss), etc. based on the tracking point data and one or more of the lookup tables 132C. For example, the object detection and tracking device 100 may execute a sigmoid function that operates on the tracking point data 1309. The sigmoid function may be used as an amplifier to enhance the spoofing response generated from the CNN process. In addition, the object detection and tracking device 100 may provide the output generated by the executed sigmoid function to the CNN process as feedback to encourage more zeros and / or ones from the generated output.

[0092] The object detection and tracking device 100 can calculate the classification loss based on the tracking point data and the lookup table 132C. In addition, the object detection and tracking device 100 can provide the classification loss and the triplet loss as feedback to the CNN process. The object detection and tracking device 100 can further determine whether one or more of the calculated losses meet the corresponding threshold to determine whether the CNN process has converged. For example, the object detection and tracking device 100 can compare each calculated loss with its corresponding threshold to determine whether each calculated loss meets or exceeds its corresponding threshold. In some examples, when each of the calculated losses meets or exceeds their corresponding thresholds, the object detection and tracking device 100 determines that the CNN process has converged and the training is complete. In addition, the object detection and tracking device 100 generates training loss data that characterizes the calculated losses, and stores the training loss data in the system memory 130.

[0093] In some examples, the object detection and tracking device 100 may provide additional data points representing the validation data set to the initially trained CNN to determine whether the initially trained CNN is sufficiently trained. For example, the object detection and tracking device 100 may apply the initially trained CNN to the data points representing the validation data set to generate action points (e.g., an improved set of action points). For example, the object detection and tracking device 100 may apply the initially trained CNN as described herein to the action points to generate additional tracking point data. Based on the tracking point data, the object detection and tracking device 100 may calculate one or more losses representing errors in detection of the hand / object (e.g., as described herein with reference to Figures 6 to 9 and 11). If the calculated losses indicate that the CNN is not sufficiently trained (e.g., one or more of the calculated losses do not satisfy their corresponding thresholds), the object detection and tracking device 100 continues to train the CNN (e.g., with additional multi-dimensional model data, gesture data, object orientation data, and tracking point data).

[0094] Although the object detection and tracking device 100 trains the CNN process as described, one or more of any suitable processing devices associated with the object detection and tracking device 100 can train the CNN process as described herein. For example, one or more servers, such as one or more cloud-based servers, can train the CNN. In some examples, one or more processors (e.g., CPU, GPU) of a distributed or cloud-based computing cluster can train the CNN process. In some specific implementations, the CNN process can be trained by another processing device associated with the object detection and tracking device 100, and the other processing device stores the configuration parameters, hyperparameters, and / or weights associated with the trained CNN process in a data repository via a network (e.g., the Internet). In addition, the object detection and tracking device 100 can obtain the stored configuration parameters, hyperparameters, and / or weights via the network and store them in the instruction memory 132 (e.g., in the detection unit 132A and / or the tracking unit 132B). The object detection and tracking device 100 can then establish a CNN based on the configuration parameters, hyperparameters, and / or weights stored in the instruction memory 132A and / or the tracking unit 132B.

[0095] In one example, the object detection and tracking device 100 implements machine learning techniques for detecting and tracking the movement of a user's object / hand in a placement area. As described above, the object detection and tracking device 100 may generate action points and object orientation data for an object (e.g., a hand), wherein the action points may be processed in combination with gesture data, multidimensional model data, and / or data points to generate tracking point data. The tracking point data may represent a sequence of points in a 3-D space spanned by the hand / object during a gesture of the object / hand. The object detection and tracking device 100 may generate a lookup table 132C based on the tracking point data as described herein. Thus, the object detection and tracking device 100 may be operable to detect and track a user's object (e.g., a hand) in a placement area of ​​a mixed environment.

[0096] Fig.10 is a flow chart of an exemplary process 1000 for detecting and tracking an object according to some implementations. The process 1000 may be performed by one or more processors executing instructions locally at a computing device, such as by Figure 1 The process 1000 may be performed by one or more of the camera processor 114, the CPU 116, and the GPU 118 of the object detection and tracking device 100. Thus, the various operations of the process 1000 may be represented by executable instructions stored in a storage medium of one or more computing platforms, such as the instruction memory 132 of the object detection and tracking device 100.

[0097] At block 1002, the object detection and tracking device 100 detects an object in a placement area of ​​a mixed environment. For example, the CPU 116 of the object detection and tracking device 100 may execute instructions stored in the detection unit 132A to detect placement of an object in the placement area of ​​the mixed environment. For example, the placement area may be the placement area 308 (as described above with reference to FIG. 1 ). Figure 3 described above).

[0098] At step 1004, in response to the detection, the object detection and tracking device 100 receives at least one parameter corresponding to the user's hand. For example, the at least one parameter may be an insertion direction, an insertion angle, a palm line, a palm contour, a shape, a fingernail size, a fingernail shape, a color, a multi-point hand contour geometry, or one or more identification marks (as described above with reference to FIG. Figures 3 to 5B In some implementations, the camera 115 can receive at least one parameter corresponding to the user's hand through one or more captured images or video sequences of the user's hand / object.

[0099] At step 1006, the object detection and tracking device 100 registers the object based on at least one parameter. For example, the object detection and tracking device 100 may generate profile data to register the object with the user, and may store the profile data in the system memory 130 and / or the instruction memory 132. The profile data may indicate that the object corresponding to the parameter is the object of the user to be used to provide gesture input.

[0100] At step 1008, the object detection and tracking device 100 tracks the movement of the object based on the registration. For example, the camera 115 may monitor the movement of the object corresponding to the user (registered at step 1006). The camera 115 may provide one or more images and / or video sequences to the CPU 116, which may execute instructions stored in the detection unit 132A and / or the tracking unit 132B to track the movement of the registered object, as described herein.

[0101] Fig.11 is a flow chart of an exemplary process 1100 for tracking an object based on a machine learning process according to some specific implementations. The process 1100 can be performed by one or more processors executing instructions locally at a computing device, such as by Figure 1 The process 1100 may be performed by one or more of the camera processor 114, the CPU 116, and the GPU 118 of the object detection and tracking device 100. Therefore, the various operations of the process 1100 may be represented by executable instructions stored in a storage medium of one or more computing platforms (such as the instruction memory 132 of the object detection and tracking device 100).

[0102] At block 1102, the object detection and tracking device 100 captures at least one image of an object in the real environment of the mixed environment. For example, the camera 115 of the object detection and tracking device 100 may capture at least one image of an object in the real environment of the mixed environment of the XR system (as described above with reference to FIG. Figure 1 , Figures 6 to 9 The camera 115 may communicate the captured image to the CPU 116, which may perform one or more operations on the image data based on instructions stored in the instruction memory 132.

[0103] Furthermore, at step 1104, the object detection and tracking device 100 generates a plurality of data points of the object based on the at least one image. For example, the CPU 116 may execute instructions stored in the detection unit 132A and / or the tracking unit 132B to generate a plurality of data points of the object based on the at least one image (as described above with reference to FIG. 1 ). Fig. 9 The CPU 116 may store the plurality of data points for the object in the instruction memory 132 and / or the system memory 130.

[0104] At block 1106, the object detection and tracking device 100 generates a multi-dimensional model of the object based on the plurality of data points. For example, the CPU 116 may execute one or more instructions stored in the detection unit 132A and / or the tracking unit 132B to generate a 3D model of the object (as described above with reference to FIG. Fig. 9 The CPU 116 may store the multi-dimensional model of the object in the instruction memory 132 and / or the system memory 130 for further operations.

[0105] Proceeding to step 1108, the object detection and tracking device 100 may generate a plurality of action points based on the multi-dimensional model of the object. For example, the CPU 116 may execute one or more instructions stored in the detection unit 132A and / or the tracking unit 132B to generate a plurality of action points (as described above with reference to FIG. Fig. 9 The plurality of action points may correspond to, for example, joints of a user's hand. The CPU 116 may store the plurality of action points of the object in the instruction memory 132 and / or the system memory 130 for further operation.

[0106] At step 1112, the object detection and tracking device 100 may track the movement of the object in the virtual environment of the hybrid environment based on the plurality of action points. For example, the CPU 116 may execute one or more instructions stored in the tracking unit 132B to track the movement of the object in the virtual environment of the hybrid environment (as described above with reference to FIG. Fig. 9 The CPU 116 may look up a sequence of tracking points and corresponding gestures stored in the lookup table 132C to track the movement and determine a gesture corresponding to the object based on the movement.

[0107] Specific implementation examples are further described in the following numbered clauses:

[0108] 1. A device, comprising:

[0109] a non-transitory machine-readable storage medium storing instructions; and

[0110] at least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to execute instructions to:

[0111] Receive image data from the camera;

[0112] detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment;

[0113] responsive to said detecting, determining a value of at least one parameter of said object;

[0114] generating profile data based on the at least one parameter value, the profile data registering the object with a user; and

[0115] Movement of the object within the hybrid environment is tracked based on the profile data.

[0116] 2. An apparatus according to clause 1, wherein the at least one processor is configured to execute the instructions to:

[0117] generating a request to place the object in the placement area of ​​the hybrid environment; and

[0118] The request is displayed within the virtual environment.

[0119] 3. An apparatus according to any of clauses 1 to 2, wherein the at least one parameter comprises one or more of the following:

[0120] an insertion angle into the placement area; and

[0121] Insertion direction into the placement area.

[0122] 4. An apparatus according to any of clauses 1 to 3, wherein the processor is further configured to execute the instructions to:

[0123] determining that the at least one parameter value is within a range, the range of values ​​being based on a field of view of the user; and

[0124] The profile data is generated based on determining that the at least one parameter value is within the range of values.

[0125] 5. An apparatus according to any one of clauses 1 to 4, the processor being further configured to execute the instructions to:

[0126] determining that the at least one parameter value is not within a range of values, the range of values ​​being based on a field of view of the user; and

[0127] Based on determining that the at least one parameter value is not within the range of values, displaying within the hybrid environment a request by the user to insert the object into the drop area of ​​the hybrid environment.

[0128] 6. An apparatus according to any of clauses 1 to 5, wherein the at least one processor is further configured to execute the instructions to:

[0129] generating a request by the user to insert the object into the placement area of ​​the hybrid environment at least one of a direction or an angle;

[0130] displaying the request within the hybrid environment; and

[0131] The object inserted into the placement area of ​​the hybrid environment is detected according to the at least one of the direction or the angle.

[0132] 7. An apparatus according to any of clauses 1 to 6, wherein the at least one processor is further configured to execute the instructions to:

[0133] generating a guidance image identifying one or more insertion angles for inserting the object into the placement area of ​​the hybrid environment; and

[0134] The guidance image is displayed within the mixed environment.

[0135] 8. An apparatus according to any one of clauses 1 to 7, wherein the object is a hand of the user.

[0136] 9. An apparatus according to clause 8, wherein the at least one parameter characterizes one or more properties of the hand of the user.

[0137] 10. The apparatus of clause 9, wherein the one or more attributes of the hand of the user include one or more of the following:

[0138] Palm lines;

[0139] palm contour;

[0140] shape;

[0141] Fingernail size;

[0142] Fingernail shape;

[0143] color;

[0144] Multi-point hand outline geometry; and

[0145] One or more identification tags.

[0146] 11. An apparatus according to any of clauses 1 to 10, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0147] 12. An apparatus according to any of clauses 1 to 11, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0148] 13. A device, comprising:

[0149] a non-transitory machine-readable storage medium storing instructions; and

[0150] at least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to execute the instructions to:

[0151] capturing at least one image of the object in a real environment of the mixed environment;

[0152] generating a plurality of data points for the object based on the at least one image;

[0153] generating a multi-dimensional model of the object based on the plurality of data points;

[0154] generating a plurality of action points based on the multi-dimensional model of the object; and

[0155] Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

[0156] 14. An apparatus according to clause 13, wherein the at least one processor is configured to execute the instructions to:

[0157] determining an orientation of the object based on the plurality of action points; and

[0158] A virtual image of the object is displayed in the virtual environment based on the determined orientation.

[0159] 15. An apparatus according to any of clauses 13 to 14, wherein the object is one of a hand of the user and a prosthesis of the user.

[0160] 16. An apparatus according to any of clauses 13 to 16, wherein the at least one processor is further configured to execute the instructions to:

[0161] generating a first request for placing the object in the hybrid environment;

[0162] displaying the first request within the virtual environment;

[0163] In response to the first request, detecting the object in the virtual environment;

[0164] generating a second request for confirmation from the user that the object is a hand of the user;

[0165] displaying the second request within the virtual environment;

[0166] In response to the second request, detecting the confirmation; generating a third request to perform at least one gesture on the user; and

[0167] The third request is displayed within the virtual environment.

[0168] 17. The apparatus of clause 16, wherein the at least one gesture comprises one or a combination of:

[0169] rotation of the hand of the user; and

[0170] A graphic of one or more shapes of the hand of the user.

[0171] 18. An apparatus according to any of clauses 16 to 17, wherein the at least one processor is further configured to execute the instructions to:

[0172] Performance of the at least one gesture is detected in response to displaying the third request.

[0173] 19. The apparatus of clause 118, wherein generating the plurality of action points is based at least in part on the at least one gesture.

[0174] 20. The apparatus of claim 19, wherein the at least one processor is further configured to execute the instructions to:

[0175] At least a portion of the plurality of action points is estimated based on the at least one gesture.

[0176] 21. An apparatus according to any of clauses 13 to 20, wherein generating the plurality of tracking points is based on a lookup table, wherein the lookup table includes a first plurality of tracking points corresponding to each gesture of a plurality of gestures.

[0177] 22. The apparatus of claim 21, wherein the at least one processor is further configured to execute the instructions to:

[0178] determining a second plurality of tracking points based on the tracked movement of the object; and

[0179] A gesture is detected based on the second plurality of tracking points and the first plurality of tracking points corresponding to each of the gestures.

[0180] 23. An apparatus according to any of clauses 13 to 22, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0181] 24. An apparatus according to any of clauses 13 to 23, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0182] 25. A method comprising:

[0183] Receive image data from the camera;

[0184] detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment;

[0185] responsive to said detecting, determining a value of at least one parameter of said object;

[0186] generating profile data based on the at least one parameter value, the profile data registering the object with a user; and

[0187] Movement of the object within the hybrid environment is tracked based on the profile data.

[0188] 26. The method according to clause 25, further comprising:

[0189] generating a request for placing the object in the placement area of ​​the hybrid environment, wherein the hybrid environment includes a real environment and a virtual environment; and

[0190] The request is provided for display within the virtual environment.

[0191] 27. A method according to any of clauses 25 to 26, wherein the at least one parameter comprises one or more of the following:

[0192] an insertion angle into the placement area; and

[0193] Insertion direction into the placement area.

[0194] 28. A method according to any of clauses 25 to 27, wherein registering the object based on the at least one parameter comprises:

[0195] The at least one parameter is determined to be within a range, wherein the range is based on a field of view of a user.

[0196] 29. A method according to any of clauses 25 to 28, wherein registering the object based on the at least one parameter comprises:

[0197] determining that the at least one parameter is not within a range, wherein the range is based on a field of view of a user; and

[0198] The user is requested to re-enter the object in a certain direction.

[0199] 30. The method according to any one of clauses 25 to 29, further comprising:

[0200] generating a request by the user to input the hand of the user in the placement area of ​​the hybrid environment at at least one of a direction and an angle;

[0201] providing the request for display within the hybrid environment; and

[0202] The hand of the user is detected within the placement area of ​​the hybrid environment at the at least one of the direction and the angle.

[0203] 31. The method of any of clauses 25 to 30, further comprising generating an insertion guide image identifying one or more insertion angles for inserting the object into the placement area of ​​the hybrid environment.

[0204] 32. A method according to any one of clauses 25 to 31, wherein the object is a hand of a user.

[0205] 33. The method of clause 32, wherein the at least one parameter characterizes one or more properties of the hand.

[0206] 34. The method of clause 33, wherein the attributes of the hand include one or more of:

[0207] Palm lines;

[0208] palm contour;

[0209] shape;

[0210] Fingernail size;

[0211] Fingernail shape;

[0212] color;

[0213] Multi-point hand outline geometry; and

[0214] One or more identification tags.

[0215] 35. A method according to any of clauses 25 to 34, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0216] 36. A method according to any of clauses 25 to 35, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0217] 37. A method comprising:

[0218] capturing at least one image of the object in a real environment of the mixed environment;

[0219] generating a plurality of data points for the object based on the at least one image;

[0220] generating a multi-dimensional model of the object based on the plurality of data points;

[0221] generating a plurality of action points based on the multi-dimensional model of the object; and

[0222] Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

[0223] 38. The method according to clause 37, further comprising:

[0224] determining an orientation of the object based on the plurality of action points; and

[0225] A virtual image of the object is displayed in the virtual environment based on the determined orientation.

[0226] 39. A method according to any of clauses 37 to 38, wherein the object is one of a hand of the user and a prosthesis of the user.

[0227] 40. The method according to any one of clauses 37 to 39, further comprising:

[0228] generating a first request for placing the object in the hybrid environment;

[0229] displaying the first request within the virtual environment;

[0230] In response to the first request, detecting the object in the virtual environment;

[0231] generating a second request for confirmation from the user that the object is a hand of the user;

[0232] displaying the second request within the virtual environment;

[0233] detecting the confirmation in response to the second request;

[0234] generating a third request to the user to perform at least one gesture; and

[0235] The third request is displayed within the virtual environment.

[0236] 41. The method of clause 40, wherein the at least one gesture comprises one or a combination of:

[0237] rotation of the hand of the user; and

[0238] A graphic of one or more shapes of the hand of the user.

[0239] 42. The method according to any one of clauses 40 to 41, further comprising:

[0240] Performance of the at least one gesture is detected in response to displaying the third request.

[0241] 43. The method of clause 42, wherein generating the plurality of action points is based at least in part on detecting the performance of the at least one gesture.

[0242] 44. A method according to any of clauses 40 to 43, comprising estimating at least a portion of the plurality of action points based on the at least one gesture.

[0243] 45. A method according to any of clauses 37 to 44, wherein generating the plurality of tracking points is based on a lookup table, wherein the lookup table includes a first plurality of tracking points corresponding to each gesture of a plurality of gestures.

[0244] 46. ​​The method according to clause 45, further comprising:

[0245] determining a second plurality of tracking points based on the tracked movement of the object; and

[0246] A gesture is detected based on the second plurality of tracking points and the first plurality of tracking points corresponding to each of the gestures.

[0247] 47. A method according to any of clauses 37 to 46, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0248] 48. A method according to any of clauses 37 to 47, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0249] 49. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

[0250] Receive image data from the camera;

[0251] detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment;

[0252] responsive to said detecting, determining a value of at least one parameter of said object;

[0253] generating profile data based on the at least one parameter value, the profile data registering the object with a user; and

[0254] Movement of the object within the hybrid environment is tracked based on the profile data.

[0255] 50. The non-transitory machine-readable storage medium of clause 49, wherein the operations further comprise:

[0256] generating a request to place the object in the placement area of ​​the hybrid environment; and

[0257] The request is displayed within the virtual environment.

[0258] 51. The non-transitory machine-readable storage medium of any of clauses 49 to 50, wherein the at least one parameter comprises one or more of:

[0259] an insertion angle into the placement area; and

[0260] Insertion direction into the placement area.

[0261] 52. The non-transitory machine-readable storage medium of any one of clauses 49 to 51, wherein the operations further comprise:

[0262] determining that the at least one parameter value is within a range of values, the range of values ​​being based on a field of view of the user; and

[0263] The profile data is generated based on determining that the at least one parameter value is within the range of values.

[0264] 53. The non-transitory machine-readable storage medium of any one of clauses 49 to 52, wherein the operations further comprise:

[0265] determining that the at least one parameter value is not within a range of values, the range of values ​​being based on a field of view of the user; and

[0266] Based on determining that the at least one parameter value is not within the range of values, displaying within the hybrid environment a request by the user to insert the object into the drop area of ​​the hybrid environment.

[0267] 54. The non-transitory machine-readable storage medium of any one of clauses 49 to 53, wherein the operations further comprise:

[0268] generating a request by the user to insert the object into the placement area of ​​the hybrid environment according to at least one of a direction or an angle;

[0269] displaying the request within the hybrid environment; and

[0270] The object inserted into the placement area of ​​the hybrid environment is detected according to the at least one of the direction or the angle.

[0271] 55. The non-transitory machine-readable storage medium of any one of clauses 49 to 54, wherein the operations further comprise:

[0272] generating a guidance image identifying one or more insertion angles for inserting the object into the placement area of ​​the hybrid environment; and

[0273] The guidance image is displayed within the mixed environment.

[0274] 56. The non-transitory machine-readable storage medium of any one of clauses 49 to 57, wherein the object is a hand of the user.

[0275] 57. A non-transitory machine-readable storage medium as described in clause 57, wherein the at least one parameter value characterizes one or more attributes of the hand of the user.

[0276] 58. The non-transitory machine-readable storage medium of clause 57, wherein the one or more attributes of the hand include one or more of:

[0277] Palm lines;

[0278] palm contour;

[0279] shape;

[0280] Fingernail size;

[0281] Fingernail shape;

[0282] color;

[0283] Multi-point hand outline geometry; and

[0284] One or more identification tags.

[0285] 59. The non-transitory machine-readable storage medium of any of clauses 49 to 58, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0286] 60. The non-transitory machine-readable storage medium of any of clauses 49 to 59, wherein the hybrid environment comprises an environment of a virtual reality (VR) system.

[0287] 61. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

[0288] capturing at least one image of the object in a real environment of the mixed environment;

[0289] generating a plurality of data points for the object based on the at least one image;

[0290] generating a multi-dimensional model of the object based on the plurality of data points;

[0291] generating a plurality of action points based on the multi-dimensional model of the object; and

[0292] Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

[0293] 62. The non-transitory machine-readable storage medium of clause 61, wherein the operations further comprise:

[0294] determining an orientation of the object based on the plurality of action points; and

[0295] A virtual image of the object is displayed in the virtual environment based on the determined orientation.

[0296] 63. The non-transitory machine-readable storage medium of any one of clauses 61 to 62, wherein the object is one of a hand of the user and a prosthesis of the user.

[0297] 64. The non-transitory machine-readable storage medium of any of clauses 61 to 63, wherein the operations further comprise:

[0298] displaying the first request within the virtual environment;

[0299] In response to the first request, detecting the object in the virtual environment;

[0300] generating a second request for confirmation from the user that the object is a hand of the user;

[0301] displaying the second request within the virtual environment;

[0302] detecting the confirmation in response to the second request;

[0303] generating a third request to the user to perform at least one gesture; and

[0304] The third request is displayed within the virtual environment.

[0305] 65. The non-transitory machine-readable storage medium of clause 64, wherein the at least one gesture comprises one or a combination of:

[0306] rotation of the hand of the user; and

[0307] A graphic of one or more shapes of the hand of the user.

[0308] 66. The non-transitory machine-readable storage medium of any of clauses 64 to 65, wherein the operations further comprise:

[0309] Performance of the at least one gesture is detected in response to displaying the third request.

[0310] 67. The non-transitory machine-readable storage medium of clause 66, wherein generating the plurality of action points is based at least in part on detecting the performance of the at least one gesture.

[0311] 68. The non-transitory machine-readable storage medium of any of clauses 64 to 67, wherein the operations further comprise estimating at least a portion of the plurality of action points based on the at least one gesture.

[0312] 69. The non-transitory machine-readable storage medium of clause 67, wherein generating the plurality of tracking points is based on a lookup table, wherein the lookup table includes a first plurality of tracking points corresponding to each gesture of a plurality of gestures.

[0313] 70. The non-transitory machine-readable storage medium of any of clauses 67 to 69, wherein the operations further comprise:

[0314] determining a second plurality of tracking points based on the tracked movement of the object; and

[0315] A gesture is detected based on the second plurality of tracking points and the first plurality of tracking points corresponding to each of the gestures.

[0316] 71. The non-transitory machine-readable storage medium of any of clauses 61 to 70, wherein the hybrid environment comprises an environment of an extended reality (XR) system.

[0317] 72. The non-transitory machine-readable storage medium of any one of clauses 61 to 71, wherein the hybrid environment comprises an environment of a virtual reality (VR) system.

[0318] 73. An object detection and tracking device comprising:

[0319] A component for receiving image data from a camera;

[0320] means for detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment;

[0321] means for determining a value of at least one parameter of said object in response to said detecting;

[0322] means for generating profile data based on the at least one parameter value, the profile data registering the object with a user; and

[0323] Means for tracking movement of the object within the hybrid environment based on the profile data.

[0324] 74. The object detection and tracking device of clause 73, further comprising:

[0325] means for generating a request for placing the object in the placement area of ​​the hybrid environment; and

[0326] Means for displaying the request within the virtual environment.

[0327] 75. An object detection and tracking apparatus according to any of clauses 73 to 74, wherein the at least one parameter comprises one or more of the following:

[0328] an insertion angle into the placement area; and

[0329] Insertion direction into the placement area.

[0330] 76. An object detection and tracking device according to any of clauses 73 to 75, further comprising:

[0331] means for determining that the at least one parameter value is within a range of values, the range of values ​​being based on a field of view of the user; and

[0332] Means for generating the profile data based on determining that the at least one parameter value is within the range of values.

[0333] 77. An object detection and tracking device according to any of clauses 73 to 76, further comprising:

[0334] means for determining that the at least one parameter value is not within a range of values, the range of values ​​being based on a field of view of the user; and

[0335] Means for displaying, within the hybrid environment, a request by the user to insert the object into the drop area of ​​the hybrid environment based on determining that the at least one parameter value is not within the range of values.

[0336] 78. An object detection and tracking device according to any of clauses 73 to 77, further comprising:

[0337] means for generating a request by the user to insert the object into the placement area of ​​the hybrid environment according to at least one of a direction or an angle;

[0338] means for displaying the request within the hybrid environment; and

[0339] Means for detecting the object inserted into the placement area of ​​the hybrid environment based on the at least one of the direction or the angle.

[0340] 79. An object detection and tracking device according to any of clauses 73 to 78, further comprising:

[0341] means for generating a guide image identifying one or more insertion angles for inserting the object into the placement area of ​​the hybrid environment; and

[0342] Means for displaying the guidance image within the hybrid environment.

[0343] 80. An object detection and tracking device as described in any of clauses 73 to 79, wherein the object is a hand of the user.

[0344] 81. An object detection and tracking device according to clause 80, wherein the at least one parameter value characterizes one or more properties of the hand of the user.

[0345] 82. An object detection and tracking device according to clause 80, wherein the attributes of the hand include one or more of the following:

[0346] Palm lines;

[0347] palm contour;

[0348] shape;

[0349] Fingernail size;

[0350] Fingernail shape;

[0351] color;

[0352] Multi-point hand outline geometry; and

[0353] One or more identification tags.

[0354] 83. An object detection and tracking device according to any of clauses 73 to 82, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0355] 84. An object detection and tracking device according to any of clauses 73 to 83, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0356] 85. An object detection and tracking device comprising:

[0357] means for capturing at least one image of the object in a real environment of the mixed environment;

[0358] means for generating a plurality of data points of the object based on the at least one image;

[0359] means for generating a multi-dimensional model of the object based on the plurality of data points;

[0360] means for generating a plurality of action points based on the multi-dimensional model of the object; and

[0361] Means for tracking movement of the object in a virtual environment of the hybrid environment based on the plurality of action points.

[0362] 86. The object detection and tracking device of clause 85, further comprising:

[0363] means for determining an orientation of the object based on the plurality of action points; and

[0364] Means for displaying a virtual image of the object in the virtual environment based on the determined orientation.

[0365] 87. An object detection and tracking device according to any of clauses 85 to 86, wherein the object is one of a hand of the user and a prosthesis of the user.

[0366] 88. An object detection and tracking device according to any one of clauses 85 to 87, further comprising:

[0367] means for generating a first request for placing the object in the hybrid environment;

[0368] means for displaying the first request within the virtual environment;

[0369] means for detecting the object in the virtual environment in response to the first request;

[0370] means for generating a second request for confirmation from a user that the object is a hand of the user;

[0371] means for displaying the second request within the virtual environment;

[0372] means for detecting said confirmation in response to said second request;

[0373] means for generating a third request to perform at least one gesture on the user; and

[0374] Means for displaying the third request within the virtual environment.

[0375] 89. An object detection and tracking device according to clause 88, wherein the at least one gesture comprises one or a combination of the following:

[0376] rotation of the hand of the user; and

[0377] A graphic of one or more shapes of the hand of the user.

[0378] 90. An object detection and tracking device according to any of clauses 88 to 89, further comprising:

[0379] Means for detecting performance of the at least one gesture in response to displaying the third request.

[0380] 91. An object detection and tracking device according to clause 90, wherein generating the plurality of action points is based at least in part on detecting the performance of the at least one gesture.

[0381] 92. An object detection and tracking device according to any of clauses 88 to 91, further comprising estimating at least a portion of the plurality of action points based on the at least one gesture.

[0382] 93. An object detection and tracking device according to any one of clauses 85 to 92, wherein generating the plurality of tracking points is based on a lookup table, wherein the lookup table includes a first plurality of tracking points corresponding to each gesture of a plurality of gestures.

[0383] 94. An object detection and tracking device according to any of clauses 85 to 93, further comprising:

[0384] means for determining a second plurality of tracking points based on the tracked movement of the object; and

[0385] Means for detecting a gesture based on the second plurality of tracking points and the first plurality of tracking points corresponding to each of the gestures.

[0386] 95. An object detection and tracking device according to any of clauses 85 to 94, wherein the mixed environment comprises an environment of an extended reality (XR) system.

[0387] 96. An object detection and tracking device according to any of clauses 85 to 95, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

[0388] Although the method described above refers to the illustrated flow chart, many other ways of performing the actions associated with the method may be used. For example, the order of some operations may be changed, and some embodiments may omit one or more operations in the described operations and / or include additional operations.

[0389] In addition, although the exemplary embodiments described herein are sometimes described with respect to an object detection and tracking device, the machine learning processes and the training of those machine learning processes can be implemented by one or more suitable devices. For example, in some examples, the object detection and tracking device can capture images or video sequences and can send the images to a distributed or cloud computing system. The distributed or cloud computing system can apply the trained machine learning processes described herein to track the movement of an object.

[0390] Additionally, the methods and systems described herein may be embodied at least in part in the form of computer-implemented processes and devices for practicing those processes. The disclosed methods may also be embodied at least in part in the form of a tangible non-transitory machine-readable storage medium encoded with computer program code. For example, the methods may be embodied in hardware, executable instructions (e.g., software) executed by a processor, or a combination of the two. The medium may include, for example, RAM, ROM, CD-ROM, DVD-ROM, BD-ROM, hard drive, flash memory, or any other non-transitory machine-readable storage medium. When the computer program code is loaded into a computer and executed by the computer, the computer becomes a device for practicing the method. The method may also be embodied at least in part in the form of a computer, the computer program code being loaded into or executed in the computer, making the computer a special-purpose computer for practicing the method. When implemented on a general-purpose processor, the computer program code segment configures the processor to create a specific logic circuit. The method may alternatively be embodied at least in part in a dedicated integrated circuit for executing the method.

Claims

1. A method, comprising: Receive image data from the camera; detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment; responsive to said detecting, determining a value of at least one parameter of said object; generating profile data based on the at least one parameter value, the profile data registering the object with a user; as well as Movement of the object within the hybrid environment is tracked based on the profile data.

2. The method according to claim 1, further comprising: generating a request for placing the object in the placement area of ​​the hybrid environment; as well as The request is displayed within the virtual environment.

3. The method of claim 1 , wherein the at least one parameter value comprises one or more of the following: an insertion angle into the placement area; and Insertion direction into the placement area.

4. The method according to claim 1, further comprising: determining that the at least one parameter value is within a range of values, the range of values ​​being based on a field of view of the user; as well as The profile data is generated based on determining that the at least one parameter value is within the range of values.

5. The method according to claim 1, further comprising: determining that the at least one parameter value is not within a range of values, the range of values ​​being based on a field of view of the user; as well as Based on determining that the at least one parameter value is not within the range of values, displaying within the hybrid environment a request by the user to insert the object into the drop area of ​​the hybrid environment.

6. The method according to claim 1, further comprising: generating a request by the user to insert the object into the placement area of ​​the hybrid environment according to at least one of a direction or an angle; displaying the request within the hybrid environment; as well as The object inserted into the placement area of ​​the hybrid environment is detected according to the at least one of the direction or the angle.

7. The method according to claim 1, further comprising: generating a guidance image identifying one or more insertion angles for inserting the object into the placement area of ​​the hybrid environment; as well as The guidance image is displayed within the mixed environment. The method of claim 1 , wherein the object is a hand of the user.

9. The method of claim 8, wherein the at least one parameter value characterizes one or more properties of the hand of the user.

10. The method of claim 9, wherein the one or more attributes of the hand of the user include one or more of the following: Palm lines; palm contour; shape; Fingernail size; Fingernail shape; color; Multi-point hand outline geometry; and One or more identification tags.

11. The method of claim 1, wherein the mixed environment comprises an environment of an extended reality (XR) system.

12. The method of claim 1, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

13. A method comprising: capturing at least one image of the object in a real environment of the mixed environment; generating a plurality of data points for the object based on the at least one image; generating a multi-dimensional model of the object based on the plurality of data points; generating a plurality of action points based on the multidimensional model of the object; as well as Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

14. The method according to claim 13, further comprising: determining an orientation of the object based on the plurality of action points; as well as A virtual image of the object is displayed in the virtual environment based on the determined orientation.

15. The method of claim 13, wherein the object is one of a hand of the user and a prosthesis of the user.

16. The method according to claim 13, further comprising: generating a first request for placing the object in the hybrid environment; displaying the first request within the virtual environment; In response to the first request, detecting the object in the virtual environment; generating a second request for confirmation from the user that the object is a hand of the user; displaying the second request within the virtual environment; detecting the confirmation in response to the second request; generating a third request to perform at least one gesture on the user; as well as The third request is displayed within the virtual environment.

17. The method of claim 16, wherein the at least one gesture comprises one or a combination of: rotation of the hand of the user; and A graphic of one or more shapes of the hand of the user.

18. The method according to claim 16, further comprising Performance of the at least one gesture is detected in response to displaying the third request.

19. The method of claim 18, wherein generating the plurality of action points is based at least in part on detecting the performance of the at least one gesture.

20. The method of claim 19, further comprising estimating at least a portion of the plurality of action points based on the at least one gesture.

21. The method of claim 13, wherein generating the plurality of tracking points is based on a lookup table, wherein the lookup table includes a first plurality of tracking points corresponding to each gesture of a plurality of gestures.

22. The method according to claim 21, further comprising: determining a second plurality of tracking points based on the tracked movement of the object; as well as A gesture is detected based on the second plurality of tracking points and the first plurality of tracking points corresponding to each of the gestures.

23. The method of claim 13, wherein the mixed environment comprises an environment of an extended reality (XR) system.

24. The method of claim 13, wherein the mixed environment comprises an environment of a virtual reality (VR) system.

25. An apparatus, comprising: a non-transitory machine-readable storage medium storing instructions; as well as at least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to: Receive image data from the camera; detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment; responsive to said detecting, determining a value of at least one parameter of said object; generating profile data based on the at least one parameter value, the profile data registering the object with a user; as well as Movement of the object within the hybrid environment is tracked based on the profile data.

26. The apparatus of claim 25, wherein the at least one processor is configured to: generating a request to place the object in the placement area of ​​the hybrid environment; and The request is displayed within the virtual environment.

27. An apparatus, comprising: a non-transitory machine-readable storage medium storing instructions; as well as at least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to: capturing at least one image of the object in a real environment of the mixed environment; generating a plurality of data points for the object based on the at least one image; generating a multi-dimensional model of the object based on the plurality of data points; generating a plurality of action points based on the multidimensional model of the object; as well as Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

28. The apparatus of claim 27, wherein the at least one processor is configured to: determining an orientation of the object based on the plurality of action points; and A virtual image of the object is displayed in the virtual environment based on the determined orientation.

29. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: Receive image data from the camera; detecting an object in a placement area of ​​a hybrid environment based on the image data, wherein the hybrid environment includes a real environment and a virtual environment; responsive to said detecting, determining a value of at least one parameter of said object; generating profile data based on the at least one parameter value, the profile data registering the object with a user; as well as Movement of the object within the hybrid environment is tracked based on the profile data.

30. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: capturing at least one image of the object in a real environment of the mixed environment; generating a plurality of data points for the object based on the at least one image; generating a multi-dimensional model of the object based on the plurality of data points; generating a plurality of action points based on the multidimensional model of the object; as well as Movement of the object in the virtual environment of the hybrid environment is tracked based on the plurality of action points.

Images