Fingernail segmentation and tracking

Abstract

An augmented reality (XR) system provides a method for displaying a virtual object in a hand-centric XR experience. The XR system provides a user with an XR user interface of the XR system. The XR system captures video frame data of a user's hand and detects the user's hand based on the video frame data and a hand detection model. The XR system generates a cropping bounding box based on the detection of the hand and the video frame data, and generates cropped video frame data based on the cropping bounding box and the video frame data. The XR system generates a 3D model of a portion of the user's hand based on the cropped video frame data, and generates a virtual object based on the 3D model and the 3D texture of the portion of the user's hand. The XR displays the virtual object in an XR user interface.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 483,403, filed on February 6, 2023, and U.S. Patent Application Serial No. 18 / 298,217, filed on April 10, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces and, more particularly, to user interfaces for extended reality. Background Art

[0004] A head wearable device can be implemented with a transparent display or a semi-transparent display, through which the user of the head wearable device can view the surrounding environment. Such a head wearable device enables the user to view the surrounding environment through the transparent display or the semi-transparent display, and also to see objects (e.g., virtual objects such as 2D or 3D graphic models, images, videos, text, etc.) that are generated to appear as part of the surrounding environment and / or superimposed on the surrounding environment. This is commonly referred to as "augmented reality" or "AR". A head wearable device can additionally completely cover the user's field of view and display a virtual environment that the user can move through or be moved by. This is commonly referred to as "virtual reality" or "VR". In a hybrid form, a view of the surrounding environment is captured using a camera device, and then displayed to the user along with augmentations on a display that covers the user's eyes. As used herein, unless the context indicates otherwise, the term extended reality (eXtended Reality, XR) refers to augmented reality, virtual reality, and any hybrid of these technologies.

[0005] Mobile devices can also be used to provide users with XR experiences. A camera of the mobile device captures video frame data of a real-world scene, and a display of the mobile device is used to display the real-world scene and a set of virtual objects to the user.

[0006] Users of head wearable devices or mobile devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with the user interface provided by the head wearable device or mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the accompanying drawings (not necessarily drawn to scale), the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest-order digit or digits in the reference numerals refer to the figure in which the element is first introduced. Some non-limiting examples are shown in the accompanying drawings, in which:

[0008] Figure 1A is a perspective view of a head-mounted device according to some examples.

[0009] Figure 1B Shown according to some examples Figure 1A Additional views of the headset.

[0010] Figure 2A is a front view of a mobile device according to some examples.

[0011] Figure 2B is a rear view of a mobile device according to some examples.

[0012] Figure 3 is a diagrammatic representation of a machine in the form of a computer system, according to some examples, within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein.

[0013] Figure 4A A collaboration diagram illustrating components of an XR system using hand tracking for user input, according to some examples.

[0014] Figure 4B A process flow diagram illustrating a method of displaying a virtual object according to some examples is shown.

[0015] Figure 4C Shown are hand tracking models and fingernail segmentation data according to some examples.

[0016] Figure 4D Shown are displays of XR experiences according to some examples.

[0017] Figure 5 A system of head wearable devices according to some examples is shown.

[0018] Figure 6 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0019] Figure 7 is a diagrammatic representation of data structures as maintained in a database, according to some examples.

[0020] Figure 8 is a diagrammatic representation of a messaging system having both client-side and server-side functionality, according to some examples.

[0021] Figure 9 is a block diagram illustrating software architecture according to some examples. DETAILED DESCRIPTION

[0022] Hand tracking is a way to provide user input from a user into an XR user interface provided by an XR system. The XR system uses a camera and computer vision methods to track one or more of the user's hands. The XR system determines hand gestures or postures made by the user based on video images captured by the camera. In some XR systems, the XR user interface generates virtual objects that appear to the user as if the user were wearing the virtual object, such as wearing a ring on a finger or applying nail polish on a fingernail.

[0023] In some cases, the illusion of wearing a virtual object is broken because the XR system may not have the processing power to calculate the position of a portion of the user's hand because the XR system must process a large amount of extraneous real-world scene image data in addition to the image data of the portion of the user's hand. This effect becomes more pronounced when the user's hand is far away from the XR system because the hand will occupy a smaller field of view of the XR system's camera used to capture image data of the user's hand.

[0024] In some examples, an extended reality (XR) system provides a method for displaying virtual objects in a hand-centric XR experience. The XR system provides an XR user interface of the XR system to a user. The XR system captures video frame data of the user's hands and detects the user's hands based on the video frame data and a hand detection model. The XR system generates a cropping bounding box based on the hand detection and the video frame data, and generates cropped video frame data based on the cropping bounding box and the video frame data. The XR system generates a 3D model of a portion of the user's hand based on the cropped video frame data, and generates a virtual object based on the 3D model and 3D texture of the portion of the user's hand. The XR displays the virtual object in the XR user interface.

[0025] Head wearable devices

[0026] Figure 1A is a perspective view of a head wearable device 100 according to some examples. The head wearable device 100 may be an XR system such as Figure 6The head wearable device 100 may be a client device of the XR computing system 602, or the head wearable device 100 may be a standalone XR system. The head wearable device 100 may include a frame 102 made of any suitable material, such as plastic or metal including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed within the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or combinations thereof. Any suitable display component may be disposed in the head wearable device 100.

[0027] Frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0028] The head wearable device 100 may include a computing device, such as a computer 120, which may be of any suitable type to be carried by the frame 102 and, in one or more examples, may be of a suitable size and shape to be partially disposed in one of the left temple piece 122 or the right temple piece 124. The computer 120 may include one or more processors with memory, wireless communication circuitry, and a power supply. As discussed below, the computer 120 includes a low-power circuit system 526, a high-speed circuit system 528, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. As shown by the machine 300 discussed herein, details of other aspects of the computer 120 may be implemented.

[0029] The computer 120 additionally includes a battery 118 or other suitable portable power source. In some examples, the battery 118 is disposed in the left temple piece 122 and electrically coupled to the computer 120 disposed in the right temple piece 124. The head wearable device 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter, or transceiver (not shown), or a combination of such devices.

[0030] The head wearable device 100 includes a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (ie, more than two) cameras.

[0031] In some examples, the head wearable device 100 may include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, and the like.

[0032] In some examples, left camera 114 and right camera 116 provide video frame data for use by head wearable device 100 to extract 3D information from a real-world scene.

[0033] The head wearable device 100 may also include a touchpad 126 mounted on or integrated with one or both of the left temple piece 122 and the right temple piece 124. In some examples, the touchpad 126 is generally arranged longitudinally, approximately parallel to the user's temple. As used herein, generally longitudinally aligned means that the touchpad is more longitudinal than horizontal, although it may be more longitudinal than this. Additional user input can be provided by one or more buttons 128, which are provided on the outer upper edges of the left optical element holder 104 and the right optical element holder 106 in the illustrated example. The one or more touchpads 126 and buttons 128 provide a means by which the head wearable device 100 can receive input from the user of the head wearable device 100.

[0034] Figure 1B The head wearable device 100 is shown from the perspective of a user wearing the head wearable device 100. Figure 1B Omitted Figure 1A Many of the components shown in FIG. Figure 1A As stated, Figure 1B The illustrated head wearable device 100 includes a left optical element 140 and a right optical element 144 secured within a left optical element holder 132 and a right optical element holder 136, respectively.

[0035] The head wearable device 100 includes a right front optical component 130 and a left front optical component 142 . The right front optical component includes a left near-eye display 150 and a right near-eye display 134 . The left front optical component includes a left projector 146 and a right projector 152 .

[0036] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective or diffractive structure (e.g., a grating and / or an optical element, such as a mirror, lens, or prism). Light 138 emitted by the right projector 152 encounters the diffractive structure of the waveguide of the right near-eye display 134, which guides the light toward the user's right eye to provide an image on or in the right optical element 144, which overlays the view of the real-world scene seen by the user. Similarly, light 148 emitted by the left projector 146 encounters the diffractive structure of the waveguide of the left near-eye display 150, which guides the light toward the user's left eye to provide an image on or in the left optical element 140, which overlays the view of the real-world scene seen by the user. The combination of the graphics processing unit, the optical image display driver, the right front optical assembly 130, the left front optical assembly 142, the left optical element 140, and the right optical element 144 provides the optical engine of the head wearable device 100. The head wearable device 100 uses the optical engine to generate an overlay with the user's view of the real-world scene, including displaying a user interface to the user of the head wearable device 100.

[0037] However, it should be understood that other display technologies or configurations can be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of a projector and waveguide, an LCD, LED or other display panel or surface can be provided.

[0038] When in use, the user of the head wearable device 100 will see information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user may use the touchpad 126 and / or buttons 128, voice input, or on an associated device (e.g., Figure 5 The device 100 may be used to interact with the head wearable device 100 by touch input on a mobile device 514 shown in FIG. 1 , and / or hand movement, position, and location recognized by the head wearable device 100.

[0039] In some examples, the optical engine of the XR system is incorporated into a lens, such as a contact lens, that contacts the user's eye. The XR system uses the contact lens to generate images with the XR experience.

[0040] In some examples, head wearable device 100 includes an XR system. In some examples, head wearable device 100 is a component of an XR system that includes additional computing components. In some examples, head wearable device 100 is a component of an XR system that includes additional user input systems or devices.

[0041] mobile device

[0042] According to some examples, Figure 2A is a front view of the mobile device 208, and Figure 2Bis a rear view of the mobile device 208. The mobile device 208 may be an XR system such as Figure 6 The mobile device 208 may be a client device of the XR computing system 602, or the mobile device 208 may be a standalone XR system. The mobile device 208 includes a screen 210 configured as a display for displaying images of the XR experience to the user. In some examples, the screen 210 is a touch screen configured to receive user input from the user. In some examples, the mobile device 208 includes one or more physical input devices (not shown), such as, but not limited to, buttons, switches, etc., configured to receive user input.

[0043] Mobile device 208 includes a computing device, such as computer 206, which may be of any suitable type to be housed in mobile device 208. Computer 206 may include one or more processors with memory, wireless communication circuitry, and a power source. Details of other aspects of computer 206 may be implemented as illustrated by machine 300 discussed herein.

[0044] The mobile device 208 includes a first camera 202 and a second camera 204. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In some examples, the head wearable device 100 includes any number of input sensors or other input / output devices in addition to the first camera 202 and the second camera 204. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, and the like.

[0045] In some examples, first camera 202 and second camera 204 provide video frame data for use by mobile device 208 to extract 3D information from a real-world scene.

[0046] The combination of a graphics processing unit (GPU), a graphics display driver, and screen 210 provides the optical engine of mobile device 208. Mobile device 208 uses the optical engine to generate an overlay with the user's view of the real-world scene, including displaying a user interface to the user of mobile device 208.

[0047] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, an LCD, LED, or other display panel or surface may be provided.

[0048] In use, information, content, and various user interfaces are presented to a user of the mobile device 208 on the screen 210. As described in more detail herein, the user can then interact with the mobile device 208 using methods and devices including, but not limited to, a touch screen, a touchpad, a set of buttons and / or a set of switches, voice input, or touch input on an associated device and / or hand movements, positions, and locations recognized by the mobile device 208.

[0049] In some examples, mobile device 208 includes an XR system. In some examples, mobile device 208 is a component of an XR system that includes additional computing components. In some examples, mobile device 208 is a component of an XR system that includes additional user input systems or devices.

[0050] Machine Architecture

[0051] Figure 3300 is a diagrammatic representation of a machine 300 within which instructions 302 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 300 to perform any one or more of the methods for a head wearable device or mobile device as discussed herein. For example, the instructions 302 may cause the machine 300 to perform any one or more of the methods described herein. The instructions 302 convert a general-purpose, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and illustrated in the manner described. The machine 300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 302 specifying actions to be taken by the machine 300. Furthermore, while a single machine 300 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 302 to perform any one or more of the methodologies discussed herein. For example, the machine 300 may include the XR computing system 602 or any of a plurality of server devices forming part of the interactive server system 612. In some examples, the machine 300 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side, and wherein certain operations of a particular method or algorithm are performed on the client side.

[0052] The machine 300 may include a processor 304, a memory 306, and input / output I / O components 308 that may be configured to communicate with each other via a bus 310. In an example, the processor 304 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 312 that executes instructions 302 and a processor 314. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 3 Multiple processors 304 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0053] The memory 306 includes a main memory 316, a static memory 340, and a storage unit 318, all of which are accessible by the processor 304 via the bus 310. The main memory 306, the static memory 340, and the storage unit 318 store instructions 302 that embody any one or more of the methodologies or functionality described herein. The instructions 302 may also reside, completely or partially, within the main memory 316, within the static memory 340, within the machine-readable medium 320 within the storage unit 318, within at least one of the processors 304 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 300.

[0054] The I / O components 308 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 308 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that the I / O components 308 may include Figure 3Many other components are not shown in the drawings. In various examples, the I / O components 308 may include user output components 322 and user input components 324. The user output components 322 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input components 324 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and force of touches or touch gestures), audio input components (e.g., microphones), etc.

[0055] In another example, the I / O component 308 may include a biometric component 326, a motion component 328, an environmental component 330, or a position component 332, as well as various other components. For example, the biometric component 326 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 328 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. In some examples, the position sensor may be incorporated into an inertial motion unit (IMU), etc.

[0056] The environmental components 330 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), a depth or distance sensor (e.g., a sensor for determining the distance to an object or for determining the depth of an object feature in a 3D coordinate system), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0057] With respect to cameras, the XR computing system 602 may have a camera system including, for example, a front-facing camera on a front surface of the XR computing system 602 and a rear-facing camera on a rear surface of the XR computing system 602. The front-facing camera may, for example, be used to capture still images and videos (e.g., “selfies”) of a user of the XR computing system 602, which may then be enhanced with the augmentation data (e.g., filters) described above. The rear-facing camera may, for example, be used to capture still images and videos in a more traditional camera mode, where these images are similarly enhanced with the augmentation data. In addition to the front and rear-facing cameras, the XR computing system 602 may also include a 360° camera for capturing 360° photos and videos.

[0058] Additionally, the camera system of the XR computing system 602 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quad, or penta rear camera configurations on the front and rear sides of the XR computing system 602. These multi-camera systems may include, for example, a wide camera, an ultra-wide camera, a telephoto camera, a macro camera, and a depth sensor.

[0059] The location component 332 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0060] A variety of technologies can be used to implement communications. The I / O components 308 also include a communications component 334 that is operable to couple the machine 300 to a network 336 or device 338 via corresponding couplings or connections. For example, the communications component 334 may include a network interface component or another suitable device that interfaces with the network 336. In other examples, the communications component 334 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Components (e.g. Low energy consumption), Components, and other communication components for providing communication via other modalities. Device 338 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0061] In addition, the communication component 334 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 334 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 334, such as a location obtained via Internet Protocol (IP) geolocation, ... Positioning derived from signal triangulation, positioning derived via detection of NFC beacon signals that can indicate a specific position, etc.

[0062] Various memories (e.g., main memory 316, static memory 340, and memory of processor 304) and storage unit 318 may store one or more sets of instructions and data structures (e.g., software) embodied or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 302), when executed by processor 304, cause various operations to implement the disclosed examples.

[0063] Instructions 302 may be sent or received over network 336 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 334) and using any one of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 302 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to device 338 using a transmission medium.

[0064] Hand-centric XR experience

[0065] According to some examples, Figure 4A A collaboration diagram showing components of an XR system that uses hand tracking to track a user's 424 hands 450 during an XR experience, and Figure 4B A process flow diagram illustrating a method of displaying a virtual object 400 in the presence of a user's hand is shown.

[0066] although Figure 4BThe method 400 for displaying a virtual object depicts a particular order of operations, but the order may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel, in a different order, or by different components of the XR system without materially affecting the functionality of the method.

[0067] The method of displaying the virtual object 400 is provided by an XR system such as ( Figure 1A ) head wearable device 100 or ( Figure 2A The mobile device 208 (e.g., a user interface engine 422) is used to provide a continuous, real-time, hand-centric XR experience to a user of the XR system via the user interface engine 422. The hand-centric XR experience includes providing virtual objects for the user to virtually try on their hands, such as, but not limited to, nail polish, nail decals, rings, synthetic nails, temporary tattoos, etc.

[0068] In operation 402, the XR system 430 generates an XR user interface 444 that is provided to the user 424. For example, the user interface engine 422 includes XR user interface control logic 454, which includes, among other things, a dialog script that specifies the user interface dialog implemented by the XR user interface 444. The XR user interface control logic 454 also includes one or more actions taken by the XR system 430 based on detecting various dialog events, such as user input.

[0069] A set of virtual objects 464 are superimposed on the image of the user's hand 450, as if the user were wearing the objects represented by the virtual objects 464. The XR user interface object model 452 also includes 3D graphics data for the virtual objects 464. The display engine 442 of the XR system 430 uses the 3D graphics data to generate an XR user interface 444 for display to the user 424.

[0070] The user interface engine 422 generates XR user interface graphical data 432 based on the XR user interface object model 452. The XR user interface graphical data 432 includes image video data of a set of virtual objects 464 of the XR user interface 444. The user interface engine 422 transmits the XR user interface graphical data 432 to the image display driver 434 of the display engine 442 of the XR system 430. The image display driver 434 receives the XR user interface graphical data 432 and generates display control signals 436 based on the XR user interface graphical data 432. The image display driver 434 uses the display control signals 436 to control the operation of one or more display devices 420 of the display engine 442. In response to the display control signals 436, the display device 420 generates a visual image of the XR user interface 444 including a rendered image of the virtual object 464, and the visual image is provided to the user 424. In some examples, when the XR system 430 includes the head-wearable device 100, the display engine 442 is the optical engine of the head-wearable device 100.

[0071] The XR system 430 displays one or more virtual object rendering displays, resulting in a binocular display facing the user 424. The binocular display of virtual objects 464 facing the user 424 provides the illusion that the virtual objects 464 are located at locations in the real-world scene that correspond to portions of the user's 424 hands 450. As the user 424 moves their hands 450 in the real-world scene, the XR system tracks the user's 424 hands 450 and updates the display of the virtual objects 464, making it appear as if the user 424 is wearing the virtual objects over the user's 424 virtual objects 464.

[0072] In operation 404, XR system 430 detects one or more of user hands 450. For example, XR system 430 uses one or more cameras 446 to capture video frame data 448 of hand gestures 438 or postures performed by user 424 using one or more of user hands 450. Video frame data 448 is transmitted to hand tracking component 458 of hand tracking pipeline 440 of XR system 430.

[0073] The hand tracking component 458 receives the video frame data 448 and detects the hands 450 of the user 424 based on the video frame data 448. In some examples, the hand tracking component 458 detects one or more of the user's hands 450 from the video frame data 448 using computer vision methods, including but not limited to Harris corner detection, Shi-Tomasi corner detection, Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Features from Speeded Up Segment Test (FAST), Orientation FAST, and Rotation Brief (ORB), among others.

[0074] In some examples, hand tracking component 458 detects one or more of the user's hands 450 from video frame data 448 using artificial intelligence methods and a hand detection model 426 previously generated using machine learning methods. In some examples, hand detection model 426 includes, but is not limited to, a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K-nearest neighbor model. In some examples, the machine learning methods used to generate hand detection model 426 may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0075] In operation 406, user interface engine 422 of XR system 430 generates a cropping bounding box based on the detection of hand 450 of user 424. The cropping bounding box is applied to video frame data 448 to mask areas of video frame data 448 that do not need to be processed because the areas do not include image data of hand 450 of user 424. For example, user interface engine 422 of XR system 430 utilizes a 2D graphics cropping method to determine a 2D cropping bounding box to apply to the frames of video frame data 448. In some examples, the 2D cropping bounding box follows the outline of hand 450. In some examples, the 2D cropping bounding box has a geometric shape, such as a polygon, a circle, etc., that defines the 2D outline of hand 450.

[0076] In operation 408 , the hand tracking component 458 generates cropped video frame data based on the cropping bounding box and the frames of the video frame data 448 by deleting portions of the frames of the video frame data that fall outside the cropping bounding box to reduce the amount of image data processed in subsequent operations of the hand tracking component 458 .

[0077] In operation 410, the hand tracking component 458 generates hand tracking and fingernail segmentation data 428 based on the cropped video frame data and the hand tracking and fingernail segmentation model 460. For example, the hand tracking component 458 extracts landmark features of one or more user hands 450 from the cropped video frame data using computer vision methods, including but not limited to Harris corner detection, Shi-Tomasi corner detection, Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Features from Speeded Up Segmentation Test (FAST), Orientation FAST, and Rotation Brief (ORB).

[0078] The hand tracking component 458 generates hand tracking and fingernail segmentation data 428 based on landmark features extracted from the cropped video frame data using artificial intelligence methods and a hand tracking and fingernail segmentation model 460 previously generated using machine learning methods. In some examples, the hand detection model 426 includes, but is not limited to, a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K-nearest neighbor model. In some examples, the machine learning methods used to generate the hand tracking and fingernail segmentation model 460 may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0079] The hand tracking and fingernail segmentation data 428 includes 3D model data of the user's hand 450 in a 3D coordinate system based on the interface features extracted from the video frame data 448. The 3D model data of the hand 450 includes ( Figure 4C The skeletal model 472 includes data for a skeletal model of the user 424, the skeletal model 472 including a set of nodes, such as node 466, and line segments, such as line segment 470, connecting the nodes. Additionally, the hand tracking and fingernail segmentation data 428 includes a 3D geometric model of a portion of the hand 450 of the user 424. In some examples, the portion of the hand 450 of the user 424 includes one or more fingernails of the hand 450 of the user 424, such as fingernail segmentation data 468a, fingernail segmentation data 468b, fingernail segmentation data 468c, fingernail segmentation data 468d, and fingernail segmentation data 468e. The 3D model data includes 3D surface data for each fingernail, which the user interface engine 422 can use to create a virtual object 464 that appears to be applied to the fingernails of the hand 450.

[0080] In some examples, the hand tracking and fingernail segmentation data 428 includes 2D model data, and the virtual object 464 is superimposed on the image data of the user's hand based on the corresponding 2D model data.

[0081] In operation 412, the XR system 430 generates a virtual object 464 based on the hand tracking and fingernail segmentation data 428 and the 3D texture. For example, the user interface engine 422 includes an XR user interface object model 452. The XR user interface object model 452 includes a set of 3D coordinate data for a virtual object 464, which includes a corresponding 3D geometric model of the virtual object. In addition, the XR user interface object model 452 includes a 3D texture that can be applied to the 3D geometric model of the virtual object during rendering of the virtual object. In some examples, the virtual object represents a physical object that can be worn by the user, such as a ring, a watch, a nail decal, etc. During the XR experience, the textured 3D virtual object is rendered and displayed to the user 424 as part of the XR user interface, making it appear as if the user 424 is wearing the virtual object.

[0082] In some examples, XR user interface object model 452 includes a 3D texture that can be applied to a 3D model of a portion of a user's hand 450. In some examples, the 3D texture simulates nail polish that can be applied to a 3D geometric model of one or more fingernails on user 424's hand 450. Then, during the XR experience, the textured 3D model of the fingernails is rendered and displayed to user 424, making it appear as if the user has applied nail polish to their fingernails. In some examples, the 3D texture represents a tattoo that can be applied to the portion of user 424's hand 450.

[0083] In some examples, the hand tracking and fingernail segmentation data 428 includes 2D model data and the virtual object 464, and the 2D texture is used to provide an image that is superimposed on the image data of the user's hand based on the corresponding 2D model data.

[0084] In operation 414, the user interface engine 422 of the XR system 430 displays the virtual object 464 to the user in the XR user interface within the context of the XR experience. Figure 4D As shown in , the XR system displays virtual objects 464 to the user 424 as if they were applied to the hand 450 of the user 424.

[0085] In operation 416, XR system 430 determines whether user 424 is still interacting with XR user interface 444. For example, user 424 may make a particular hand gesture or pose to signal to XR system 430 that XR user interface 444 should be terminated, XR system 430 may not detect user 424's hands for a specified period of time and may time out the XR user interface, user 424 may interact with a physical input device to signal that they are finished with XR user interface 444, etc.

[0086] In response to determining that user 424 is still interacting with XR user interface 444, XR system 430 transitions 456 to operation 404 and repeats the process of detecting the hand of user 424 as described herein. In response to determining that user 424 is no longer interacting with the XR user interface, XR system 430 transitions to operation 418 and ends.

[0087] In some examples, fingernail segmentation is added to a hand tracking model that performs hand segmentation, keypoint localization, and other auxiliary tasks. The model consists of a backbone trained on a dataset of hand images and a neck module for further feature processing, as well as multiple independent heads, such as but not limited to segmentation, landmark identification, etc. Fingernail segmentation is formulated as a binary segmentation task, and one or more channels are added to the full segmentation output of the existing model.

[0088] In some examples, hand tracking component 458 operates on images of size 256 pixels by 256 pixels in video frame data 448. In some examples, other image resolutions are used, such as, but not limited to, 256 pixels by 256 pixels, 320 by 320, etc.

[0089] In some examples, camera 446 is an infrared camera or a black and white camera and generates monochrome video data as video frame data 448. In some examples, camera 446 is a color camera and generates red, green, blue (RGB) or red, green, blue, alpha (RGBA) video data as video frame data 448.

[0090] In some examples, two or more or more hand tracking and nail segmentation models 460 are generated. The XR system 430 selects a hand tracking and nail segmentation model 460 to use based on the computing resources available to process the video frame data 448. For example, when the XR system 430 is a mobile device or component of a head wearable device that does not have a powerful processor, the XR system 430 selects a hand tracking and nail segmentation model 460 that requires a small amount of computing resources. Additionally, when the XR system 430 is a mobile device or component of a head wearable device that has a powerful processor, the XR system 430 selects a hand tracking and nail segmentation model 460 that requires many computing resources. In some examples, the models differ in size, with larger models utilizing more processing power and smaller models utilizing less processing power.

[0091] In some examples, two or more hand detection models 426 are made available, and the XR system selects between the two based on the processing capabilities of the mobile device or head wearable device.

[0092] In some examples, hand tracking and nail segmentation model 460 is trained using skip connections from early layers of the machine learning model. In some examples, skip connections are provided from the first layer onwards. In some examples, skip connections are added to hand tracking and nail segmentation model 460 that is intended for use on mobile devices or head wearables with powerful processors.

[0093] In some examples, because the amount of image area occupied by the fingernails of the user's 424 hand 450 is a small fraction of the total image area, the Dice coefficient loss function is used to train the hand tracking and nail segmentation model 460 .

[0094] In some examples, to improve fingernail segmentation, the video frame data 448 is upsampled. For example, after cropping, the image can be significantly reduced in size to a size smaller than an acceptable size for processing. Therefore, upsampling the image, for example, by upsampling from a 128 pixel by 128 pixel image to a 256 pixel by 256 pixel image, does not adversely affect performance.

[0095] In some examples, an image having a size of 288 pixels by 288 pixels may be utilized to improve fingernail segmentation. In some examples, other image resolutions such as, but not limited to, 256 pixels by 256 pixels, 320 by 320, etc. are used.

[0096] In some examples, during training of the hand tracking and nail segmentation model 460, a model training system (not shown) applies color dithering enhancement to frames of the training dataset based on the hand and fingernail masks, for example, by applying enhancement to either the hand or the mask, but not both. By doing so, the edges of the detected fingernail segments are enhanced. For example, the model training system randomly perturbs the color, brightness, and contrast of the hand and fingernail regions of the training images.

[0097] In some examples, XR system 430 utilizes various APIs and system libraries of a mobile device or head wearable device to perform the functions of hand tracking pipeline 440, user interface engine 422, and display engine 442.

[0098] System with head-worn device

[0099] Figure 5 Shown is a system 500 including a head wearable device 100 with a selector input device according to some examples. Figure 5 is a high-level functional block diagram of an example head wearable device 100 communicatively coupled to a mobile device 514 and various server systems 504 (eg, an interactive server system 612 ) via various networks 610 .

[0100] The head wearable device 100 includes one or more cameras, each of which may be, for example, one or more cameras 508 , a light emitter 510 , and one or more wide-band cameras 512 .

[0101] The mobile device 514 is connected to the head wearable device 100 using both a low power wireless connection 516 and a high speed wireless connection 518. The mobile device 514 is also connected to the server system 504 and the network 506.

[0102] The head wearable device 100 also includes two image displays in the image display 520 of the optical assembly. The two image displays 520 of the optical assembly include one image display associated with the left lateral side of the head wearable device 100 and one image display associated with the right lateral side of the head wearable device 100. The head wearable device 100 also includes an image display driver 522 and a GPU 524. The image display 520 of the optical assembly, the image display driver 522, and the GPU 524 constitute the optical engine of the head wearable device 100. The image display 520 of the optical assembly is used to present images and videos to the user of the head wearable device 100, including images that may include a graphical user interface.

[0103] The image display driver 522 commands and controls the image display 520 of the optical assembly. The image display driver 522 can deliver image data directly to the image display 520 of the optical assembly for presentation or can convert the image data into a signal or data format suitable for delivery to an image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data can be formatted according to a compression format such as Portable Network Graphics (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (EXIF).

[0104] The head wearable device 100 includes a frame and rods (or temples) extending from the lateral sides of the frame. The head wearable device 100 also includes a user input device 530 (e.g., a touch sensor or a push button), including an input surface on the head wearable device 100. The user input device 530 (e.g., a touch sensor or a push button) is used to receive input selections from the user to manipulate the graphical user interface of the presented image.

[0105] Figure 5The components for the head wearable device 100 shown in FIG are located on one or more circuit boards, such as PCBs or flexible PCBs, in the bezel or temples. Alternatively or additionally, the depicted components can be located in a block, frame, hinge, or nosepiece of the head wearable device 100. The left and right cameras 508 can include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible or light capturing elements that can be used to capture data, including images of a scene with an unknown object.

[0106] The head wearable device 100 includes a memory 502 that stores instructions for performing a subset or all of the functions described herein. The memory 502 may also include a storage device.

[0107] like Figure 5 As shown, high-speed circuitry 528 includes a high-speed processor 532, memory 502, and high-speed wireless circuitry 534. In some examples, image display driver 522 is coupled to high-speed circuitry 528 and operated by high-speed processor 532 to drive the left and right image displays of image display 520 of the optical assembly. High-speed processor 532 can be any processor capable of managing high-speed communications and operations required by any general-purpose computing system for head wearable device 100. High-speed processor 532 includes the processing resources required to manage high-speed data transmission over high-speed wireless connection 518 to a wireless local area network (WLAN) using high-speed wireless circuitry 534. In some examples, high-speed processor 532 executes an operating system for head wearable device 100, such as a Linux operating system or other such operating system, and the operating system is stored in memory 502 for execution. In addition to any other responsibilities, high-speed processor 532, which executes the software architecture for head wearable device 100, is used to manage data transmission with high-speed wireless circuitry 534. In some examples, high-speed wireless circuitry 534 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, high-speed wireless circuitry 534 can implement other high-speed communication standards.

[0108] The low-power wireless circuit system 536 and the high-speed wireless circuit system 534 of the head wearable device 100 may include a short-range transceiver (Bluetooth TM) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Mobile device 514—including a transceiver that communicates via low-power wireless connection 516 and high-speed wireless connection 518—can be implemented using details of the architecture of head wearable device 100, as well as other elements of network 506.

[0109] The memory 502 comprises any storage device capable of storing various data and applications, including camera data generated by the left and right cameras 508, the wideband camera 512, and the GPU 524, as well as images generated by the image display driver 522 for display on the image display 520 of the optical assembly. Although the memory 502 is shown as being integrated with the high-speed circuitry 528, in some examples, the memory 502 can be a separate, independent component of the head wearable device 100. In some such examples, electrical wiring can provide a connection from the GPU 524 or the low-power processor 538 to the memory 502 through a chip including the high-speed processor 532. In some examples, the high-speed processor 532 can manage addressing of the memory 502 so that the low-power processor 538 will initiate the high-speed processor 532 whenever a read or write operation involving the memory 502 is required.

[0110] like Figure 5 As shown, the low-power processor 538 or the high-speed processor 532 of the head wearable device 100 can be coupled to a camera (camera 508, light emitter 510 or wide-band camera 512), an image display driver 522, a user input device 530 (for example, a touch sensor or button) and a memory 502.

[0111] The head-worn device 100 is connected to a host computer. For example, the head-worn device 100 is paired with a mobile device 514 via a high-speed wireless connection 518 or is connected to a server system 504 via a network 506. The server system 504 can be one or more computing devices that are part of a service or network computing system, for example, including a processor, memory, and a network communication interface to communicate with the mobile device 514 and the head-worn device 100 via the network 506.

[0112] The mobile device 514 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via the network 506, the low-power wireless connection 516, or the high-speed wireless connection 518. The mobile device 514 may also store at least a portion of the instructions for generating two-channel audio content in a memory of the mobile device 514 to implement the functionality described herein.

[0113] The output components of the head wearable device 100 include visual components, such as a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide. The image display of the optical assembly is driven by the image display driver 522. The output components of the head wearable device 100 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 100, the mobile device 514, and the server system 504, such as the user input device 530, may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0114] The head wearable device 100 may also include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 100. For example, the peripheral elements may include any I / O components including output components, motion components, positioning components, or any other such components described herein.

[0115] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a positioning sensor component for generating positioning coordinates (e.g., a global positioning system (GPS) receiver component), a Wi-Fi or Bluetooth receiver component for generating positioning system coordinates, etc. TM transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor components (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 514 via the low-power wireless connection 516 and the high-speed wireless connection 518 via the low-power wireless circuitry 536 or the high-speed wireless circuitry 534.

[0116] Networked computing environment

[0117] Figure 66 is a block diagram illustrating an example interactive system 600 for facilitating interactions over a network (e.g., exchanging text messages, making text, audio, and video calls, or playing games). The interactive system 600 includes one or more XR systems, such as an XR computing system 602, each of which hosts a plurality of applications, including an interactive client 606 and an application 608, other applications 608. Each interactive client 606 is communicatively coupled to other instances of the interactive client 606 (e.g., hosted on a corresponding other computing system, such as computing system 604), an interactive server system 612, and a third-party server 614 via one or more communication networks, including a network 610 (e.g., the Internet). The interactive client 606 can also communicate with the locally hosted application 608 using an application programming interface (API).

[0118] Each XR computing system 602 may include one or more user devices, such as a mobile device 514 , a head wearable device 100 , and a computer client device 616 , that are communicatively connected to exchange data and messages.

[0119] The interaction clients 606 interact with other interaction clients 606 and with an interaction server system 612 via a network 610. Data exchanged between the interaction clients 606 (e.g., interaction 618) and between the interaction clients 606 and the interaction server system 612 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0120] The interaction server system 612 provides server-side functionality to the interaction clients 606 via the network 610. Although certain functions of the interaction system 600 are described herein as being performed by either the interaction clients 606 or the interaction server system 612, the location of certain functions within the interaction clients 606 or within the interaction server system 612 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the interaction server system 612, but later migrate the technologies and functions to the interaction clients 606 where the XR computing system 602 has sufficient processing power.

[0121] The interactive server system 612 supports various services and operations provided to the interactive clients 606. Such operations include sending data to the interactive clients 606, receiving data from the interactive clients 606, and processing data generated by the interactive clients 606. The data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the interactive system 600 is activated and controlled by functions available through the user interface (UI) of the interactive client 606.

[0122] Turning now specifically to the interaction server system 612, an application program interface (API) server 620 is coupled to the interaction server 622 and provides a programming interface thereto, making the functionality of the interaction server 622 accessible to the interaction clients 606, applications 608, other applications 608, and third-party servers 614. The interaction server 622 is communicatively coupled to a database server 624, thereby facilitating access to a database 626 that stores data associated with interactions processed by the interaction server 622. Similarly, a web server 628 is coupled to the interaction server 622 and provides a web-based interface to the interaction server 622. To this end, the web server 628 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0123] The application program interface (API) server 620 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 622 and the XR computing system 602 (and, for example, the interaction client 606 and the application 608 or other applications 608 ) and the third-party server 614. Specifically, the application program interface (API) server 620 provides a set of interfaces (e.g., routines and protocols) that the interaction client 606 and the application 608 or other applications 608 can call or query to activate the functions of the interaction server 622. The application program interface (API) server 620 exposes various functions supported by the interaction server 622, including account registration; login functionality; sending interaction data from a particular interaction client 606 to another interaction client 606 via the interaction server 622; transferring media files (e.g., images or videos) from the interaction client 606 to the interaction server 622; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the XR computing system 602; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to the interaction client 606).

[0124] Interactive server 622 hosting the following reference Figure 8Multiple systems and subsystems described. Returning to the interactive client 606, the features and functionality of the external resource (e.g., a linked application 608 or applet) are made available to the user via the interface of the interactive client 606. In this context, "external" refers to the fact that the application 608 or applet is external to the interactive client 606. External resources are typically provided by a third party, but may also be provided by the creator or provider of the interactive client 606. The interactive client 606 receives a user selection of an option to launch or access features of such an external resource. The external resource can be an application 608 installed on the XR computing system 602 (e.g., a "local app"), or a small-scale version of an application (e.g., a "mini-program") hosted on the XR computing system 602 or located remotely from the XR computing system 602 (e.g., on a third-party server 614). The small-scale version of the application includes a subset of the features and functionality of the application (e.g., the full-scale local version of the application) and is implemented using a markup language document. In some examples, a small-scale version of an application (e.g., a "mini-program") is a web-based markup language version of the application and is embedded in the interactive client 606. In addition to using markup language documents (e.g., .*ml files), mini-programs can include scripting languages ​​(e.g., .*js files or .json files) and style sheets (e.g., .*ss files).

[0125] In response to receiving a user selection of an option to launch or access a feature of an external resource, the interactive client 606 determines whether the selected external resource is a web-based external resource or a locally installed application 608. In some cases, the application 608 locally installed on the XR computing system 602 can be independent of the interactive client 606 and launched separately from the interactive client 606, for example, by selecting an icon corresponding to the application 608 on a home screen of the XR computing system 602. A small-scale version of such an application can be launched or accessed via the interactive client 606, and in some examples, no portion of the small-scale application can be accessed outside of the interactive client 606 or only a limited portion of the small-scale application can be accessed outside of the interactive client 606. The small-scale application can be launched by the interactive client 606 receiving, for example, a markup language document associated with the small-scale application from a third-party server 614 and processing such a document.

[0126] In response to determining that the external resource is a locally installed application 608, the interactive client 606 instructs the XR computing system 602 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 606 communicates with the third-party server 614 (for example) to obtain a markup language document corresponding to the selected external resource. The interactive client 606 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 606.

[0127] The interaction client 606 can notify the user of the XR computing system 602 or other users related to such user (e.g., "friends") about activities occurring in one or more external resources. For example, the interaction client 606 can provide participants in a conversation (e.g., a chat session) within the interaction client 606 with notifications about external resources currently or recently used by one or more members of a group of users. One or more users can be invited to join an active external resource or a recently used but currently inactive external resource (in the group of friends) can be launched. The external resource can provide participants in the conversation, each using a corresponding interaction client 606, with the ability to share items, conditions, states, or locations in the external resource with one or more members of the group of users in the chat session. Shared items can be interactive chat cards that members of the chat can interact with to, for example, launch the corresponding external resource, view specific information within the external resource, or take members of the chat to a specific location or state within the external resource. Within a given external resource, a response message can be sent to the user on the interaction client 606. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0128] The interactive client 606 can present a list of available external resources (e.g., applications 608 or applets) to the user to launch or access a given external resource. The list can be presented in a context-sensitive menu. For example, the icons representing different applications (or applets) of the application 608 (or applets) can change based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0129] Data Architecture

[0130] Figure 7 is a diagram illustrating a data structure 700 that may be stored in a database 704 of the interactive server system 612 according to certain examples. Although the contents of the database 704 are shown as including a plurality of tables, it will be appreciated that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0131] The database 704 includes message data stored in a message table 706. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Figure 7

[0076] Additional details regarding information that may be included in a message and included within the message data stored in message table 706 are described.

[0132] The entity table 708 stores entity data and is (e.g., referentially) linked to the entity graph 710 and the profile data 702. Entities for which records are maintained within the entity table 708 may include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of the entity type, any entity for which the interactive server system 612 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0133] The entity graph 710 stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based. Some relationships between entities may be unidirectional, such as a subscription by an individual user to digital content from a business or publishing user (e.g., a newspaper or other digital media outlet or brand). Other relationships may be bidirectional, such as a "friend" relationship between various users of the interactive system 600.

[0134] Certain permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a bidirectional relationship (e.g., a friend relationship between respective users) can include authorization for the publication of digital content items between the respective users, but can impose certain restrictions or filters on the publication of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between a personal user and a business user can impose varying degrees of restrictions on the publication of digital content from the business user to the personal user, and can significantly restrict or prevent the publication of digital content from the personal user to the business user. A particular user, as an example of an entity, can have certain restrictions recorded (e.g., via privacy settings) in its record for that entity within entity table 708. Such privacy settings can apply to all types of relationships within the context of interactive system 600, or can selectively apply to only certain types of relationships.

[0135] Profile data 702 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, profile data 702 can be selectively used and presented to other users of the interactive system 600. In the case where the entity is a person, profile data 702 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 600 and on a map interface displayed to other users by the interactive client 606. The collection of avatar representations can include a "status avatar," which presents a graphical representation of the user's status or activity that they may choose to transmit at a particular time.

[0136] Where the entity is a group, profile data 702 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings for the relevant group (eg, notifications).

[0137] Database 704 also stores enhancement data, such as overlays or filters, in enhancement table 712. Enhancement data is associated with and applied to videos (video data is stored in video table 714) and images (image data is stored in image table 716).

[0138] In some examples, a filter is displayed as an overlay on an image or video during presentation to a message recipient. Filters can be of various types, including user-selected filters from a set of filters presented to the message sender by the interaction client 606 while the message sender is composing the message. Other types of filters include geolocation filters (also known as geofilters), which can be presented to the message sender based on geolocation. For example, a geolocation filter specific to a nearby or special location can be presented within the user interface by the interaction client 606 based on geolocation information determined by a global positioning system (GPS) unit of the XR computing system 602.

[0139] Another type of filter is a data filter, which can be selectively presented to the message sender by the interaction client 606 based on other input or information collected during the message creation process by the XR computing system 602. Examples of data filters include the current temperature at a particular location, the current speed at which the message sender is traveling, the battery life of the XR computing system 602, or the current time.

[0140] Other augmented data that can be stored in the image table 716 include augmented reality content items (eg, corresponding to application lenses or augmented reality experiences). Augmented reality content items can be real-time special effects and sounds that can be added to images or videos.

[0141] As described above, augmented data includes AR, VR, and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., video or images). This includes real-time modifications, which modify the image as it is captured using the device sensors (e.g., one or more cameras) of the XR computing system 602 and then displayed on the screen of the XR computing system 602 with the modifications. This also includes modifications to stored content, such as video clips in a collection or group, that can be modified. For example, in an XR computing system 602 that accesses multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clip. Similarly, real-time video capture can use modifications to show how the video image currently being captured by the sensors of the XR computing system 602 will modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device sensors can be recorded with or without modifications (or both) and stored in memory. In some systems, a preview feature can simultaneously display how different augmented reality content items will look in different windows in the display. For example, this can enable multiple windows with different pseudo-random animations to be viewed on the display at the same time.

[0142] Thus, using data from an augmented reality content item and various systems or other such transformation systems that use this data to modify content can involve: detecting various objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in a video frame; tracking such objects as they leave, enter, and move around the field of view; and modifying or transforming such objects while being tracked. In various examples, different methods for implementing such transformations can be used. Some examples can involve: generating a three-dimensional mesh model of one or more objects; and using transformations of the model and animated textures within the video to implement the transformations. In some examples, tracking points on the objects can be used to place an image or texture (which can be two-dimensional or three-dimensional) at the tracked locations. In yet another example, neural network analysis of video frames can be used to place the image, model, or texture within the content (e.g., an image or video frame). Thus, an augmented reality content item involves both the images, models, and textures used to create the transformations within the content, as well as the additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0143] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or a device's sensors can be used to generate a video stream. In addition, computer animation models can be used to process any object, such as a human face and body parts, an animal, or an inanimate object (e.g., a chair, a car, or other objects).

[0144] In some examples, when a specific modification is selected along with the content to be transformed, the element to be transformed is identified by a computing device and then detected and tracked if the element to be transformed is present in a frame of the video. The elements of the object are modified according to the modification request, thereby transforming the frame of the video stream. For different types of transformations, the frames of the video stream can be transformed by different methods. For example, for frame transformations that primarily involve changing the form of the elements of the object, characteristic points are calculated for each element of the object (e.g., using an active shape model (ASM) or other known methods). A grid based on the characteristic points is then generated for each element of the object. This grid is used in subsequent stages of tracking the elements of the object in the video stream. During the tracking process, the grid for each element is aligned with the position of each element. Additional points are then generated on the grid.

[0145] In some examples, a transformation that uses elements of an object to change some areas of an object can be performed by calculating characteristic points for each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas based on these points are generated. The elements of the object are then tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific requirements of the modification, the properties of the mentioned area can be transformed in different ways. Such modifications can involve: changing the color of the area; removing some parts of the area from the frame of the video stream; including new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some characteristic points can be selected as control points for the entire state space to be used to determine the options for model animation.

[0146] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the facial region of the image to detect facial feature reference points.

[0147] Other methods and algorithms suitable for face detection can be used. For example, in some examples, visual features are located using landmarks, which represent distinguishable points that are present in most of the images under consideration. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmarks are not identifiable (for example, if the person has an eye mask), secondary landmarks can be used. Such a landmark identification step can be used for any such object. In some examples, a set of landmarks forms a shape. The coordinates of the points in the shape can be used to represent the shape as a vector. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The mean shape is the average of the aligned training shapes.

[0148] The transformation system can capture an image or video stream on a client device (e.g., XR computing system 602) and perform complex image manipulations locally on the XR computing system 602 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transformations, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the XR computing system 602.

[0149] In some examples, a computer-animated model for transforming image data can be used by a system in which a user can capture an image or video stream of the user (e.g., a selfie) using an XR computing system 602 having a neural network operating as part of an interactive client 606 operating on the XR computing system 602. The transformation system operating within the interactive client 606 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer-animated model to transform the image data, or the computer-animated model can exist in association with an interface described herein. The modification icon includes changes that serve as the basis for modifying the user's face within the image or video stream as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process that transforms the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the XR computing system 602. The transformation system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream, and once a modification icon has been selected, the modified result can be presented to the user in real time or near real time. Furthermore, when the video stream is being captured, the modification can be persistent, and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0150] The graphical user interface presenting the modifications performed by the transformation system can provide additional interactive options to the user. Such options can be based on the interface used to initiate selection of a particular computer-animated model and content capture (e.g., from a content creator user interface). In various examples, the modifications can be persistent after the initial selection of the modification icon. The user can toggle the modifications on or off by tapping or otherwise selecting a face modified by the transformation system, and store them for later review or browsing to other areas of the imaging application. In the event that multiple faces are modified by the transformation system, the user can globally toggle the modifications on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some examples, each face in a group of multiple faces can be modified individually, or such modifications can be individually toggled by tapping or selecting a single face or series of faces displayed within the graphical user interface.

[0151] The story table 718 stores data about a collection of messages and associated images, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 708). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by that user. To this end, the user interface of the interactive client 606 can include a user-selectable icon that enables the message sender to add specific content to his or her personal story.

[0152] The collection can also constitute a "live story" that is a collection of content from multiple users, which is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time can be presented with the option of contributing content to a particular live story, for example, via the user interface of the interactive client 606. Live stories can be identified to the user by the interactive client 606 based on his or her location. The end result is a "live story" told from a group perspective.

[0153] Another type of content collection is called a "location story," which enables users whose XR computing system 602 is located within a specific geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0154] As mentioned above, video table 714 stores video data that, in some examples, is associated with messages for which records are maintained within message table 706. Similarly, image table 716 stores image data associated with messages whose message data is stored in entity table 708. Entity table 708 can associate various enhancements from enhancement table 712 with various images and videos stored in image table 716 and video table 714.

[0155] Database 704 also includes social networking information collected by social networking system 822 .

[0156] System Architecture

[0157] Figure 86 is a block diagram illustrating additional details regarding an interactive system 600 according to some examples. Specifically, the interactive system 600 is shown as including an interactive client 606 and an interactive server 622. The interactive system 600 includes a plurality of subsystems that are supported on the client side by the interactive client 606 and on the server side by the interactive server 622. Example subsystems are discussed below.

[0158] Image processing system 802 provides various functions that enable a user to capture and enhance (eg, enhance or otherwise modify or edit) media content associated with a message.

[0159] The camera system 804 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the XR computing system 602 (e.g., directly or via operating system control) to modify and enhance real-time images captured and displayed via the interactive client 606.

[0160] The augmentation system 806 provides functionality related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera of the XR computing system 602 or retrieved from the memory of the XR computing system 602. For example, in operation, the augmentation system 806 selects, presents, and displays media overlays (e.g., image filters or image lenses) for interactive clients 606 for enhancing real-time images received via the camera system 804 or stored images retrieved from the memory 502 of the XR computing system 602. These augmentations are selected and presented to the user of the interactive client 606 by the augmentation system 806 based on inputs and data such as:

[0161] The geolocation of the XR computing system 602; and

[0162] Social network information of users of the XR computing system 602 .

[0163] Enhancements can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at the XR computing system 602 for transmission in messages, or applied to video content such as a video content stream or feed sent from the interactive client 606. Thus, the image processing system 802 can interact with and support various subsystems of the communication system 808, such as the messaging system 810 and the video communication system 812.

[0164] The media overlay may include text or image data that can be superimposed on a photo taken by the XR computing system 602 or a video stream produced by the XR computing system 602. In some examples, the media overlay may be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In another example, the image processing system 802 uses the geolocation of the XR computing system 602 to identify a media overlay that includes the name of a business at the geolocation of the XR computing system 602. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 626 and accessed through the database server 624.

[0165] Image processing system 802 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which specific media overlays should be provided to other users. Image processing system 802 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0166] The augmented reality creation system 814 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive client 606. The augmented reality creation system 814 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0167] In some examples, the enhancement creation system 814 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geolocation via a bidding process. For example, the enhancement creation system 814 associates the highest bidding merchant's media overlay with the corresponding geolocation for a predefined amount of time.

[0168] The communication system 808 is responsible for enabling and processing various forms of communication and interaction within the interactive system 600 and includes a messaging system 810, an audio communication system 816, and a video communication system 812. The messaging system 810 is responsible for enforcing temporary or time-limited access to content by the interactive clients 606. The messaging system 810 includes multiple timers (e.g., within a transient timer system 818) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive clients 606 based on duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 818 are provided below. The audio communication system 816 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 606. Similarly, the video communication system 812 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 606.

[0169] The user management system 820 is operationally responsible for managing user data and profiles, and includes a social networking system 822 that maintains social networking information about relationships between users of the interactive system 600 .

[0170] The collection management system 824 is operationally responsible for managing collections or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, video, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 824 is also responsible for publishing an icon providing notification of a particular collection to the user interface of the interactive client 606. The collection management system 824 includes curation functionality that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate a collection of content related to a specific event (e.g., removing inappropriate content or redundant messages). In addition, the collection management system 824 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users can be compensated for including user-generated content in a collection. In such cases, the collection management system 824 operates to automatically pay such users for use of their content.

[0171] The mapping system 826 provides various geolocation functions and supports the presentation of map-based media content and messages by the interactive client 606. For example, the mapping system 826 enables the display of a user icon or avatar (e.g., stored in the profile data 702) on a map to indicate the current or past locations of the user's "friends" and media content generated by these friends (e.g., a collection of messages including photos and videos) within the context of the map. For example, a message posted by a user to the interactive system 600 from a particular geolocation can be displayed to the "friends" of a particular user within the context of that particular location on the map on the map interface of the interactive client 606. A user can also share his or her location and status information with other users of the interactive system 600 via the interactive client 606 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to the selected user within the context of the map interface of the interactive client 606.

[0172] The gaming system 828 provides various gaming functions within the context of the interactive client 606. The interactive client 606 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 606 and played with other users of the interactive system 600. The interactive system 600 also enables a particular user to invite other users to play a particular game by sending an invitation to such other users from the interactive client 606. The interactive client 606 also supports audio, video, and text messaging (e.g., chatting) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0173] The external resource system 830 provides an interface for the interactive client 606 to communicate with a remote server (e.g., a third-party server 614) to launch or access external resources (i.e., applications or applets). Each third-party server 614 hosts, for example, an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5). The interactive client 606 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 614 associated with the web-based resource. The applications hosted by the third-party server 614 are programmed in JavaScript using a software development kit (SDK) provided by the interactive server 622. The SDK includes an application program interface (API) with functions that can be called or activated by a web-based application. The interactive server 622 hosts a JavaScript library that provides access to a given external resource for specific user data of the interactive client 606. HTML5 is an example of a technology used to program games, but applications and resources programmed based on other technologies can be used.

[0174] To integrate the SDK's functionality into a web-based resource, the SDK is downloaded from the interaction server 622 by the third-party server 614, or is otherwise received by the third-party server 614. Once downloaded or received, the SDK is included as part of the application code of the external web-based resource. The code of the web-based resource can then call or activate the SDK's functionality to integrate the features of the interaction client 606 into the web-based resource.

[0175] The SDK stored on the interactive server system 612 effectively provides a bridge between external resources (e.g., applications 608 or applets) and the interactive client 606. This gives users a seamless experience communicating with other users on the interactive client 606, while also preserving the look and feel of the interactive client 606. In order to bridge the communication between the external resources and the interactive client 606, the SDK facilitates communication between the third-party server 614 and the interactive client 606. The WebViewJavaScriptBridge running on the XR computing system 602 establishes two one-way communication channels between the external resources and the interactive client 606. Messages are sent asynchronously between the external resources and the interactive client 606 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0176] By using the SDK, not all information from the interactive client 606 is shared with the third-party server 614. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 614 provides an HTML5 file corresponding to the web-based external resource to the interactive server 622. The interactive server 622 can add a visual representation of the web-based external resource (e.g., a box design or other graphics) in the interactive client 606. Once the user selects the visual representation or instructs the interactive client 606 to access a feature of the web-based external resource through the GUI of the interactive client 606, the interactive client 606 obtains the HTML5 file and instantiates a resource for accessing the feature of the web-based external resource.

[0177] The interactive client 606 presents a graphical user interface (e.g., a login page or title screen) for the external resource. During, before, or after presenting the login page or title screen, the interactive client 606 determines whether the launched external resource has previously been authorized to access the user data of the interactive client 606. In response to determining that the launched external resource has previously been authorized to access the user data of the interactive client 606, the interactive client 606 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access the user data of the interactive client 606, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 606 slides up a menu (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen) for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource is authorized to use. In response to receiving a user selection of the accept option, the interactive client 606 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 606. External resources are authorized by the interactive client 606 to access user data under the OAuth2 framework.

[0178] The interaction client 606 controls the type of user data shared with the external resource based on the type of external resource authorized. For example, an external resource comprising a full-scale application (e.g., application 608) is provided with access to a first type of user data (e.g., a two-dimensional avatar of the user with or without different avatar characteristics). As another example, an external resource comprising a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the look and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0179] The advertising system 832 operatively enables third parties to purchase advertisements for presentation to end users via the interactive client 606 and also handles the delivery and presentation of these advertisements.

[0180] Software Architecture

[0181] Figure 99 is a block diagram 900 illustrating a software architecture 902 that can be installed on any one or more of the devices described herein. The software architecture 902 is supported by hardware such as a machine 904 including a processor 906, a memory 908, and I / O components 910. In this example, the software architecture 902 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 902 includes layers such as an operating system 912, libraries 914, frameworks 916, and applications 918. In operation, the applications 918 invoke API calls 920 through the software stack and receive messages 922 in response to the API calls 920.

[0182] The operating system 912 manages hardware resources and provides common services. The operating system 912 includes, for example, a kernel 924, services 926, and drivers 928. The kernel 924 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 924 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 926 can provide other common services to other software layers. Drivers 928 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 928 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (e.g., USB drivers), drivers, audio drivers, power management drivers, etc.

[0183] The libraries 914 provide a common low-level infrastructure used by the applications 918. The libraries 914 may include system libraries 930 (e.g., C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 914 may include API libraries 932, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphical content on a display in two dimensions (2D) and three dimensions (3D), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. The libraries 914 may also include various other libraries 934 to provide many other APIs to the applications 918.

[0184] The framework 916 provides a common high-level infrastructure used by applications 918. For example, the framework 916 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 916 can provide a wide range of other APIs that can be used by applications 918, some of which may be specific to a particular operating system or platform.

[0185] In an example, applications 918 may include a home application 936, a contacts application 938, a browser application 940, a book reader application 942, a location application 944, a media application 946, a messaging application 948, a game application 950, and a variety of other applications such as third-party applications 952. Applications 918 are programs that perform functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 918 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C language or assembly language). In a specific example, third-party applications 952 (e.g., those written by entities other than the vendor of a particular platform using ANDROID) TM or IOS TM Applications developed with a software development kit (SDK) can be developed for mobile operating systems such as IOS TM ANDROID TM , WINDOWS Phone, or other mobile operating systems. In this example, third-party applications 952 can activate API calls 920 provided by the operating system 912 to facilitate the functions described herein.

[0186] in conclusion

[0187] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

[0188] Glossary

[0189] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media to facilitate communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0190] "Client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, multiple laptop computers, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0191] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0192] "Component" refers to a device, physical entity, or logic with the following boundaries: the boundaries are defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and can be part of a program that generally performs a specific function among related functions. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor group) can be configured by software (e.g., an application or an application portion) to be a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component can be a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be appreciated that it can be decided whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured circuit system (for example, configured by software) for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (for example, hardwired) or temporarily configured (for example, programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (for example, programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components.Therefore, the hardware components described can be considered to be coupled in communication. In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components in the hardware components (for example, by appropriate circuits and buses). In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a storage device coupled in communication with it. Then, other hardware components can access the storage device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information). The various operations of the example methods described herein can be performed at least in part by a temporary configuration (for example, by software) or a permanent configuration to one or more processors performing related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein specific one or more processors are examples of hardware. For example, at least some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the " cloud computing " environment or operate as " software as a service " (SaaS). For example, at least some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of certain operations in the operation can be distributed between the processors, not only resides in a single machine, but also deployed across multiple machines. In some examples, the parts implemented by the processor or the processor can be located in a single geographic location (for example, in a home environment, an office environment or a server farm). In other examples, the parts implemented by the processor or the processor can be distributed across multiple geographic locations.

[0193] "Machine-readable storage media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "computer-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0194] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0195] “Non-transitory machine-readable storage medium” refers to a tangible medium capable of storing, encoding, or carrying instructions for execution by a machine.

[0196] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0197] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

1. A computer-implemented method comprising: providing, by the one or more processors, an XR user interface of the XR system to the user; capturing, by the one or more processors, video frame data of the user's hand; detecting, by the one or more processors, the hand of the user based on the video frame data and a hand detection model; generating, by the one or more processors, a cropping bounding box based on the detection of the hand and the video frame data; generating, by the one or more processors, cropped video frame data based on the cropping bounding box and the video frame data; generating, by the one or more processors, a 3D model of a portion of the hand of the user based on the cropped video frame data and a hand tracking and nail segmentation model; generating, by the one or more processors, a virtual object based on the 3D model and 3D texture of the portion of the hand of the user; as well as Display of the virtual object is provided in the XR user interface by the one or more processors.

2. The computer-implemented method of claim 1 , wherein: The clipping bounding box is a polygon.

3. The computer-implemented method of claim 1 , wherein: The cropping bounding box follows the outline of the hand of the user.

4. The computer-implemented method of claim 1 , wherein: The portion of the hand of the user is a fingernail of the hand of the user.

5. The computer-implemented method of claim 1 , wherein: The 3D texture represents fingernail polish.

6. The computer-implemented method of claim 1 , wherein: The XR system includes a head wearable device.

7. The computer-implemented method of claim 1 , wherein: The XR system includes a mobile device.

8. A machine comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: Provide users with an XR user interface for the XR system; capturing video frame data of the user's hand; detecting the hand of the user based on the video frame data and a hand detection model; generating a cropping bounding box based on the hand detection and the video frame data; generating cropped video frame data based on the cropping bounding box and the video frame data; generating a 3D model of a portion of the hand of the user based on the cropped video frame data and a hand tracking and nail segmentation model; generating a virtual object based on the 3D model and 3D texture of the portion of the hand of the user; and A display of the virtual object is provided in the XR user interface.

9. The machine according to claim 8, wherein The clipping bounding box is a polygon.

10. The machine according to claim 8, wherein The cropping bounding box follows the outline of the hand of the user.

11. The machine according to claim 8, wherein The portion of the hand of the user is a fingernail of the hand of the user.

12. The machine according to claim 8, wherein The 3D texture represents fingernail polish.

13. The machine according to claim 8, wherein: The XR system includes a head wearable device.

14. The machine of claim 8, wherein: The XR system includes a mobile device.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by a machine, cause the machine to perform operations comprising: Provide users with an XR user interface for the XR system; capturing video frame data of the user's hand; detecting the hand of the user based on the video frame data and a hand detection model; generating a cropping bounding box based on the hand detection and the video frame data; generating cropped video frame data based on the cropping bounding box and the video frame data; generating a 3D model of a portion of the hand of the user based on the cropped video frame data and a hand tracking and nail segmentation model; generating a virtual object based on the 3D model and 3D texture of the portion of the hand of the user; as well as A display of the virtual object is provided in the XR user interface.

16. The non-transitory machine-readable storage medium of claim 15, wherein: The clipping bounding box is a polygon.

17. The non-transitory machine-readable storage medium of claim 15, wherein: The cropping bounding box follows the outline of the hand of the user.

18. The non-transitory machine-readable storage medium of claim 15, wherein: The portion of the hand of the user is a fingernail of the hand of the user.

19. The non-transitory machine-readable storage medium of claim 15, wherein: The 3D texture represents fingernail polish.

20. The non-transitory machine-readable storage medium of claim 15, wherein: The XR system includes a head wearable device.

21. The non-transitory machine-readable storage medium of claim 15, wherein: The XR system includes a mobile device.

Images