Hand surface normal estimation

Abstract

A system for enhancing an image using hand surface normal estimation is provided. In a model training phase, 3D data of the hand at various positions are used to generate a 3D model of the hand. In one embodiment, target normal training data is generated, the target normal training data including a normal of a surface of a 3D model, and synthetic 2D image training data corresponding to the 3D model and the normal is generated. The target normal training data and the composite image training data are used for training a normal estimation model. The interactive application uses normal estimation to generate enhancements applied to the hand image data.

Description

[0001] Priority Statement

[0002] This application claims the benefit of U.S. Patent Application Serial No. 18 / 179,717, filed on March 7, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates generally to image processing, and more specifically to the processing of 2D and 3D images. Background Technology

[0004] Users of interactive platforms enjoy sharing images of themselves and their friends. Their enjoyment is amplified when they can add enhancements to the images. Attached Figure Description

[0005] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the figures:

[0006] Figure 1 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0007] Figure 2A It is a data flow diagram based on some examples of the data flow during the normal estimation model generation process.

[0008] Figure 2B It is an activity graph of a method for generating synthetic images based on training data from some examples.

[0009] Figure 2C This is an activity graph of a normal estimation model training method based on some examples.

[0010] Figure 3 This is a sequence of diagrams illustrating enhancement methods for generating enhanced images using normal estimation models, based on some examples.

[0011] Figure 4 A system of head-worn wearable devices based on some examples is shown.

[0012] Figure 5 It is a graphical representation of the data structures maintained in the database based on some examples.

[0013] Figure 6It is a diagrammatic representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0014] Figure 7 It is a graphical representation of a messaging system with both client-side and server-side functionalities, based on some examples.

[0015] Figure 8 It is a block diagram showing the software architecture in which examples can be implemented. Detailed Implementation

[0016] When users of an interactive platform share images of themselves and / or their friends, they can add enhancements to those images. These enhancements can take many forms, such as adding earrings, fun hats or other clothing to a user's image, or changing the appearance of the outer surface of a part of the user's image. For example, a user might want to look as if they are made of shiny metallic material or covered in fur. To achieve this effect, one or more parts of the user's 3D body model are rendered using 3D textures applied to the surfaces of the 3D model. To achieve a realistic appearance rendering of the surface using textures, 3D rays (called "normals" in the singular or "normals" in the plural) are determined perpendicular to or orthogonal to the modeled 3D surface. The normals used during the rendering process to determine lighting and shadow effects are used to give the rendered image a realistic 3D effect when displayed as a 2D image on a 2D display, etc. Conventional systems rely on having a 3D model available for rendering. In some cases, the 3D model requires a large amount of memory storage, which is impractical for mobile devices such as smartphones. Furthermore, the generation and manipulation of 3D models may require a large amount of computing resources, making it difficult to generate rendered images for video and other purposes in real time.

[0017] This disclosure relates to a method for estimating normals of human body parts (e.g., but not limited to, hands) based on 2D image data. Therefore, sufficient 3D information to generate 2D image enhancements on 2D images of 3D objects can be readily extracted from 2D image data. Enhancements can then be created for the image quickly while reducing the computational load on the processing system used to generate the enhancements. The estimated normals are used to generate renders of the hand using 3D textures to give the appearance of a hand composed of different materials. These renders are used to create enhancements to the hand, which can be displayed to users on interactive platforms, etc.

[0018] In some examples, the method for generating synthetic image training data includes: receiving 3D data of a hand and generating a 3D model of the hand based on the 3D data, wherein the 3D model includes a set of surfaces. The method also includes: generating target normal training data, which includes a set of normals of the set of surfaces of the 3D model; and generating synthetic 2D image training data based on the 3D model and the set of normals, which includes a set of synthetic 2D images. The method further includes: training a normal estimation model based on the synthetic 2D image training data and the target normal training data.

[0019] In use, the interactive system captures image data of a hand in a real-world scene and generates a set of estimated normals based on the image data and a normal estimation model. The interactive system then generates enhanced image data based on this set of estimated normals and the image data, and presents the enhanced image to the user in the user interface based on this enhanced image data.

[0020] Other technical features may be readily apparent to those skilled in the art from the accompanying drawings, description and claims.

[0021] Machine architecture

[0022] Figure 1This is a schematic representation of machine 100, within which instructions 102 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 100 to perform any or more of the methods discussed herein. For example, instructions 102 can cause machine 100 to perform any or more of the methods described herein. Instructions 102 transform the general, unprogrammed machine 100 into a specific machine 100 programmed to perform the described and illustrated functions in the described manner. Machine 100 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 100 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 102 specifying actions to be taken by machine 100. Furthermore, while a single machine 100 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 102 to perform any one or more of the methods discussed herein. For example, machine 100 may include computing system 602 or any one of a plurality of server devices forming part of interactive server system 610. In some examples, machine 100 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0023] Machine 100 may include a processor 104, a memory 106, and input / output (I / O) components 108 that can be configured to communicate with each other via bus 110. In the example, processor 104 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 112 and 114 that execute instruction 102. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although... Figure 1 Multiple processors 104 are shown, but machine 100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0024] Memory 106 includes main memory 116, static memory 118, and storage cells 120, all of which are accessible by processor 104 via bus 110. Main memory 106, static memory 118, and storage cells 120 store instructions 102 that implement any one or more of the methods or functions described herein. During execution of instructions 102 by machine 100, instructions 102 may also reside wholly or partially in main memory 116, in static memory 118, in machine-readable medium 122 within storage cell 120, within at least one processor in processor 104 (e.g., within the processor's cache memory), or in any suitable combination thereof.

[0025] I / O component 108 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurement results, etc. The specific I / O component 108 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It will be appreciated that I / O component 108 may include... Figure 1 Many other components are not shown. In various examples, I / O component 108 may include user output component 124 and user input component 126. User output component 124 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 126 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0026] In another example, I / O component 108 may include biometric component 128, motion component 130, environmental component 132, or position component 134, and various other components. For example, biometric component 128 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., authentication). To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to the data occurs). Any use of biometric data may be strictly limited to identification and verification purposes, and the biometric data may not be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures have been implemented to ensure the security and confidentiality of this sensitive information.

[0027] The moving part 130 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope). In some examples, the moving part 130 is incorporated into an inertial measurement unit (IMU).

[0028] For example, environmental component 132 may include one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), depth or distance sensors (e.g., sensors for determining the distance to an object or the depth of an object's features in a 3D coordinate system), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0029] Regarding the camera device, the computing system 602 may have a camera device system including, for example, a front-facing camera on the front surface of the computing system 602 and a rear-facing camera on the rear surface of the computing system 602. The front-facing camera may be used, for example, to capture still images and videos (e.g., "selfies") of the user of the computing system 602, which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, which are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, the computing system 602 may also include a 360° camera for capturing 360° photos and videos.

[0030] Furthermore, the camera system of computing system 602 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even include triple, quadruple, or quintuple rear camera configurations on the front and rear sides of computing system 602. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0031] The position component 134 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0032] A wide variety of technologies can be used to implement communication. I / O component 108 also includes a communication component 136 operable to couple machine 100 to network 138 or device 140 via a corresponding coupling or connection. For example, communication component 136 may include a network interface component or another suitable device that interfaces with network 138. In further examples, communication component 136 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, etc. Components (e.g.) (Low energy consumption) Components and other communication components that provide communication via other modes. Device 140 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0033] Furthermore, communication component 136 can detect identifiers, or include components operable to detect identifiers. For example, communication component 136 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be obtained via communication component 136, such as location via Internet Protocol (IP) geolocation, etc. Location can be obtained through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0034] Various memories (e.g., main memory 116, static memory 118, and the memory of processor 104) and storage units 120 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any or more of the methods or functions described herein. When executed by processor 104, these instructions (e.g., instruction 102) cause various operations to implement the disclosed examples.

[0035] Instructions 102 can be sent or received over network 138 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 136) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 102 can be sent or received via a transmission medium coupled to device 140 (e.g., peer-to-peer coupling).

[0036] Based on some examples, Figure 2A This is a data flow graph of the normal estimation model generation data flow 200a during the normal estimation model generation process. Figure 2B It is the activity graph of the synthetic image training data generation method 200b, and Figure 2C This is an activity graph of the normal estimation model training method 200c. The normal estimation model generation process is used by a data processing system such as machine 100 to generate a normal estimation model 228, which is used to estimate the hand normals for generating an enhanced image of the hand.

[0037] Reference Figure 2BIn operation 202, the synthetic image generation component 218 of machine 100 receives 3D data 230 of a hand 232 in various poses (positions). In some examples, the 3D data includes a set of 3D scans of the hand 232 in various poses. The 3D scans can be generated using various methods. In some examples, the 3D scan is a point cloud comprising a set of 3D points identified in a 3D coordinate system. The point cloud can be generated by any number of 3D scanning methods, such as, but not limited to, contact-based scanning using physical contact probes, laser triangulation (scanning laser or point laser), structured light scanning, time-of-flight scanning, photogrammetry, etc.

[0038] The composite image generation component 218 generates a set of 3D models based on a set of 3D scans of 3D data 230. For example, for each 3D scan of 3D data 230, the composite image generation component 218 generates a corresponding 3D model, which includes a set of surfaces or faces and a set of vertices defined by the intersections of the faces. For example, the 3D scan includes a point cloud comprising a set of 3D points captured from the hand 232, and the composite image generation component 218 generates a mesh based on the 3D points of the point cloud. Each surface in the set of surfaces is generated based on a subset of 3D points selected from the point cloud. In some examples, the 3D model is a point cloud comprising 3D points. In some examples, the 3D points of the 3D model comprise the average of a subset of 3D points from the point cloud containing the 3D scans.

[0039] In operation 204, the synthetic image generation component 218 generates target normal training data 222 based on the set of 3D models, including multiple sets of normals (e.g., a set of normals 236). For example, for each of the sets of surfaces constituting the 3D model, the synthetic image generation component 218 generates a ray with a 3D point on the surface of the 3D model as its origin, wherein the ray is perpendicular to the surface and projects outward from the 3D model. In some examples, the 3D model includes a point cloud, and the synthetic image generation component 218 generates a ray for each 3D point of the point cloud, with its origin at the 3D point of the point cloud, wherein the ray is perpendicular to a plane defined by at least three 3D points of the point cloud constituting the 3D model and projects outward away from the 3D model.

[0040] In operation 206, the synthetic image generation component 218 generates synthetic 2D image training data 220 based on the set of 3D models, target normal training data 222, and a set of camera device and lighting parameters 216. This synthetic 2D image training data 220 includes a set of synthetic 2D images of the hand 232, such as synthetic 2D image 234. For example, the synthetic image generation component 218 applies texture to the 3D model based on a specified lighting angle and lighting level. The synthetic image generation component 218 projects the textured 3D model onto a 2D plane based on the camera device distance relative to the 3D model and a specified camera device angle to generate a 2D image. In some examples, multiple combinations of lighting level, lighting angle, camera device angle, and camera device distance are used to create multiple synthetic 2D images for each 3D model. In some examples, the values ​​of lighting level, lighting angle, camera device angle, and camera device distance are randomized. When the generation process is complete, the synthetic 2D image training data 220 simulates images of various hands as if the images were captured using a physical camera device from various camera angles, camera distances, lighting angles, and lighting levels.

[0041] The target normal training data 222 generated by the synthetic image generation component 218 and the synthetic 2D image training data 220 are used to train the normal estimation model 228, as shown in the reference. Figure 2C A more complete description.

[0042] In some examples, 3D data 230 includes scans of physical objects other than the hand. Therefore, the target normal training data 222 and the synthetic 2D image training data 220 are not related to the hand, but rather to the physical object. The physical object can be any type of physical object suitable for 3D scanning.

[0043] In some examples, 3D data 230 is not 3D data of the hand, but 3D data of other appendages or parts of the human body, such as, but not limited to, arms, legs, feet, head, etc. Therefore, the target normal training data 222 and the synthetic 2D image training data 220 are not related to the hand, but to that part of the human body.

[0044] Reference Figure 2CIn operation 208, the training component 226 of machine 100 receives synthetic 2D image training data 220 comprising a set of synthetic 2D images (such as synthetic 2D image 234) and detects the position of the hand in each synthetic 2D image. For example, the training component 226 uses computer vision methods to detect the position of the hand in the synthetic 2D images, including but not limited to Harris corner detection, Shi-Tomasi corner detection, Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Speed-Up Segment Test Features (FAST), Oriented FAST, and Rotation BRIEF (ORB).

[0045] In some examples, training component 226 uses artificial intelligence methods and a detection model 224 previously generated using machine learning methods to detect the position of the hand. In some examples, the detection model 224 includes, but is not limited to, neural networks, learned vector quantization networks, logistic regression models, support vector machines, random decision forests, Naive Bayes models, linear discriminant analysis models, K-nearest neighbor models, etc. In some examples, the machine learning methods may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, anomaly detection, etc.

[0046] In operation 210, training component 226 generates a set of cropping boundaries, such as, but not limited to, bounding boxes, based on the detected position of the hand in the synthetic 2D image. For example, in a synthetic 2D image comprising pixels organized by rows and columns, training component 226 determines the leftmost pixel among the pixels constituting the hand and sets the left boundary in a pixel column to the left of the leftmost pixel. In a similar manner, training component 226 determines the topmost pixel among the pixels constituting the hand and sets the upper cropping boundary in a row above the topmost pixel, determines the bottommost pixel and sets the lower cropping boundary in a row below the bottommost pixel, and determines the rightmost pixel and sets the right cropping boundary in a pixel column to the right of the rightmost pixel.

[0047] In operation 212, training component 226 generates a cropped synthetic 2D hand image 240 by cropping the synthetic 2D image based on cropping boundaries. For example, training component 226 crops all pixels to the left of the left cropping boundary, to the right of the right cropping boundary, above the upper cropping boundary, and below the lower cropping boundary in the synthetic 2D image.

[0048] In operation 214, training component 226 generates a normal estimation model 228 based on training data from a pairwise input-output training dataset comprising a cropped synthetic 2D hand image 240 and target normal training data 222. For example, model parameters 238 provide the parameters or coefficients of the normal estimation model 228. During training, these model parameters 238 are tuned based on the input-output training pairs of the cropped synthetic 2D hand image 240 and target normal training data 222. After the model parameters 238 are tuned (after training), they are used by training component 226 to generate a trained machine learning normal estimation model 228.

[0049] Training component 226 trains normal estimation model 228 based on multiple input-output pairs of the cropped synthetic 2D hand image 240 and target normal training data 222. For example, training component 226 can train normal estimation model 228 by minimizing the loss function based on comparing multiple estimated normals with multiple normals 236 of the target normal training data 222. The normal estimation model 228 may include any one or a combination of classifiers or neural networks, such as artificial neural networks, convolutional neural networks, adversarial networks, generative adversarial networks, deep feedforward networks, radial basis function networks, recurrent neural networks, long / short-term memory networks, gated recurrent units, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep belief networks, deep convolutional networks, deconvolutional networks, deep convolutional inverse graph networks, liquid machines, extreme learning machines, echo state networks, deep residual networks, Kohonen networks, support vector machines, neural Turing machines, etc.

[0050] The model parameters 238 of the normal estimation model 228 are trained, tuned, or optimized by minimizing the loss function for multiple sets of paired, cropped, synthetic 2D hand images 240 and target normal training data 222. In this way, the normal estimation model 228 is trained to generate multiple sets of estimated normals based on image data of one or more hands.

[0051] The normal estimation model and hand detection model 224 generated by training component 226 are used in image enhancement methods, as shown in reference. Figure 3 A more complete description.

[0052] Figure 3This is a sequence diagram illustrating an enhancement method 300 for generating enhanced images (such as enhanced image 336 for a user interface) based on a normal estimation model 228, using some examples. An interactive system 312 uses enhancement method 300 to provide an enhanced user interface 310 of an interactive application 328 to a user 316. The interactive application 328 provides the enhanced user interface 310 to the user 316 to allow the user to create and view enhanced images, such as enhanced image 336.

[0053] In operation 302, the interactive application 328 uses one or more camera devices 330 of the interactive system 312 to capture image data 320 of the real-world scene 318, including one or more hands 338 within the real-world scene 318. The one or more camera devices 330 transmit the image data 320 to the normal estimation component 326 and the interactive application 328.

[0054] In operation 304, normal estimation component 326 receives image data 320. Normal estimation component 326 generates a set of estimated normals 322 for each image in image data 320, including a hand, based on image data 320, detection model 224, and normal estimation model 228. For example, normal estimation component 326 uses detection model 224 to detect the location of the hand in the image of image data 320, as described with reference to training normal estimation model 228. Normal estimation component 326 generates a set of cropping boundaries and uses this set of cropping boundaries to crop the image, as described with reference to training normal estimation model 228. Normal estimation component 326 then uses artificial intelligence methods to generate a set of estimated normals 322 based on the cropped image data and normal estimation model 228, as described more fully with reference to training normal estimation model 228. Normal estimation component 326 transmits this set of estimated normals 322 to interactive application 328.

[0055] In operation 306, the interactive application 328 receives a set of estimated normals 322 and uses the enhancement generation component 340 to generate enhanced image data 324 based on the set of estimated normals 322. For example, the set of estimated normals 322 are estimated normals of hand image data of hand 338 in real-world scene 318. Each normal in the set of estimated normals 322 includes a ray with a 3D point as its origin. The set of origins for the normals includes a 3D point cloud defining the surface of hand 338 in a 3D coordinate system. The vector of the ray indicates a direction orthogonal to the surface of hand 338 in the 3D coordinate system. Therefore, the set of estimated normals 322 includes a partial 3D model, to which the enhancement generation component 340 applies a texture and projects a 2D rendering of the textured 3D model onto a 2D plane of the image data at the location of hand 338 in image data 320, thereby applying enhancement to the image data of hand to create enhanced image 336. The interactive application 328 transmits the enhanced image data 324 of the enhanced image 336 to the image processing component 314 of the interactive system 312.

[0056] In some examples, image data 320 includes an image with pixels, and normal estimation component 326 generates a set of estimated normals 322, wherein each pixel of the image data of hand 338 has a corresponding normal in the set of estimated normals 322.

[0057] In operation 308, image processing component 314 receives enhanced image data 324 from interactive application 328. Image processing component 314 generates user interface data 332 for enhanced user interface 310 based on enhanced image data 324. Image processing component 314 provides enhanced user interface 310, including enhanced image 336, to user 316 via display 334 of interactive system 312 based on user interface data 332.

[0058] In some examples, the enhanced image 336 includes an image generated and overlaid on the image of the original image data 320 at the location within the image containing the hand 338. For example, the enhancement generation component 340 can generate an image of a hand covered by a glove or similar material. In some examples, the hand 338 is overlaid with a generated image in the form of a cartoon animal paw for comedic effect. In some examples, the hand 338 is enlarged or reduced in size. In some examples, the hand 338 is overlaid with an image of the hand 338 having a different surface texture other than skin (such as a shiny metallic texture).

[0059] In some examples, the interactive system 312 continuously repeats the operation of the enhancement method 300 to provide the user 316 with an enhanced user interface 310 in real time.

[0060] In some examples, the operation of interactive application 328 is distributed across a network. For instance, interactive application 328 is a web application that connects to a server via a network such as the Internet. Interactive application 328 transmits image data 320 to the server, and the server generates enhanced image data 324 and transmits the enhanced image data 324 to interactive application 328 via the network.

[0061] In some examples, the interactive application 328 synthesizes image data 320 and enhanced image data 324, and transmits the synthesized video frame data to the image processing component 314. The image processing component 314 receives the synthesized video frame data and generates user interface data 332 based on the synthesized video frame data.

[0062] Systems with head-mounted devices

[0063] Figure 4 A system 400 according to some examples is shown, which includes a head-worn wearable device 418 having a selector input device. Figure 4 This is a high-level functional block diagram of an example head-mounted wearable device 418 that is communicatively coupled to mobile devices 414 and various server systems 404 (e.g., interactive server system 610) via various networks 608.

[0064] The head-mounted wearable device 418 includes one or more camera devices, each of which may be, for example, a visible light camera 408, an infrared emitter 410, and an infrared camera 412.

[0065] Mobile device 414 connects to head-mounted wearable device 418 using both low-power wireless connection 416 and high-speed wireless connection 420. Mobile device 414 also connects to server system 404 and network 406.

[0066] The head-worn wearable device 418 also includes two image displays of an optical component image display 422. The two image displays 422 of the optical component include an image display associated with the left lateral side of the head-worn wearable device 418 and an image display associated with the right lateral side of the head-worn wearable device 418. The head-worn wearable device 418 also includes an image display driver 424, an image processor 426, a low-power circuitry system 428, and a high-speed circuitry system 430. The image displays 422 of the optical component are used to present images and videos to the user of the head-worn wearable device 418, including images that may include a graphical user interface.

[0067] The image display driver 424 commands and controls the image display 422 of the optical components. The image display driver 424 can directly deliver image data to the image display 422 of the optical components for presentation, or it can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data can be formatted according to compression formats such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), or Exchangeable Image File Format (EXIF).

[0068] The head-worn device 418 includes a frame and a pole (or temple) extending laterally from the frame. The head-worn device 418 also includes a user input device 432 (e.g., a touch sensor or a press button), comprising an input surface on the head-worn device 418. The user input device 432 (e.g., a touch sensor or a press button) is used to receive input selections from the user through a graphical user interface that manipulates the presented image.

[0069] Figure 4 The components shown for the head-worn wearable device 418 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in the blocks, frames, hinges, or nose bridge of the head-worn wearable device 418. The left and right visible light camera devices 408 may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0070] The head-mounted wearable device 418 includes a memory 402 that stores instructions for performing a subset or all of the functions described herein. The memory 402 may also include a storage device.

[0071] like Figure 4As shown, the high-speed circuit system 430 includes a high-speed processor 434, a memory 402, and a high-speed wireless circuit system 436. In some examples, an image display driver 424 is coupled to the high-speed circuit system 430 and operated by the high-speed processor 434 to drive the left and right image displays of the image display 422 of the optical components. The high-speed processor 434 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the head-worn device 418. The high-speed processor 434 includes the processing resources required to manage high-speed data transmission over the high-speed wireless connection 420 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436. In some examples, the high-speed processor 434 executes the operating system of the head-worn device 418 (e.g., a LINUX operating system) or other such operating system, and this operating system is stored in the memory 402 for execution. Among other duties, the high-speed processor 434, which executes the software architecture of the head-worn device 418, manages data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, the high-speed wireless circuit system 436 may implement other high-speed communication standards.

[0072] The low-power wireless circuit system 438 and high-speed wireless circuit system 436 of the head-worn wearable device 418 may include a short-range transceiver (Bluetooth). TM The device 414 includes a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). The mobile device 414, including transceivers communicating via low-power wireless connection 416 and high-speed wireless connection 420, can be implemented using the architectural details of the head-mounted wearable device 418, and other components of the network 406 can also be implemented in this way.

[0073] Memory 402 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 408, the infrared camera 412, and the image processor 426, as well as the generated images for display by the image display driver 424 on an image display 422 in the optical components. While memory 402 is shown as integrated with the high-speed circuitry 430, in some examples, memory 402 may be a separate, independent component of the head-mounted wearable device 418. In some such examples, electrical wiring may provide a connection from the image processor 426 or the low-power processor 440 to memory 402 via a chip including the high-speed processor 434. In some examples, the high-speed processor 434 may manage addressing of memory 402 such that the low-power processor 440 will activate the high-speed processor 434 whenever a read or write operation involving memory 402 is required.

[0074] like Figure 4 As shown, the low-power processor 440 or high-speed processor 434 of the head-mounted wearable device 418 may be coupled to a camera device (visible light camera 408, infrared emitter 410 or infrared camera 412), an image display driver 424, a user input device 432 (e.g., a touch sensor or a press button), and a memory 402.

[0075] The head-mounted wearable device 418 is connected to a host computer. For example, the head-mounted wearable device 418 is paired with the mobile device 414 via a high-speed wireless connection 420 or connected to the server system 404 via a network 406. The server system 404 may be one or more computing devices as part of a service or network computing system, for example, it includes a processor, memory, and network communication interfaces to communicate with the mobile device 414 and the head-mounted wearable device 418 via the network 406.

[0076] Mobile device 414 includes a processor and a network communication interface coupled to the processor. The network communication interface enables communication via network 406, low-power wireless connection 416, or high-speed wireless connection 420. Mobile device 414 may also store at least a portion of the instructions for generating binaural audio content in the memory of mobile device 414 to implement the functions described herein.

[0077] The output components of the head-worn wearable device 418 include visual components, such as displays (e.g., liquid crystal displays (LCDs), plasma display panels (PDPs), light-emitting diode (LED) displays, projectors, or waveguides). The image display of the optical components is driven by an image display driver 424. The output components of the head-worn wearable device 418 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input devices 432) of the head-worn wearable device 418, mobile device 414, and server system 404 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0078] The head-mounted wearable device 418 may also include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted wearable device 418. For example, peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such elements described herein.

[0079] For example, biometric components include those for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include positioning sensor components (e.g., GPS receiver components) for generating position coordinates, and Wi-Fi or Bluetooth for generating positioning system coordinates. TM Transceivers, altitude sensor components (e.g., altimeters or barometers that detect air pressure and determine altitude), orientation sensor components (e.g., magnetometers), etc. The coordinates of such a positioning system can also be received from the mobile device 414 via low-power wireless connection 416 and high-speed wireless connection 420 through low-power wireless connection 416 or high-speed wireless connection 436, either via low-power wireless circuitry 438 or high-speed wireless circuitry 436.

[0080] Data Architecture

[0081] Figure 5This is a schematic diagram illustrating a data structure 500 that can be stored in a database 504 of an interactive server system 610, according to certain examples. Although the contents of the database 504 are shown as including multiple tables, it will be appreciated that the data can be stored in other types of data structures (e.g., object-oriented databases).

[0082] Database 504 includes message data stored in message table 506. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and a payload. See below for reference. Figure 5 Further details are provided regarding information that can be included in the message and is contained within the message data stored in message table 506.

[0083] Entity table 508 stores entity data and (e.g., by reference) links to entity diagram 510 and profile data 502. Entities for which records are maintained within entity table 508 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of entity type, any entity whose data is stored in the interactive server system 610 can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0084] Entity graph 510 stores information about relationships and associations between entities. As an example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as an individual user subscribing to digital content from a business or publishing user (e.g., a newspaper or other digital media channel, or a brand). Other relationships can be two-way, such as the "friendship" between individual users of interactive system 600.

[0085] Certain licenses and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friendship between individual users) can include authorization for the posting of digital content items between the individual users, but certain restrictions or filters can be imposed on the posting of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user can impose varying degrees of restrictions on the posting of digital content from the business user to the individual user, and can significantly restrict or prevent the posting of digital content from the individual user to the business user. As an example of an entity, a specific user can record certain restrictions (e.g., through privacy settings) in the entity's records within entity table 508. Such privacy settings can be applied to all types of relationships within the context of interaction system 600, or selectively applied only to certain types of relationships.

[0086] Profile data 502 stores various types of profile data about a specific entity. Based on privacy settings specified by the specific entity, profile data 502 can be selectively used and presented to other users of the interactive system 600. In the case of an individual, profile data 502 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected by the user. The specific user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 600 and on a map interface displayed to other users through the interactive client 604. The set of avatar representations may include a "situation avatar," which presents a graphical representation of a situation or activity that the user can choose to communicate at a specific time.

[0087] In the case that the entity is a group, in addition to the group name, members and various settings of the associated group (e.g., notifications), the group profile data 502 may similarly include one or more avatars associated with the group.

[0088] Database 504 also stores enhancement data, such as overlays or filters, in enhancement table 512. Enhancement data is associated with and applied to videos (video data is stored in video table 514) and images (image data is stored in image table 516).

[0089] In some examples, filters are displayed as overlays on images or videos during presentation to the message recipient. Filters can be of various types, including user-selected filters from a set of filters presented to the message sender by the interactive client 604 when the message sender is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the message sender based on geographic location. For example, a geolocation filter specific to a nearby or particular location can be presented by the interactive client 604 within the user interface based on geolocation information determined by the Global Positioning System (GPS) unit of the computing system 602.

[0090] Another type of filter is a data filter, which can be selectively presented to the message sender by the interactive client 604 based on other inputs or information collected by the computing system 602 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the message sender, the battery life of the computing system 602, or the current time.

[0091] Other augmented data that can be stored in image table 516 includes augmented reality content items (e.g., corresponding to an applied lens or augmented reality experience). Augmented reality content items can be real-time special effects and / or sounds that can be added to images or videos.

[0092] As described above, augmented data includes augmented reality (AR), virtual reality (VR), and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., video or images). This includes real-time modifications that modify images as they are captured by the device sensors (e.g., one or more cameras) using computing system 602 and then display the modified image on the screen of computing system 602. This also includes modifications to stored content such as video clips that can be modified within a set or group. For example, in computing system 602 that accesses multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clip. Similarly, real-time video capture can be modified to show how the video image currently captured by the sensors of computing system 602 will modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device sensors can be recorded and stored in memory with or without modification (or both). In some systems, preview features can show how different augmented reality content items will look simultaneously in different windows on the display. For example, this can make it possible to view multiple windows with different pseudo-random animations on the monitor at the same time.

[0093] Therefore, using augmented reality content items and various systems, or other transformation systems that use that data to modify the content, can involve the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking such objects as they leave and enter the field of view in a video frame and as they move around within the field of view, and modifying or transforming such objects while they are being tracked. In various examples, different methods can be used to implement such transformations. Some examples may involve generating a 3D mesh model of one or more objects, and using transformations of the model and animated textures within the video to implement the transformation. In some examples, tracking points on the object can be used to place an image or texture (which can be two-dimensional or three-dimensional) at the tracked location. In yet another example, neural network analysis of video frames can be used to place images, models, or textures within the content (e.g., images or video frames). Therefore, augmented reality content items refer both to the images, models, and textures used to create transformations within the content, and to the additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0094] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and store it in the device's memory, or the device's sensors can be used to generate a video stream. Furthermore, computer-animated models can be used to process any object, such as a human face and body parts, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0095] In some examples, when a specific modification is selected along with the content to be transformed, the computing device identifies the element to be transformed, and then, if the element to be transformed exists in the video frame, detects and tracks it. The elements of the object are modified according to the request for modification, thereby transforming the frames of the video stream. For different types of transformations, the transformation of video stream frames can be performed using different methods. For example, for frame transformations that primarily involve changing the form of the elements of an object, feature points for each element of the object are calculated (e.g., using an Active Shape Model (ASM) or other known methods). A feature-point-based mesh is then generated for each element of the object. This mesh is used in subsequent stages of tracking the elements of the object in the video stream. During tracking, the mesh for each element is aligned with the position of each element. Additional points are then generated on the mesh.

[0096] In some examples, transforming certain regions of an object using its elements can be performed by calculating feature points for each element of the object and generating a mesh based on those calculated feature points. Points are generated on the mesh, and then various regions are generated based on those points. The elements of the object are then tracked by aligning the regions for each element with positions for each of at least one element, and the properties of the regions can be modified based on modification requests, thereby transforming frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications can involve: changing the color of the region; removing a portion of the region from a frame of the video stream; including a new object in the region based on the modification request; and modifying or distorting the elements of the region or object. In various examples, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space of options for model animation.

[0097] In some examples of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. The Active Shape Model (ASM) algorithm is then applied to the facial regions of the image to detect facial feature reference points.

[0098] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate virtual features, which represent distinguishable points present in most of the images considered. For example, for a face landmark, the location of the left pupil could be used. If the initial landmark is not recognizable (e.g., if the person is wearing an eye patch), secondary landmarks can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation that minimizes the average Euclidean distance between the points of the shape (which allows for translation, scaling, and rotation). The mean shape is the average of the aligned training shapes.

[0099] The transformation system can capture image or video streams on a client device (e.g., computing system 602) and perform complex image manipulations locally on computing system 602 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, mood shifts (e.g., changing a face from frowning to smiling), state shifts (e.g., aging a subject, reducing apparent age, changing gender), style shifts, application of graphical elements, and any other suitable image or video manipulation implemented by a convolutional neural network that has been configured to execute efficiently on computing system 602.

[0100] In some examples, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream (e.g., a selfie) using a computing system 602 that has a neural network operating as part of an interactive client 604 operating on the computing system 602. A transformation system operating within the interactive client 604 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model for transforming the image data, or the computer animation model may exist in association with the interface described herein. This modification icon will be included as part of the modification operation, serving as a basis for modifying the user's face within the image or video stream. Once a modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the computing system 602. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, users can capture image or video streams, and once an edit icon is selected, the modified result can be presented to the user in real-time or near real-time. Furthermore, the changes can be persistent while the video stream is being captured and the selected edit icon continues to toggle. Machine learning neural networks can be used to achieve such modifications.

[0101] A graphical user interface (GUI) presenting modifications performed by a transformation system can provide users with additional interactive options. These options can be based on the interface used to initiate content capture and select specific computer animation models (e.g., initiated from a content creator GUI). In various examples, modifications can be persistent after the initial selection of the modification icon. Users can toggle the modification on or off by tapping or otherwise selecting the face being modified by the transformation system and save it for later viewing or browsing other areas of the imaging application. In cases where the transformation system modifies multiple faces, users can globally toggle the modification on or off by tapping or selecting a single face being modified and displayed within the GUI. In some examples, individual faces within a set of multiple faces can be modified separately, or such modifications can be toggled individually by tapping or selecting a single face or a series of individual faces displayed within the GUI.

[0102] Story table 518 stores data about collections of messages, along with associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user maintaining records in entity table 508). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the user interface of interactive client 604 can include user-selectable icons that allow message senders to add specific content to their personal stories.

[0103] The collection can also constitute a "live story" as a collection of content from multiple users, created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a common location event at a specific time can be presented with the option to contribute content to a specific live story, for example, via the user interface of interactive client 604. Live stories can be identified to a user by interactive client 604 based on their location. The end result is a "live story" told from a community perspective.

[0104] Another type of content collection is called a "location story," which allows users whose computing system 602 is located within a specific geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributing to a location story may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0105] As mentioned above, video table 514 stores video data, which in some examples is associated with messages whose records are maintained within message table 506. Similarly, image table 516 stores image data, which is associated with messages whose message data is stored in entity table 508. Entity table 508 can associate various enhancements from enhancement table 512 with various images and videos stored in image table 516 and video table 514.

[0106] Database 504 also includes entity relationship information collected by entity relationship system 722.

[0107] Networked computing environment

[0108] Figure 6This is a block diagram illustrating an example interactive system 600 for facilitating interactions on a network, such as exchanging text messages, making text-to-audio and video calls, or playing games. Interactive system 600 includes multiple computing systems 602, each hosting multiple applications including an interactive client 604 and other applications 606. Each interactive client 604 is communicatively coupled to other instances of the interactive client 604 (e.g., hosted on corresponding other computing systems 602), an interactive server system 610, and a third-party server 612 via one or more communication networks including a network 608 (e.g., the Internet). The interactive client 604 can also communicate with the locally hosted applications 606 using an application programming interface (API).

[0109] Each computing system 602 may include one or more user devices, such as mobile devices 414, head-mounted wearable devices 418, and computer client devices 614, which are communicatively connected to exchange data and messages.

[0110] Interactive client 604 interacts with other interactive clients 604 and with interactive server system 610 via network 608. The data exchanged between interactive clients 604 (e.g., interaction 616) and between interactive client 604 and interactive server system 610 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0111] Interactive server system 610 provides server-side functionality to interactive client 604 via network 608. While some functions of interactive system 600 are described herein as being performed by interactive client 604 or interactive server system 610, the location of certain functions—whether within interactive client 604 or interactive server system 610—may be a design choice. For example, it may be technically preferred that specific technologies and functions are initially deployed within interactive server system 610, but later migrated to interactive client 604, which has sufficient processing power within computing system 602.

[0112] The interactive server system 610 supports various services and operations provided to the interactive client 604. Such operations include sending data to and receiving data from the interactive client 604, and processing data generated by the interactive client 604. This data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchange within the interactive system 600 is invoked and controlled via functions available through the user interface (UI) of the interactive client 604.

[0113] The focus now shifts to the interactive server system 610. An application programming interface (API) server 618 is coupled to the interactive server 620 and provides it with a programming interface, making the functionality of the interactive server 620 accessible to the interactive client 604, other applications 606, and third-party servers 612. The interactive server 620 is communicatively coupled to a database server 622, thereby facilitating access to a database 624 that stores data associated with the interactions processed by the interactive server 620. Similarly, a web server 626 is coupled to the interactive server 620 and provides a web-based interface to it. For this purpose, the web server 626 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0114] Application Programming Interface (API) server 618 receives and sends interactive data (e.g., command and message payloads) between interactive server 620 and computing system 602 (and, for example, interactive client 604 and other applications 606) and third-party server 612. Specifically, API server 618 provides a set of interfaces (e.g., routines and protocols) that interactive client 604 and other applications 606 can invoke or query to activate the functionality of interactive server 620. Application Programming Interface (API) server 618 exposes various functions supported by interaction server 620, including account registration; login functionality; sending interactive data from one interaction client 604 to another interaction client 604 via interaction server 620; transferring media files (e.g., images or videos) from interaction client 604 to interaction server 620; setting media data sets (e.g., stories); retrieving the friend list of users of computing system 602; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends within the entity graph; and opening (e.g., application events associated with interaction client 604).

[0115] Interactive server 620 hosts multiple systems and subsystems, as shown below. Figure 7 Describe it.

[0116] Application of links

[0117] Returning to interactive client 604, the features and functionality of the external resource (e.g., a linked application 606 or applet) become available to the user via the interface of interactive client 604. In this context, "external" refers to the fact that application 606 or applet is outside of interactive client 604. External resources are typically provided by a third party, but may also be provided by the creator or provider of interactive client 604. Interactive client 604 receives user selections regarding options for launching or accessing the features of such external resources. The external resource may be application 606 (e.g., a "native app") installed on computing system 602, or a smaller version (e.g., an "app") of an application hosted on computing system 602 or located remotely on computing system 602 (e.g., on a third-party server 612). A smaller version of an application includes a subset of the features and functionality of the application (e.g., a full-scale, local version of the application) and is implemented using markup language documentation. In some examples, a smaller version of an application (e.g., an "app") is a web-based markup language version of the application and is embedded in interactive client 604. In addition to using markup language documents (e.g., .*ml files), mini-programs can include scripting languages ​​(e.g., .*js files or .json files) and style sheets (e.g., .*ss files).

[0118] In response to receiving a user selection of an option for launching or accessing an external resource, interactive client 604 determines whether the selected external resource is a web-based external resource or a locally installed application 606. In some cases, application 606, locally installed on computing system 602, can be launched independently of interactive client 604 and separately from interactive client 604, for example, by selecting the icon corresponding to application 606 on the home screen of computing system 602. A smaller version of such an application can be launched or accessed via interactive client 604, and in some examples, no part of the smaller application can be accessed outside of interactive client 604, or only a limited portion of the smaller application can be accessed outside of interactive client 604. A smaller application can be launched by receiving, for example, a markup language document associated with the smaller application from third-party server 612 and processing such a document.

[0119] In response to determining that the external resource is a locally installed application 606, the interactive client 604 instructs the computing system 602 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 604 communicates with, for example, a third-party server 612 to obtain a markup language document corresponding to the selected external resource. The interactive client 604 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 604.

[0120] Interactive client 604 can notify users of computing system 602 or other users (e.g., "friends") associated with such users of one or more external resources of ongoing activities. For example, interactive client 604 can provide participants in a conversation (e.g., a chat session) within interactive client 604 with notifications related to the current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join an active external resource or to activate (in a group of friends) a recently used but currently inactive external resource. External resources can provide participants in the conversation, each using their respective interactive client 604, with the ability to share items, conditions, states, or locations within the external resource with one or more members of a group of users during the chat session. Shared items can be interactive chat cards that chat members can use to interact, for example, to activate the corresponding external resource, view specific information within the external resource, or take chat members to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on interactive client 604. Based on the current context of the external resource, the external resource can selectively include different media items in the response.

[0121] Interactive client 604 can present a list of available external resources (e.g., application 606 or mini-program) to the user to launch or access a given external resource. This list can be presented as a context-sensitive menu. For example, the icons representing different applications (or mini-programs) of application 606 (or mini-program) can change based on how the user launches the menu (e.g., from a conversational interface or a non-conversational interface).

[0122] System Architecture

[0123] Figure 7 This is a block diagram illustrating further details of an interactive system 600 based on some examples. Specifically, the interactive system 600 is shown as including an interactive client 604 and an interactive server 620. The interactive system 600 comprises multiple subsystems, which are supported on the client side by the interactive client 604 and on the server side by the interactive server 620. Example subsystems are discussed below.

[0124] The image processing system 702 provides various functions that enable users to capture and enhance (e.g., enhance or otherwise modify or edit) media content associated with a message.

[0125] The camera device system 704 includes (e.g., in a camera device application) control software that (e.g., directly or via an operating system) interacts with and controls the hardware camera device hardware of the computing system 602 to modify and enhance real-time images captured and displayed via the interactive client 604.

[0126] Enhancement system 706 provides functionality related to the generation and distribution of enhancements (e.g., media overlays) for images captured in real time by the camera device of computing system 602 or retrieved from the memory of computing system 602. For example, enhancement system 706 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for interactive client 604 to enhance real-time images received via camera device system 704 or stored images retrieved from memory 402 of computing system 602. These enhancements are selected and presented to the user of interactive client 604 by enhancement system 706 based on some inputs and data, such as:

[0127] • The geographic location of computing system 602; and

[0128] • Entity relationship information of users of computing system 602.

[0129] Enhancements may include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) at computing system 602 for transmission in messages, or to video content such as video content streams or feeds sent from interactive client 604. Therefore, image processing system 702 can interact with various subsystems of communication system 708 and support various subsystems of communication system 808, such as messaging system 710 and video communication system 712.

[0130] Media overlays may include text or image data that can be superimposed on photographs taken by computing system 602 or video streams produced by computing system 602. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of live events, or business names (e.g., Beach Cafe). In other examples, image processing system 702 uses the geolocation of computing system 602 to identify media overlays that include the name of a business at the geolocation of computing system 602. Media overlays may include additional tags associated with the business. Media overlays may be stored in database 624 and accessed through database server 622.

[0131] Image processing system 702 provides a user-based publishing platform that allows users to select a geographic location on a map and upload content associated with that location. Users can also specify which media overlays should be provided to other users. Image processing system 702 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0132] The Augmented Creation System 714 supports augmented reality developer platforms and includes applications for content creators (e.g., artists and developers) to create and publish interactive clients 604, such as augmented reality experiences. The Augmented Creation System 714 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates.

[0133] In some examples, the enhancement creation system 714 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geolocation through a bidding process. For example, the enhancement creation system 714 associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time.

[0134] Communication system 708 is responsible for enabling and processing various forms of communication and interaction within interactive system 600, and includes messaging system 710, audio communication system 716, and video communication system 712. Messaging system 710 is responsible for enabling temporary or time-limited access to content by interactive client 604. Messaging system 710 includes (e.g., within short-lived timer system 718) multiple timers that selectively enable access (e.g., for presentation and display) of messages and associated content via interactive client 604 based on duration and display parameters associated with a message or set of messages (e.g., a story). Further details regarding the operation of short-lived timer system 718 are provided below. Audio communication system 716 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 604. Similarly, video communication system 712 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 604.

[0135] The user management system 720 is operationally responsible for managing user data and profiles, and includes an entity relationship system 722 that maintains information about the relationships between users of the interaction system 600.

[0136] The collection management system 724 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into “event galleries” or “event stories.” Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be made available as a “story” for the duration of the concert. The collection management system 724 can also be responsible for publishing icons that notify the user interface of the interactive client 604 of the availability of specific collections. The collection management system 724 includes curatorial functions that enable collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Furthermore, the collection management system 724 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be compensated for including user-generated content in a collection. In such cases, the collection management system 724 operates to automatically pay such users for using their content.

[0137] Map system 726 provides various geolocation functions and supports the presentation of map-based media content and messages by interactive client 604. For example, map system 726 enables the display (e.g., stored in profile data 502) of user icons or avatars on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content (e.g., a collection of messages including photos and videos) generated by such friends. For example, on the map interface of interactive client 604, messages posted by a user from a specific geolocation to interactive system 600 can be displayed to the specific user's "friends" within the context of that specific location on the map. Users can also share their location and status information with other users of interactive system 600 via interactive client 604 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 604.

[0138] Game system 728 provides various game functions within the context of interactive client 604. Interactive client 604 provides a game interface that offers a list of available games that can be initiated by a user within the context of interactive client 604 and played with other users of interactive system 600. Interactive system 600 also enables specific users to invite other users to play specific games by sending invitations from interactive client 604. Interactive client 604 also supports sending and receiving audio, video, and text messages (e.g., chat) within the context of playing the game, provides leaderboards for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0139] External resource system 730 provides interactive client 604 with an interface to communicate with remote servers (e.g., third-party server 612) to launch or access external resources (i.e., applications or applets). Each third-party server 612 hosts applications or smaller versions of applications (e.g., game applications, utility applications, payment applications, or ride-sharing applications) based on markup languages ​​(e.g., HTML5). Interactive client 604 can launch web-based resources (e.g., applications) by accessing HTML5 files from the third-party server 612 associated with the web-based resource. The application hosted by third-party server 612 is programmed in JavaScript using a software development kit (SDK) provided by interactive server 620. The SDK includes application programming interfaces (APIs) with functions that can be called or triggered by the web-based application. Interactive server 620 hosts a JavaScript library that provides access to a given external resource for specific user data of interactive client 604. HTML5 is an example of a technology used for programming games, but applications and resources programmed using other technologies can be used.

[0140] To integrate the SDK's functionality into a web-based resource, a third-party server 612 downloads the SDK from an interactive server 620, or receives the SDK from the third-party server 612 in another manner. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 604 into the web-based resource.

[0141] The SDK stored on the interactive server system 610 effectively bridges the gap between external resources (e.g., application 606 or applet) and the interactive client 604. This provides users with a seamless experience communicating with other users on the interactive client 604 while preserving the look and feel of the interactive client 604. To bridge communication between the external resources and the interactive client 604, the SDK facilitates communication between the third-party server 612 and the interactive client 604. The WebViewJavaScriptBridge running on the computing system 602 establishes two unidirectional communication channels between the external resources and the interactive client 604. Messages are sent asynchronously between the external resources and the interactive client 604 via these communication channels. Each SDK function call is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0142] By using the SDK, not all information from the interactive client 604 is shared with the third-party server 612. The SDK restricts which information is shared based on the needs of the external resource. Each third-party server 612 provides the interactive server 620 with an HTML5 file corresponding to the web-based external resource. The interactive server 620 can add a visual representation (e.g., box design or other graphics) of the web-based external resource to the interactive client 604. Once the user selects the visual representation or instructs the interactive client 604 to access the features of the web-based external resource through the interactive client 604's GUI, the interactive client 604 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0143] Interactive client 604 presents a graphical user interface (GUI) for an external resource (e.g., a login page or title screen). During, before, or after presenting the login page or title screen, interactive client 604 determines whether the initiated external resource has previously been authorized to access interactive client 604's user data. In response to determining that the initiated external resource has previously been authorized to access interactive client 604's user data, interactive client 604 presents another GUI of the external resource, including its functions and features. In response to determining that the initiated external resource has not previously been authorized to access interactive client 604's user data, after displaying the external resource's login page or title screen for a threshold time period (e.g., 3 seconds), interactive client 604 slides up a menu for authorizing the external resource to access user data (e.g., animating the menu to appear from the bottom of the screen to the middle of the screen or other parts). The menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, interactive client 604 adds the external resource to the list of authorized external resources and enables the external resource to access user data from interactive client 604. External resources are authorized by the interactive client with a 604 error to access user data under the OAuth2 framework.

[0144] Interactive client 604 controls the type of user data shared with external resources based on the type of authorized external resource. For example, it provides access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics) to external resources including full-scale applications (e.g., application 606). As another example, it provides access to a second type of user data (e.g., payment information, a user's two-dimensional avatar, a user's three-dimensional avatar, and avatars with various avatar characteristics) to external resources including smaller versions of an application (e.g., a web-based version of an application). Avatar characteristics include different ways of customizing the appearance and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0145] The advertising system 732 is operationally designed to enable third parties to purchase advertisements to be presented to end users via the interactive client 604, and also handles the delivery and presentation of these advertisements.

[0146] Software Architecture

[0147] Figure 8This is a block diagram 800 illustrating a software architecture 802 that can be installed on any or more of the devices described herein. The software architecture 802 is supported by hardware, such as a machine 804 including a processor 806, memory 808, and I / O components 810. In this example, the software architecture 802 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 802 includes layers such as an operating system 812, libraries 814, frameworks 816, and applications 818. Operationally, application 818 activates API calls 820 via the software stack and receives messages 822 in response to API calls 820.

[0148] Operating system 812 manages hardware resources and provides public services. Operating system 812 includes, for example, a core 824, services 826, and drivers 828. Core 824 serves as an abstraction layer between hardware and other software layers. For example, core 824 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 826 can provide other public services to other software layers. Drivers 828 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 828 may include display drivers, camera drivers, etc. or Low-power drivers, flash drives, serial communication drivers (e.g., USB drives), Drivers, audio drivers, power management drivers, etc.

[0149] Library 814 provides common low-level infrastructure used by application 818. Library 814 may include system libraries 830 (e.g., the C standard library) that provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 814 may include API libraries 832, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG) or Portable Web Graphics (PNG)), graphics libraries (e.g., an OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing capabilities), etc. Library 814 may also include a wide variety of other libraries 834 to provide many other APIs to application 818.

[0150] Framework 816 provides a common high-level infrastructure for use by application 818. For example, framework 816 provides a variety of graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 816 can provide a wide range of other APIs that can be used by application 818, some of which may be specific to a particular operating system or platform.

[0151] In the example, application 818 may include a home application 836, a contacts application 838, a browser application 840, a book reader application 842, a location application 844, a media application 846, a messaging application 848, a game application 850, and a variety of other applications such as a third-party application 852. Application 818 is a program that performs the functions defined in the program. One or more applications 818 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 852 (e.g., an entity other than the vendor of a particular platform using Android) TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or another mobile operating system. In this example, a third-party application 852 can activate API call 820 provided by the operating system 812 to facilitate the functions described herein.

[0152] in conclusion

[0153] Changes and modifications may be made to the disclosed examples without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

[0154] Glossary

[0155] "Carrier signal" refers to any intangible medium or other intangible medium capable of storing, encoding, or carrying machine-executable instructions, including digital or analog communication signals. Instructions can be transmitted or received over a network using a transmission medium via a network interface device.

[0156] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0157] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0158] A “component” refers to a device, physical entity, or logic with boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function in a related function. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more hardware components (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It should be recognized that the decision to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., software-configured) circuit system can be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components.Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute components of a processor implementation that operates to perform one or more operations or functions described herein. As used herein, a “processor-implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some of the operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as a “Software as a Service” (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of a machine including processors), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across several machines. In some examples, the processor or processor-implemented component can reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component can be distributed across several geographic locations.

[0159] "Machine-readable storage medium" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "computer-readable medium," "machine-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably within the scope of this disclosure.

[0160] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0161] "Non-transitory machine-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0162] "Signal medium" means any intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible medium facilitating the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" refers to a signal whose various characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0163] Changes and modifications may be made to the disclosed examples without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

Claims

1. A computer-implemented method, comprising: A set of processors receives 3D data from the hand; The set of processors uses the 3D data to generate a 3D model of the hand, the 3D model including a set of surfaces; The set of processors generates target normal training data, which includes a set of normals of the set of surfaces of the 3D model. The set of processors uses the 3D model and the set of normals to generate synthetic 2D image training data, which includes a set of synthetic 2D images. as well as The set of processors uses the synthetic 2D image training data and the target normal training data to train a normal estimation model.

2. The computer-implemented method according to claim 1, wherein, The generation of the synthetic 2D image training data also uses a set of camera devices and lighting parameters.

3. The computer-implemented method according to claim 2, wherein, The set of camera devices and lighting parameters includes randomized values.

4. The computer-implemented method according to claim 1, wherein, Training the normal estimation model includes: The synthetic 2D image training data and detection model are used to determine a set of cropping boundaries; and The set of composite 2D images is cropped using the set of cropping boundaries.

5. The computer-implemented method according to claim 1, in, The synthetic 2D image training data includes a set of pixels, and The set of normals includes a corresponding normal for each pixel in the set of pixels.

6. The computer-implemented method according to claim 1, further comprising: The image data of the second hand is captured by the second set of processors; The second set of processors uses the image data and the normal estimation model to generate a set of estimated normals; The second set of processors uses the estimated normals and the image data to generate enhanced image data; as well as The enhanced image is then used by the second set of processors to provide the enhanced image to the user in the user interface.

7. The computer-implemented method according to claim 6, wherein, Generating the set of estimated normals includes: The second set of processors uses the image data and detection model to determine a set of cropping boundaries; and The image data is cropped by the second set of processors using the set of cropping boundaries.

8. The computer-implemented method according to claim 6, in, The image data of the second hand includes a set of pixels, and The estimated set of normals includes the corresponding normal for each pixel in the set of pixels.

9. A machine comprising: One or more processors; as well as The memory stores instructions that, when executed by the one or more processors, cause the machine to perform operations, including: Receive 3D data from the hand; The 3D data is used to generate a 3D model of the hand, the 3D model including a set of surfaces; Generate target normal training data, which includes a set of normals of the set of surfaces of the 3D model; The 3D model and the set of normals are used to generate synthetic 2D image training data, which includes a set of synthetic 2D images; and The synthetic 2D image training data and the target normal training data are used to train the normal estimation model.

10. The machine according to claim 9, wherein, The generation of the synthetic 2D image training data also uses a set of camera devices and lighting parameters.

11. The machine according to claim 10, wherein, The set of camera devices and lighting parameters includes randomized values.

12. The machine according to claim 9, wherein, Training the normal estimation model includes: The synthetic 2D image training data and detection model are used to determine a set of cropping boundaries; and The set of composite 2D images is cropped using the set of cropping boundaries.

13. The machine according to claim 9, in, The synthetic 2D image training data includes a set of pixels, and The set of normals includes a corresponding normal for each pixel in the set of pixels.

14. An interactive system, comprising: One or more processors; as well as The memory stores instructions that, when executed by the one or more processors, cause the interactive system to perform operations, including: The camera device of the interactive system is used to capture image data of the hand; A set of estimated normals is generated using the image data and the normal estimation model; Enhanced image data is generated using the estimated set of normals and the image data; and The enhanced image data is used to provide the enhanced image to the user in the user interface.

15. The interactive system according to claim 14, wherein, Generating the set of estimated normals includes: Using the image data and detection model, a set of cropping boundaries is determined; and The image data is cropped using the set of cropping boundaries.

16. The interactive system according to claim 14, in, The image data of the hand includes a set of pixels, and The estimated set of normals includes the corresponding normal for each pixel in the set of pixels.

17. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform an operation, the operation comprising: Receive 3D data from the hand; The 3D data is used to generate a 3D model of the hand, the 3D model including a set of surfaces; Generate target normal training data, which includes a set of normals of the set of surfaces of the 3D model; The 3D model and the set of normals are used to generate synthetic 2D image training data, which includes a set of synthetic 2D images. as well as The synthetic 2D image training data and the target normal training data are used to train the normal estimation model.

18. The computer-readable storage medium according to claim 17, wherein, The generation of the synthetic 2D image training data also uses a set of camera devices and lighting parameters.

19. The computer-readable storage medium according to claim 18, wherein, The set of camera devices and lighting parameters includes randomized values.

20. The computer-readable storage medium of claim 17, wherein, Training the normal estimation model includes: The synthetic 2D image training data and detection model are used to determine a set of cropping boundaries; and The set of composite 2D images is cropped using the set of cropping boundaries.

Images