Generating user interface displaying augmented reality graphics

Abstract

An augmented reality (AR) graphics system is provided. The AR graphics system may detect an object in a real-world scene that corresponds to the AR graphics display surface. The AR graphics system may generate AR graphics as an overlay display of the AR graphics display surface. The AR graphics system may track motion of the graphics input tool relative to the AR graphics display surface to generate AR graphics based on the motion of the graphics input tool. The AR graphics may include a plurality of indicia generated based on motion of the graphics input tool.

Description

[0001] Priority claim

[0002] This patent application claims the benefit of priority to U.S. application serial number 17 / 845,626, filed on June 21, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to generating a user interface that displays augmented reality graphics. Background Art

[0004] A head-mounted device may be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and also to see objects (e.g., virtual objects such as renderings of 2D or 3D graphics models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is commonly referred to as "augmented reality" or "AR". A head-mounted device may also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is commonly referred to as "virtual reality" or "AR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.

[0005] Users of head mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with a user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to one or more examples.

[0008] Figure 2 Based on one or more examples Figure 1 Additional views of the head-mounted device.

[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to one or more examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4is a diagram of an environment including multiple computing systems and devices for detecting a surface and causing a user interface to be displayed relative to the surface including augmented reality graphics according to one or more examples.

[0011] Figure 5 is a diagram of an architecture including multiple computing components for generating a user interface that displays augmented reality graphics relative to a surface based on detecting and tracking motion of a graphical input tool according to one or more examples.

[0012] Figure 6 is a flow diagram of a process for causing a user interface to be displayed with augmented reality graphics superimposed on a surface located in an environment according to one or more examples.

[0013] Figure 7 is a flow diagram of a process for activating an augmented reality content item using data captured by a camera of a head mounted device to generate a user interface including augmented reality graphics superimposed on surfaces located in an environment according to one or more examples.

[0014] Figure 8 is a user interface including augmented reality graphics superimposed on a surface according to one or more examples.

[0015] Fig. 9 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to some examples.

[0016] Fig.10 is shown according to some examples Figure 1 A block diagram of the details of the head mounted device.

[0017] Fig.11 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples. DETAILED DESCRIPTION

[0018] In many augmented reality systems, users can interact with virtual objects displayed in their environment. An input modality that can be used with an AR system is hand tracking combined with direct manipulation of virtual objects (DMVO), where the user is provided with a user interface displayed to the user in an AR overlay with a two-dimensional (2D) or three-dimensional (3D) rendering. The rendering is a graphical model in 2D or 3D, where the virtual objects located in the model correspond to interactive elements of the user interface. In this way, the user perceives the virtual objects as objects within the overlay in the user's field of view of the real world scene when wearing the AR system, or perceives the virtual objects as objects within the virtual world as seen by the user when wearing the AR system. In order to allow the user to manipulate virtual objects, the AR system detects the user's hands and tracks the movement, position, and / or positioning of the hands to determine the user's interaction with the virtual objects.

[0019] However, in existing systems, user interactions with virtual objects in an AR environment lack tactile feedback. To illustrate, when a user touches and interacts with a virtual object in an AR environment, the user's fingers, hands, or limbs typically do not feel anything in response to touching the virtual object. Therefore, although the user can see that the virtual object is being touched, without providing tactile feedback in response to the user's interaction, the user's interaction with the virtual object may be awkward or cause frustration to the user. For example, it may be challenging for the user to judge how the virtual object will move or otherwise react to their interaction with the virtual object.

[0020] Additionally, in scenarios where the state of a virtual object depends on the degree or amount of interaction with the virtual object, some interactions performed by the user may not activate the desired state of the virtual object. To illustrate, in the case of a user attempting to write, paint, or draw on a virtual surface, without tactile feedback, the stroke widths produced by the user's interactions with an input tool (e.g., a pen, brush, stylus, or finger) may not correspond to the user's expected results. As a result, the graphics generated in response to these interactions may not correspond to the expected graphics. As a result, the user's experience in the AR environment will be different from the user's experience painting, drawing, or writing using a physical tool that interacts with a physical medium.

[0021] In at least some cases, unless a particular state of a virtual object is activated based on the proximity of the input tool to the virtual object, a drawing, writing, or painting function may not be activated. For example, if the input tool is not touching the virtual object to be marked, a function to draw, paint, write, or otherwise mark on the virtual object may not be activated. However, without tactile feedback, the user may not be able to tell whether the input tool is touching the virtual object, which results in graphics generated by the user's interaction with the virtual object being different from the expected graphics.

[0022] In one or more examples, an implementation of an augmented reality graphics system may detect a surface in a real-world scene that can be used as an AR graphics display surface. The augmented reality graphics system may also detect a graphics input tool in a real-world scene and determine when the graphics input tool is within a threshold distance of the surface. In at least some examples, the augmented reality graphics system may determine when the graphics input tool is touching the surface. Drawing, writing, and / or painting functions may be activated in response to the graphics input tool being within a threshold distance of the surface. The augmented reality graphics system may then track the movement of the graphics input tool and generate an augmented reality graphic corresponding to the movement of the graphics input tool. In this way, a user may use a physical surface (e.g., a wall, table, floor, etc.) in the environment to generate at least one of a drawing, painting, handwriting, or design to generate a graphic without physically modifying the surface. That is, a user may create a painting using a surface included in the environment without actually applying paint to change the color of the surface.

[0023] By detecting the surface in the real world scene and the movement of the graphic input tool relative to the surface, tactile feedback can be provided to the user when the graphic input tool touches the surface. In this way, the mark generated by using the graphic input tool corresponds more closely to the mark expected by the user than when the virtual object is used as the surface on which the AR graphics are generated. In addition, in a scene where the amount of pressure applied by the graphic input tool has an effect on the appearance of the AR graphics, such as when pressing the brush harder on the surface causes a thicker mark to be made, the implementation described herein can also more accurately generate AR graphics with the appearance expected by the user, which is related to the case where the virtual object is used as the surface on which the AR graphics are generated. In addition, in the case where the state of the augmented reality content item is based on the proximity of the graphic input tool to the surface, the implementation described herein can more accurately detect the trigger condition for activating the state of the augmented reality content item or the client application. In particular, the graphic input tool can change shape in response to touching the surface in the real world scene, such as the tip of the brush or the bend of the finger. Thus, the accuracy of determining a trigger condition or selection of a user interface element corresponding to the location of a surface in a real-world environment can be more accurate and / or consume less computing and memory resources than where the state or activation of a user interface element is based on the proximity of a graphical input tool to a virtual object.

[0024] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0025] Figure 1 According to some examples, a head mounted device (e.g., Figure 1100). The glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0026] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0027] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 902 discussed below.

[0028] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0029] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0030] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.

[0031] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, and the touchpad 124 is approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide the following means by which the glasses 100 can receive input from the user of the glasses 100.

[0032] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0033] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0034] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image superimposed on or in the right optical element 110 with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image superimposed on or in the left optical element 108 with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.

[0035] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to a user within the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.

[0036] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Fig. 9 The user may interact with the glasses 100 via voice input or touch input on a client device 926 shown in FIG. 1 , and / or hand movements, positions, and locations detected by the glasses 100 .

[0037] Figure 3 is a diagrammatic representation of a computing device 300 within which instructions 310 (e.g., software, programs, applications, applet, app, other executable code) may be executed that cause the computing device 300 to perform any one or more of the methodologies discussed herein. The computing device 300 may be used as Figure 1The computer 120 of the glasses 100. For example, the instructions 310 can cause the computing device 300 to perform any one or more of the methods described herein. The instructions 310 transform a general, unprogrammed computing device 300 into a specific computing device 300 that is programmed to perform the functions described and shown in the described manner. The computing device 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the computing device 300 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The computing device 300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a notebook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 310 specifying actions to be taken by the computing device 300. In addition, while a single computing device 300 is shown, the term "machine" may also be understood to include a collection of machines that individually or collectively execute the instructions 310 to perform any one or more of the methodologies discussed herein.

[0038] Computing device 300 may include processor 302, memory 304, and I / O components 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but computing device 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0039] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the computing device 300, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a cache memory of a processor), or within any suitable combination thereof.

[0040] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 306 may include Figure 3 306. In various examples, the I / O component 306 may include an output component 328 and an input component 332. The output component 328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 332 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides a touch position and / or force or a touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0041] In some examples, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, and a positioning component 340, as well as various other components. For example, the biometric component 334 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit, an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 338 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning components 340 may include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), orientation sensor components (e.g., an inertial measurement unit (IMU)), etc.

[0042] A variety of technologies may be used to implement communications. I / O components 306 also include communication components 342 that are operable to couple computing device 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0043] In addition, the communication component 342 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0044] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0045] Instructions 310 may be transmitted or received over network 322 via a network interface device (e.g., a network interface component included in communications component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0046] Figure 4is a diagram of an environment 400 including multiple computing systems and devices for detecting a surface and causing a user interface to be displayed relative to the surface including augmented reality graphics according to one or more examples. Environment 400 may include client device 402. Client device 402 may be operated by user 404. Client device 402 may include multiple computing devices having processing resources and memory resources. For example, client device 402 may include at least one of head mounted device 406, wearable device 408, or mobile computing device 410 (e.g., a smartphone). In various examples, client device 402 may include multiple computing devices operating in conjunction with one another. For illustration, head mounted device 406 may operate in conjunction with at least one of wearable device 408 or mobile computing device 410, or wearable device 408 may operate in conjunction with mobile computing device 410. In one or more illustrative examples, client device 402 may include Figure 1 100 for glasses.

[0047] The processing resources and memory resources of client device 402 can execute multiple applications such as client application 412. In one or more examples, client application 412 can include messaging functionality that enables a user of client application 412 to send messages to and receive messages from other users of client application 412. In one or more additional examples, client application 412 can include social networking functionality that enables a user of client application 412 to share content with other users of client application 412 and / or access content created by other users of client application 412. In one or more illustrative examples, client application 412 can include information about Fig.10 At least one of the messaging client 1002 or the application 1004 is described in more detail.

[0048] Client device 402 may also include one or more cameras, such as camera 414. Camera 414 may capture images of the environment in which client device 402 is located. In one or more examples, camera 414 may capture video of the environment in which client device 402 is located. The video may include at least one of a series of images or a stream of images captured during a period of time. In various examples, camera 414 may capture video of a real-world scene in response to input from user 404. The images captured by camera 414 may be within a field of view of camera 414. The field of view may correspond to a portion of the environment that may be imaged by camera 414 at a given time, and may be based on the focal length of the lens of camera 414 and the size of the sensor of camera 414. Although in Figure 4402. Although not shown in the illustrative example of FIG. 404, client device 402 may also include multiple audio capture devices. For illustration, client device 402 may include multiple microphones to capture audio content generated in the environment in which client device 402 is located. In various examples, client device 402 may include one or more microphones to capture audio content in conjunction with video content captured by camera 414.

[0049] Environment 400 may also include an augmented reality (AR) graphics system 416. AR graphics system 416 may generate augmented reality graphics that are displayed in the environment in which client device 402 is located. For example, AR graphics system 416 may generate augmented reality graphics that are generated within a real-world scene captured by camera 414. The augmented reality graphics and the real-world scene may be visible via a display device of client device 402, but the augmented reality graphics may not be visible outside of the display device of client device 402. Although the display device 414 may be visible to the user, the augmented reality graphics may not be visible outside of the display device of client device 402. Figure 4 In the illustrative examples of FIG. 4 , AR graphics system 416 is shown as being external to client device 402, but in one or more implementations, at least a portion of the operations performed by AR graphics system 416 may be performed by client device 402. In one or more illustrative examples, client device 402 may include at least a portion of AR graphics system 416.

[0050] Augmented reality graphics can be generated in conjunction with augmented reality content items. Augmented reality content items can include program code that can be executed to perform one or more functions. In various examples, augmented reality content items can be executable within client application 412. For example, an instance of client application 412 can be activated by client device 402, and one or more user interfaces of client application 412 can be displayed via client device 402. An augmented reality content item can be selected while viewing one or more user interfaces of client application 412, and the augmented reality content item is executed to activate one or more functions corresponding to the selected augmented reality content item.

[0051] In one or more examples, the augmented reality content item can be executed to modify the appearance of one or more objects included in the image captured by the camera 414. For illustration, the augmented reality content item can be executed to change the appearance of a face included in the image captured by the camera 414. Additionally, the augmented reality content item can be executed to generate one or more augmented reality graphics related to one or more objects included in the image captured by the camera 414. In one or more illustrative examples, the augmented reality content item can be executed to cause a crown to appear on the head of an individual included in the image captured by the camera 414. Additionally, the augmented reality content item can be executed to generate one or more animations related to one or more objects included in the image captured by the camera 414. In one or more additional illustrative examples, the augmented reality content item can be executed to cause stars to hover above the head of an individual included in the image captured by the camera 414. In one or more additional illustrative examples, the augmented reality content item can be executed to display information obtained from one or more sources. For example, the augmented reality content item can be executed to display movie times and locations within one or more user interfaces of the client application 412 .

[0052] In one or more examples, the AR graphics system 416 can obtain camera data 418 generated by the camera 414. The camera data 418 can include one or more images captured by the camera 414 of the real-world scene. The AR graphics system 416 can analyze the camera data 418 to generate AR graphics display data 420. The AR graphics display data 420 may include one or more AR graphics displayed with one or more objects included in the real-world scene captured by the camera 414. In one or more illustrative examples, the AR graphics display data 420 can at least one of include user interface data, or be used to generate user interface data, the user interface data corresponding to one or more user interfaces including the AR graphics display data 420. In various examples, the operation of the AR graphics system 416 can be performed in response to activation of the augmented reality content item of the client application 412.

[0053] The AR graphics system 416 may include a surface detection system 422 that can detect one or more surfaces included in the camera data 418. The surface detection system 422 may analyze the camera data 418 to determine one or more target objects that may correspond to the surface. For example, the surface detection system 422 may analyze the camera data 418 to identify objects having a variety of characteristics. For illustration, the surface detection system 422 may analyze the camera data 418 to identify objects having one or more shapes, one or more dimensions, one or more colors, one or more textures, one or more combinations thereof, and the like. In various examples, the characteristics of the target object may correspond to the characteristics of the AR graphics display surface. In one or more examples, the AR graphics display surface may have characteristics that facilitate writing, drawing, painting, marking, one or more combinations thereof, and the like. In one or more illustrative examples, the AR graphics display surface may have at least a minimum dimension such as a minimum length and a minimum width and / or a maximum dimension such as a maximum length or a maximum width. In one or more illustrative examples, the AR graphics display surface may have one or more textures. In various examples, the AR graphics display surface can have a relatively flat texture or less than a maximum amount of surface roughness. In one or more additional illustrative examples, the AR graphics display surface can have one or more colors that facilitate the display of at least one of writing, drawing, painting, or other markings. In one or more further illustrative examples, the surface detection system 422 can analyze the camera data 418 to identify at least one of at least a portion of a wall, at least a portion of a floor, at least a portion of a table, at least a portion of a canvas, at least a portion of a whiteboard, at least a portion of a screen, or at least a portion of another surface on which an AR graphic can be clearly displayed. Figure 4 In the illustrative example of , surface detection system 422 may analyze camera data 418 to identify AR graphics display surface 424 .

[0054] In the case where multiple target objects correspond to an AR graphics display surface, the surface detection system 422 can analyze the camera data 418 to determine the candidate object that is most likely to be the AR graphics display surface. For example, the surface detection system 422 can determine the ranking of the candidate objects based on the characteristics of the candidate objects relative to one or more standards representing the AR graphics display surface. In these scenarios, the surface detection system 422 determines the candidate object with the highest ranking as the AR graphics display surface 424. In various examples, the surface detection system 422 can estimate the value of each candidate object relative to the specified value of the characteristic of the AR graphics display surface to determine the corresponding score of each candidate object. The score can indicate a similarity measure between the value of the characteristic of the candidate object and the value of the characteristic of one or more predetermined AR graphics display surfaces. In one or more illustrative examples, the scores of each candidate object can be used to rank each candidate object relative to each other.

[0055] In one or more additional examples, the surface detection system 422 can analyze additional inputs to identify an AR graphics display surface. For example, the surface detection system 422 can analyze the camera data 418 in conjunction with data indicating the gaze direction of the user 404 to identify the AR graphics display surface 424. For illustration, in the case where the surface detection system 422 analyzes the camera data 418 to determine multiple candidate AR graphics display surfaces, the surface detection system 422 can also analyze at least one of the orientation or position of at least one of the camera 414 or the client device 402 to determine the gaze direction of the user 404. In various examples, the surface detection system 422 can determine a candidate AR graphics display surface aligned with the gaze of the user 404 to identify the AR graphics display surface 424. In one or more additional examples, the surface detection system 422 can identify the AR graphics display surface 424 based on input received from the user 404. In one or more examples, the AR graphics display surface 424 can appear in a user interface generated by the client device 402 where there is a user interface element that can be actuated to select the AR graphics display surface 424.

[0056] The AR graphics system 416 may also include an object detection system 426 for analyzing the camera data 418 to identify one or more additional objects included in the camera data 418. In one or more examples, the object detection system 426 may analyze the camera data 418 to identify a graphic input tool 428 within the field of view of the camera 414. The graphic input tool 428 may include a device that can be used for at least one of writing, drawing, or painting. In one or more illustrative examples, the graphic input tool 428 may include a paintbrush. In at least some examples, the paintbrush is free of paint. In one or more additional examples, the graphic input tool 428 may include a stylus. In one or more additional examples, the graphic input tool 428 may include a finger. In addition, the object detection system 426 may analyze the camera data 418 to identify a tip 430 of the graphic input tool 428. In addition, the object detection system 426 may analyze the camera data 418 to identify a hand 432 in the field of view of the camera 414. In one or more examples, object detection system 426 can analyze camera data 418 to identify hand 432 grasping graphic input tool 428 .

[0057] In one or more examples, the object detection system 426 can analyze the camera data 418 to determine at least one of a plurality of edges, a plurality of contours, or a plurality of shapes that may correspond to an object, either individually or in combination. In various examples, the object detection system 426 can determine one or more target regions corresponding to at least a portion of an object in a real-world scene. In at least some examples, the object detection system 426 can identify one or more objects in a scene based on a similarity measure between features included in the target region and features of one or more objects included in a template image. The template image may include a previously captured image and include one or more objects of interest. The template image may be used to compare with subsequently captured images to identify an object of interest. In one or more illustrative examples, the object detection system 426 may implement one or more machine learning techniques to identify at least one of an object included in a scene, such as a graphic input tool 428, a tip 430 of a graphic input tool 428, or a hand 432. In one or more additional illustrative examples, object detection system 426 can determine a set of pixels included in one or more images captured by camera 414 that correspond to an object of interest, such as at least one of graphics input tool 428, tip 430 of graphics input tool, or hand 432.

[0058] In addition, the AR graphics system 416 may include an object tracking system 434. The object tracking system 434 may analyze the camera data 418 to determine a motion path of one or more objects included in the field of view of the camera 414. For example, the object tracking system 434 may analyze the camera data 418 to determine a motion path of at least one of the graphics input tool 428, the tip 430 of the graphics input tool 428, or the hand 432. The object tracking system 434 may determine the motion path of the object of interest by analyzing the camera data 418 using one or more difference techniques for determining a difference in positioning of the object between multiple images included in the camera data 418. In one or more additional examples, the object tracking system 434 may determine the motion path of the object of interest by determining a background included in multiple images of the camera data 418 and determining a change in position of a portion of the multiple images that is not included in the background. In various examples, the object tracking system 434 may analyze the intensities of pixels included in multiple images included in the camera data 418 to determine a motion path of one or more objects of interest (e.g., at least one of the graphics input tool 428, the tip 430 of the graphics input tool, or the hand 432).

[0059] In addition, the AR graphics system 416 may include an AR graphics display system 436. The AR graphics display system 436 may generate AR graphics using data generated by at least one of the surface detection system 422, the object detection system 426, or the object tracking system 434. In one or more examples, the AR graphics display system 436 may generate user interface data corresponding to one or more user interfaces including AR graphics. In various examples, the AR graphics generated by the AR graphics display system 436 may correspond to the movement of at least one of the graphic input tool 428, the tip 430 of the graphic input tool 428, or the hand 432. In at least some examples, the AR graphics display system 436 may generate an AR graphic corresponding to at least one of writing, drawing, painting, or another type of marking.

[0060] In one or more illustrative examples, AR graphics display system 436 may determine a location of an AR graphic to be displayed based on the location of AR graphics display surface 424. In one or more examples, surface detection system 422 may generate coordinates in real-world space corresponding to AR graphics display surface 424 and provide the coordinates in real-world space to AR graphics display system 436. In various examples, the coordinates in real-world space obtained by AR graphics display system 436 may correspond to a boundary of AR graphics display surface 424 and cause AR graphics to be generated and displayed within the boundary of AR graphics display surface 424. The AR graphics generated by AR graphics display system 436 may be included in a user interface displayed as an overlay of AR graphics display surface 424.

[0061] In addition, the AR graphics display system 436 may obtain a motion path of at least one of the graphic input tool 428, the tip 430 of the graphic input tool 428, or the hand 432 from the object tracking system 434. The AR graphics display system 436 may generate an AR graphic corresponding to the motion path of at least one of the graphic input tool 428, the tip 430 of the graphic input tool 428, or the hand 432. In one or more illustrative examples, when the tip 430 of the graphic input tool 428 moves within the boundary of the AR graphics display surface 424, the AR graphics display system 436 may cause a marker to be displayed within the user interface, wherein the marker corresponds to the motion of the tip 430 of the graphic input tool 428. In this way, the AR graphics display system 436 may cause the AR graphics 438 to be displayed in the user interface including the AR graphics display surface 424.

[0062] The AR graphics generated by the AR graphics display system 436 can have one or more colors. In various examples, the color of the mark generated by the AR graphics display system 436 can be selected from a color menu based on input from the user 404. For example, the AR graphics display system 436 can generate a user interface including a menu that includes a plurality of user interface elements corresponding to different colors that can be used to generate AR graphics. In response to the selection of the user interface element corresponding to a given color, the AR graphics display system 436 can generate an AR graphic with the selected color according to the movement of the tip 430 of the AR graphics input tool 428.

[0063] The AR graphics display system 436 can also generate a mark of the AR graphics based on additional input provided by the user 404. For example, the AR graphics display system 436 can generate a user interface including a menu, which includes multiple line widths. In these scenarios, the AR graphics display system 436 displays an AR graphics including one or more marks with a width selected by the user 404. In one or more additional examples, the width of the mark generated by the AR graphics display system 436 can be based on the state of the tip 430 of the AR graphics input tool 428. For illustration, as the tip 430 is bent to a greater extent, the thickness of the mark generated by the AR graphics display system 436 will increase. In one or more illustrative examples, the tip 430 can be associated with a first state, a second state, and a third state, the first state corresponding to the first bending amount of the tip 430 and related to the first marking width, the second state corresponding to the second bending amount of the tip 430 and related to the second marking width, and the third state corresponding to the third bending amount of the tip 430 and related to the third marking width. In various examples, the second bending amount can be greater than the first bending amount, and the third bending amount can be greater than the second bending amount. In these cases, the second mark width can be thicker than the first mark width, and the third mark width can be thicker than the second mark width.

[0064] In one or more examples, the AR graphics display system 436 may detect an input indicating that at least a portion of the AR graphics 438 is to be erased. In addition, the AR graphics display system 436 may detect an input to apply one or more additional augmented reality content items to a user interface including the AR graphics 438. For example, the AR graphics display system 436 may display a menu including a plurality of user interface elements that can be selected to execute one or more additional augmented reality content items. In various examples, the one or more additional augmented reality content items may cause an additional predetermined graphic to be displayed in the user interface together with the AR graphics 438. In addition, the one or more additional augmented reality content items may cause an animation to be displayed in the user interface together with the AR graphics 438.

[0065] In at least some examples, the AR graphics display system 436 may cause the AR graphics 438 to be saved in at least one of a memory of the client device 402 or a data storage device remotely located relative to the client device 402. The saved AR graphics 438 may be reproduced in response to the detection of the AR graphics display surface 424. For example, the surface detection system 422 may determine that the AR graphics display surface 424 is no longer detected within the field of view of the camera 414. To illustrate, the user 404 may move at least a threshold distance away from the AR graphics display surface 424, or turn the client device 402 so that the camera 414 is no longer pointed in a manner that the AR graphics display surface 424 is within the field of view of the camera 414. In these cases, the AR graphics 438 is deleted from the user interface displayed by the AR graphics display system 436. The surface detection system 422 may determine at another time that the AR graphics display surface 424 has returned to the field of view of the camera 414. Therefore, the AR graphics display system 436 may cause the AR graphics 438 to be redisplayed in the user interface including the AR graphics display surface 424. In various examples, in response to surface detection system 422 determining that at least a minimum amount of AR graphics display surface 424 is within the field of view of camera 414 , AR graphics display system 436 can cause AR graphics 438 to be displayed in the user interface.

[0066] In one or more additional examples, the AR graphics display system 436 can detect input to display the AR graphics 438 in a user interface including the AR graphics display surface 424. In one or more additional examples, the AR graphics display system 436 can detect input to display the AR graphics 438 on one or more additional surfaces. For illustration, the client application 412 can cause a user interface including a user interface element to be displayed, and the user interface element can be selected to view one or more AR graphics generated using the client application 412. A single AR graphic can be associated with one or more additional user interface elements, and the one or more additional user interface elements can be selected to cause the corresponding AR graphics to be displayed in a user interface including one or more surfaces located in the field of view of the camera 414. In various examples, the client application 412 can also generate one or more user interfaces that enable the user 404 to include the AR graphics 438 in one or more messages. One or more messages can be sent to other users of the client application 412. In one or more additional examples, the client application 412 can generate one or more user interfaces that enable the user 404 to share the AR graphics 438 with individuals included in the social network of the user 404. For example, the client application 412 may generate one or more user interfaces including user interface elements that are selectable to enable the user 404 to include the AR graphic 438 in a social media post or story.

[0067] Figure 5 4 is a diagram of an architecture 500 including multiple computing components for generating a user interface for displaying augmented reality graphics relative to a surface based on detecting and tracking the movement of a graphical input tool according to one or more examples. The architecture 500 may include a client device 402 operated by a user 404. The client device 402 may execute an instance of a client application 412. The client device 402 may also include a camera 414 that generates camera data 418. The client device 402 may provide the camera data 418 to an AR graphics system 416. The AR graphics system 416 may analyze the camera data 418 to generate AR graphics display data 420. In various examples, at least a portion of the operations performed by the AR graphics system 416 may be performed by the client device 402. In one or more additional examples, at least a portion of the operations performed by the AR graphics system 416 may be performed by one or more additional computing devices remotely located relative to the client device 402.

[0068] The AR graphics display data 420 may correspond to one or more AR graphics displayed within a user interface generated by the client application 412. In one or more examples, the AR graphics display data 420 may be generated in conjunction with an AR graphics display content item 502. The AR graphics display content item 502 may include computer-readable code that is executable to activate features of the AR graphics system 416 to generate AR graphics that may be displayed in one or more user interfaces of the client application 412. For example, the AR graphics display content item 502 may be executable to cause an AR graphic to be displayed in an AR graphics user interface 504. In one or more illustrative examples, the AR graphics display content item 502 may be executable to display the AR graphics as one or more AR graphics display surfaces (e.g., Figure 4 An overlay of the AR graphics display surface 424 ) is displayed in the AR graphical user interface 504 .

[0069] The AR graphics system 416 may include an AR graphics service 506. The AR graphics service 506 may perform a plurality of computational operations for analyzing the camera data 418 to generate AR graphics display data 420. In at least some examples, the AR graphics service 506 may implement one or more machine learning techniques. In one or more examples, the AR graphics service 506 may be activated in response to activation of the AR graphics display content item 502.

[0070] The AR graphics service 506 may include a camera service 508 that obtains camera data 418. For example, the camera service 508 may obtain a plurality of images captured by the camera 414, such as a plurality of video frames captured by the camera 414. In one or more examples, the camera service 508 may continuously receive the camera data 418 over a period of time. In various examples, the camera service 508 may obtain the camera data 418 during a period of time when the AR graphics display content item 502 is activated. In one or more additional examples, the camera service 508 may be activated and deactivated in response to user input received via the AR graphics display content item 502. In at least some examples, the camera service 508 may enable the camera data 418 to be used for one or more additional components of the AR graphics service 506.

[0071] The AR graphics service 506 may also include an object detection service 510. The object detection service 510 may obtain the camera data 418 from the camera service 508. In one or more examples, the object detection service 510 may analyze the camera data 418 to identify one or more objects in the real-world scene. The object detection service 510 may analyze the camera data 418 to identify at least a portion of a graphical input tool that may be used to generate a markup including an AR graphic displayed in the AR graphical user interface 504. The object detection service 510 may also analyze the camera data 418 to determine one or more AR graphics display surfaces within the real-world scene corresponding to the camera data 418. In addition, the object detection service 510 may analyze the camera data 418 to identify one or more additional objects included in the real-world scene, such as a hand of the user 404, a finger of the user 404, or a portion of the arm of the user 404.

[0072] In one or more examples, object detection service 510 can implement one or more artificial neural networks to analyze camera data 418 to identify objects within a real-world scene. For example, object detection service 510 can implement one or more convolutional neural networks on camera data 418 to identify objects within a real-world scene. Additionally, object detection service 510 can implement one or more classification machine learning techniques to analyze camera data 418 to identify one or more objects in a real-world scene. To illustrate, object detection service 510 can implement one or more support vector machines on camera data 418 to identify one or more objects in a real-world scene.

[0073] In one or more illustrative examples, the object detection service 510 can analyze the camera data 418 to determine the number of at least one of contours, edges, or shapes, which can be used to determine one or more candidate regions that may include one or more objects of interest (e.g., a graphics input tool, an AR graphics display surface, a tip of a graphics input tool, a finger, or one or more combinations thereof). The object detection service 510 can also implement a convolutional neural network for extracting features from one or more candidate regions. In addition, the object detection service 510 can implement one or more support vector machines to classify one or more objects included in the one or more candidate regions based on features extracted from the one or more candidate regions by the convolutional neural network. In various examples, one or more machine learning techniques implemented by the object detection service 510 can be trained using previously captured images that include one or more of the objects of interest and are marked as including one or more objects of interest.

[0074] The AR graphics service 506 may also include an object tracking service 512 that can detect movement of one or more of the objects of interest identified by the object detection service 510. In one or more examples, the object tracking service 512 may implement one or more trackers that determine movement of one or more of the objects of interest within a frame of the camera data 418. In various examples, the object tracking service 512 may mark the object of interest as an object to be tracked and determine a difference in position of the marked object of interest within multiple video frames. In one or more illustrative examples, the object tracking service 512 may implement one or more of a convolutional neural network, a recurrent neural network, an autoencoder, or a generative adversarial network to track the movement of the object of interest across video frames.

[0075] In addition, the AR graphics service 506 may include a depth determination service 514 that determines a distance between one or more objects included in the camera data 418 and the camera 414. In various examples, the depth determination service 514 may determine coordinates in real-world space for one or more objects included in the real-world scene corresponding to the camera data 418. In one or more illustrative examples, the depth determination service 514 may determine the coordinates in the real-world space using data obtained from multiple cameras of the client device 402. In one or more additional illustrative examples, the depth determination service 514 may implement one or more convolutional neural networks to determine the coordinates in the real-world space for one or more objects of interest included in the camera data 414. The one or more convolutional neural networks may be trained using images that include objects that have been labeled as being at different distances from one or more cameras that captured the training images.

[0076] In addition, the AR graphics system 416 can implement an application programming interface (API) 516. The API 516 can include a plurality of calls that can be used to obtain information generated by the AR graphics service 506. For example, one or more first calls of the API 516 can be provided to obtain information from the camera service 508. Additionally, one or more second calls of the API 516 can be provided to obtain information from the object detection service 510. Furthermore, one or more third calls of the API 516 can be provided to obtain information from the object tracking service 512. In various examples, one or more fourth calls of the API 516 calls can be provided to obtain information from the depth determination service 514.

[0077] exist Figure 5In the illustrative example of, calls to API 516 can be used to provide information generated by the AR graphics service 506 to one or more computing models. For example, the AR graphics system 416 may include one or more graphics input tool tracking models 518. In one or more examples, the graphics input tool tracking model 518 can be implemented to determine the position of the graphics input tool relative to the AR graphics display surface. In various examples, the one or more graphics input tool tracking models 518 can use one or more calls to API 516 to obtain information from the object detection service 510 and the object tracking service 512 to track the position of the graphics input tool relative to the AR graphics display surface. The one or more graphics input tool tracking models 518 can use the position of the graphics input tool to determine the position of the marker of the AR graphics to be displayed in the AR graphical user interface 504 relative to the AR graphics display surface.

[0078] In addition, the AR graphics system 416 may include one or more hand tracking models 520. The one or more hand tracking models 520 may be implemented to determine the position of the hand of the user of the graphical input tool. The one or more hand tracking models 520 may use one or more calls of the API 516 to obtain information from the object detection service 510 and the object tracking service 512 to track the position of the hand of the user 404 that is holding the graphical input tool, such as a brush, a stylus, another marking tool (implement), etc. By tracking the movement of the hand of the user 404 that is holding the graphical input tool, additional information may be obtained to improve the accuracy of the position of the mark formed in response to the movement of the graphical input tool held by the hand of the user 404. For example, not only can the accuracy of the position of the graphical input tool be increased by detecting and tracking the movement of the hand of the user 404 that is holding the graphical input tool, but the angle at which the graphical input tool is being held may have an impact on the appearance of the mark generated in response to the movement of the graphical input tool and in response to the movement of the hand of the user 404.

[0079] In addition, the AR graphics system 416 may include one or more graphics input tool state models 522. One or more graphics input tool state models 522 may be implemented to determine the state of a graphics input tool used to generate a mark displayed in the AR graphical user interface 504 relative to the AR graphics display surface. The state of the graphics input tool may affect the characteristics of the mark generated in response to the movement of the graphics input tool. The state of the graphics input tool may also correspond to whether the marking function of the AR graphics display content item 502 has been activated. In various examples, the one or more graphics input tool state models 522 may use one or more calls of the API 516 to obtain information from the object detection service 510, the object tracking service 512, and the depth determination service 514.

[0080] For example, the graphics input tool may include a brush, and the amount of curvature of the tip of the brush may correspond to the width of a marker displayed in response to the movement of the graphics input tool. The one or more graphics input tool state models 522 may obtain information from at least one of the object detection service 510 or the depth determination service 514 to determine the amount of curvature of the tip of the brush in response to contact with the AR graphics display surface. In at least some examples, one or more first curvatures of the tip of the brush may correspond to one or more first widths of a marker displayed in the AR graphical user interface 504, one or more second curvatures of the tip of the brush may correspond to one or more second widths of a marker displayed in the AR graphical user interface 504, and one or more third curvatures of the tip of the brush may correspond to one or more third widths of a marker displayed in the AR graphical user interface 504. In this way, when the graphics input tool moves in conjunction with the AR graphics display surface, the marker may be displayed in the AR graphical user interface 504 based on the brush state determined using the one or more graphics input tool state models 522.

[0081] In various examples, one or more graphic input tool state models 522 may use information obtained from at least one of the object detection service 510, the object tracking service 512, or the depth determination service 514 to determine whether the graphic input tool is within a threshold distance of the AR graphics display surface. For example, one or more graphic input tool state models 522 may be implemented to determine that the brush is within a threshold distance of a wall including the AR graphics display surface. In one or more illustrative examples, one or more graphic input tool state models 522 may be implemented to determine that the brush is touching the wall. In one or more examples, when the graphic input tool is within a threshold distance of the AR graphics display surface, the marking function of the graphic input tool may be in an activated state. In response to the graphic input tool being in an active state, a mark may be generated in the AR graphical user interface 504 based on the movement of the graphic input tool. In a scene where the graphic display tool is greater than a threshold distance from the AR graphics display surface, the graphic input tool may be in a deactivated state. In response to the graphic input tool being in a deactivated state, the movement of the graphic input tool may not cause a mark to be generated in the AR graphical user interface 504. In one or more illustrative examples, the threshold distance may be no greater than 50 mm, no greater than 40 mm, no greater than 30 mm, no greater than 20 mm, no greater than 10 mm, no greater than 5 mm, or no greater than 1 mm.

[0082] Despite Figure 5In the illustrative example of FIG. 4 , one or more graphical input tool tracking models 518, one or more hand tracking models 520, and one or more graphical input tool state models 522 are included in the AR graphics system 416 and are shown as being separate from the client device 402, but in one or more implementations, at least a portion of the operations performed by the one or more graphical input tool tracking models 518, one or more hand tracking models 520, one or more graphical input tool state models 522, or one or more combinations thereof, can be performed by the client device 402. In various examples, the architecture 500 is arranged to access information from the AR graphics service 506 using the API 516 through the models 518, 520, 522 to minimize computational resources and memory resources utilized by the client device 402 to generate the AR graphical user interface 504. That is, since the client device 402 may include a head mounted device or a wearable device with limited processing resources and memory resources, one or more calls of the API 516 may be used to perform at least a portion of the operations performed for the AR graphical display content item 502 using computing resources and memory resources located remotely from the client device 402 to generate the AR graphical user interface 504. In one or more illustrative examples, at least one of the one or more graphical input tool tracking models 518, the one or more hand tracking models 520, and the one or more graphical input tool state models 522 may be stored in the memory of the client device 402, while the AR graphics service 506 is stored and executed using computing resources and memory resources located remotely from the client device 402. In these scenarios, the calls of the API 516 are used to obtain information from the AR graphics service 506, which is used by at least one of the one or more graphical input tool models 518, the one or more hand tracking models 520, or the one or more graphical input tool state models 522 to generate AR graphics within the AR graphical user interface 504. In one or more additional examples, the surface detection system 422, the object detection system 426, the object tracking system 434, and the AR graphics display system 436 may include and / or implement components of the AR graphics service 506, the API 516, the graphics input tool tracking model 518, one or more hand tracking models 520, and one or more graphics input tool state models 522.

[0083] In one or more examples, the AR graphics system 416 can obtain user input 524. The user input 524 can be generated in response to an action taken by the user 404 with a graphical input tool. In various examples, one or more menus can be displayed in conjunction with the AR graphical user interface 504. In these scenarios, the user input 524 corresponds to the selection of one or more menus. One or more menus can be displayed in the AR graphical user interface 504 as an overlay of the AR graphical display surface. In one or more illustrative examples, one or more menus can include a plurality of graphical user interface elements, which can be selected to activate one or more functions of the AR graphical display content item 502. For example, one or more user interface elements of one or more menus can be selected so that the mark generated in the AR graphical user interface 504 has a specified color. In addition, one or more user interface elements of one or more menus can be selected to generate a mark with a specified width in the AR graphical user interface 504. In one or more additional examples, one or more user interface elements of one or more menus can be selected so that one or more additional AR content items are executed for one or more objects included in the AR graphical user interface 504.

[0084] In at least some examples, a function corresponding to a given user interface element can be activated in response to determining that a graphical input tool is within a threshold distance of a user interface element displayed in the AR graphical user interface 504 as an overlay of the AR graphical display surface. In one or more illustrative examples, the appearance of the user interface element can be modified in response to a selection of the user interface element in response to user input 524. To illustrate, a user interface element corresponding to generating a mark with a given color can be displayed as having a given color, and can become a lighter shade of the color in response to a selection based on user input 524. In one or more additional illustrative examples, a mesh collider can be generated for a single user interface element, and user input 524 indicated by contact between an input device (e.g., a finger, a stylus, or a graphical input tool) and the mesh collider can activate a function corresponding to the single user interface element.

[0085] Figure 6 and Figure 7A flowchart of a process for generating an AR graphic as an overlay of an AR graphic display surface located in a real-world scene is shown. The process may be embodied in computer-readable instructions that are executed by one or more processors such that operations of the process may be performed in part or in whole by functional components of at least one of one or more client devices or one or more server systems. Thus, in some cases, the process described below is by way of reference to examples thereof. However, in other implementations, Figure 6 and Figure 7 At least some of the operations of the described processes may be deployed on various other hardware configurations. Figure 6 and Figure 7 The described process is not intended to be limited to being performed by one or more server systems or one or more client devices described herein, and may be implemented in whole or in part by one or more additional components. Although the described flowchart may show the operation as a sequential process, many operations in the operation may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operation of the process is completed, the process terminates. The process may correspond to a method, a program, an algorithm, etc. The operation of the method may be performed in whole or in part, may be performed in combination with some or all of the operations in other methods, and may be performed by any number of different systems (e.g., the system described herein) or any part thereof (e.g., a processor included in any one of the systems).

[0086] Figure 6 6 is a flow diagram of a process 600 according to one or more examples that causes a user interface to be displayed with augmented reality graphics superimposed on a surface located in an environment. The process 600 may include obtaining camera data corresponding to one or more images of a field of view captured by a camera at operation 602. The camera may be located in a client device. The client device may include a head mounted device. Additionally, the client device may include at least one of a wearable device or a mobile phone. The camera data may include multiple images of a real world scene. In one or more examples, the camera data may include multiple video frames. In various examples, the camera data may include video frames captured from multiple cameras of the client device.

[0087] Process 600 may include analyzing the camera data to determine an augmented reality (AR) graphics display surface included in the field of view at operation 604. In one or more examples, determining the AR graphics display surface may be performed in response to launching of an augmented reality content item executed within a client application. For example, the functionality of generating an AR graphic as an overlay of the AR graphics display surface may be performed based at least in part on computer readable code of the augmented reality content item executed within the client application.

[0088] Multiple characteristics of an object of interest included in the camera data may be analyzed to identify an AR graphics display surface. For example, at least one of the size, color, texture, or shape of the object of interest may be analyzed to identify the AR graphics display surface. One or more object recognition computing techniques may be used to analyze the camera data to identify an AR graphics display surface in a real-world scene. In one or more illustrative examples, one or more neural networks may be executed to identify an AR graphics display surface. In various examples, one or more neural networks may be trained using previously captured images of an object corresponding to the AR graphics display surface. In one or more additional examples, an AR graphics display surface may be identified based at least in part on input obtained from a user of a client device indicating an AR graphics display surface in a field of view.

[0089] The AR graphics display surface may be a two-dimensional surface in a real-world scene having at least a minimum length and at least a minimum width. In one or more examples, the AR graphics display surface may include a wall, a floor, a table, a projection screen, a piece of furniture, a canvas, a sketchpad, one or more sheets of paper, a poster, one or more combinations thereof, and the like. In various examples, the AR graphics display surface may be fixed to a wall or other structure. In one or more illustrative examples, the AR graphics display surface may have at least one of a length or width of at least 8 cm, at least 12 cm, at least 15 cm, at least 18 cm, at least 20 cm, at least 30 cm, at least 50 cm, or at least 100 cm. In the case where a surface such as a wall or floor includes the entire field of view of a camera device, at least a portion of the surface included in the field of view is identified as an AR graphics display surface.

[0090] Additionally, process 600 may include analyzing the camera data to detect a graphical input tool included in the field of view at operation 606. The graphical input tool may include a handheld tool that can be manipulated by a user in a manner corresponding to the motion produced when making marks on a surface (e.g., handwriting, drawing, painting, one or more combinations thereof, etc.). In one or more additional examples, the graphical input tool may include a finger of a user.

[0091] Process 600 may also include determining that the graphics input tool is within a threshold distance of the AR graphics display surface at operation 608. In one or more examples, determining that the graphics display tool is within a threshold distance of the AR graphics display surface may include determining that the graphics input tool is touching the AR graphics display surface based on the camera data. In various examples, determining that the graphics input tool is within a threshold distance of the AR graphics display surface may include determining depth information indicating a distance of the graphics input tool from the AR graphics display surface.

[0092] Additionally, process 600 may include determining, based on the camera data, movement of the graphics input tool when the graphics input tool is within a threshold distance of the AR graphics display surface at operation 610. The movement of the graphics input tool may be determined based on changes in the position of the graphics input tool in multiple video frames.

[0093] In one or more additional examples, the camera data may be analyzed to detect a hand holding a graphics input tool included in the field of view. In various examples, a first position of the hand relative to a second position of the graphics input tool may be determined. For example, the angle of the graphics input tool may be determined based on the first position of the hand relative to the second position of the graphics input tool. In one or more scenarios, the movement of the hand when the graphics input tool is within a threshold distance of the AR graphics display surface may be determined based on the camera data. The characteristics of the AR graphics may be based on the movement of the graphics input tool, the movement of the hand, and the first position of the hand relative to the second position of the graphics input tool. For illustration, the movement of the graphics input tool, the movement of the hand, and the angle of the graphics input tool may be used to determine the width of a mark generated in response to the movement of the graphics input tool.

[0094] In one or more examples, an application programming interface (API) may be provided that enables information to be obtained from one or more services to detect objects in a real-world scene and track the movement of objects in the real-world scene. For example, analysis of camera data for detecting a graphical input tool in a real-world scene may be performed in response to one or more first API calls to an object detection service. Additionally, determining the distance between the graphical input tool and the AR graphics display surface and / or determining whether the graphical input tool is within a threshold distance of the AR graphics display surface may be performed in response to one or more second API calls to a depth determination service. Furthermore, determining the movement of the graphical input tool may be performed in response to one or more third API calls to an object tracking service.

[0095] Process 600 may also include causing a user interface superimposed on the AR graphics display surface to be displayed at operation 612. The user interface may include an AR graphic corresponding to the movement of the graphic input tool. In one or more examples, one or more machine learning models may be implemented to determine the amount of curvature of the tip of the graphic input tool. The size of the mark may then be determined based on the amount of curvature of the tip of the graphic input tool. In various examples, one or more machine learning models may be trained based on a training data set using supervised machine learning training techniques. The training data set may include: a first number of graphic input tool images corresponding to a first curvature of the tip of the graphic input tool, associated with a first label; and a second number of graphic input tool images corresponding to a second curvature of the tip of the graphic input tool, associated with a second label.

[0096] In various examples, the AR graphics menu may be displayed within a user interface. The AR graphics menu may include multiple user interface elements. Each user interface element can be selected to activate one or more features of the augmented reality content item, thereby performing at least one of the following: creating or modifying at least a portion of the AR graphics. For example, the AR graphics menu may include a user interface element that can be selected to create an AR graphic marker with a specified color. In one or more examples, the feature corresponding to the corresponding user interface element may be activated in response to determining that an object included in the field of view is within an additional threshold distance of a user interface element in a plurality of user interface elements of the AR graphics menu. The additional threshold distance for determining when to activate the feature of the user interface element may be the same or different from the threshold distance between the graphic input tool and the AR graphics display surface that triggers the generation of a marker for creating an AR graphic.

[0097] In one or more examples, at least one of the user interface or the AR graphics can be modified. For example, additional AR graphics can be added to the user interface. In addition, the existing AR graphics can be modified by adding additional markers to the AR graphics. In addition, at least one of the user interface or the AR graphics can be modified by executing additional augmented reality content items for at least one of the AR graphics or AR graphics display surfaces. The additional augmented reality content items can be executed so that one or more additional AR graphics are displayed in the user interface. For example, the additional augmented reality content items can cause at least one of the images, text content, animations, or additional computer-generated graphics (e.g., emoticons, stickers, or avatars) of the items to be displayed in the user interface. The additional augmented reality content items can also be executed to modify the appearance of the AR graphics, such as modifying at least one of the shape, color, or size of the AR graphics. In one or more illustrative examples, a user input for selecting an additional augmented reality content item from the AR graphics menu can cause the user interface to be modified according to the additional augmented reality content items.

[0098] In at least some examples, the AR graphics can be saved in association with the AR graphics display surface. Therefore, as the client device including the camera moves away from the AR graphics display surface, the user interface can be modified to no longer show the AR graphics. In addition, when the client device including the camera moves so that the AR graphics display surface is within the field of view of the camera, the AR graphics may reappear in the user interface. For example, the field of view of the camera can change from a first field of view including the AR graphics display surface to a second field of view in which the AR graphics display surface does not exist. In these scenarios, the user interface is modified by removing the AR graphics from the user interface. In various examples, when the field of view of the camera has changed back from the second field of view to the first field of view, the user interface can be modified by adding the AR graphics to the user interface.

[0099] In addition, additional camera data corresponding to one or more additional images of the second field of view captured by the camera may be obtained. The additional camera data may include one or more additional images captured during a time period after the initial time period of capturing one or more images of the initial camera data. In these cases, the additional camera data is analyzed to determine the additional AR graphic display surface included in the second field of view. In addition, the additional camera data may be analyzed to detect the graphic input tool in the second field of view, and determine that the graphic input tool is within a threshold distance of the additional AR graphic display surface. Additional movement of the graphic input tool when the graphic input tool is within a threshold distance of the additional AR graphic display surface may also be determined based on the additional camera data. Thus, an additional user interface superimposed on the additional AR graphic display surface may be displayed, wherein the additional user interface includes an additional AR graphic corresponding to the additional movement of the graphic input tool. In various examples, the additional AR graphics may be stored together with the additional AR graphic display surface.

[0100] In one or more additional examples, the distance information can be used to adjust the appearance of the AR graphics displayed in the user interface. For example, the coordinates in the real world space for the AR graphics display surface can be determined, and the coordinates in the real world space can be used to determine the distance from the camera to the AR graphics display surface. The position data of the camera, such as the position coordinates of the client device that accommodates the camera, can also be used to determine the distance between the camera and the AR graphics display surface. In one or more illustrative examples, the distance from the camera to the AR graphics display surface can be changed from a first distance to a second distance. In these scenarios, the magnification level of the AR graphics in the user interface is modified based on the distance from the camera to the AR graphics display being changed from a first distance to a second distance. For illustration, as the camera moves further away from the AR graphics display surface, the magnification of the AR graphics increases. In addition, when the camera moves closer to the AR graphics display surface, the magnification of the AR graphics decreases. In this way, the appearance of the AR graphics can be relatively consistent as the distance between the camera and the AR graphics display surface changes.

[0101] In one or more additional examples, information can be obtained from one or more sensors of the client device to determine the position of the camera relative to the AR graphics display surface. For example, gyroscope sensor information can be used to generate position information of the camera relative to the AR graphics display surface. In various examples, gyroscope sensor data can be used to calibrate the position of the camera relative to the coordinates of the AR graphics display surface in the world space. In one or more illustrative examples, the origin of the world space coordinate system can be identified as a point of the AR graphics display system, and the gyroscope sensor data can be calibrated relative to the origin. Since the gyroscope sensor data indicates a change in at least one of the position or orientation of the camera, the position data derived from the modified gyroscope sensor data can be used to generate a change in the coordinates of the camera in the world space. In this way, when at least one of the position or orientation of the camera and / or the client device changes, the AR graphics display surface or at least one of the AR graphics generated for the AR graphics display surface may appear to be fixed in the real world space.

[0102] Figure 7 7 is a flow chart of a process 700 for activating an augmented reality content item using data captured by a camera of a head mounted device to generate a user interface including augmented reality graphics superimposed on a surface located in an environment according to one or more examples. Process 700 may include capturing, by a camera of a head mounted device, one or more images within a field of view of the camera at operation 702. The head mounted device may include at least one display device that displays a user interface viewable by a user of the head mounted device.

[0103] Process 700 may include activating an augmented reality (AR) content item executed within the client application at operation 704. For example, a user may select an icon of an augmented reality content item from a menu of augmented reality content items to activate the AR content item. In one or more illustrative examples, the AR content item may include: Figure 5 The AR graphic display content item 502 is shown.

[0104] Additionally, process 700 may include providing camera data including one or more images to one or more AR graphics services using one or more application programming interface (API) calls in response to activating the augmented reality content item at operation 706. The camera data may correspond to one or more images captured by a camera of the head mounted device. In various examples, the camera data may be analyzed by the one or more AR graphics services to generate AR graphics within a user interface displayed by the head mounted device.

[0105] Process 700 may also include obtaining AR graphics display data from one or more AR graphics services at operation 708. The AR graphics display data may indicate at least one of an AR graphics display surface or a graphics input tool included in the field of view. In one or more examples, the camera data may be analyzed by an object detection service that is a component of at least one of a surface detection system or an object detection system to determine an object of interest corresponding to the AR graphics display surface included in the field of view. In various examples, the characteristics of one or more objects of interest included in the field of view may be analyzed to determine the AR graphics display surface. In various examples, the AR graphics display data may indicate at least one of the position, size, or shape of the AR graphics display surface. In addition, the AR graphics display data may indicate at least one of the position, size, shape, or size of the graphics input tool.

[0106] In addition, process 700 may include determining, at operation 710, based on the AR graphics display data, the movement of the graphics input tool included in the field of view when the graphics input tool is within a threshold distance of the AR graphics display surface. The graphics input tool may be identified by analyzing the camera data by an object detection service to determine an object of interest having characteristics corresponding to the characteristics of the graphics input tool. The movement of the graphics input tool may be determined by analyzing the camera data by an object tracking service and providing information to one or more graphics input tool tracking models. At least one of the object tracking service or the graphics input tool tracking model may be a component of an object tracking system. The movement of the graphics input tool may be identified by determining the difference in the position of the graphics input tool in multiple video frames captured by the camera.

[0107] Determining that the graphics input tool is within a threshold distance of the AR graphics display surface may include determining the distance between the graphics input tool and the AR graphics display surface. In one or more examples, at least one of the depth determination service or object tracking service 512, which may be components of the AR graphics display system, may provide information to one or more graphics input tool state models to determine that the marking feature of the augmented reality content item is activated and determine that the graphics display tool is within a threshold distance of the AR graphics display surface. In various examples, a determination that the graphics input tool is outside a threshold distance of the AR graphics display surface may be made. In these cases, the marking function feature of the augmented reality content item is deactivated so that the mark is no longer generated based on the movement of the graphics input tool.

[0108] Process 700 may also include causing a user interface superimposed on the AR graphics display surface to be displayed at operation 712. The user interface may include an AR graphic corresponding to the movement of the graphical input tool. The user interface may also include a menu including a plurality of user interface elements that can be selected to generate a marker of the AR graphic with a corresponding color or width.

[0109] Figure 8 800 includes an augmented reality graphic superimposed on a surface according to one or more examples. The user interface 800 may include an AR graphics display surface 802. The AR graphics display surface 802 may include a portion of a wall, a table, a floor, or another surface suitable as a background for the display of AR graphics. In one or more examples, the AR graphics display surface 802 may have a size that is larger than the field of view of a camera that captures an image of the AR graphics display surface 802. In these examples, the user interface 800 may display a portion of the AR graphics display surface 802.

[0110] The user interface 800 may also display an AR graphic 804. The AR graphic 804 may be displayed as an overlay of the AR graphics display surface 802. The AR graphic 804 may be displayed in response to a motion path of a graphical input tool across the AR graphics display surface 802. In one or more examples, the AR graphic 804 may be part of a painting, text content, a drawing, one or more combinations thereof, and the like.

[0111] The user interface 800 may also include a menu 806. The menu 806 may be displayed as an overlay of the AR graphics display surface 802. The menu 806 may include a plurality of user interface elements that can be selected to perform at least one of the following: create or modify the AR graphics 804. For example, the menu 806 may include a first user interface element 808 that can be selected to cause a marker of an AR graphic having a corresponding color corresponding to the selected first user interface element 808 to be generated. In addition, the menu 806 may include a second user interface element 810 that can be selected to cause a marker of an AR graphic having a line width corresponding to the corresponding second user interface element 810 to be generated. The menu 806 may also include a third user interface element 812 that can be selected to execute an augmented reality content item corresponding to the corresponding third user interface element 812. In one or more illustrative examples, each third user interface element 812 may correspond to a corresponding sticker that can be displayed with the AR graphics 804.

[0112] In addition, menu 806 may include a fourth user interface element 814, which can be selected to display additional selectable options. Additional selectable options may correspond to additional augmented reality content items, additional colors, additional line widths, additional features, one or more combinations thereof, etc. of the AR graphics. Additional options displayed in response to the selection of the fourth user interface element 814 may also include a red, green, and blue (RGB) color picker. The RGB color picker may include a user interface element that enables the selection of a variety of different colors to be used to generate a mark for the AR graphics 804. In at least some examples, the RGB color picker may enable the selection of colors using the red, green, and blue coordinates input into the user interface element. In one or more additional examples, the RGB color picker may include a visual representation of a variety of colors that can be selected using a graphic input tool at one or more granularity levels.

[0113] Additionally, the menu 806 may include a fifth user interface element 816 that can be selected to save the AR graphic 804. In one or more examples, the AR graphic 804 may be saved in association with the AR graphics display surface 802 such that the AR graphic 804 is displayed in response to the AR graphics display surface 802 being in the field of view of the client device displaying the user interface 800.

[0114] In one or more examples, the third user interface element 812 may include the following interface elements, which can be selected to add augmented reality content items to the AR graphics, such as one or more emoticons, one or more pictures, or one or more shapes. In various examples, one or more shapes may include multiple shapes with given boundaries. The boundaries of a given shape can be adjusted. In addition, the color of the shape can also be selected by the user. In one or more illustrative examples, the third user interface element 812 can be selected by the user in response to determining that the positioning of the graphic input tool corresponds to the position of the third user interface element 812. In response to detecting the selection of the third user interface element 812, a visual representation of the item associated with the third user interface element is displayed in conjunction with the position of the graphic input tool. When the graphic input tool moves relative to the AR graphics display surface 802, the position of the visual representation of the item associated with the selected third user interface element 812 can also be moved in a corresponding manner. The position of the selected third user interface element 812 can be determined in response to detecting that a touch and grip operation has occurred relative to the graphic input tool and the point on the AR graphics display surface 802. The touch and grip operation may be detected in response to determining that the position of the graphical input tool is relatively the same for at least 0.5 seconds, at least 1 second, at least 1.5 seconds, or at least 2 seconds. In response to determining the position of the selected third user interface element 812, the selected third user interface element 812 may be displayed in the position after the graphical input tool has been moved to another position. In one or more additional illustrative examples, an augmented reality content item may be selected in response to a touch and grip input detected using the designated third user interface element 812 for selecting an augmented reality item, followed by dragging the input to a position on the AR graphics display surface 804 relative to the selected third user interface element 812. The drag operation may be performed by the user gripping the graphical input tool to touch the AR graphics display surface 802 for a period of time, and then removing the graphical input tool from the AR graphics display surface 802 at the position on the AR graphics display surface where the selected augmented reality content item is to be placed.

[0115] Fig. 9is a block diagram showing a networked system 900 including details of the glasses 100 according to some examples. The networked system 900 includes the glasses 100, a client device 926, and a server system 932. The client device 926 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 936 and / or a high-speed wireless connection 934. The client device 926 is connected to the server system 932 via a network 930. The network 930 may include any combination of wired and wireless connections. The server system 932 may be one or more computing devices that are part of a service or network computing system. The client device 926 and any elements of the server system 932 and the network 930 may use Fig.11 and Figure 3 The details of the software architecture 1104 or machine 300 described respectively are implemented.

[0116] The glasses 100 include a data processor 902, a display 910, one or more cameras 908, and additional input / output elements 916. The input / output elements 916 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 902. Fig.11 and Figure 3 Examples of input / output elements 916 are further discussed. For example, input / output elements 916 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 910 are described in the discussion above. In the specific examples described herein, display 910 includes displays for the left and right eyes of a user.

[0117] Data processor 902 includes image processor 906 (eg, video processor), GPU and display driver 938, tracking module 940, interface 912, low power circuitry 904, and high speed circuitry 920. Components of data processor 902 are interconnected by bus 942.

[0118] The interface 912 refers to any source of user commands provided to the data processor 902. In one or more examples, the interface 912 is a physical button that, when pressed, sends a user input signal from the interface 912 to the low-power processor 914. The low-power processor 914 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 914 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 912 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 908. In other examples, the interface 912 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 926.

[0119] Image processor 906 includes circuitry for receiving signals from camera 908 and processing those signals from camera 908 into a format suitable for storage in memory 924 or for transmission to client device 926. In one or more examples, image processor 906 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 908, and volatile memory used by the microprocessor in operation.

[0120] The low power circuit system 904 includes a low power processor 914 and a low power wireless circuit system 918. These elements of the low power circuit system 904 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 914 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 914 can accept user input signals from the interface 912. The low power processor 914 can also be configured to receive input signals or command communications from a client device 926 via a low power wireless connection 936. The low power wireless circuit system 918 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 918. In other examples, other low power communication systems may be used.

[0121] High-speed circuit system 920 includes a high-speed processor 922, a memory 924, and a high-speed wireless circuit system 928. High-speed processor 922 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 902. High-speed processor 922 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 934 using high-speed wireless circuit system 928. In some examples, high-speed processor 922 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.11 The high-speed processor 922 uses the software architecture of the data processor 902 to manage data transmission with the high-speed wireless circuit system 928, in addition to any other duties. In some examples, the high-speed wireless circuit system 928 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to as Wi-Fi herein. In other examples, the high-speed wireless circuit system 928 can implement other high-speed communication standards.

[0122] The memory 924 includes any storage device capable of storing camera data generated by the camera 908 and the image processor 906. Although the memory 924 is shown as being integrated with the high-speed circuitry 920, in other examples, the memory 924 may be a separate, independent element of the data processor 902. In some such examples, electrical wiring may provide a connection from the image processor 906 or the low-power processor 914 to the memory 924 through a chip including the high-speed processor 922. In other examples, the high-speed processor 922 may manage the addressing of the memory 924 so that the low-power processor 914 will direct the high-speed processor 922 whenever a read or write operation involving the memory 924 is desired.

[0123] The tracking module 940 estimates the pose of the glasses 100. For example, the tracking module 940 uses image data and associated inertial data from the camera 908 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 940 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking module 940 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 910.

[0124] The GPU and display driver 938 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 910 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 938 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real scene environment.

[0125] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 926 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1106, such as the messaging application 1146.

[0126] Fig.10 is a block diagram illustrating an example messaging system 1000 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1000 includes multiple instances of a client device 926 that host several applications including a messaging client 1002 and other applications 1004. The messaging client 1002 is communicatively coupled to other instances of the messaging client 1002 (e.g., hosted on respective other client devices 926), a messaging server system 1006, and a third-party server 1008 via a network 930 (e.g., the Internet). The messaging client 1002 may also communicate with the locally hosted application 1004 using an application program interface (API).

[0127] The messaging clients 1002 are able to communicate and exchange data with other messaging clients 1002 and with a messaging server system 1006 via the network 930. The data exchanged between the messaging clients 1002 and between the messaging clients 1002 and the messaging server system 1006 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0128] The messaging server system 1006 provides server-side functionality to a particular messaging client 1002 via the network 930. Although some functions of the messaging system 1000 are described herein as being performed by the messaging client 1002 or by the messaging server system 1006, the location of some functions within the messaging client 1002 or within the messaging server system 1006 may be a design choice. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 1006, but then migrate the technologies and functions to the messaging client 1002 where the client device 926 has sufficient processing power.

[0129] The messaging server system 1006 supports various services and operations provided to the messaging clients 1002. Such operations include transmitting data to the messaging clients 1002, receiving data from the messaging clients 1002, and processing data generated by the messaging clients 1002. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1000 is activated and controlled by functions available via the user interface (UI) of the messaging client 1002.

[0130] Turning now specifically to the messaging server system 1006, an application program interface (API) server 1010 is coupled to and provides a programming interface to an application server 1014. The application server 1014 is communicatively coupled to a database server 1016, which facilitates access to a database 1020 that stores data associated with messages processed by the application server 1014. Similarly, a web server 1024 is coupled to and provides a web-based interface to the application server 1014. To this end, the web server 1024 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0131] The application program interface (API) server 1010 receives and transmits message data (e.g., commands and message payloads) between the client device 926 and the application server 1014. Specifically, the application program interface (API) server 1010 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1002 to activate the functions of the application server 1014. The application program interface (API) server 1010 exposes various functions supported by the application server 1014, including: account registration; login functionality; sending messages from a particular messaging client 1002 to another messaging client 1002 via the application server 1014, sending media files (e.g., images or videos) from the messaging client 1002 to the messaging server 1012, and possible access by another messaging client 1002; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 926; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1002).

[0132] The application server 1014 hosts several server applications and subsystems, including, for example, a messaging server 1012, an image processing server 1018, and a social network server 1022. The messaging server 1012 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1002. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1002. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1012.

[0133] The application server 1014 also includes an image processing server 1018 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1012.

[0134] The social network server 1022 supports various social networking functions and services and makes these functions and services available to the messaging server 1012. To this end, the social network server 1022 maintains and accesses an entity graph within the database 1020. Examples of functions and services supported by the social network server 1022 include identifying other users in the messaging system 1000 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0135] The messaging client 1002 may notify the user of the client device 926 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1002 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1002 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0136] Fig.111100 is a block diagram illustrating a software architecture 1104 that may be installed on any one or more of the devices described herein. The software architecture 1104 is supported by hardware, such as a machine 1102, including a processor 1120, a memory 1126, and an I / O component 1138. In this example, the software architecture 1104 may be conceptualized as a stack of layers, in which each layer provides specific functionality. The software architecture 1104 includes layers such as an operating system 1112, a library 1108, a framework 1110, and an application 1106. In operation, the application 1106 invokes an API call 1150 through the software stack and receives a message 1152 in response to the API call 1150.

[0137] The operating system 1112 manages hardware resources and provides public services. The operating system 1112 includes, for example, a kernel 1114, services 1116, and drivers 1122. The kernel 1114 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1114 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1116 can provide other public services to other software layers. Drivers 1122 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1122 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0138] The library 1108 provides a low-level common infrastructure used by the application 1106. The library 1108 may include a system library 1118 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1108 may include an API library 1124, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) image content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1108 may also include various other libraries 1128 to provide many other APIs to the application 1106 .

[0139] The framework 1110 provides a high-level common infrastructure used by the applications 1106. For example, the framework 1110 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1110 may provide a wide range of other APIs that may be used by the applications 1106, some of which may be specific to a particular operating system or platform.

[0140] In an example, applications 1106 may include a home application 1136, a contacts application 1130, a browser application 1132, a book reader application 1134, a location application 1142, a media application 1144, a messaging application 1146, a game application 1148, and a variety of other applications such as third-party applications 1140. Applications 1106 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1106 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1140 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1140 can call the API call 1150 provided by the operating system 1112 to facilitate the functions described in this article.

[0141] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0142] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0143] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0144] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0145] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0146] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media".

[0147] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0148] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0149] A numbered, non-limiting list of examples of the present subject matter is presented below.

[0150] Example 1. A computing device comprising: a camera; one or more processors; and a memory storing instructions, which, when executed by the one or more processors, cause the computing device to perform operations, the operations comprising: capturing, by the camera, one or more images within a field of view of the camera; activating an augmented reality (AR) content item executed within a client application; in response to activating the augmented reality content item, providing camera data including one or more images to one or more AR graphics services using one or more application programming interface (API) calls; obtaining AR graphics display data from one or more AR graphics services, the AR graphics display data indicating an AR graphics display surface included in the field of view and a graphics input tool included in the field of view; determining a movement of the graphics input tool based on at least one of the AR graphics display data and the camera data; and causing a user interface to be displayed superimposed on the AR graphics display surface, the user interface including an AR graphic corresponding to the movement of the graphics input tool.

[0151] Example 2. A computing device according to Example 1, wherein the computing device is a head-mounted device including at least one display device, and the user interface is displayed by the at least one display device.

[0152] Example 3. A computing device according to Example 1 or 2, wherein the memory stores instructions that, when executed by one or more processors, cause the computing device to perform operations, the operations comprising: in response to determining that a graphics input tool is within a threshold distance of an AR graphics display surface, activating one or more features of an augmented reality content item to create an AR graphic or modify at least one of the AR graphics; determining that the graphics input tool is outside the threshold distance of the AR graphics display surface; and in response to determining that the graphics input tool is outside the threshold distance of the AR graphics display surface, deactivating one or more features of the augmented reality content item.

[0153] Example 4. A computing device according to any one of Examples 1 to 3, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: causing an AR graphics menu to be displayed within a user interface, the AR graphics menu comprising multiple user interface elements, each of the multiple user interface elements being selectable to activate one or more features of an augmented reality content item to create a portion of the AR graphics or modify at least one of a portion of the AR graphics.

[0154] Example 5. A computing device according to any one of Examples 1 to 4, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that a graphical input tool is within an additional threshold distance of a user interface element among multiple user interface elements of an AR graphical menu; and causing a feature of the user interface element to be activated.

[0155] Example 6. A computing device according to any one of Examples 1 to 5, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: obtaining coordinates of an AR graphics display surface in real-world space based on camera device data; and determining a distance from a camera device to the AR graphics display surface based on the camera device data.

[0156] Example 7. A computing device according to Example 6, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the distance from the camera device to the AR graphics display surface has changed from a first distance to a second distance; and modifying the magnification level of the AR graphics within the user interface based on the distance from the camera device to the AR graphics display changing from the first distance to the second distance.

[0157] Example 8. A computer-implemented method, comprising: obtaining, by a computing system comprising one or more processors and a memory, camera data corresponding to one or more images of a field of view captured by a camera; analyzing, by the computing system, the camera data to determine an augmented reality (AR) graphics display surface included in the field of view, the AR graphics display surface being a two-dimensional surface having at least a minimum length and at least a minimum width; analyzing, by the computing system, the camera data to detect a graphics input tool included in the field of view; determining, by the computing system and based on the camera data, a movement of the graphics input tool when the graphics input tool is within a threshold distance of the AR graphics display surface; and causing, by the computing system, to display a user interface superimposed on the AR graphics display surface, the user interface comprising AR graphics corresponding to the movement of the graphics input tool.

[0158] Example 9. The computer-implemented method according to Example 8 further includes: implementing one or more machine learning models by a computing system to determine an amount of curvature of the tip of the graphic input tool; and determining, by the computing system, a size of a marker of the AR graphic based on the amount of curvature of the tip of the graphic input tool.

[0159] Example 10. The computer-implemented method according to Example 9 further includes: training one or more machine learning models based on a training data set by a computing system using supervised machine learning training techniques, wherein the training data set includes: a first number of graphic input tool images, the first number of graphic input tool images corresponding to a first bending amount of the tip of the graphic input tool and associated with a first label; and a second number of graphic input tool images, the second number of graphic input tool images corresponding to a second bending amount of the tip of the graphic input tool and associated with a second label.

[0160] Example 11. A computer-implemented method according to any one of Examples 8 to 10, wherein: analysis of camera device data for detecting a graphics input tool is performed in response to one or more first application programming interface (API) calls corresponding to a first AR graphics service; determination of the distance between the graphics input tool and the AR graphics display surface is performed by the computing system in response to one or more second API calls corresponding to a second AR graphics service; and determination of movement of the graphics input tool is performed in response to one or more third API calls corresponding to a third AR graphics service.

[0161] Example 12. The computer-implemented method of any one of Examples 8 to 11 further includes: determining, by the computing system, that the graphics input tool is within a threshold distance of the AR graphics display surface; and wherein determining that the graphics input tool is within the threshold distance of the AR graphics display surface includes determining, by the computing system and based on camera device data, that the graphics input tool is touching the AR graphics display surface.

[0162] Example 13. A computer-implemented method according to any one of Examples 8 to 12, further comprising: analyzing camera data by a computing system to detect a hand grasping a graphics input tool included in a field of view; determining, by the computing system and based on the camera data, the movement of the hand when the graphics input tool is within a threshold distance of an AR graphics display surface; and determining, by the computing system and based on the camera data, a first position of the hand relative to a second position of the graphics input tool; wherein the characteristics of the AR graphics are based on the movement of the graphics input tool, the movement of the hand, and the first position of the hand relative to the second position of the graphics input tool.

[0163] Example 14. A computer-implemented method according to any one of Examples 8 to 13, wherein the computing system begins determining the AR graphics display surface in response to launching an augmented reality content item executed within the client application.

[0164] Example 15. The computer-implemented method of any one of Examples 8 to 14 further includes: obtaining input by the computing system to execute additional augmented reality content items for at least one of the AR graphics or the AR graphics display surface; and causing the user interface to be modified according to the additional augmented reality content items by the computing system and based on the input, wherein executing the additional augmented reality content items causes at least one of the additional AR graphics to be displayed in the user interface or causes the appearance of the AR graphics to be modified.

[0165] Example 16. A computing device comprising: one or more processors; and a memory storing instructions, which, when executed by the one or more processors, cause the computing device to perform operations, the operations comprising: obtaining camera data corresponding to one or more images of a field of view captured by a camera; analyzing the camera data to determine an augmented reality (AR) graphics display surface included in the field of view, the AR graphics display surface being a two-dimensional surface having at least a minimum length and at least a minimum width; analyzing the camera data to detect a graphics input tool included in the field of view; determining, based on the camera data, movement of the graphics input tool when the graphics input tool is within a threshold distance of the AR graphics display surface; and causing a user interface to be displayed superimposed on the AR graphics display surface, the user interface comprising AR graphics corresponding to the movement of the graphics input tool.

[0166] Example 17. A computing device according to Example 16, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the field of view of the camera device has changed from a first field of view that includes an AR graphics display surface to a second field of view in which no AR graphics display surface exists; and in response to the field of view of the camera device changing from the first field of view to the second field of view, causing the user interface to be modified by removing the AR graphics from the user interface.

[0167] Example 18. A computing device according to Example 17, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: obtaining additional camera data corresponding to one or more additional images of a second field of view captured by the camera, wherein the additional camera data includes one or more additional images captured during a time period after an initial time period in which the one or more images of the camera data are captured; analyzing the additional camera data to determine an additional AR graphics display surface included in the second field of view; analyzing the additional camera data to detect a graphical input tool in the second field of view; determining that the graphical input tool is within a threshold distance of the additional AR graphics display surface; determining, based on the additional camera data, additional movement of the graphical input tool when the graphical input tool is within the threshold distance of the additional AR graphics display surface; and causing an additional user interface to be displayed superimposed on the additional AR graphics display surface, the additional user interface including additional AR graphics corresponding to additional movement of the graphical input tool.

[0168] Example 19. A computing device according to Example 18, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: causing AR graphics to be stored in the memory together with the AR graphics display surface; and causing additional AR graphics to be stored in the memory together with the additional AR graphics display surface.

[0169] Example 20. A computing device according to Example 19, wherein the memory stores additional instructions that, when executed by one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the field of view of the camera device has changed from the second field of view to the first field of view; and in response to the field of view of the camera device changing from the second field of view to the first field of view, modifying the user interface by adding AR graphics to the user interface.

[0170] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computing device, comprising: Camera device; one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: capturing, by the camera device, one or more images within a field of view of the camera device; Activating an augmented reality (AR) content item executed within a client application; In response to activating the augmented reality content item, providing camera data including the one or more images to one or more AR graphics services using one or more application programming interface (API) calls; obtaining AR graphics display data from the one or more AR graphics services, the AR graphics display data indicating an AR graphics display surface included in the field of view and a graphics input tool included in the field of view; determining a motion of the graphics input tool based on at least one of the AR graphics display data and the camera data; and A user interface is caused to be displayed that is superimposed on the AR graphics display surface, the user interface including an AR graphics corresponding to the movement of the graphic input tool.

2. The computing device of claim 1, wherein: The computing device is a head mounted device including at least one display device, and the user interface is displayed by the at least one display device.

3. The computing device of claim 1, wherein: The memory stores instructions that, when executed by the one or more processors, cause the computing device to perform operations including: In response to determining that the graphic input tool is within a threshold distance of the AR graphics display surface, activating one or more features of the augmented reality content item to at least one of create the AR graphics or modify the AR graphics; determining that the graphic input tool is outside a threshold distance of the AR graphics display surface; and In response to determining that the graphical input tool is outside a threshold distance of the AR graphics display surface, one or more features of the augmented reality content item are deactivated.

4. The computing device of claim 1, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: An AR graphics menu is displayed within the user interface, wherein the AR graphics menu includes multiple user interface elements, each of which can be selected to activate one or more features of the augmented reality content item to create a portion of the AR graphics or modify at least one of the portions of the AR graphics.

5. The computing device of claim 1, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the graphical input tool is within an additional threshold distance of a user interface element in the plurality of user interface elements of the AR graphical menu; and Causes a feature of the user interface element to be activated.

6. The computing device of claim 1, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: Obtaining coordinates of the AR graphics display surface in real world space based on the camera data; and A distance from the camera to the AR graphics display surface is determined based on the camera data.

7. The computing device of claim 6, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the distance from the camera to the AR graphics display surface has changed from a first distance to a second distance; and The magnification level of the AR graphic within the user interface is modified based on a change in the distance from the camera to the AR graphic display from the first distance to the second distance.

8. A computer-implemented method comprising: obtaining, by a computing system including one or more processors and memory, camera data corresponding to one or more images of a field of view captured by a camera; analyzing, by the computing system, the camera data to determine an augmented reality (AR) graphics display surface included in the field of view, the AR graphics display surface being a two-dimensional surface having at least a minimum length and at least a minimum width; analyzing, by the computing system, the camera data to detect a graphical input tool included in the field of view; determining, by the computing system and based on the camera data, a movement of the graphics input tool when the graphics input tool is within a threshold distance of the AR graphics display surface; as well as The computing system causes display of a user interface superimposed on the AR graphics display surface, the user interface including an AR graphic corresponding to the movement of the graphic input tool.

9. The computer-implemented method of claim 8, further comprising: implementing, by the computing system, one or more machine learning models to determine an amount of curvature of the tip of the graphic input tool; as well as The size of the marker of the AR graphic is determined by the computing system based on the amount of bending of the tip of the graphic input tool.

10. The computer-implemented method of claim 9, further comprising: The one or more machine learning models are trained by the computing system based on a training data set using a supervised machine learning training technique, wherein the training data set includes: a first number of graphic input tool images, the first number of graphic input tool images corresponding to a first amount of curvature of the tip of the graphic input tool and associated with a first label; and a second number of graphic input tool images, the second number of graphic input tool images corresponding to a second amount of curvature of the tip of the graphic input tool and associated with a second label.

11. The computer-implemented method of claim 8, wherein: performing analysis of the camera data for detecting the graphical input tool in response to one or more first application programming interface (API) calls corresponding to a first AR graphics service; determining, by the computing system, a distance between the graphical input tool and the AR graphics display surface in response to one or more second API calls corresponding to a second AR graphics service; as well as Determining the motion of the graphical input tool is performed in response to one or more third API calls corresponding to a third AR graphics service.

12. The computer-implemented method of claim 8, further comprising: determining, by the computing system, that the graphic input tool is within a threshold distance of the AR graphics display surface; as well as Wherein, determining that the graphics input tool is within a threshold distance of the AR graphics display surface includes determining, by the computing system and based on the camera data, that the graphics input tool is touching the AR graphics display surface.

13. The computer-implemented method of claim 8, further comprising: analyzing, by the computing system, the camera data to detect a hand grasping the graphical input tool included in the field of view; determining, by the computing system and based on the camera data, movement of the hand when the graphical input tool is within a threshold distance of the AR graphics display surface; as well as determining, by the computing system and based on the camera data, a first position of the hand relative to a second position of the graphical input tool; The characteristic of the AR graphic is based on the movement of the graphic input tool, the movement of the hand, and a first position of the hand relative to a second position of the graphic input tool.

14. The computer-implemented method of claim 8, wherein: The computing system begins determining the AR graphics display surface in response to launching an augmented reality content item executing within a client application.

15. The computer-implemented method of claim 8, further comprising: obtaining, by the computing system, input to execute an additional augmented reality content item for at least one of the AR graphic or the AR graphic display surface; as well as The user interface is modified, by the computing system and based on the input, according to the additional augmented reality content item, wherein execution of the additional augmented reality content item causes at least one of additional AR graphics to be displayed in the user interface or causes the appearance of the AR graphic to be modified.

16. A computing device comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: obtaining camera data corresponding to one or more images of a field of view captured by the camera; analyzing the camera data to determine an augmented reality (AR) graphics display surface included in the field of view, the AR graphics display surface being a two-dimensional surface having at least a minimum length and at least a minimum width; analyzing the camera data to detect a graphical input tool included in the field of view; determining, based on the camera data, a movement of the graphics input tool when the graphics input tool is within a threshold distance of the AR graphics display surface; and A user interface is caused to be displayed that is superimposed on the AR graphics display surface, the user interface including an AR graphics corresponding to the movement of the graphic input tool.

17. The computing device of claim 16, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the field of view of the camera device has changed from a first field of view including the AR graphics display surface to a second field of view in which the AR graphics display surface does not exist; and In response to the field of view of the camera changing from the first field of view to the second field of view, the user interface is modified by removing the AR graphic from the user interface.

18. The computing device of claim 17, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: obtaining additional camera data corresponding to one or more additional images of the second field of view captured by the camera, wherein the additional camera data includes one or more additional images captured during a time period subsequent to an initial time period during which the one or more images of the camera data were captured; analyzing the additional camera data to determine an additional AR graphics display surface included in the second field of view; analyzing the additional camera data to detect the graphical input tool in the second field of view; determining that the graphic input tool is within a threshold distance of the additional AR graphic display surface; determining, based on the additional camera data, additional movement of the graphics input tool when the graphics input tool is within a threshold distance of the additional AR graphics display surface; and An additional user interface is caused to be displayed superimposed on the additional AR graphics display surface, the additional user interface including an additional AR graphic corresponding to the additional movement of the graphic input tool.

19. The computing device of claim 18, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: causing the AR graphics to be stored in the memory together with the AR graphics display surface; and The additional AR graphics are caused to be stored in the memory together with the additional AR graphics display surface.

20. The computing device of claim 19, wherein: The memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations, the additional operations comprising: determining that the field of view of the camera device has changed from the second field of view to the first field of view; and In response to the field of view of the camera changing from the second field of view to the first field of view, the user interface is modified by adding the AR graphic to the user interface.

Images