Virtual selfie stick selfie

Abstract

Methods for generating virtual selfie stick images are described. In one aspect, the method includes generating, at a device, an original selfie image with an optical sensor of the device, the optical sensor aimed at a face of a user of the device, the device held at an arm's length from the face of the user; displaying, on a display of the device, an indication, the indication directing the user to move the device around the face of the user at the arm's length within a limited range in a plurality of poses; accessing, at the device, image data generated by the optical sensor in the plurality of poses; and generating a virtual selfie stick selfie image based on the original selfie image and the image data.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Application Serial No. 17 / 851,448, filed June 28, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein generally relates to imaging systems. Specifically, this disclosure presents systems and methods for generating selfie images. Background Technology

[0004] Augmented reality (AR) devices allow users to observe a scene while seeing related virtual content that can be aligned with items, images, objects, or the environment within the device's field of view. Virtual reality (VR) devices offer a more immersive experience than AR devices. VR devices utilize virtual content displayed based on the VR device's positioning and orientation to obscure the user's field of view.

[0005] Both AR and VR devices rely on motion tracking systems that track the device's posture (e.g., orientation, orientation, position). Motion tracking systems (also known as visual tracking systems) use images captured by the AR / VR device's optical sensors to track its posture. However, images can become blurry when the AR / VR device moves quickly. Therefore, high motion blur leads to degraded tracking performance. Alternatively, high motion blur results in higher computational operations to maintain sufficient tracking accuracy and image quality under high dynamic conditions. Attached Figure Description

[0006] To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced.

[0007] Figure 1 An example environment for operating the device is shown according to an example implementation.

[0008] Figure 2 This is a block diagram illustrating a device according to an example implementation.

[0009] Figure 3 This is a block diagram illustrating a selfie application according to an example implementation.

[0010] Figure 4 This is a block diagram showing the image area of ​​a selfie image taken with a virtual selfie stick according to an example implementation.

[0011] Figure 5 An example operation of the device according to an example implementation is shown.

[0012] Figure 6 An example graphical user interface for a selfie app, according to an example implementation, is shown.

[0013] Figure 7 An example graphical user interface for a selfie app, according to an example implementation, is shown.

[0014] Figure 8 This is a block diagram illustrating a visual tracking system according to an example implementation.

[0015] Figure 9 This is a block diagram illustrating an example processing of a selfie application according to an example implementation.

[0016] Figure 10 This is a flowchart illustrating a method for generating a virtual selfie stick selfie image according to an example implementation.

[0017] Figure 11 This is a flowchart illustrating a method for applying a computer vision algorithm to generate a virtual selfie stick selfie image according to an example implementation.

[0018] Figure 12 An aspect of the subject matter according to an example implementation is shown.

[0019] Figure 13 An aspect of the subject matter according to an example implementation is shown.

[0020] Figure 14 An area of ​​a virtual selfie stick selfie image is shown according to an example implementation.

[0021] Figure 15 This is a block diagram illustrating a software architecture in which the present disclosure can be implemented according to an example implementation.

[0022] Figure 16 It is a schematic representation of a machine in the form of a computer system according to an example implementation, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0023] The following description illustrates systems, methods, techniques, instruction sequences, and computer program products that demonstrate exemplary embodiments of the subject matter. In this description, numerous specific details are set forth for illustrative purposes to provide an understanding of various embodiments of the subject matter. However, it will be apparent to those skilled in the art that embodiments of the subject matter can be practiced without some or more of these specific details. The examples merely represent possible variations. Unless explicitly stated otherwise, structures (e.g., structural components such as modules) are optional and can be combined or subdivided, and operations (e.g., in processing, algorithms, or other functions) can vary in order or be combined or subdivided.

[0024] The term "selfie" is used in this document to refer to a person taking a photograph of themselves (e.g., a "selfie image"). For example, capturing a selfie image using a camera device / smartphone held by the user with their outstretched arm. The term "selfie stick" refers to a stick / pole on which a camera device can be mounted so that a person holding the stick can take a photo of themselves from a greater distance than if they were holding the camera device or smartphone in their hand.

[0025] The term "visual tracking system" is used herein to refer to a computer-operated application or system that enables the system to track visually identified features in images captured by one or more camera devices of the visual tracking system. The visual tracking system constructs a model of a real-world environment based on the tracked visual features. Non-limiting examples of visual tracking systems include Visual Simultaneous Localization and Mapping (VSLAM) systems and Visual Inertial Odometry (VIO) systems. VSLAM can be used to construct a target from an environment or scene based on one or more camera devices of the visual tracking system. VIO (also known as a visual inertial tracking system) determines the latest posture (e.g., localization and orientation) of the device based on data acquired from multiple sensors (e.g., optical sensors, inertial sensors) of the device.

[0026] The term "Inertial Measurement Unit" (IMU) is used herein to refer to a device capable of reporting the inertial state of a moving subject, including its acceleration, velocity, orientation, and position. An IMU tracks the movement of a subject by integrating the acceleration and angular velocity measured by the IMU. An IMU can also refer to a combination of an accelerometer and a gyroscope, which can determine and quantify linear acceleration and angular velocity, respectively. Values ​​obtained from the IMU's gyroscopes can be processed to obtain the IMU's pitch, roll, and heading, thereby obtaining the pitch, roll, and heading of the subject associated with the IMU. Signals from the IMU's accelerometers can also be processed to obtain the IMU's velocity and displacement.

[0027] When taking a selfie, the user's face may be distorted due to the proximity of the camera to the face. On the other hand, selfies or portraits taken with a selfie stick have less distortion compared to selfies taken at arm's length. Without a selfie stick, achieving the ideal distance for capturing portrait images can be difficult.

[0028] This application describes a method for optimizing image processing by dividing regions on a portrait image into invisible, occluded, and interpolated regions and applying computer vision algorithms to only specific portions of the portrait image, rather than calculating every pixel of the portrait image. For example, computer vision algorithms such as Multi-View Stereo (MVS) or Neural Radiation Field (NeRF) techniques are used to calculate pixels only in the invisible and occluded regions. Other regions of the portrait image are calculated using image interpolation (e.g., scaling), which is significantly faster and more efficient than any 3D computer vision algorithm.

[0029] Therefore, this application describes an efficient process for generating a portrait image (e.g., approximately 1.5m from the camera device to the face) by utilizing a selfie image of a person (approximately 0.5m from the camera device to the face) and several images taken around the face at similar distances (approximately 0.5m). The portrait image is generated using a computer vision algorithm that divides pixels into three different types (missing, occluded, and interpolable). The first two types of pixels can be computed using standard 3D composite view generation methods (e.g., multi-view stereo (MVS) or neural radiation field (NeRF) or different variations such as MVSneRF). Interpolable pixels can be computed using interpolation of pixels from the selfie image. The interpolation process is faster than that of 3D computer vision algorithms.

[0030] The present invention describes a method for generating a virtual selfie stick image. In one aspect, the method includes: generating an original selfie image at a device using an optical sensor of the device, the optical sensor being aligned with the user's face on the device, the device being held at an arm's length distance from the user's face; displaying instructions on a display of the device guiding the user to move the device around the user's face at arm's length within a limited range in multiple poses; accessing image data generated by the optical sensor in the multiple poses at the device; and generating a virtual selfie stick selfie image based on the original selfie image and the image data.

[0031] Therefore, one or more of the methods described herein help solve the technical problems of processing distorted and rendering selfie images. The currently described methods provide improvements in the operation of device functionality by limiting the computational operations of computer vision algorithms to specific regions in the image. Therefore, one or more of the methods described herein can avoid the need for certain workloads or computational resources. Examples of such computational resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0032] Figure 1 An example environment 102 for operating device 106 is shown according to an example implementation. Environment 102 includes user 104, device 106, and background 110. User 104 operates device 106.

[0033] Device 106 may be a computing device with a display, such as a smartphone or tablet. Device 106 may include a front-facing camera 118 and a rear-facing camera (not shown). User 104 (using an extended user arm 116) grasps device 106 which has a front-facing camera 118 aligned with the user's head 114 to capture a selfie image. The front-facing camera 118 has a field of view 112 that captures an image of the user 104's face and a background 110. The background 110 includes any scenery located behind the user 104. Device 106 includes a screen 120 that displays the selfie image captured by the front-facing camera 118 of device 106.

[0034] In one example implementation, device 106 includes a posture tracking system 210. The posture tracking system 210 uses, for example, optical sensors (e.g., depth-enabled 3D cameras, image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track the posture (e.g., positioning and orientation) of device 106 relative to environment 102. In one example, device 106 displays virtual content based on the posture of device 106 relative to user head 114 and / or background 110.

[0035] Figure 1 Any of the machines, databases, or devices shown can be implemented in a general-purpose computer that has been software-modified (e.g., configured or programmed) to perform one or more of the functions described herein for that machine, database, or device. For example, see below. Figures 10 to 11The discussion focuses on computer systems capable of implementing any one or more of the methods described herein. As used herein, a "database" is a data storage resource and can store data structured as text files, tables, spreadsheets, relational databases (e.g., object-relational databases), triplet storage, hierarchical data storage, or any suitable combination thereof. Furthermore, device 106 can be assembled into a single machine, and the functionality described herein for any single machine, database, or device can be subdivided among multiple machines, databases, or devices.

[0036] Device 106 can operate via a computer network. A computer network can be any network that enables communication between or within machines, databases, and devices. Therefore, a computer network can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. A computer network may include one or more components constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0037] Figure 2 This is a block diagram illustrating the modules (e.g., components) of a device 106 according to some example embodiments. Device 106 includes a sensor 202, a display 204, a processor 206, and a storage device 208. Examples of device 106 include mobile computing devices or smartphones.

[0038] Sensor 202 includes, for example, optical sensors 214 (e.g., imaging devices such as color cameras, thermal cameras, depth sensors, and one or more grayscale, global / rolling shutter tracking cameras) and inertial sensors 212 (e.g., gyroscopes, accelerometers, magnetometers). Other examples of sensor 202 include proximity sensors or position sensors (e.g., near-field communication, GPS, Bluetooth, Wi-Fi), audio sensors (e.g., microphones), thermal sensors, pressure sensors (e.g., barometers), or any suitable combination thereof. Note that sensor 202 described herein is for illustrative purposes and therefore sensor 202 is not limited to the sensors described above.

[0039] Display 204 includes a screen or monitor configured to display images generated by processor 206. In another example, display 204 includes a touchscreen display configured to receive user input via a touchscreen display.

[0040] Processor 206 includes a selfie app 216 and a pose tracking system 210. Selfie app 216 uses a combination of computer vision algorithms and scaling algorithms to generate virtual selfie stick selfie images. In one example implementation, selfie app 216 accesses selfie images taken from different angles and multiple other selfie images, and generates virtual selfie stick selfie images based on these images. Selfie app 216 segments or divides the virtual selfie stick selfie images into three different types (missing, occluded, and interpolable). Selfie app 216 applies computer vision algorithms (e.g., 3D synthetic view generation methods such as Multi-View Stereo (MVS) or Neural Radiation Field (NeRF) or different variations such as MVS / NeRF) to compute pixels in missing and occluded regions. Selfie app 216 uses interpolation of pixels on the selfie images to compute interpolable pixels in the interpolated regions.

[0041] The pose tracking system 210 estimates the pose of the device 106. For example, the pose tracking system 210 uses image data and corresponding inertial data from the optical sensor 214 and the inertial sensor 212 to track the position and pose of the device 106 relative to a reference frame (e.g., the real-world environment). (Refer to below...) Figure 8 The pose tracking system 210 is described in more detail.

[0042] Storage device 208 stores selfie images taken from different angles, multiple other selfie images, the pose of device 106 corresponding to different angles, and selfie images taken with a virtual selfie stick.

[0043] Any one or more modules described herein may be implemented using hardware (e.g., a machine's processor) or a combination of hardware and software. For example, any module described herein may configure a processor to perform the operations described herein for that module. Furthermore, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Moreover, according to various example implementations, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0044] Figure 3 This is a block diagram illustrating a selfie application 216 according to an example implementation. The selfie application 216 includes a user guidance module 302, a region segmentation module 304, an extended region 306, an occlusion region 308, an interpolation region 310, a computer vision module 312, a zoom module 314, and a virtual selfie stick selfie module 316.

[0045] User guidance module 302 generates the following instructions to user 104: take an original selfie image and then move device 106 in different directions to capture other selfies at those different angles. In one example, user guidance module 302 generates a graphical user interface that displays directional indicators or provides written instructions to the user to move his device 106 in the specified direction. User guidance module 302 accesses images captured by optical sensor 214 (e.g., front-facing camera device 118) at the original selfie location and at other angles. User guidance module 302 provides original selfie image data from the original selfie image to region segmentation module 304. User guidance module 302 provides extended selfie image data from selfie images captured at other angles to region segmentation module 304. For example, extended selfie image data is two or more images captured from two or more different angles.

[0046] Region segmentation module 304 identifies portions of the virtual selfie stick selfie image: extended region 306, occlusion region 308, and interpolation region 310. Extended region 306 includes the boundary region of the virtual selfie stick selfie image. For example, the boundary region includes content that continuously extends from the peripheral region of the original selfie image. An example of extended region 306 is shown in... Figure 4 and Figure 14 As shown in the diagram. The occlusion region 308 includes an occlusion area in the virtual selfie stick selfie image adjacent to the face of user 104. The occlusion region includes background content (e.g., a portion of background 110) in the original selfie image that is obscured by the user's head 114. An example of the occlusion region 308 is shown in... Figure 4 and Figure 14 As shown in the diagram. The interpolation region 310 includes a remapping area that encompasses the content of the user 104's face and background 110 displayed in both the original selfie image and the virtual selfie stick selfie image. The remapping area does not include boundary areas and occlusion areas. An example of the interpolation region 310 is shown in... Figure 4 and Figure 14 As shown in the image.

[0047] Computer vision module 312 includes computer vision algorithms (e.g., 3D synthetic view generation methods such as multi-view stereo (MVS) or neural radiation field (NeRF) or different variations such as MVS / NeRF) to compute pixels in extended region 306 and occluded region 308. Scaling module 314 includes a scaling / mapping engine that uses interpolation of pixels on the selfie image to compute interpolable pixels in interpolated region 310.

[0048] The virtual selfie stick selfie module 316 forms a virtual selfie stick selfie image based on the combination of the processed extended region 306, the occlusion region 308 and the interpolation region 310.

[0049] Figure 4 This is a block diagram illustrating the image area of ​​a selfie image taken with a virtual selfie stick according to an example embodiment. User 104 grasps device 106 with his outstretched arm to capture the original selfie image. Front-facing camera device 118 includes an original field of view 408 that captures an image of user 104's face (e.g., face region 424) and background 110. User's head 114 obscures background 110 in obscured regions 406 and 404. Device 106 is positioned along an original plane 430, which is positioned to have an original foreground depth 426 and an original background depth 416. Device 106 is positioned along the original plane 430 having the original foreground depth 426 and the original background depth 416.

[0050] The virtual selfie stick selfie module 316 generates a virtual selfie stick selfie image to make it appear as if it were taken from the selfie stick position 402. The selfie stick position 402 positions the device 106 along a virtual plane 432 having a virtual selfie stick foreground depth 428 and a selfie stick background depth 418. The device 106, located at the selfie stick position 402, has a selfie stick field of view 410 that captures an image of the user 104's face (e.g., face region 424) and the background 110. Because the selfie stick position 402 is further away from the user's head 114, a larger portion of the background 110 is captured: extended regions 412 and 414.

[0051] The computer vision module 312 performs calculations on the extended region 412, extended region 414, occluded region 404, and occluded region 406. The scaling module 314 remaps or rescales the interpolated region 420, interpolated region 422, and face region 424 to match the scaling of the virtual selfie stick selfie image relative to the original selfie image.

[0052] Figure 5 An example operation of device 106 according to an example implementation is shown. User 104 grasps device 106 with an extended user arm 116 to capture a raw selfie image in posture A506. Device 106 captures a first image (e.g., the raw selfie image) in posture A506 and registers the first image in posture A506.

[0053] Device 106 instructs the user to move device 106 to different angles / poses (e.g., pose B 502 and pose C 504) using his / her extended user arm 116. Device 106 captures a second image in pose B 502 and registers the second image in pose B 502. Device 106 captures a third image in pose C 504 and registers the third image in pose C 504.

[0054] Figure 6An example graphical user interface for a selfie application according to an exemplary implementation is shown. User 104 grips device 106 with an extended user arm 116 to capture a raw selfie image against a background 110. Device 106 includes a graphical user interface that displays instructions 604 and / or directional indicators 602 to instruct or guide user 104 to move device 106 in a specified direction.

[0055] Figure 7 An example graphical user interface for a selfie application according to an exemplary implementation is shown. User 104 grips device 106 with an extended user arm 116 to capture an original selfie image against a background 110. Device 106 includes a graphical user interface displaying instructions 704 and a slider 702, allowing user 104 to adjust the distance of a virtual selfie stick relative to user 104 (e.g., virtual selfie stick foreground depth 428). Once the new foreground depth is set, computer vision module 312 performs calculations on extended regions 412, extended regions 414, occlusion regions 404, and occlusion regions 406 based on the new foreground depth. Scaling module 314 remaps or rescales interpolation regions 420, interpolation regions 422, and face region 424 to match the scaling of the virtual selfie stick selfie image relative to the original selfie image based on the new foreground depth.

[0056] Figure 8 This is a block diagram illustrating a posture tracking system 210 according to an example embodiment. The posture tracking system 210 includes an inertial sensor module 802, an optical sensor module 804, and a posture estimation module 806. The inertial sensor module 802 accesses inertial sensor data from inertial sensor 212. The optical sensor module 804 accesses optical sensor data (e.g., images, camera device settings / operation parameters) from optical sensor 214. Examples of camera device operation parameters include, but are not limited to, the exposure time of optical sensor 214, the field of view of optical sensor 214, the ISO value of optical sensor 214, and the image resolution of optical sensor 214.

[0057] The pose estimation module 806 determines the pose (e.g., position, orientation, orientation) of the device 106 relative to a reference frame (e.g., user head 114 or background 110). In one example embodiment, the pose estimation module 806 includes a VIO system that estimates the pose of the device 106 based on a 3D map of feature points from a current image captured by the optical sensor 214 and inertial sensor data captured by the inertial sensor 212.

[0058] In one example implementation, the pose estimation module 806 calculates the position and orientation of the device 106. The device 106 includes one or more optical sensors 214 mounted on a rigid platform (the frame of the device 106), and one or more inertial sensors 212. The optical sensors 214 may be mounted with non-overlapping (distributed aperture) or overlapping (stereo or more) fields of view.

[0059] In some example implementations, the pose estimation module 806 includes an algorithm that combines inertial information from the inertial sensor 212 and image information from the pose estimation module 806, the inertial sensor 212 and the pose estimation module 806 being coupled to a rigid platform (e.g., device 106) or a rig. In one implementation, the rig may consist of multiple camera devices mounted on a rigid platform having an inertial navigation unit (e.g., inertial sensor 212). Thus, the rig may have at least one inertial navigation unit and at least one camera device.

[0060] Figure 9 This is a block diagram illustrating an example process according to an example implementation. The pose tracking system 210 receives sensor data from sensor 202 to determine the pose of device 106. The pose estimation module 806 identifies the pose of the pose tracking system 210 and provides the pose data to the selfie application 216.

[0061] Selfie app 216 retrieves image data from optical sensor 214 and applies a combination of computer vision and scaling algorithms to different parts of the image to generate a virtual selfie stick selfie image. The virtual selfie stick selfie image is displayed on display 204 and can be stored in storage device 208.

[0062] Figure 10 This is a flowchart illustrating a method 1000 for applying a computer vision algorithm to generate a virtual selfie stick selfie image according to an example implementation. The operations in method 1000 can be performed by a selfie application 216 using the methods described above. Figure 3 The described components (e.g., modules, engines) are used to perform this operation. Therefore, method 1000 is described by way of example with reference to selfie application 216. However, it should be understood that at least some of the operations of method 1000 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0063] In block 1002, user guidance module 302 instructs user 104 to capture a selfie image (e.g., an original selfie image) in a first pose. In block 1004, user guidance module 302 captures first image data in the first pose. In block 1006, user guidance module 302 instructs user 104 to move device 106 within a limited range. In block 1008, user guidance module 302 captures additional image data from the pose based on the movement of device 106. In block 1010, virtual selfie stick selfie module 316 generates a virtual selfie stick selfie image based on the first image data and the additional image data.

[0064] It should be noted that other implementations may use different orderings, additional or fewer operations, and different nomenclature or terminology to accomplish similar functionality. In some implementations, various operations may be executed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein have been chosen to illustrate some operational principles in a simplified form.

[0065] Figure 11 This is a flowchart illustrating a method for applying a computer vision algorithm to generate a virtual selfie stick selfie image according to an example embodiment. The operations in method 1100 can be used by selfie application 216 as described above. Figure 3 The described components (e.g., modules, engines) are used to perform this. Therefore, method 1100 is described by way of example with reference to selfie application 216. However, it should be understood that at least some of the operations in method 1100 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0066] In block 1102, the region segmentation module 304 identifies the extended region 306, the occluded region 308, and the interpolated region 310 based on the first image data and the supplementary image data. In block 1104, the computer vision module 312 applies a computer vision algorithm to determine the pixels in the extended region 306 and the occluded region 308. In block 1106, the scaling module 314 remaps the interpolated region 310 based on the first image data and the supplementary image data.

[0067] It should be noted that other implementations may use different orderings, additional or fewer operations, and different nomenclature or terminology to accomplish similar functionality. In some implementations, various operations may be executed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein have been chosen to illustrate some operational principles in a simplified form.

[0068] Figure 12 An aspect of the subject matter according to an example implementation is shown.

[0069] Figure 13 An aspect of the subject matter according to an example implementation is shown.

[0070] Figure 14 An area of ​​a virtual selfie stick selfie image is shown according to an example implementation.

[0071] Figure 15 This is a block diagram 1500 illustrating a software architecture 1504, which can be installed on any one or more of the devices described herein. The software architecture 1504 is supported by hardware such as machine 1502, which includes a processor 1520, memory 1526, and I / O components 1538. In this example, the software architecture 1504 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1504 includes layers such as an operating system 1512, libraries 1510, frameworks 1508, and applications 1506. Operationally, application 1506 invokes API calls 1550 and receives messages 1552 in response to API calls 1550 via the software stack.

[0072] Operating system 1512 manages hardware resources and provides public services. Operating system 1512 includes, for example, kernel 1514, services 1516, and drivers 1522. Kernel 1514 serves as an abstraction layer between hardware and other software layers. For example, kernel 1514 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1516 can provide other public services to other software layers. Driver 1522 is responsible for controlling or interfacing with the underlying hardware. For example, driver 1522 may include a display driver, a camera driver, etc. or Low-power drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0073] Library 1510 provides low-level public infrastructure used by application 1506. Library 1510 may include system library 1518 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1510 may include API library 1524, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., an OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing capabilities), and so on. Library 1510 may also include various other libraries 1528 to provide many other APIs to application 1506.

[0074] Framework 1508 provides advanced public infrastructure for use by Application 1506. For example, Framework 1508 provides various graphical user interface (GUI) functionalities, advanced resource management, and advanced location services. Framework 1508 can provide a wide range of other APIs that can be used by Application 1506, some of which may be specific to a particular operating system or platform.

[0075] In an example implementation, application 1506 may include a home application 1536, a contacts application 1530, a browser application 1532, a book reader application 1534, a location application 1542, a media application 1544, a messaging application 1546, a game application 1548, and a wide variety of other applications such as third-party application 1540. Application 1506 is a program that performs the functions defined in the program. Various programming languages ​​can be used to create one or more applications of application 1506 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1540 (e.g., an application developed by an entity other than a vendor of a particular platform using an ANDROID™ or IOS™ software development kit (SDK)) may be an application developed using an ANDROID™ or IOS™ software development kit. Mobile software running on the phone's mobile operating system or another mobile operating system. In this example, a third-party application 1540 can invoke API calls 1550 provided by the operating system 1512 to facilitate the functions described herein.

[0076] Figure 16 This is a schematic representation of machine 1600, within which instructions 1608 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1600 to perform any or more of the methods discussed herein. For example, instructions 1608 can cause machine 1600 to perform any or more of the methods described herein. Instructions 1608 transform the general, unprogrammed machine 1600 into a specific machine 1600 programmed to perform the described and illustrated functions in the described manner. Machine 1600 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1600 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1600 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1608 specifying actions to be taken by machine 1600. Furthermore, although only a single machine 1600 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1608 to perform any or more of the methods discussed herein.

[0077] Machine 1600 may include processor 1602, memory 1604, and I / O components 1642 configured to communicate with each other via bus 1644. In an example embodiment, processor 1602 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, processor 1606 and processor 1610 executing instruction 1608. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. AlthoughFigure 16 Multiple processors 1602 are shown, but machine 1600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0078] Memory 1604 includes main memory 1612, static memory 1614, and memory cell 1616, all of which are accessible by processor 1602 via bus 1644. Main memory 1604, static memory 1614, and memory cell 1616 store instructions 1608 embodying any one or more of the methods or functions described herein. Instructions 1608 may also reside wholly or partially in main memory 1612, in static memory 1614, in machine-readable medium 1618 within memory cell 1616, within at least one processor in processor 1602 (e.g., within the processor's cache memory), or in any suitable combination thereof during execution by machine 1600.

[0079] I / O component 1642 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurement results, etc. The specific I / O component 1642 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be understood that I / O component 1642 may be included in... Figure 16 Many other components are not shown. In various example embodiments, I / O component 1642 may include output component 1628 and input component 1630. Output component 1628 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component 1630 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0080] In other example implementations, I / O component 1642 may include biometric component 1632, motion component 1634, environmental component 1636 or positioning component 1638, and various other components. For example, biometric component 1632 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1634 includes accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. Environmental component 1636 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an hearing sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1638 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0081] A wide variety of technologies can be used to implement communication. I / O component 1642 also includes communication component 1640, which is operable to couple machine 1600 to network 1620 or device 1622 via coupling 1624 and coupling 1626, respectively. For example, communication component 1640 may include a network interface component or another suitable device to interface with network 1620. In other examples, communication component 1640 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, etc. Components (e.g.) (low power consumption) Components and other communication components for providing communication via other modes. Device 1622 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0082] Furthermore, the communication component 1640 can detect identifiers or include components operable to detect identifiers. For example, the communication component 1640 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, data symbols, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be obtained via the communication component 1640, such as location via Internet Protocol (IP) geolocation, etc. The location of signal triangulation, the location of NFC beacon signals that can be detected to indicate a specific location, and so on.

[0083] Various memories (e.g., memory 1604, main memory 1612, static memory 1614, and / or the memory of processor 1602) and / or storage units 1616 may store one or more sets of instructions and data structures (e.g., software) embodying any or more of the methods or functions described herein, or used by any or more of the methods or functions described herein. These instructions (e.g., instruction 1608) cause various operations to implement the disclosed embodiments when executed by processor 1602.

[0084] Instructions 1608 can be sent or received over network 1620 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 1640) and using any of a plurality of known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1608 can be sent or received to device 1622 via a transmission medium using coupling 1626 (e.g., peer-to-peer coupling).

[0085] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and may be used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed in the term “signal medium” discussed below.

[0086] The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be understood to include any intangible medium capable of storing, encoding, or carrying instructions 1416 for execution by machine 1400, and include digital or analog communication signals or other intangible media facilitating communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal.

[0087] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” refer to the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals.

[0088] Although embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader scope of this disclosure. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The accompanying drawings, which form part of this invention, illustrate specific embodiments in which the subject matter can be practiced by way of illustration rather than limitation. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments can be utilized and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, these specific embodiments should not be construed as restrictive, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

[0089] An abstract of this disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted in the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen in the foregoing detailed description, various features are combined in a single embodiment for the purpose of simplifying the disclosure. This method of the disclosure should not be construed as reflecting an intention to require more features than expressly stated in each claim. Rather, as reflected in the appended claims, the subject matter of the invention lies in fewer than all features of a single disclosed embodiment. Therefore, the claims are hereby incorporated into the detailed description, wherein each claim exists independently as a separate embodiment.

[0090] Example

[0091] Example 1 is a method comprising: generating an original selfie image at a device using the device's optical sensor, the optical sensor being aligned with the user's face on the device, the device being held at arm's length from the user's face; displaying instructions on the device's display guiding the user to move the device around the user's face at arm's length in multiple poses within a limited range; accessing image data generated by the optical sensor in multiple poses at the device; and generating a virtual selfie stick selfie image based on the original selfie image and the image data.

[0092] Example 2 includes the method of Example 1, and further includes: identifying an extended region, an occluded region, and an interpolated region of a virtual selfie stick selfie image, wherein the extended region includes a boundary region of the virtual selfie stick selfie image, the boundary region including content continuously extending from the peripheral region of the original selfie image, wherein the occluded region includes an occluded region in the virtual selfie stick selfie image adjacent to the user's face, the occluded region including background content in the original selfie image occluded by the user's face, and wherein the interpolated region includes a remapping region, the remapping region including the user's face and background content displayed in both the original selfie image and the virtual selfie stick selfie image, the remapping region excluding the boundary region and the occluded region.

[0093] Example 3 includes the method of Example 2, and further includes: calculating pixels in the extended region and the occluded region by applying a computer vision algorithm to the original selfie image and image data corresponding to the extended region and the occluded region.

[0094] Example 4 includes the method of Example 3, further comprising: running a computer vision algorithm at a first resolution based on multiple poses of the device; identifying a first foreground depth and a first background depth of the original selfie image based on running the computer vision algorithm at the first resolution; identifying a second foreground depth and a second background depth of the virtual selfie stick selfie image; and running the computer vision algorithm at a second resolution to calculate pixels in an extended region and an occluded region based on the second foreground depth and the second background depth, wherein the second resolution is higher than the first resolution, wherein the second foreground depth is higher than the first foreground depth, and wherein the second background depth is higher than the first background depth.

[0095] Example 5 includes the method of Example 4, and further includes: receiving a request to change a second foreground depth to a third foreground depth; calculating a third background depth based on the third foreground depth; and running a computer vision algorithm at a second resolution to calculate pixels in the extended region and the occluded region based on the third foreground depth and the third background depth.

[0096] Example 6 includes the method of Example 5, and further includes: generating a slider graphical user interface element that allows a user to request a change in the depth of a second foreground.

[0097] Example 7 includes the method of Example 3, wherein the computer vision algorithm includes at least one of the neural radiation field algorithm, multi-view stereo vision algorithm, and 3D reconstruction algorithm.

[0098] Example 8 includes the method of Example 2, and further includes: remapping the contents of the remapping region of the original selfie image to the interpolation region of the virtual selfie stick selfie image.

[0099] Example 9 includes the method of Example 1, and further includes: accessing pose data corresponding to image data in multiple poses, wherein the device includes a visual tracking system that generates pose data based on the corresponding poses of the device, and wherein a virtual selfie stick selfie image is based on the pose data.

[0100] Example 10 includes the method of Example 1, wherein the indication includes a graphical user interface that indicates the orientation of the user's mobile device.

[0101] Example 11 is a computing device including: a processor; and a memory storing instructions that, when executed by the processor, configure the device to perform operations including: generating an original selfie image at the device using an optical sensor aligned with the user's face, the optical sensor being held at arm's length from the user's face; displaying instructions on a display of the device guiding the user to move the device around the user's face at arm's length in multiple poses within a limited range; accessing image data generated by the optical sensor in multiple poses at the device; and generating a virtual selfie stick selfie image based on the original selfie image and the image data.

[0102] Example 12 includes the computing device of Example 11, wherein the instructions further configure the device to: identify an extended region, an occlusion region, and an interpolation region of a virtual selfie stick selfie image, wherein the extended region includes a boundary region of the virtual selfie stick selfie image, the boundary region including content continuously extending from the peripheral region of the original selfie image, wherein the occlusion region includes an occlusion region in the virtual selfie stick selfie image adjacent to the user's face, the occlusion region including background content in the original selfie image occluded by the user's face, and wherein the interpolation region includes a remapping region including the user's face and background content displayed in both the original selfie image and the virtual selfie stick selfie image, the remapping region excluding the boundary region and the occlusion region.

[0103] Example 13 includes the computing device of Example 12, wherein the instructions further configure the device to: calculate pixels in the extended region and the occluded region by applying a computer vision algorithm to the original selfie image and image data corresponding to the extended region and the occluded region.

[0104] Example 14 includes the computing device of Example 13, wherein the instructions further configure the device to: run a computer vision algorithm at a first resolution based on multiple poses of the device; identify a first foreground depth and a first background depth of an original selfie image based on running the computer vision algorithm at the first resolution; identify a second foreground depth and a second background depth of a virtual selfie stick selfie image; and run the computer vision algorithm at a second resolution to calculate pixels in an extended region and an occluded region based on the second foreground depth and the second background depth, wherein the second resolution is higher than the first resolution, wherein the second foreground depth is higher than the first foreground depth, and wherein the second background depth is higher than the first background depth.

[0105] Example 15 includes the computing device of Example 14, wherein the instructions further configure the device to: receive a request to change a second foreground depth to a third foreground depth; calculate a third background depth based on the third foreground depth; and run a computer vision algorithm at a second resolution to calculate pixels in an extended region and an occluded region based on the third foreground depth and the third background depth.

[0106] Example 16 includes the computing device of Example 15, wherein the instructions further configure the device to generate a slider graphical user interface element that allows a user to request a change in the depth of a second foreground.

[0107] Example 17 includes the computing device of Example 13, wherein the computer vision algorithm includes at least one of a neural radiation field algorithm, a multi-view stereo vision algorithm, and a 3D reconstruction algorithm.

[0108] Example 18 includes the computing device of Example 12, wherein the instructions further configure the device to: remap the contents of the remapping region of the original selfie image to the interpolation region of the virtual selfie stick selfie image.

[0109] Example 19 includes the computing device of Example 11, wherein the instructions further configure the device to: access pose data corresponding to image data in multiple poses, wherein the device includes a visual tracking system that generates pose data based on the corresponding poses of the device, and wherein a virtual selfie stick selfie image is based on the pose data.

[0110] Example 20 is a non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations including: generating an original selfie image at a device using an optical sensor of the device, the optical sensor being aligned with the user's face of the device, the device being held at arm's length from the user's face; displaying instructions on a display of the device guiding the user to move the device around the user's face at arm's length in a limited range of multiple poses; accessing image data generated by the optical sensor in multiple poses at the device; and generating a virtual selfie stick selfie image based on the original selfie image and the image data.

Claims

1. A method for generating virtual selfie stick selfie images, comprising: The device generates an original selfie image using its optical sensor, which is aligned with the user's face and held at arm's length from the user's face. A directional graphical user interface is displayed on the device's screen, the directional graphical user interface including arrows pointing in a direction that guides the user to move the device within a limited range around the user's face at arm's length in multiple postures; A slider graphical user interface is displayed on the device's screen, which allows the user to adjust the distance of the virtual selfie stick relative to the user's face; Access image data generated by the optical sensor under the multiple postures at the device; A virtual selfie stick selfie image is generated based on the distance between the virtual selfie stick and the user's face, the original selfie image, and the image data; The virtual selfie stick selfie image is segmented into an extended region, an occluded region, and an interpolated region; The pixels in the extended region and the occluded region are calculated using computer vision algorithms; The scaling engine is used to calculate the pixels in the interpolation region; as well as In response to segmenting the virtual selfie stick selfie image into the extended region, the occluded region, and the interpolated region, the computer vision algorithm is applied only to the extended region and the occluded region, and the scaling engine is applied only to the interpolated region.

2. The method according to claim 1, in, The extended region includes the boundary area of ​​the virtual selfie stick selfie image, and the boundary area includes content that continuously extends from the peripheral area of ​​the original selfie image. The obscured area includes the area adjacent to the user's face in the virtual selfie stick selfie image, and the obscured area includes the background content in the original selfie image that is obscured by the user's face; and The interpolation region includes a remapping region, which includes the user's face and background content displayed in both the original selfie image and the virtual selfie stick selfie image. The remapping region does not include the boundary region and the occlusion region.

3. The method according to claim 2, further comprising: The computer vision algorithm is applied to the original selfie image and the image data corresponding to the extended region and the occluded region to calculate the pixels in the extended region and the occluded region.

4. The method according to claim 3, further comprising: The computer vision algorithm is run at a first resolution based on the multiple poses of the device; The first foreground depth and the first background depth of the original selfie image are identified by running the computer vision algorithm at the first resolution. The second foreground depth and the second background depth are identified in the selfie image taken with the virtual selfie stick; as well as The computer vision algorithm is run at a second resolution to calculate pixels in the extended region and the occluded region based on the second foreground depth and the second background depth. The second resolution is higher than the first resolution. Wherein, the second foreground depth is higher than the first foreground depth, and The second background depth is higher than the first background depth.

5. The method according to claim 4, further comprising: Receive a request to change the second foreground depth to a third foreground depth; Calculate the third background depth based on the third foreground depth; as well as The computer vision algorithm is run at the second resolution to calculate pixels in the extended region and the occluded region based on the third foreground depth and the third background depth.

6. The method according to claim 1, further comprising: The foreground depth of the virtual selfie stick selfie image is adjusted based on the distance of the virtual selfie stick relative to the user's face selected in the slider graphical user interface.

7. The method according to claim 3, wherein, The computer vision algorithm includes at least one of the following: neural radiation field algorithm, multi-view stereo vision algorithm, and 3D reconstruction algorithm.

8. The method according to claim 1, further comprising: Access the pose data corresponding to the image data under the multiple poses. The device includes a visual tracking system, which generates the posture data based on the corresponding posture of the device. The selfie image taken with the virtual selfie stick is based on the posture data.

9. The method according to claim 1, wherein, The directional graphical user interface includes text instructions that direct the user to move the device in the direction of the arrow.

10. A computing device, comprising: monitor; Optical sensors; processor; as well as A memory for storing instructions that, when executed by the processor, configure the computing device to perform operations including: The optical sensor is used to generate an original selfie image, the optical sensor being aligned with the user's face on the computing device, which is held at arm's length from the user's face; A directional graphical user interface is displayed on the display, the directional graphical user interface including arrows pointing in a direction that guides the user to move the computing device around the user's face at arm's length in multiple postures within a limited range; A slider graphical user interface is displayed on the display, which allows the user to adjust the distance of the virtual selfie stick relative to the user's face; Access image data generated by the optical sensor under the multiple postures at the computing device; A virtual selfie stick selfie image is generated based on the distance between the virtual selfie stick and the user's face, the original selfie image, and the image data; The virtual selfie stick selfie image is segmented into an extended region, an occluded region, and an interpolated region; The pixels in the extended region and the occluded region are calculated using computer vision algorithms; The scaling engine is used to calculate the pixels in the interpolation region; and In response to segmenting the virtual selfie stick selfie image into the extended region, the occluded region, and the interpolated region, the computer vision algorithm is applied only to the extended region and the occluded region, and the scaling engine is applied only to the interpolated region.

11. The computing device according to claim 10, in, The extended region includes the boundary area of ​​the virtual selfie stick selfie image, and the boundary area includes content that continuously extends from the peripheral area of ​​the original selfie image. The obscured area includes the area adjacent to the user's face in the selfie image taken with the virtual selfie stick. The obscured area includes the background content in the original selfie image that is obscured by the user's face. The interpolation region includes a remapping region, which includes the user's face and background content displayed in both the original selfie image and the virtual selfie stick selfie image. The remapping region does not include the boundary region and the occlusion region.

12. The computing device according to claim 11, wherein, The instructions also configure the computing device to: The computer vision algorithm is applied to the original selfie image and the image data corresponding to the extended region and the occluded region to calculate the pixels in the extended region and the occluded region.

13. The computing device according to claim 12, wherein, The instructions also configure the computing device to: The computer vision algorithm is run at a first resolution based on the multiple poses of the computing device; The first foreground depth and the first background depth of the original selfie image are identified by running the computer vision algorithm at the first resolution. The second foreground depth and the second background depth are identified in the selfie image taken with the virtual selfie stick; as well as The computer vision algorithm is run at a second resolution to calculate pixels in the extended region and the occluded region based on the second foreground depth and the second background depth. The second resolution is higher than the first resolution. Wherein, the second foreground depth is higher than the first foreground depth, and The second background depth is higher than the first background depth.

14. The computing device according to claim 13, wherein, The instructions also configure the computing device to: Receive a request to change the second foreground depth to a third foreground depth; Calculate the third background depth based on the third foreground depth; and The computer vision algorithm is run at the second resolution to calculate pixels in the extended region and the occluded region based on the third foreground depth and the third background depth.

15. The computing device according to claim 10, wherein, The instructions also configure the computing device to: The foreground depth of the virtual selfie stick selfie image is adjusted based on the distance of the virtual selfie stick relative to the user's face selected in the slider graphical user interface.

16. The computing device according to claim 12, wherein, The computer vision algorithm includes at least one of the following: neural radiation field algorithm, multi-view stereo vision algorithm, and 3D reconstruction algorithm.

17. The computing device according to claim 10, wherein, The instructions also configure the computing device to: Access the pose data corresponding to the image data under the multiple poses. The computing device includes a visual tracking system that generates the posture data based on the corresponding posture of the computing device. The selfie image taken with the virtual selfie stick is based on the posture data.

18. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions that, when executed by a device, cause the device to perform an operation, the operation comprising: An original selfie image is generated at the device using the device's optical sensor, which is aligned with the user's face, and the device is held at arm's length from the user's face. A directional graphical user interface is displayed on the device's screen, the directional graphical user interface including arrows pointing in a direction that guides the user to move the device within a limited range around the user's face at arm's length in multiple postures; A slider graphical user interface is displayed on the device's screen, which allows the user to adjust the distance of the virtual selfie stick relative to the user's face; Access image data generated by the optical sensor under the multiple postures at the device; A virtual selfie stick selfie image is generated based on the distance between the virtual selfie stick and the user's face, the original selfie image, and the image data; The virtual selfie stick selfie image is segmented into an extended region, an occluded region, and an interpolated region; The pixels in the extended region and the occluded region are calculated using computer vision algorithms; The scaling engine is used to calculate the pixels in the interpolation region; as well as In response to segmenting the virtual selfie stick selfie image into the extended region, the occluded region, and the interpolated region, the computer vision algorithm is applied only to the extended region and the occluded region, and the scaling engine is applied only to the interpolated region.

Images