Method and system for generating three-dimensional models

Abstract

In various example embodiments, systems and methods for generating and manipulating three-dimensional (3D) models are presented. The systems and methods cause an interface frame to be presented that surrounds a field of view of an image acquisition device. The systems and methods detect an object of interest within the interface frame, generate motion instructions for the object of interest, and detect a first change in position and a second change in position of the object of interest. The systems and methods generate a 3D model of the object of interest based on the first change in position and the second change in position.

Description

[0001] This application is a divisional application of Chinese patent application (201680030189.6). The application date of the original application is (March 25, 2016), the priority date is (March 27, 2015), and the name of the invention is “Automatic 3D model generation”.

[0002] Priority claim

[0003] This application claims the benefit of priority to U.S. patent application serial number 15 / 080,367, filed on March 24, 2016, which claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 139,009, filed on March 27, 2015, each of which is hereby claimed and is incorporated herein by reference in its entirety. Technical Field

[0004] Embodiments of the present disclosure relate generally to three-dimensional model generation, and more particularly, but not by way of limitation, to generating a three-dimensional model based on directional motion of an object to be modeled with respect to an image acquisition device. Background Art

[0005] Typically, systems and methods for generating three-dimensional models are third-party systems located remotely from the intended recipient of the model. The generation of three-dimensional models typically utilizes multiple sets of static images acquired prior to model generation. Systems and methods for generating three-dimensional images often perform computationally intensive operations to generate and animate the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The figures in the drawings depict only exemplary embodiments of the present disclosure and are not to be considered limiting of its scope.

[0007] Figure 1 is a block diagram illustrating a networking system according to some example embodiments.

[0008] Figure 2 is a block diagram of an example model generation system in accordance with various embodiments.

[0009] Figure 3 is a flow chart illustrating an example method of generating a three-dimensional model within a graphical user interface in accordance with various embodiments.

[0010] Figure 4A is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0011] Figure 4B is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0012] Figure 5is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0013] Figure 6 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0014] Figure 7 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0015] Figure 8 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0016] Fig. 9 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0017] Fig.10 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0018] Fig.11 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0019] Fig.12 is an example interface diagram showing a three-dimensional model according to various embodiments.

[0020] Fig.13 is an example interface diagram showing a three-dimensional model according to various embodiments.

[0021] Fig.14 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0022] Fig.15 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0023] Fig.16 is a flow chart illustrating an example method for generating a three-dimensional model within a graphical user interface in accordance with various embodiments.

[0024] Fig.17 is a flow chart illustrating an example method of generating a three-dimensional model within a graphical user interface according to various embodiments.

[0025] Fig.18 is a flow chart illustrating an example method of generating a three-dimensional model within a graphical user interface according to various embodiments.

[0026] Fig.19is a flow chart illustrating an example method of generating a three-dimensional model within a graphical user interface in accordance with various embodiments.

[0027] Fig. 20 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0028] Fig.21 is an example interface diagram illustrating a user interface screen of a model generation system according to various embodiments.

[0029] Fig. 22 is a block diagram illustrating an example of a software architecture that may be installed on a machine according to some example embodiments.

[0030] Fig.23 A diagram is shown of a machine in the form of a computer system within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein, according to an example embodiment.

[0031] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the terms used. DETAILED DESCRIPTION

[0032] The following description includes systems, methods, techniques, instruction sequences, and computer program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject matter of the present invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the present invention can be practiced without these specific details. Typically, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0033] The systems and methods described herein enable an untrained individual to use a computing device (e.g., a smartphone or tablet computer) with one or more image acquisition devices (e.g., a camera) attached to create a fitted canonical three-dimensional (3D) mesh model of an object without the help of another person. In some embodiments, the object can be a person's face, hand, arm, leg, or other body part; an object of a recognized type (e.g., a car, cup, sofa, table, or other determined object). The systems and methods allow the 3D facial model to be imported, visualized, and / or manipulated in other software programs and computing systems that contain or display 3D graphics, such as video games, virtual reality environments, and business platforms. The 3D facial model can be used for personalization of products, services, experiences, games, graphic content, recognition, and data analysis.

[0034] In some embodiments, the systems and methods described herein may form all or part of an end-user application (EUA). An EUA may be a software program including user interface elements that enables an untrained end user to launch the EUA on a computing device with one or more cameras attached, such as (but not limited to) a computer, laptop, IPAD, smart phone, tablet computer, or any other mobile computing device available to a user. In some embodiments, the EUA may be a stand-alone program. The EUA may also be part of or embedded within another software program.

[0035] EUA allows untrained users to create a fitted, canonical 3D mesh model of their own face without the help of others. EUA may include one or more components described below for generating depth maps, generating fused depth maps for 3D models, object mesh fitting, model evaluation, and data transmission. The EUA components that provide 3D data acquisition, model creation, and completed model data transmission are described in detail below. These components manipulate the raw data from the camera and create a surface model of the object (e.g., face) within the camera's field of view, fit the surface model to a canonical mesh, store the results locally on the device, and optionally send it to a data storage platform.

[0036] The systems and methods disclosed herein can enable the EUA to determine whether conditions are suitable for modeling before acquiring a set of images. For example, the EUA can detect whether the environment is too dark based on the camera exposure and / or International Standards Organization (ISO) sensitivity increasing or exceeding a predetermined threshold. Additional checks can also be performed, such as ensuring that pixel intensities fall within an acceptable range to avoid overexposed areas of the image. If the system or EUA determines that the current acquisition conditions are not suitable, a message is given to the user instructing the user to improve the conditions.

[0037] In some embodiments, in preparation for acquisition, the EUA may generate a message instructing the user to align the user's face relative to the camera. The message may be provided in various ways. In some cases, the message is provided visually. The EUA may cause a camera image to be presented on the screen of the mobile computing device along with alignment guidelines. Based on the guidelines, the EUA may generate a message after the face is positioned. Facial detection may be used to verify that the user has correctly positioned the user's face according to the guidelines. In some embodiments, the message may be provided audibly. The EUA may generate and cause presentation of audio (e.g., voice played through an audio device or speaker of the mobile computing device) that includes instructions on how to position the user's face relative to the camera. Facial detection may be used to determine the current position of the user's face, which may then be used to issue instructions for correct positioning (e.g., "move closer," "move slightly to the left," or "move down").

[0038] After the EUA detects facial alignment, the systems and methods of the present disclosure automatically initiate the acquisition and modeling process. In some cases, the acquisition and modeling process can be manually initiated. In either case, once the system and method of the present disclosure is signaled, an acquisition component or a group of components are initiated to acquire a set of images and track the 3D pose of the camera relative to the user's face in real time. Based on the tracked pose and the group of images, the system and method of the present disclosure generates a message (e.g., visual, audio, or tactile feedback) to guide the movement of the camera or object (e.g., the user's face) to a position that enables the camera to observe all sides of the object. For example, in the case where the object is the user's face, the message may indicate the movement of the face and / or image acquisition device to acquire the face, forehead, chin, neck, and other aspects of the face. In some embodiments, the message may indicate the movement of the image acquisition device that acquires the image of the object. For example, in the case where the object is a car, a table, or other object, the message may indicate the movement of the image acquisition device relative to the object. The EUA can provide feedback to ensure that the user has acquired all relevant areas of the object (e.g., the user's face) so that a 3D model of the object can be built. The EUA also generates messages or alerts indicating potential problems with the scanned data acquisition. For example, the EUA may generate a message or alert indicating motion blur caused by moving the camera or rotating the head too quickly.

[0039] In some embodiments, the camera of the mobile computing device that captures a set of images for 3D modeling is a rear-facing camera. The rear-facing camera can be located on the side of the mobile computing device opposite the display device. Visual feedback (e.g., alerts and messages) displayed by the EUA can be displayed in a mirrored form so that if the user stands in front of the mirror, the user will be able to read and follow the instructions on the screen.

[0040] refer to Figure 1 , shows an example embodiment of a high-level client-server based network architecture 100. A network system 102 in an example form of a network-based model generation system provides server-side functionality to one or more client devices 110 via a network 104 (e.g., the Internet or a wide area network (WAN)). Figure 1 For example, a web client 112 (e.g., such as provided by Microsoft Corporation of Redmond, Washington) is shown. The company developed such as INTERNET The client device 110 includes a browser, an application 114, and a programming client 116 executed on the client device 110.

[0041] The client device 110 may include, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a personal digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronic product, a game console, a set-top box, or any other communication device that a user may utilize to access the network system 102. In some embodiments, the client device 110 may include a display component (not shown) to display information (e.g., in the form of a user interface). In other embodiments, the client device 110 may include one or more of a touch screen, an accelerometer, a gyroscope, a camera, a microphone, a global positioning system (GPS) device, and the like.

[0042] The client device 110 may be a device of a user for collecting images and sending images and modeling data over a network. One or more users 106 may be individuals, machines, or other components that interact with the client device 110. In an embodiment, the user 106 is not part of the network architecture 100, but may interact with the network architecture 100 via the client device 110 or another component. For example, one or more parts of the network 104 may be a self-organizing network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a WAN, a wireless WAN (WWAN), a metropolitan area network (MAN), a part of the Internet, a part of the public switched telephone network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, another type of network, or a combination of two or more such networks. Each of the client devices 110 may include one or more applications (also referred to as "application software"), such as, but not limited to, a web browser, a messaging application, an email application, etc.

[0043] One or more users 106 may be a person, machine, or other component that interacts with client device 110. In an example embodiment, user 106 is not part of network architecture 100, but may interact with network architecture 100 via client device 110 or other component. For example, a user provides input (e.g., touch screen input or alphanumeric input) to client device 110, and the input is transmitted to network system 102 via network 104. In this case, network system 102 transmits information to client device 110 via network 104 for presentation to the user in response to receiving the input from the user. In this way, a user may interact with network system 102 using client device 110.

[0044] Application program interface (API) server 120 and web server 122 are respectively coupled to one or more application servers 140 and provide programming and web interfaces, respectively. Application server 140 can host one or more publishing systems 142 and model generation system 150, each of which can include one or more components or applications, and each of which can be implemented as hardware, software, firmware, or any combination thereof. Application server 140 is further shown as being coupled to one or more database servers 124 that facilitate access to one or more information repositories or databases 126. In an example embodiment, database 126 is a storage device that stores information (e.g., publications or lists) to be published to publishing system 142. Database 126 can also store object data, historical data, and 3D model data according to an example embodiment.

[0045] In some cases, the database and database server may act as a data storage platform for 3D models generated within the client device 110. The database server 124 may store information about each end user (e.g., user 106) that uploads a model to the database 126. The information collected and retained about each user may include, but is not limited to, name, email address, password, and any additional location or identification data that may be requested by the database server 124. Each model stored in the database server 124 or database 126 may be associated with the end user 106 who created it, or with any third party to whom the end user 106 creator has granted access rights. Metadata is also collected and maintained for each facial model for business and system analysis purposes, such as, but not limited to, the number and originator of the downloads.

[0046] In some embodiments, the API server 120 includes a third-party data access interface (TPDAI). TPDAI is an API protocol that enables authorized third-party programs and systems to access models in the database server 124 or database 126 on behalf of the user 106. TPDAI enables third-party programs and systems to authenticate the access credentials of the user 106 to download or otherwise access the model. The TPDAI request can also include a valid access token, which enables the database server 124 to ensure that the request is initiated from the authorized user 106 of the database server 124. The valid access token also allows the end-user application to track the usage of each third-party system, as well as allowing other tracking and management operations. In response to a request from a third-party program or system, the model data sent to the third-party program or system may include all elements of the model that are imported, visualized, or manipulated in a software program or physical component and a computing system that contains or displays 3D graphics.

[0047] Additionally, third-party applications 132 executing on third-party servers 130 are shown as having programmatic access to network system 102 via a programming interface provided by API server 120. For example, using information retrieved from network system 102, third-party applications 132 support one or more features or functions on a website hosted by a third party.

[0048] The publishing system 142 can provide a plurality of publishing, archiving and data storage functions and services to the users 106 accessing the network system 102. For example, the publishing system 142 can collect, publish and store object data, 3D model data, 2.5 dimensional model data and other data related to the generated model and the application of the model to one or more platforms or programs. The publishing system 142 can publish the object data, image data and model data to an internal database or a publicly available database to enable the generation of a 3D model based on the object data, image data and motion data. In some embodiments, the publishing system 142 accesses one or more third-party servers or databases (e.g., third-party server 130) to retrieve, modify and provide object data within the database 126.

[0049] The model generation system 150 may provide functionality operable to perform various model generation and manipulation functions and functions for generating objects, instructions, images, and graphical representations of model editing effects and model modifications. For example, the model generation system 150 generates a 3D model of an object, body part, or scene based on image data provided by an image acquisition device associated with or coupled to the client device 110. The 3D model may be generated based on the movement of an object or face, or the movement of a camera acquisition device. In some example embodiments, the model generation system 150 communicates with the publishing system 142 to access or provide 3D modeling data for use with a third party program or system.

[0050] In addition, although Figure 1 The client-server based network architecture 100 shown in utilises a client-server architecture, but the subject matter of the present invention is certainly not limited to such an architecture and may equally well find application in, for example, distributed or peer-to-peer architecture systems. The various publishing systems 142 and model generation systems 150 may also be implemented as standalone software programs that do not necessarily have networking capabilities.

[0051] Web clients 112 may access various publishing and model generation systems 142 and 150 via web interfaces supported by web server 122. Similarly, programmatic clients 116 access various services and functions provided by publishing and model generation systems 142 and 150 via programmatic interfaces provided by API server 120.

[0052] Additionally, third-party applications 132 executing on third-party servers 130 are shown as having programmatic access to network system 102 via a programmatic interface provided by API server 114. For example, third-party applications 132 may support one or more features or functions on a website hosted by a third party using information retrieved from network system 102. For example, a third-party website may, for example, provide one or more promotional, marketing, data repository, company interaction, or object tracking functions supported by an associated application of network system 102.

[0053] Figure 2 is a block diagram illustrating components of a model generation system 150 according to some example embodiments. The model generation system 150 is shown to include a presentation component 210, a detection component 220, a notification component 230, a location component 240, a modeling component 250, and a model viewing component 260, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of the components described herein may be implemented using hardware (e.g., one or more processors of a machine) or a combination of hardware and software. For example, any component described herein may configure a processor (e.g., in one or more processors of a machine) to perform an operation for which the component is designed. In addition, any two or more of these components may be combined into a single component, and the functionality described herein for a single component may be subdivided between multiple components.

[0054] The presentation component 210 causes presentation of user interfaces and user interface elements generated by the detection component 220, notification component 230, location component 240, and model viewing component 260. The presentation component 210 also causes presentation of 3D models generated by the modeling component 250. In some embodiments, the presentation component 210 performs presentation by visualizing the user interfaces and user interface elements at the client device 110 via a display device. The presentation component 210 can operate in conjunction with one or more other components described herein.

[0055] Detection component 220 detects an object of interest within a field of view of an image acquisition device associated with client device 110. In some cases, detection component 220 detects the object of interest and the location of the object of interest within a graphical user interface element presented on a display device.

[0056] Notification component 230 generates user interface elements, selectable user interface elements, and instruction sets based on interactions, selections, motions, and operations received by client device 110. In some embodiments, notification component 230 determines the appropriate user interface element, instruction set, or combination thereof based on the operation and the completion status of the operation performed by one or more other components described herein.

[0057] Once the object of interest is properly positioned within the field of view of the image acquisition device, position component 240 detects changes in the position of the object of interest. Position component 240 can cooperate with notification component 230 to direct movement of the object of interest with respect to client device 110 or movement of client device 110 with respect to the object of interest.

[0058] Modeling component 250 generates a 3D model based on a set of image frames or video received from an image acquisition device and the movement of an object of interest or the image acquisition device corresponding to the instructions generated by notification component 230. In some cases, modeling component 250 generates a 3D model together with an image that incorporates the 3D model.

[0059] Model viewing component 260 can cooperate with presentation component 210 to generate user interface elements representing possible modifications to the 3D model. In some cases, model viewing component 260 receives modification input from a user interface device and causes modeling component 250 to generate a modified model based on the input.

[0060] Figure 3 is a flow chart of operations of model generation system 150 in performing method 300 of generating a 3D model within a graphical user interface, according to some example embodiments. The operations in method 300 may be performed by model generation system 150 using components described herein.

[0061] In some cases, prior to initiation of method 300, model generation system 150 may receive a launch instruction. Model generation system 150 may launch an end-user application on one or more image acquisition devices associated with a mobile computing device (e.g., client device 110), a mobile device, a tablet computer, an Internet computer, a mobile phone, or other device having the capability to capture video. Model generation system 150 receives a launch selection to initiate a new capture session. Initiation of a capture session may cause one or more image acquisition devices to begin video capture or enable video capture to begin by accessing one or more modules or components associated with the image acquisition devices.

[0062] In operation 310, presentation component 210 causes presentation of a graphical user interface frame that surrounds a graphical representation of a field of view of an image acquisition device of a mobile computing device. The graphical user interface frame may be generated as an overlay that is presented on a representation of content of the field of view of an image acquisition device of a mobile computing device. For example, Figure 4A As shown in , the graphical user interface frame 410 may include one or more selectable user interface elements 420, which are configured to enable selection between two or more image acquisition devices of the mobile computing device. The graphical user interface frame may also include a notification interface element 430 that operates as a message, alert, or instruction for EUA operation.

[0063] like Figure 4A As shown in , the graphical user interface frame 410 may be configured with a visual framing element 440 that is sized and shaped based on the field of view of the image acquisition device and the intended object. In FIG. 4 , the visual framing element 440 is sized and shaped to surround a face. The visual framing element 440 is represented as an ellipse located within the field of view of the image acquisition device. The ellipse is sized so that a face located within the ellipse and occupying a predetermined portion of the ellipse is appropriately positioned for scanning and generation of a 3D model. Although the visual framing element is shown as an ellipse, it should be understood that the visual framing element 440 can be other suitable shapes (e.g., square, rectangular, circular, or polygonal) configured based on the dimensions (e.g., size or shape) of the object to be modeled.

[0064] One or more selectable user interface elements 420 may be located on a portion of the display outside of the visible framing element 440. Figure 4A As shown in , one or more selectable user interface elements 420 can be located at the bottom of the display below the visible framing element 440 and the appearance of the field of view of the image capture device. The one or more selectable user interface elements 420 can be a single user interface element, the selection of which causes the mobile computing device to switch between the first image capture device and the second image capture device.

[0065] In some cases, switching between the first image acquisition device and the second image acquisition device may cause the presentation component 210 to visualize changes in the visible framing element 440. For example, Figure 4A and 4B As shown in , selectable user interface element 420 may be selectable to switch between a first image acquisition device (e.g., a rear camera) for acquisition and modeling of scenes or objects in a wide field of view and a second image acquisition device (e.g., a front camera) for acquisition and modeling of faces. When selectable user interface element 420 switches to the first image acquisition device for modeling faces, as shown in Figure 4A As shown in FIG. 4 , visible framing element 440 (e.g., framing element 440A) may be an ellipse or other suitable shape for framing a face depicted within the field of view. Figure 4B As shown in , when selectable user interface element 420 is switched to a second image acquisition device for modeling a scene, visible framing element 440 (e.g., framing element 440B) may be rectangular and enclose a different portion of the field of view than visible framing element 440A.

[0066] The one or more selectable user interface elements 420 may include a set of user interface elements. The set of user interface elements may be selected to configure one or more aspects of image acquisition of the EUA. For example, the user interface element may select an image acquisition device, an image acquisition feature (e.g., flash memory), an image import (e.g., retrieve a previously acquired image), or cancel. Selection of the cancel element may stop the image acquisition mode and cause the presentation component 210 to display a menu or other graphical user interface screen.

[0067] The notification interface element 430 may include instructions for interacting with one or more of the image acquisition device, the visual framing element 440, or the EUA. Figure 4A As shown, notification interface element 430 instructs the user to “Put face into area below” so that the face is within visible framing element 440 .

[0068] In operation 320, the detection component 220 detects objects of interest within the graphical user interface frame and the field of view of the image acquisition device. For example, in the case where the EUA is in a mode for modeling faces, the detection component 220 can detect faces within the graphical user interface frame and the field of view. In some embodiments, the detection component 220 is configured with a predetermined detection algorithm associated with a given acquisition mode switched by the selectable user interface element 420. For example, in the case where the selectable user interface element 420 is switched to a second position facing the face, the detection component 220 can be configured to run a face detection algorithm such as a Viola-Jones detection algorithm. In some cases, the detection component 220 dynamically selects a detection algorithm at the start of operation 320. In order to dynamically select a detection algorithm, the detection component can be configured with a set of object detection algorithms. For example, the object detection algorithm can include an edge detection algorithm, a point detection algorithm, a shape detection algorithm, an object recognition algorithm, a face detection algorithm, or any other suitable algorithm or method set for identifying objects of interest within the field of view. The detection component 220 can run one or more object detection algorithms until an object type is identified for an object within the field of view. For example, the detection component 220 can step through the object detection algorithms in a predetermined or random order until an object detection algorithm in the set of object detection algorithms identifies the type of object (eg, classifies).

[0069] The detection component 220 may use one or more detection methods to detect objects of interest. For example, the detection component 220 may use one or more edge detection operations, point detection operations, or face detection operations. The object detection operation may be selected based on the position of the selectable user interface element 420. In the case where the selectable user interface element is positioned for collecting and modeling scenes, the detection component 220 may select a point detection operation or an edge detection operation for object recognition. In the case where the selectable user interface element is positioned for collecting and modeling faces, the detection component 220 may select and utilize a face detection operation to detect faces within the field of view.

[0070] In operation 330, the notification component 230 generates motion instructions that direct the movement of an object within the field of view of the image acquisition device. In the case where the object to be modeled is a scene, the notification component 230 may generate motion instructions that direct the movement of the image acquisition device or the mobile computing device. In the case where the object to be modeled is a face, the notification component 230 generates motion instructions that direct the movement of the face.

[0071] like Figure 4A , Figure 5 and Figure 6 As shown, in the event that the detection component 220 detects a face, the notification component 230 can initially generate a motion element including motion instructions that direct the motion to position the face within the visible framing element. Figure 5 As shown in FIG. 1 , the notification component 230 may initially generate a first motion element including an instruction to “move your face closer.” Based on the detection component 220 detecting the motion of a face within the field of view, but not properly positioning the face for scanning and modeling, the notification component 230 may generate a first motion element including an instruction to “move your face closer.” Figure 5 Alternate to Figure 4A , change the first motion element and motion instruction to the second motion element. Figure 4A As shown in , the second motion element includes instructions to "put the face into the following area".

[0072] Once the detection component 220 detects a face within the visible framing element 440, the notification component 230 can generate a third motion element. The third motion element can be a user element configured as a focus element. The focus element is configured and positioned to guide the gaze of the eye. The focus element can be located near the image acquisition device of the mobile computing device. For example, as shown in the figure, the focus element is located in a graphical user interface on the display device and below the image acquisition device of the mobile computing device. In some cases, the third motion element includes Figure 6 In addition to the third motion element, the notification component 230 can generate a gesture element. Figure 6In the embodiment of the present invention, the gesture element is separated from the third motion element and can guide a different motion or gesture than the instruction of the third motion element. As shown in the figure, the gesture element includes an instruction of "observation point" and contains a direction element that visually guides the indicated gesture. The gesture element instruction can instruct the user to attach the gaze to the focus element to align the eyes to a position close to the image acquisition device. When the gaze is directed to the focus element and the focus element moves with the face, the model generation system 150 can capture the image and generate a 3D model centered on the pupil and iris of the eye.

[0073] In operation 340, the position component 240 detects a first change in the position of an object (e.g., a face) within the field of view of the image acquisition device. When the first change in the position of the object is detected, the position component 240 causes the image acquisition device to acquire image data for the first change in position. In some cases, the position component 240 may acquire sensor data where sensor data is detected in addition to the image data and the sensor data indicates the first change in position.

[0074] In some embodiments, the input data may include raw image frames obtained from an image acquisition device and sensor data from sensors associated with the mobile computing device, such as accelerometer data and gyroscope data. In order to provide appropriate input data for the subsequent 3D reconstruction (e.g., modeling) stage, the position component 240 and the detection component 220 specify a set of target 3D poses to be observed by the camera. During acquisition, the detection component 220 and the position component 240 monitor the current 3D pose of the camera and track the 3D pose of the image acquisition device relative to the object.

[0075] In some embodiments, the first change in facial position is an expected change in position. The expected change in position may include a first initial position and a first final position. The first change in position may also include a first set of intermediate positions. Each of the first initial position, the first set of intermediate positions, and the first final position may be associated with a first side of the face, wherein the model generation system 150 is configured to model the face captured by the image capture device.

[0076] Upon detecting a face within the visible framing element 440, the detection component 220 may initiate image acquisition and modeling operations. Figure 7 As shown in , initiation of an image acquisition operation may cause notification component 230 to generate a position element 710 that includes instructions 720 for a change in position of a face (e.g., an object within the field of view). Figure 7, the position element 710 may be a first position element 730 indicating a first change in position. The first position element 730 may include a first starting position 740 and a first ending position 750. For example, the position element may include an instruction 720 of “slowly turn your head to the right.”

[0077] like Figure 8 As shown in , during the first change in position detected in operation 340, the position component 240 continuously monitors (e.g., tracks) the movement of the object (e.g., face) within the field of view. The position component 240 can pass a portion of the data representing the movement to the notification component 230. The notification component 230 can continuously modify the graphical user interface in real time based on the movement data. Figure 8 , the notification component 230 continuously adjusts or modifies the position element 710. The notification component 230 can modify the position element 710 by moving the first position element 730 relative to the current position of the object within the field of view and the travel distance detected for the object. The first position element 730 can move a distance between a first starting position 740 and a first ending position 750. The position of the first position element 730 along the distance can be proportional to the amount of motion detected in the object between the starting position and the expected ending position.

[0078] In operation 350, position component 240 detects a second change in position of an object (e.g., a face) within the field of view of the image capture device. The second change in the position of the object can be in a trajectory similar to or the same trajectory as the first change in the position of the object described in operation 340. Upon detecting the second change in position, position component 240 causes the image capture device to capture image data indicating the second change in position. In some cases, as described above, in addition to the image data, position component 240 captures sensor data indicating the second change in position.

[0079] In some cases, the second change in facial position is an expected change in position. The expected change in position may include a second initial position and a second final position. In some embodiments, the second change in position includes a second set of intermediate positions. Each of the second initial position, the second set of intermediate positions, and the second final position may be associated with a second side of the face, wherein the model generation system 150 is configured to model the face captured by the image capture device.

[0080] like Figure 9-11 As shown in , upon completion of the first change in position, the notification component 230 generates a position element 910 including instructions 920 for a second change in position of the object (eg, face). Fig. 9, position element 910 is a second position element 930 that includes a second starting position 940 and a second ending position 950. As shown, position element 910 may include an instruction 920 to "slowly turn your head to the left," indicating a desired direction for the second change in position. In some cases, second starting position 940 may be positioned at or near first ending position 750, such that the first change in position and the second change in position are linked to form a fluid motion. Figure 9-11 As shown in , notification component 230 can monitor a second change in position detected by position component 240 and continually modify position element 910 to proportionally represent the position of an object (e.g., a face) between an initial position of the second change in position and an expected end of the second change in position.

[0081] Although described as collecting images and sensor data in response to a first change in position and a second change in position, it should be understood that the position component 240 can collect an image stream (e.g., a set of image frames) and sensor data representing the first change in position, the second change in position, and the motion between the first change in position and the second change in position. In some cases, as will be explained in more detail below, the first change in position, the second change in position, and one or more intermediate changes in position (e.g., along the first change in position and the second change in position) can be identified as key frames within the image stream for generating a 3D model of the object.

[0082] In some embodiments, the position component 240 detects a first change in position and a second change in position in a single sweep motion. For example, in the case where the model generation system 150 is capturing and modeling a face, the face may be initially positioned so that the outline of the face is within the field of view. In some cases, the position component 240 detects a single change in position as a rotation across two or more planes. For example, a single change in position may be a rotation of the face that exposes the right side profile, the left side profile, the uppermost portion of the forehead, and the lower side of the chin or jaw. The position component 240 may perform the above operation 340 based on the change in position. As the face passes through the center line, the position component 240 may perform operation 350. In some embodiments, the position component 240 detects more than two changes in position. For example, the position component 240 may detect a first change in position of a face rotated to the right from a center position, a second change in position of a face rotated to the left from a center position or from a right side profile position, a third change in position of a face rotated downward from a center position, and a fourth change in position rotated upward from a center position or a downward position.

[0083] Although described with respect to capturing data of an object of interest in the form of a face, it will be appreciated that the present methods and systems are capable of capturing and dimensional modeling scenes and objects other than faces or human body parts.

[0084] In operation 360, the modeling component 250 generates a 3D model of the object (e.g., a face). The 3D model can be generated based on image data collected from the first change in position and the second change in position. As will be explained in more detail below, the 3D model can be generated based on various keyframes and trajectories for tracking points detected on or around the object.

[0085] After the second change in position is complete, the notification component 230 generates a processing element. The processing element can be presented with an obscured representation of the end position of the second change in position. The processing element can be visualized as a representation of the modeling component 250 that generates a 3D model of the object. The presentation component 210 can cause the presentation of the processing element for a period of time corresponding to the generation of the 3D model (e.g., by presenting an animated drawing point of light traveling around a circle, an hourglass, or an element sweeping across a portion of the display device).

[0086] Once the modeling component 250 generates a 3D model of the object, such as Fig.12 and Fig.13 As shown in , the presentation component 210 can present the 3D model in the isolated interface 1210. In some embodiments, the isolated interface 1210 includes the 3D model 1220 in an interactive representation without the background captured in the image data. The interactive representation can be linked to one or more sensors of the mobile computing device to enable interaction with the 3D model 1220. Fig.12 and Fig.13 As shown in , the motion of the mobile computing device is detected within a motion sensor (e.g., a gyroscope or accelerometer) and causes representative manipulations of the 3D model 1220. For example, moving the left side of the mobile computing device can be translated by the motion sensor and rendering component 210 into a rotation of the 3D model 1220 that exposes the left side of the 3D model 1220 relative to the viewer. Moving the right side, top side, or bottom side of the mobile computing device can be translated by the motion sensor and rendering component 210 into a rotation of the 3D model 1220 that exposes the right side, top side, or bottom side of the 3D model 1220, respectively, relative to the viewer.

[0087] In some embodiments, in response to generating the 3D model, Fig.12 and Fig.13The presentation of the 3D model in is initiated by the model viewing component 260. The model viewing component 260 can enable viewing of the 3D model generated by the facial acquisition session. In some cases, the model viewing component 260 generates one or more user interface elements for manipulating, modifying, and interacting with the 3D model. The modeling component 250 can generate multiple models in a scanning and acquisition session. Each acquired and created model can be stored in a digital storage device (e.g., a non-transitory processor-readable storage medium associated with a mobile computing device) that resides locally on the device. The user can choose to discard a given 3D model or send it from the local digital storage to a data storage platform located outside the scanning acquisition device for later retrieval and / or deployment to other applications, mobile devices, or computing systems.

[0088] like Fig.14 and Fig.15 As shown in FIG. 1 , in conjunction with the presentation component 210, the model viewing component 260 can cause a viewing interface 1410 to be presented. The viewing interface 1410 can include one or more viewing elements 1420 (e.g., first, second, and third viewing elements 1430, 1440, and 1450). The one or more viewing elements 1420 enable the 3D model to be modified within a set of predefined modification options. Fig.14 and Fig.15 As shown in , the first viewing element 1430 enables selection of one or more color filters, color effects, color temperature filters, and other filters, effects, and color value adjustments configured to modify one or more of the color, sharpness, hue, saturation, or hue of the 3D model. In some cases, selection of the first viewing element 1430 causes the model viewing component 260 and the rendering component 210 to generate and cause the rendering of the effect element. The effect element can include a thumbnail or preview of the 3D model that appears with the effect corresponding to the specified interface element.

[0089] like Fig.15 As shown in , the selection of the second viewing element 1440 causes the model viewing component 260 and the rendering component 210 to generate a depth adjustment element 1510. The movement of the depth adjustment element 1510 causes the model viewing component 260 and the modeling component 250 to modify the depth of the 3D model. Fig.15 As shown in , relative to the user, the movement of the depth adjustment element 1510 to the left causes the modeling component 250 and the rendering component 210 to display the 3D model at a relatively shallower depth. Relative to the user, the movement of the depth adjustment element 1510 to the right causes the modeling component 250 and the rendering component 210 to display the 3D model at a relatively deeper depth.

[0090] Selection of the third viewing element 1450 enables adjustment of the apparent depth of field of the 3D model by enabling modification of the focus within the visualized image containing the 3D model. In some embodiments, upon selection of the third viewing element 1450, the notification component 230 and the presentation component 210 generate and cause presentation of instructions for interacting with the visualized image and 3D model. For example, the instructions may indicate that modification of the focus is performed by clicking at a desired focus point of the touch screen interface. Upon receiving a selection of a portion of the visualized image and 3D model, the model viewing component 260 and the modeling component 250 modify the focus level of the selected point and the portion of the visualized image and 3D model that surrounds the selected point within a predetermined proximity.

[0091] Fig.16 is a flow chart of the operations of the model generation system 150 in performing the method 1600 of generating a 3D model within a graphical user interface, according to some example embodiments. Fig.16 The operations shown in can be performed by the model generation system 150 using the components described herein. Fig.16 As shown in , in some embodiments, method 1600 may be performed as a part or sub-operation of the above-described method 300 .

[0092] In operation 1610, detection component 220 identifies a set of facial tracking points on a face within a field of view of an image acquisition device. In some embodiments, identification of the set of facial tracking points may be a coarse facial detection prior to image acquisition. Coarse facial detection is performed to identify that an object (e.g., a face) is properly aligned within the field of view. The set of facial tracking points may be pixels from one or more images from the image acquisition device suitable for tracking over time. Operation 1610 or a similar method may be performed each time a reference keyframe is updated during acquisition to generate these points for each reference keyframe.

[0093] In some cases, the set of facial tracking points are feature points of the face, such as points that define the shape of the face, eyebrows, mouth, nose, eyes, and other suitable feature points. The set of facial tracking points may initially be two-dimensional points of interest detected on the user's face in a reference image. The points of interest may be detected using a point of interest detection algorithm. In some cases, the points of interest are detected using a standard point of interest detection algorithm, such as features of the accelerated segment test (FAST), adaptive and general angle detection based on the accelerated segment test (AGAST), Harris, Shi-Tomasi, etc. In some embodiments, in order to establish a more uniform distribution of points of interest, the area of ​​the field of view or the image expected to contain a face (e.g., a portion of the field of view or an image within a visible framing element) may be divided into a grid. The points of interest for each grid cell in the grid may be determined according to a point of interest scoring function such as Harris, Shi-Tomasi, etc. The method of determining points of interest for a grid cell may be repeated at different scales for the image data to identify points of interest at multiple scales.

[0094] In some embodiments, in response to one or more of detecting the first change in position in operation 340 and detecting the second change in position in operation 350, notification component 230 generates a graphical representation of the set of facial tracking points on the face within the graphical user interface frame. Figure 7 As shown, during the duration of detecting the first change in position and the second change in position, the presentation component 210 causes presentation of a graphical representation of the set of facial tracking points on the face.

[0095] In operation 1620, the position component 240 identifies a first keyframe where the set of facial tracking points has a first set of positions. When the face is close enough to the target pose, the detection component 220 and the position component 240 record the information associated with the camera frame as a keyframe for later processing. The keyframe includes the camera image and any sensor data such as from an accelerometer and / or gyroscope associated with the mobile scanning data acquisition device, an estimated 3D pose of the camera, and a set of 2D measurements of points of interest on the user's face.

[0096] In operation 1630, the position component 240 determines a change in the position of the one or more facial tracking points from the set of first positions. After identifying the points of interest, the position component 240 can track the two-dimensional position of each point of interest over time in subsequent image frames received from the image acquisition device. This tracking by the position component 240 can be performed in response to detecting a change in the position of the one or more facial tracking points or in response to the first change in position or the second change in position (as described in operations 340 and 350, respectively). In some embodiments, the position component 240 can track the two-dimensional position of each point of interest using a suitable tracking algorithm such as Kanade-Lucas-Tomasi tracking.

[0097] In some embodiments, operation 1630 is performed by one or more sub-operations. In operation 1632, in response to initiation of a change in the position of one or more facial tracking points, position component 240 identifies a trajectory for each of the one or more facial tracking points. The trajectory may be identified as a two-dimensional vector extending away from an initial position of a given facial tracking point. In the case where the initial position of the facial tracking point is a keyframe, the vector may extend between the initial position and a subsequent position indicating a direction of travel over time.

[0098] As the object moves, the track is tracked and increases in length. In embodiments where the object is a face, the position component 240 tracks the track of one or more facial tracking points as the face rotates with respect to the image capture device as the track length increases. Based on tracking the one or more facial tracking points across multiple image frames, the track can grow from image data captured by the image capture device with each successive image frame with respect to an initial image frame.

[0099] In operation 1634, the position component 240 determines an average length of the track of the one or more facial tracking points. The average length of the track can be determined for each of the one or more facial tracking points based on a distance of a vector extending between an initial position and a subsequent position. The distances associated with each of the one or more facial tracking points can then be averaged across the one or more facial tracking points. Averaging the distances of the one or more facial tracking points can be performed by calculating a median track length to provide robustness to erroneous tracks.

[0100] In operation 1636, the position component 240 determines that the average length exceeds the track threshold. As described above, once the average length of the track exceeds the track threshold in the direction indicated by the instruction of the position element, the position component 240 can identify the target 3D pose of the image acquisition device and record a keyframe as described below.

[0101] In some cases, the track threshold may be a predetermined segment of the average distance traveled by facial tracking points between an initial position and an expected final position. The track threshold may be determined prior to initiating image acquisition and modeling operations. In some cases, the track threshold may be determined during a configuration process of the model generation system 150. In some embodiments, the track threshold may be determined heuristically based on the object being modeled. In some cases, the track threshold is determined as a fraction of the image width. For example, the track threshold may be a distance equal to between 0.5% and 10% of the image width or field of view of the image acquisition device.

[0102] In operation 1640, based on the change in position, the position component 240 identifies a second keyframe in which the one or more facial tracking points have the second position. The second keyframe may be determined based on the one or more facial tracking points being within a predetermined proximity of an expected position (e.g., the second position). As in operation 1636, in embodiments where the position component 240 determines that the average length exceeds the track threshold, the second keyframe may be identified based on a determination that the average length exceeds the track threshold.

[0103] In some embodiments, one or more facial tracking points having a trajectory that continues (e.g., extends) from an initial keyframe to a second keyframe (e.g., a new keyframe) are used to record measurements in both the initial keyframe and the second keyframe. The continued measurements can specify the observed two-dimensional position of a particular point of interest in the keyframe. An initial estimate of the 3D position of the point of interest is calculated by triangulating the two measurements associated with each point of interest using the assumed 3D pose of the image acquisition device for the first two keyframes.

[0104] Using the initial estimate of the keyframe pose and 3D point of interest position, the position component 240 can perform bundle adjustment to refine the pose of the second keyframe and the 3D position of the point of interest. During bundle adjustment, a robust nonlinear least squares optimization can be performed to jointly optimize the 3D keyframe pose and the 3D point of interest position in a manner that theoretically minimizes the reprojection error of the point of interest in the keyframe. The reprojection error can be defined as the distance between the two-dimensional projected position of the point of interest in the keyframe and the observed actual two-dimensional measurement.

[0105] Fig.17 is a flow chart of the operations of the model generation system 150 in performing the method 1700 of generating and manipulating a 3D model within a graphical user interface, according to some example embodiments. Fig.17 The operations shown in can be performed by the model generation system 150 using the components described herein. In some embodiments, Fig.17 As shown in , method 1700 can be performed as a part of the above-mentioned method 300 or as a sub-operation of method 300.

[0106] In operation 1710, the position component 240 identifies one or more subsequent keyframes based on one or more changes in the position of one or more facial tracking points along the trajectory. The one or more subsequent keyframes may be associated with the Fig.16 The first keyframe and the second keyframe described are similarly or identically determined. In some embodiments, the number of subsequent keyframes identified by the position component 240 can be predetermined based on the object. For example, where the object is a face, the face can have an expected range of motion (e.g., rotation of the neck) corresponding to the distance between the initial position and the final position of the set of facial tracking points. The distance between the initial position and the final position can be divided equally to determine the number of subsequent keyframes for a given side of the face.

[0107] In some cases, the latest keyframe (e.g., the second most recently generated keyframe or subsequent keyframe) can be used as the current reference keyframe. The position component 240 initializes the track for each existing measurement in the reference keyframe. The same point of interest detection method described above can also be used to initialize a new track. The new track can be initialized in a point of interest at a predetermined distance from an existing track.

[0108] In operation 1720, the position component 240 generates a set of relative 3D position estimates for the mobile computing device based on the first keyframe, the second keyframe, and one or more subsequent keyframes. The set of relative position estimates may indicate the relative position of the mobile computing device with respect to the modeled object (e.g., a face). The position component 240 may generate a relative position estimate in the set of relative position estimates for each keyframe (e.g., the first keyframe, the second keyframe, and one or more subsequent keyframes). The set of relative position estimates may use the 3D camera pose estimate from the previous frame to identify points of interest with known 3D positions that are not tracked in the current image frame. This discovery operation may enable recapture of points of interest that were lost between two frames.

[0109] In some cases, the set of relative position estimates can be used to identify keyframes. In these cases, the relative position estimate for the current frame can be compared with the next target relative position estimate. When the relative position estimate enters the predetermined proximity of the next target relative position estimate, the position component 240 identifies a new keyframe and records the new keyframe using the relative position estimates of the current image frame and the adjacent target relative position estimates. Measurements can be recorded for each tracked point of interest in the new keyframe. For points of interest that did not previously have a 3D position (e.g., newly identified points of interest in the most recent keyframe), the track (e.g., the trajectory of each new point of interest) can be filtered to ensure that the epipolar constraints between the reference keyframe and the new keyframe are met. Tracks that meet the epipolar constraints can be used to create measurements in both the reference keyframe (e.g., the previously identified keyframe) and the new keyframe. The 3D position of the associated point of interest can be initialized using triangulation between the measurements in the reference keyframe and the new keyframe.

[0110] In some embodiments, after identifying a new keyframe, one or more adjustment processes may be performed on the keyframes preceding and including the new keyframe. The one or more adjustment processes may be bundle adjustment as described above. Bundle adjustment may improve or refine estimates of the relative 3D positions of the keyframes and the 3D positions of the points of interest. Whenever a new keyframe is recorded, bundle adjustment may be performed to collectively refine estimates of all keyframe poses and 3D point of interest positions preceding and including the new keyframe.

[0111] In operation 1730, the modeling component 250 generates a set of depth maps based on the first keyframe, the second keyframe, one or more subsequent keyframes, and a set of relative position estimates. In some embodiments, the set of relative depth maps includes a depth map for each keyframe. Given the set of keyframes and 3D points of interest (e.g., facial tracking points and tracks), the modeling component 250 generates a 2.5-dimensional depth map for one or more keyframes using a multi-view stereo technique.

[0112] In some embodiments, depth map estimation is performed by planar swept stereo. In these embodiments, photo consistency between an image frame of a reference keyframe and one or more other image frames of one or more other keyframes is calculated for each pixel in the depth map at a predetermined number of discrete depths. A final depth map may be generated by selecting the depth of each depth map pixel that produces the best photo consistency between views.

[0113] In some cases, before generating the set of depth maps, the modeling component 250 pre-processes the image frame of the key frame. The pre-processing can generate a representation of the image frame that encodes both texture and brightness. For a given pixel, the modeling component 250 samples a 5×5 patch around the pixel. The modeling component 250 calculates a 24-bit binary signature by comparing the brightness of each non-center pixel with the average brightness of the patch. The comparison of brightness and average brightness is combined with the 8-bit brightness of the center pixel to give a 32-bit value that can be stored in a red, green, blue, alpha (RGBA) texture and sampled with a single texture that is looked up on a graphics processing unit (GPU). The modeling component 250 can calculate the photo consistency between two such values ​​by combining the Hamming distance on the binary signature and the Euclidean distance on the brightness.

[0114] Photo consistency values ​​can be pooled from neighboring pixels to generate a robust photo consistency value. For example, the modeling component 250 can pool photo consistency values ​​from neighboring pixels within a 3×3 patch at a given depth. Robust photo consistency values ​​can be determined during a plane sweep. In some embodiments, the modeling component 250 generates a two-pass plane swept stereo. In these cases, as described above, the modeling component 250 uses a first-pass plane sweep to generate a depth map. The first pass can then be used by the modeling component 250 as a starting point for subsequent plane sweeps over a smaller depth range around the previously generated depth map. Subsequent plane sweeps can modify the depth range for each pixel in the depth map.

[0115] In some embodiments, measurements of 3D points of interest in a given keyframe may be used to guide depth map generation. Modeling component 250 may provide initialization or constraints for selected regions during depth map generation. The depth map may also be post-processed to reduce noise. Post-processing by modeling component 250 may include consistency checks between keyframes. Post-processing operations of modeling component 250 may further include median filtering, bilateral filtering, total variation based denoising, and conditional random fields that form labels between keyframes for depth map values.

[0116] In some cases, operation 1730 includes a set of sub-operations. In generating the set of depth maps, the sub-operations of operation 1730 may be performed on each depth map generated by modeling component 250. In operation 1732, modeling component 250 generates a first depth map having a first resolution.

[0117] In operation 1734, the modeling component 250 generates a second depth map having a second resolution. In some cases, the second resolution is a higher resolution than the first resolution of the first depth map. In some embodiments, the modeling component 250 can create more than two depth maps. The modeling component 250 can create depth maps with increasingly higher resolutions a predetermined number of times until a predetermined resolution is reached, or any suitable number of iterations for the resolution.

[0118] Fig.18 is a flowchart of the operations of the model generation system 150 in performing the method 1800 of generating and manipulating a 3D model within a graphical user interface, according to some example embodiments. Fig.18 The operations shown in can be performed by the model generation system 150 using the components described herein. In some embodiments, Fig.18 As shown in , method 1800 can be performed as a part of the above-mentioned method 1700 or as a sub-operation of method 1700.

[0119] In operation 1810, the modeling component 250 fuses the set of depth maps to generate a 3D model of the face. In some cases, the modeling component 250 combines the 2.5-dimensional depth maps into a single 3D surface model of the object (eg, face).

[0120] In operation 1820, the modeling component 250 defines a volumetric 3D grid for the 3D model of the face. In the combined depth map, the modeling component 250 defines a truncated signed distance function (TSDF) on the volumetric 3D grid. Although described in separate operations, in some cases, operations 1810 and 1820 are performed in a single operation, such that the volumetric 3D grid with the TSDF is a single method for fusing the depth maps into a single 3D model.

[0121] In operation 1830, the modeling component 250 represents the 3D surface of the 3D model within a volumetric 3D grid. Each element (e.g., voxel) in the grid is identified as being inside or outside the 3D surface and is stored as a scalar value between -1 and 1. The sign of the value defines whether the voxel is inside or outside the 3D surface. The 3D surface can be implicitly defined by the boundary between positive and negative values.

[0122] In some embodiments, the depth map generated for each key frame can provide an estimate of the surface of the 3D model. The modeling component 250 can convert the depth map into a view-specific TSDF through a volume 3D grid. The TSDF of each depth map can be aggregated to aggregate a histogram of TSDF values ​​at each voxel in the volume 3D grid. The histogram of TSDF values ​​plays a role in summarizing the estimates of the generated depth map. The modeling component 250 can perform theoretical optimization on the histogram of TSDF values ​​to estimate a single TSDF value at each voxel, which considers both histogram data and regularization strategies to promote smoothness. In some embodiments, the modeling component 250 uses L1 data terms and total variation regularization, resulting in a surface that is robust to noisy depth map data. In addition, the surface is smooth and minimizes small isolated surface areas. The optimization method is also applicable to GPU implementations, making the optimization actions effective in practice. In some cases, the surface defined by the optimized TSDF can be converted to a polygonal surface grid using an implicit surface polygon algorithm (such as a marching cube or Bloomenthal's method).

[0123] In some cases, the depth map may be a dense depth map and represent the 3D surface of the model as a mesh. The 3D surface may be constructed using a Poisson surface reconstruction operation.

[0124] Fig.19 is a flow chart of the operations of the model generation system 150 in performing the method 1900 of generating and manipulating a 3D model within a graphical user interface, according to some example embodiments. Fig.19 The operations shown in can be performed by the model generation system 150 using the components described herein. In some embodiments, Fig.19 As shown in , method 1900 can be performed as a part of the above-mentioned method 1800 or as a sub-operation of method 1800.

[0125] In operation 1910, the modeling component 250 identifies a facial mesh. The facial mesh may include a set of polygons and a set of vertices connecting the polygons. The set of vertices may represent a set of facial tracking points. In some embodiments, the facial mesh includes a fixed number of vertices and polygon faces. Certain vertices and faces in the mesh may be labeled as corresponding to specific facial features, such as "tip of nose", "left edge of mouth", "inside left eye", and other suitable features. These features may be manually or automatically labeled in some or all of the keyframe images.

[0126] In operation 1920, the modeling component 250 deforms one or more portions of the facial mesh to fit the 3D model of the face. The modeling component 250 may deform the facial mesh by moving one or more vertices connecting two or more polygons in the set of polygons. The one or more vertices may be moved to correspond to the position of one or more facial tracking points of the set of facial tracking points within the volumetric 3D grid defined in operation 1820 above.

[0127] In some embodiments, deformation of portions of the facial mesh may be performed by the modeling component 250 as an optimization. The deformation adjusts the positions of vertices in the facial mesh subject to one or more constraints. For example, the surface of the facial mesh may be deformed so that the surface is within a predetermined proximity of the surface of the 3D model generated from the depth map. In some cases, facial mesh vertices (e.g., facial landmarks) are deformed to project to image frames of keyframes proximate to facial tracking points (e.g., points of interest) on the 3D model. The facial mesh may also be modified so that the overall shape of the face remains within the 3D model and the facial mesh. In some embodiments, the facial mesh includes a UV map. The UV map allows a texture map of the face to be constructed by projecting image frames of keyframes onto the fitted facial mesh.

[0128] In some embodiments, after generating the 3D model, the model generation system 150 may send the 3D model to a data storage platform (e.g., the network system 102). The device for collecting the scanned image data is associated with an end-user application, such as a mobile computing device (e.g., the client device 110). The end-user application directs the model generation system 150 to transmit the 3D model data to the network system 102 via the network 104. The model generation system 150 may also retrieve the 3D facial model data from the network system 102 for use by the end-user application on the client device 110. In some cases, all of the data used to generate the 3D model may also be sent, enabling the model reconstruction process to be performed at a future time, or using a more powerful computing device, computer architecture (such as, in a non-limiting example, within a cloud server or cloud network), or computing method. The 3D model data may be optimized for transmission to minimize transmission time through known encoding, optimization, or other data transmission techniques that will not be further discussed herein. The data may be linked to one or more account identifiers associated with the end user 106 for easy retrieval by the end user 106 or an authorized third party on behalf of the end user 106 at a future time.

[0129] In some embodiments, the model generation system 150 can also retrieve 3D model data for the end user to populate the end user application. If the end user 106 installs the end user application on a second device different from the client device 110 used to acquire the one or more facial models sent to the network system 102, the model generation system 150 can retrieve the identified model data. When retrieving the model data, the model generation system 150 can cause the retrieved model to be presented, or can access the facial data to manipulate the facial model retrieved on the second device.

[0130] In addition to the 3D model, the model generation system 150 can generate a 2.5 dimensional model. In some cases, the 2.5D model can be generated in conjunction with a 3D model of a specified object within the field of view. Fig. 20 and Fig.21 As shown, the detection component 220 can detect objects or points of interest within the scene in the field of view of the image acquisition device. The notification component 230 can generate user interface elements and instructions to indicate the movement of the client device 110 to enable the acquisition of multiple angles of the scene. As shown, in some cases, the instructions indicate movement in a single direction. In some embodiments, the instructions indicate movement in multiple directions.

[0131] The position component 240 can detect changes in the position of the client device 110 with respect to one or more points of interest identified within the scene. The modeling component 250 can generate a 3D or 2.5D model of the scene. In the case where the modeling component 250 generates a 2.5D model, the 2.5D model can be a single depth map generated for the field of view. As described in the above method 300 and other methods, Fig. 20 and Fig.21 The operations described may be similar or identical to the operations described with respect to objects of interest or faces.

[0132] In embodiments where a model for a scene is generated together with a 3D model for an object, the model for the scene may be generated simultaneously or separately from the 3D model for the object. For example, the instructions generated by the notification component 230 may instruct the client device 110 to move in one or more directions while the object and scene remain in a static position. In some embodiments, as described above, the model generation system 150 may use the movement of the image acquisition device instead of the movement of the object to identify and acquire the object within the field of view. The model generation system 150 may then acquire the scene. The model generation system 150 may then generate a 3D model for the object and a 2.5D model for the scene as a background for the object.

[0133] In some cases, the model generation system 150 uses the motion of the object of interest to identify and capture objects within the field of view as described in method 300. After capturing the object of interest, the notification component 230 can generate instructions for the client device 110 to move and capture the scene. The model generation system 150 can then generate a 3D model for the object of interest and a 2.5D model for the scene in the background behind the object of interest, respectively.

[0134] According to various example embodiments, one or more methods described herein can assist in the generation and manipulation of 3D models based on guided motion of a mobile computing device or an image acquisition device. The method for generating and modifying a 3D model automatically determines modeling parameters associated with the appropriate motion of an object to acquire and generate a 3D model. The method further automatically generates an instruction set and user interface elements that are configured to guide and correct user interactions to ensure that the 3D model is correctly generated. Therefore, one or more methods described herein can have the effect of allowing a user to generate clear and accurate 3D models of objects, faces, and scenes without sending the underlying modeling data to a third-party modeling program or organization. In addition, the methods described herein can have the effect of reducing the time, cost, and computing resources required to generate a 3D model. The methods described herein can also enable the visualization and transmission of the completed 3D model for direct integration into a third-party system or product.

[0135] Components and Logic

[0136] Certain embodiments are described herein as including logic or multiple components or mechanisms. A component may constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that is capable of performing certain operations and that can be configured or arranged in some physical manner. In various exemplary embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein.

[0137] In some embodiments, the hardware components can be implemented mechanically, electronically, or any suitable combination thereof. For example, the hardware components may include a dedicated circuit or logic that is permanently configured to perform certain operations. For example, the hardware components may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware components may also include programmable logic or circuits that are temporarily configured by software to perform certain operations. For example, the hardware components may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of machines) that are specifically customized for performing the configured functions and are no longer general-purpose processors. It should be understood that the decision to physically implement the hardware components in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0138] Thus, the phrase "hardware component" should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some manner or perform certain operations described herein. As used herein, a "hardware-implemented component" refers to a hardware component. Contemplating embodiments in which a hardware component is temporarily configured (e.g., programmed), each of the hardware components need not be configured or instantiated at any one time. For example, where the hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. Accordingly, software configures a particular processor, for example, to constitute a particular hardware component at one point in time and to constitute a different hardware component at a different point in time.

[0139] Hardware components can provide information to other hardware components and receive information therefrom. Therefore, the described hardware components can be considered to be communicatively coupled. In the case where multiple hardware components exist simultaneously, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, the communication between such hardware components can be achieved, for example, by the storage and retrieval of information in a memory structure accessed by multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component can access the memory device at a later time to retrieve and process the output of the storage. The hardware component can also initiate communication with an input or output device, and can operate on a resource (e.g., a collection of information).

[0140] The various operations of the example methods described herein are performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Such processors, whether temporarily or permanently configured, may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refers to hardware components that are implemented using one or more processors.

[0141] Similarly, the methods described herein may be at least partially processor-implemented, where a particular processor or processors are examples of hardware. For example, at least some of the operations of the methods may be performed by one or more processors or processor-implemented components. In addition, one or more processors may also operate to support the performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0142] The performance of certain operations may be distributed among processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, the processor or processor-implemented components may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented components may be distributed in multiple geographic locations.

[0143] Machine and software architecture

[0144] Combination Figure 2-21 The components, methods, applications, etc. described are implemented in some embodiments in the context of a machine and associated software architecture. In various embodiments, the components, methods, applications, etc. described above are implemented in the context of multiple machines and one or more associated software architectures distributed on and transmitted via a network. The following section describes representative software architectures and machine (e.g., hardware) architectures suitable for the disclosed embodiments.

[0145] Software architectures are used together with hardware architectures to create devices and machines that are tailored for specific purposes. For example, a specific hardware structure combined with a specific software architecture will create a mobile device, such as a mobile phone, a tablet computer device, etc. Slightly different hardware and software architectures may produce smart devices for the "Internet of Things", while another combination produces a server computer for use within a cloud computing architecture. Not all such combinations of software and hardware architectures are presented here, as those skilled in the art can readily understand how to implement the present embodiments in contexts other than those disclosed herein.

[0146] Software Architecture

[0147] Fig. 22 is a block diagram 2200 illustrating a representative software architecture 2202 that may be used in conjunction with the various hardware architectures described herein. Fig. 22 This is merely a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 2202 may be implemented in a system such as Fig.23 2300, which includes, among other things, a processor 2310, a memory 2330, and an input / output (I / O) component 2350. A representative hardware layer 2204 is shown and may represent, for example, Fig. 22 The block diagram 2200 of the machine is shown. The representative hardware layer 2204 includes one or more processing units 2206 with associated executable instructions 2208. The executable instructions 2208 represent executable instructions of the software architecture 2202, including Figure 2-21 The hardware layer 2204 also includes a memory and / or storage component 2210 that also has executable instructions 2208. The hardware layer 2204 may also include other hardware as shown at 2212, which represents any other hardware of the hardware layer 2204, such as other hardware shown as part of the machine 2300.

[0148] exist Fig. 22 In the example architecture of , software architecture 2202 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 2202 may include layers such as operating system 2214, library 2216, framework / middleware 2218, application 2220, and presentation layer 2244. In operation, application 2220 and / or other components within a layer may call API call 2224 through a software stack, and receive responses, return values, etc. as shown in message 2226 in response to API call 2224. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware layer 2218, while others may provide such a layer. Other software architectures may include additional layers or different layers.

[0149] The operating system 2214 can manage hardware resources and provide public services. The operating system 2214 may include, for example, a kernel 2228, services 2230, and drivers 2232. The kernel 2228 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 2228 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 2230 may provide other public services to other software layers. Drivers 2232 may be responsible for controlling or interfacing with the underlying hardware. For example, drivers 2232 may include display drivers, camera drivers, drives, flash drives, serial communications drivers (such as Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

[0150] The library 2216 can provide a public infrastructure that can be used by the application 2220 and / or other components and / or layers. The library 2216 generally provides functions that allow other software components to perform tasks in a simpler way than the direct interface connection of the underlying operating system 2214 functions (e.g., kernel 2228, service 2230 and / or driver 2232). The library 2216 may include a system 2234 library (e.g., C standard library) that can provide functions such as memory allocation functions, string operation functions, mathematical functions, etc. In addition, the library 2216 may include an API library 2236 such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as Moving Picture Experts Group 4 (MPEG4), H.264, MP3, Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework that can be used to display two-dimensional and three-dimensional graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a Web library (e.g., WebKit that can provide Web browsing functions), etc. The library 2216 may also include a variety of other libraries 2238 to provide many other APIs to the application 2220 and other software components.

[0151] The framework 2218 (also sometimes referred to as middleware) can provide a higher level common infrastructure that can be utilized by the application 2220 and / or other software components. For example, the framework 2218 can provide various graphical user interface functions, high-level resource management, advanced location services, etc. The framework 2218 can provide a wide range of other APIs that can be used by the application 2220 and / or other software components, some of which may be specific to a particular operating system or platform. In some example embodiments, the model generation component 2219 (e.g., one or more components of the model generation system 150) can be implemented at least partially within the middleware / framework 2218. For example, in some cases, at least a portion of the presentation component 210 that provides graphical and non-graphical user interface functions can be implemented in the middleware / framework 2218. Similarly, in some example embodiments, a portion of one or more of the presentation component 210, the notification component 230, the modeling component 250, and the model viewing component 260 can be implemented in the middleware / framework 2218.

[0152] Applications 2220 include built-in applications 2240, third-party applications 2242, and / or model generation components 2243 (e.g., user-facing portions of one or more components of model generation system 150). Examples of representative built-in applications 2240 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and / or a game application. Third-party applications 2242 may include any built-in application as well as a wide variety of other applications. In a specific example, third-party applications 2242 (e.g., applications created by an entity other than the vendor of a particular platform using Android TM or iOS TM Applications developed using a software development kit (SDK) can be mobile software that runs on a mobile operating system, such as iOS. TM 、Android TM , Phone or other mobile operating system. In this example, the third-party application 2242 can call API calls 2224 provided by a mobile operating system such as operating system 2214 to facilitate the functions described herein. In various example embodiments, the user-facing portion of the model generation component 2243 can include information about Figure 2 For example, in some cases, presentation component 210, detection component 220, notification component 230, location component 240, modeling component 250, and portions of model viewing component 260 associated with user interface elements (e.g., data input and data output functionality) can be implemented in the form of an application.

[0153] The application 2220 may utilize built-in operating system functionality (e.g., kernel 2228, services 2230, and / or drivers 2232), libraries (e.g., system 2234, API 2236, and other libraries 2238), and framework / middleware 2218 to create a user interface that interacts with a user of the system. Alternatively or in addition, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 2244. In these systems, the application / component "logic" may be separated from the aspects of the application / component that interact with the user.

[0154] Some software architectures make use of virtual machines. Fig. 22 In the example of , this is illustrated by virtual machine 2248. A virtual machine creates a software environment where applications / components can be run as if on a hardware machine (e.g., such as a Fig.23 The virtual machine is run by the host operating system ( Fig. 22 The virtual machine 2248 is hosted by an operating system 2214 in the host computer and typically (although not always) has a virtual machine monitor 2246 that manages the operation of the virtual machine and interfaces with the host operating system (i.e., operating system 2214). The software architecture executes within the virtual machine such as an operating system 2250, libraries 2252, frameworks / middleware 2254, applications 2256, and / or a presentation layer 2258. These software architecture layers executed within the virtual machine 2248 may be the same as the corresponding layers previously described or may be different.

[0155] Example machine architecture and machine-readable medium

[0156] Fig.23 2300 is a block diagram illustrating components of a machine 2300 according to some example embodiments, which is capable of reading instructions (e.g., processor executable instructions) from a machine-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Fig.23 A diagrammatic representation of a machine 2300 in the example form of a computer system is shown within which instructions 2316 (e.g., software, programs, applications, applet, application software, or other executable code) may be executed to cause the machine 2300 to perform any one or more of the methodologies discussed herein. For example, the instructions may cause the machine to perform Figure 3 and 16 -19. Additionally or alternatively, the instructions may implement Figure 2-21 The instructions transform a general-purpose non-programmed machine into a specific machine that is programmed to perform the functions described and shown in the manner described.

[0157] In an alternative embodiment, the machine 2300 operates as a standalone device or can be coupled (e.g., networked) to other machines in a network system. In a network deployment, the machine 2300 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a point-to-point (or distributed) network environment. The machine 2300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box, an entertainment media system, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 2316 of the actions that the specified machine 2300 will take. In some example embodiments, in a network deployment, one or more machines may implement at least a portion of the above-mentioned components. One or more machines interacting with the machine 2300 may include, but are not limited to, a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), and other smart devices. Further, while a single machine 2300 is illustrated, the term "machine" shall also be taken to include any collection of machines 2300 that individually or jointly execute instructions 2316 to perform any one or more of the methodologies discussed herein.

[0158] The machine 2300 may include a processor 2310, a memory 2330, and an I / O component 2350 that may be configured to communicate with each other, for example, via a bus 2302. In an example embodiment, the processor 2310 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 2312 and a processor 2314 that may execute instructions 2316. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.23 Multiple processors are shown, but the machine 2300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0159] The memory / storage 2330 may include a memory 2332 (such as a main memory or other memory storage) and a storage unit 2336, both of which may be accessible to the processor 2310 via the bus 2302. The storage unit 2336 and the memory 2332 store instructions 2316 that embody any one or more of the methods or functions described herein. The instructions 2316 may also reside, in whole or in part, within the memory 2332, within the storage unit 2336, within at least one of the processors 2310 (e.g., within a cache memory of the processor), or any suitable combination thereof during execution thereof by the machine 2300. Thus, the memory 2332, the storage unit 2336, and the memory of the processor 2310 are examples of machine-readable media.

[0160] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions 2316. The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., instructions 2316) executed by a machine (e.g., machine 2300), such that when the instructions are executed by one or more processors (e.g., processor 2310) of machine 2300, machine 2300 performs any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0161] I / O components 2350 may include a variety of components for receiving input, providing output, generating output, sending information, exchanging information, collecting measurements, etc. The specific I / O components 2350 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 2350 may include Fig.23Many other components not shown in the figure. The I / O components 2350 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 2350 may include output components 2352 and input components 2354. The output components 2352 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistor mechanisms), other signal generators, etc. The input components 2354 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0162] In further exemplary embodiments, the I / O component 2350 may include a biometric component 2356, a motion component 2358, an environmental component 2360, or a position component 2362, among a wide range of other components. For example, the biometric component 2356 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 2358 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 2360 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor to detect safe concentrations of hazardous gases or measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 2362 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects the altitude from which air pressure can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0163] A variety of technologies may be used to achieve communication. I / O components 2350 may include a communication component 2364 that is operable to couple machine 2300 to network 2380 or device 2370 via coupling 2382 and coupling 2372, respectively. For example, communication component 2364 may include a network interface component or other suitable means for interfacing with network 2380. In another example, communication component 2364 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low energy consumption), Components and other communication components that provide communication via other modalities. Device 2370 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0164] In addition, the communication component 2364 can detect an identifier or include a component that can be operated to detect an identifier. For example, the communication component 2364 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode; a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data map, a MaxiCode, a PDF417, a Supercode, a Uniform Commercial Code Reduced Space Symbol (UCC RSS)-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 2364, such as positioning via Internet Protocol (IP) geolocation, location ... Positioning via signal triangulation, positioning via detection of NFC beacon signals that can indicate a specific location, etc.

[0165] Transmission medium

[0166] In various example embodiments, one or more portions of network 2380 may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, a portion of the PSTN, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 2380, or a portion of network 2380, may include a wireless or cellular network, and coupling 2382 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, coupling 2382 may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0167] Instructions 2316 may be sent or received over network 2380 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 2364), and utilizing any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 2316 may be sent or received to device 2370 using a transmission medium via coupling 2372 (e.g., a peer-to-peer coupling). The term "transmission medium" shall be deemed to include any intangible medium capable of storing, encoding, or carrying instructions 2316 for execution by machine 2300, and includes digital or analog communication signals or other intangible media to facilitate the communication of such software.

[0168] language

[0169] Throughout the specification, multiple instances can be implemented as components, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as individual operations, one or more individual operations can be performed simultaneously, and it is not necessary to perform the operations in the order shown. The structure and function presented as the individual components in the example configuration can be implemented as a combined structure or component. Similarly, the structure and function presented as a single component can be implemented as an individual component. These and other variations, modifications, additions and improvements fall within the scope of this paper theme.

[0170] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein, individually or collectively, by the term "invention", merely for convenience and without intending to voluntarily limit the scope of the present application to any single disclosure or inventive concept, if in fact more than one is disclosed.

[0171] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered restrictive, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents granted by these claims.

[0172] As used herein, the term "or" may be interpreted in an inclusive or exclusive manner. In addition, multiple instances may be provided as a single instance for the resources, operations, or structures described herein. In addition, the boundaries between the various resources, operations, components, engines, and data stores are somewhat arbitrary, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions may be envisioned and may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as separate resources in an example configuration may be implemented as a combined structure or resource. Similarly, structures and functions presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.

Claims

1. A method for generating a three-dimensional model, comprising: detecting an object within a field of view of an image acquisition device of a mobile computing device; displaying a motion element in a graphical user interface along with an image of the object within the field of view of the image acquisition device, the motion element comprising one or more visual movement instructions for locating the object within the field of view of the image acquisition device; displaying a position element in the graphical user interface along with the image of the object, wherein a position of the position element is proportional to an amount of motion detected in the object; continuously moving the position element between a starting position and an ending position by an amount determined based on a current position of the object within the field of view and a travel distance detected for the object; identifying, within a predetermined number of key frames, a first key frame and a second key frame corresponding to a first position change and a second position change, respectively, of the object within the field of view of the image acquisition device relative to a starting position, wherein the predetermined number of key frames are identified by equally dividing a distance between an initial position and a final position of a portion of the object; generating a first depth map and a second depth map having a first resolution and a second resolution, respectively, the first depth map being generated based on the identified first key frame and the second depth map being generated based on the second key frame; and A three-dimensional model of the object is generated based on the first depth map and the second depth map.

2. The method according to claim 1, wherein: The object comprises a face, and the method further comprises: identifying a set of facial tracking points on the face; In response to detecting one or more of the first change in position and the second change in position, generating a graphical representation of the set of facial tracking points on the face within a graphical user interface frame; and The graphical representation of the set of facial tracking points on the face is caused to be presented for a duration of detecting the first change in position and the second change in position.

3. The method according to claim 1, wherein: The first position change is an expected position change having a first initial position and a first final position, and wherein the first position change has a first set of intermediate positions, the first initial position, the first set of intermediate positions, and the first final position being associated with a first side of the object.

4. The method according to claim 3, wherein: The second position change is an expected position change having a second initial position and a second final position, and wherein the second position change has a second set of intermediate positions, the second initial position, the second set of intermediate positions, and the second final position are associated with a second side of the face opposite to the first side of the object.

5. The method according to claim 1, further comprising: identifying a set of tracking points on the object within the field of view of the image acquisition device; identifying the first keyframe in which the set of tracking points has a first set of positions; determining a change in position of one or more tracking points in the set of tracking points from the set of first positions; as well as Based on the change in position, a second keyframe is identified in which the one or more tracking points have a second position.

6. The method according to claim 5, wherein: Determining the change in position of the one or more tracking points comprises: responsive to the initiation of the change in position of the one or more tracking points, identifying a trajectory of each of the one or more tracking points; determining an average length of the trajectories of the one or more tracking points; and It is determined that the average length exceeds a trajectory threshold.

7. The method according to claim 6, wherein: The second keyframe is identified based on determining that the average length exceeds the trajectory threshold.

8. The method according to claim 6, further comprising: identifying one or more subsequent keyframes based on one or more changes in positions of the one or more tracking points along the trajectory; as well as A set of relative position estimates of the mobile computing device relative to the object is generated based on the first keyframe, the second keyframe, and the one or more subsequent keyframes, a relative position estimate in the set of relative position estimates being generated for each keyframe.

9. The method according to claim 8, further comprising: A set of depth maps is generated based on the first keyframe, the second keyframe, the one or more subsequent keyframes, and the set of relative position estimates, the set of depth maps including a depth map for each keyframe.

10. The method according to claim 9, further comprising: fusing the set of depth maps to generate the three-dimensional model of the object; defining a volumetric three-dimensional mesh of the three-dimensional model for the object; as well as A three-dimensional surface representing the three-dimensional model.

11. The method according to claim 1, wherein: The second resolution of the second depth map is greater than the first resolution of the first depth map, wherein the motion elements overlay the image of the object, and wherein the motion elements include focus elements and gesture elements.

12. A system for generating a three-dimensional model, comprising: one or more processors; an image acquisition device operably coupled to the one or more processors; as well as a non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting an object within a field of view of an image acquisition device of a mobile computing device; displaying a motion element in a graphical user interface along with an image of the object within the field of view of the image acquisition device, the motion element comprising one or more visual movement instructions for locating the object within the field of view of the image acquisition device; displaying a position element in the graphical user interface along with the image of the object, wherein a position of the position element is proportional to an amount of motion detected in the object; continuously moving the position element between a starting position and an ending position by an amount determined based on a current position of the object within the field of view and a travel distance detected for the object; identifying, within a predetermined number of key frames, a first key frame and a second key frame corresponding to a first position change and a second position change, respectively, of the object within the field of view of the image acquisition device relative to a starting position, wherein the predetermined number of key frames are identified by equally dividing a distance between an initial position and a final position of a portion of the object; generating a first depth map and a second depth map having a first resolution and a second resolution, respectively, the first depth map being generated based on the identified first key frame and the second depth map being generated based on the second key frame; and A three-dimensional model of the object is generated based on the first depth map and the second depth map.

13. The system according to claim 12, wherein: The object includes a face, and the operations further include: identifying a set of facial tracking points on the face; In response to detecting one or more of the first change in position and the second change in position, generating a graphical representation of the set of facial tracking points on the face within a graphical user interface frame; and The graphical representation of the set of facial tracking points on the face is caused to be presented for a duration of detecting the first change in position and the second change in position.

14. A method for generating a three-dimensional model, comprising: causing presentation of a graphical user interface frame surrounding a graphical representation of a field of view of an image acquisition device of a mobile computing device; detecting a face within the graphical user interface frame and the field of view of the image acquisition device; generating movement instructions to direct movement of the face within the field of view of the image acquisition device; detecting a first change in position of the face within the field of view of the image capture device; identifying a first keyframe corresponding to the first change; detecting a second change in the position of the face within the field of view of the image capture device; identifying one or more subsequent keyframes corresponding to the second change; generating a set of depth maps including a depth map for each key frame based on the first key frame and the one or more subsequent key frames, the set of depth maps including a first depth map having a first resolution and a second depth map having a second resolution higher than the first resolution, wherein the one or more subsequent key frames are identified and the set of depth maps are generated until a predetermined resolution is reached; as well as A three-dimensional model of the face is generated based on the set of depth maps.

15. A system for generating a three-dimensional model, comprising: one or more processors; an image acquisition device operably coupled to the one or more processors; as well as a non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: causing presentation of a graphical user interface frame surrounding a graphical representation of a field of view of the image acquisition device; detecting a face within the graphical user interface frame and the field of view of the image acquisition device; generating movement instructions to direct movement of the face within the field of view of the image acquisition device; detecting a first change in position of the face within the field of view of the image capture device; identifying a first keyframe corresponding to the first change; detecting a second change in the position of the face within the field of view of the image capture device; identifying one or more subsequent keyframes corresponding to the second change; generating a set of depth maps including a depth map for each key frame based on the first key frame and the one or more subsequent key frames, the set of depth maps including a first depth map having a first resolution and a second depth map having a second resolution higher than the first resolution, wherein the one or more subsequent key frames are identified and the set of depth maps are generated until a predetermined resolution is reached; and A three-dimensional model of the face is generated based on the set of depth maps.

16. A method for generating a three-dimensional model, comprising: detecting an object within a field of view of an image acquisition device of a mobile computing device; identifying a first keyframe corresponding to a first change in position of the object within the field of view of the image acquisition device relative to a starting position and one or more subsequent keyframes corresponding to a second change in position of the object within the field of view of the image acquisition device relative to the starting position; generating a set of depth maps based on the identified first keyframe and the one or more subsequent keyframes, the set of depth maps comprising a first depth map having a first resolution and a second depth map having a second resolution higher than the first resolution, wherein the one or more subsequent keyframes are identified and the set of depth maps are generated until a predetermined resolution is reached; as well as A three-dimensional model of the object is generated based on the set of depth maps.

17. A system for generating a three-dimensional model, comprising: one or more processors; an image acquisition device operably coupled to the one or more processors; as well as a non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting an object within a field of view of an image acquisition device of a mobile computing device; identifying a first keyframe corresponding to a first change in position of the object within the field of view of the image acquisition device relative to a starting position and one or more subsequent keyframes corresponding to a second change in position of the object within the field of view of the image acquisition device relative to the starting position; generating a set of depth maps based on the identified first keyframe and the one or more subsequent keyframes, the set of depth maps comprising a first depth map having a first resolution and a second depth map having a second resolution higher than the first resolution, wherein the one or more subsequent keyframes are identified and the set of depth maps are generated until a predetermined resolution is reached; and A three-dimensional model of the object is generated based on the set of depth maps.

Images