Image-based avatar customization

Abstract

Image-based customization includes extracting feature parameters of a subject in a digital image using one or more neural networks trained with a machine learning algorithm configured to determine the feature parameters of the subject. The feature parameters are then applied to a virtual model of the subject. Aspects of the present disclosure relate to importing real-world objects into virtual applications. More specifically, aspects of the present disclosure relate to scanning and importing real-world objects and human body features into applications for avatar customization.

Description

Technical Field

[0001] Various aspects of the present disclosure relate to importing real-world objects into virtual applications. More specifically, various aspects of the present disclosure relate to scanning and importing real-world objects and human features into applications for avatar customization. Background of the Invention

[0003] Video games and other applications sometimes include detailed characters that represent users, often called avatars. Users often choose to have their avatars match the user's physical features in the real world. This can provide a more satisfying experience for users because they can feel as if they are part of the universe of the video game or application.

[0004] In the past, importing objects into virtual environments was done through sophisticated 3D scanning equipment that uses lasers or other surface scanning methods (such as structured light or coaxial holography) to create a map of the surface of the subject. Other methods (such as photogrammetry) use photos from multiple angles or wide-angle photos to generate multiple views of the subject. Points can be established from the photos to generate a 3D mesh of the subject. All of these methods require specialized equipment or expensive high-quality cameras and are therefore out of reach for the average person who wishes to create a 3D model of themselves or an object at home.

[0005] It is within this context that aspects of the present disclosure are presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The teachings of the present disclosure may be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0007] Figure 1 is a pictorial diagram depicting a method for image-based avatar customization using facial images, according to aspects of the present disclosure.

[0008] Figure 2 is a flow chart depicting a method for image-based avatar customization using facial images according to aspects of the present disclosure.

[0009] Figure 3 is a pictorial diagram depicting a method for image-based avatar customization using body images, according to aspects of the present disclosure.

[0010] Figure 4 is a flow chart depicting a method for image-based avatar customization using body images according to aspects of the present disclosure.

[0011] Figure 5 is a pictorial illustration of a method for image-based virtual object customization using an object image according to aspects of the present disclosure.

[0012] Figure 6is a flow chart illustrating a method for image-based virtual object customization using an object image according to aspects of the present disclosure.

[0013] Figure 7 is a pictorial diagram of another method for image-based virtual object customization using an object image according to aspects of the present disclosure.

[0014] Figure 8 is a flow chart illustrating another method of image-based virtual object customization using object images according to aspects of the present disclosure.

[0015] Fig. 9A is a diagram depicting the basic form of an RNN having a layer of nodes, each of which is characterized by an activation function, an input weight, recurrent hidden node transition weights, and output transition weights, in accordance with aspects of the present disclosure.

[0016] Fig. 9B is a simplified diagram illustrating that an RNN according to aspects of the present disclosure can be considered as a series of nodes with the same activation function over time.

[0017] Fig. 9C Depicted are example layouts of convolutional neural networks, such as CRNNs, in accordance with aspects of the present disclosure.

[0018] Fig.9D A flow chart depicting a method for supervised training of a machine learning neural network in accordance with aspects of the present disclosure is shown.

[0019] Fig.10 is a system block diagram of a system for image-based customization of virtual models according to aspects of the present disclosure. DETAILED DESCRIPTION

[0020] Although the following detailed description contains many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are set forth without losing the generality of the claimed invention and without imposing limitations on the claimed invention.

[0021] According to aspects of the present disclosure, a subject of a digital image may be imported into a virtual space using one or more neural networks trained with a machine learning algorithm and configured to determine characteristic parameters of a subject within the image. Here, a subject may be a human body, a human face, an animal, any type of inanimate object, a building, a plant, or the like. A scene within an image may contain multiple subjects set on a background.

[0022] As used herein, a digital image refers to a visual representation of data. Such data can be obtained by using a sensor to transduce light from a source into electrical pulses, converting the electrical pulses into digital data, and storing or transmitting the digital data. Digital image data can be obtained using an appropriate imaging system (such as a digital camera, a video camera, a light detection and ranging (LIDAR) system, etc.).

[0023] Alternatively, digital image data can be synthetically generated. For example, a scene can be represented by digital data representing a vertex set corresponding to an object in the scene. Each vertex can have associated parameter values, such as position values ​​(e.g., XY coordinates and Z depth values), color values, lighting values, texture coordinates, etc. Vertex shading calculations, such as tessellation and geometry shader calculations, are performed, which can optionally be used to generate new vertices and new geometry in virtual space. Tessellation calculations subdivide scene geometry and geometry shading calculations to generate new scene geometry that exceeds the scene geometry initially set in the application. The scene geometry represented by the vertex is then converted into screen space and converted into a set of discrete picture elements (i.e., pixels) in a process called rasterization. In this process, the virtual space geometry (which can be three-dimensional) is transformed into screen space geometry (usually two-dimensional) by grouping vertex subsets to define a primitive set in screen space and projecting the vertices from virtual space to the viewing window (or "viewport") of the scene in screen space. The pixel data of the resulting image can then be transmitted or stored.

[0024] In some implementations, a digital image may be synthetically generated from two or more images containing the same subject. Such images may be obtained from different angles. Three-dimensional data may be generated by analyzing two images, and a synthetic digital image may be generated from the three-dimensional data, such as described above. In some implementations, if a particular subject is identifiable, the scene containing the subject may be viewed from different angles. Figure 1 The composite image of the subject can be stitched together. Image analysis software can be used to identify a specific subject and isolate it from other parts of the image. In addition, in some implementations, the composite image can be obtained by post-capture modification of the digital image. Such modification can involve manipulating the three-dimensional data representing the image containing the subject to distort or change the proportions of the subject. For example, the user interface can be configured to allow a user to manually stretch, pull, push or bend the image or portions of the image. The resulting image can then be transmitted or stored.

[0025] As used herein, the subject of an image generally refers to a person or thing depicted in the image that is distinct from the background of the image. According to aspects of the present disclosure, a user can easily create an avatar with the user's likeness, for example, using an image of the user's face as a starting point for customization. In a similar manner, a user can create a customized avatar for a virtual character, virtual object, or even a virtual animal or plant using an image with a corresponding object, person, animal, or plant as the subject.

[0026] Figure 1 1 is a pictorial diagram of an example of a method for image-based avatar customization using facial images according to aspects of the present disclosure. Here, a user 103 captures an image 104 of his face. In the illustrated implementation, the image 104 is captured using a camera 102 integrated into a mobile device 101. The image may be a single photograph, a frame from a video, or a composite image. In some implementations, the camera 102 may store the image 104 in digital form. Alternatively, the camera may generate a non-digital image that may be converted to digital form, for example, by a scanner (not shown). The digital data corresponding to the image 104 is then analyzed by a neural network that is trained with a machine learning algorithm and configured to determine feature parameters within the image. Here, the feature parameters characterize facial landmarks and may include, but are not limited to, eye width 107, eye shape, eye spacing, eye color, eye size 113, nose width 109, nose length 114, mouth shape, mouth position, mouth width 108, beard shape, beard position, beard width 110, beard length 117, hair shape, hair size, hair color, hair texture, hairline position 111, head shape, head length 115, head width, jaw shape 116, or any combination thereof or the like. In some implementations, the avatar customization system may present the user with a dashboard of other customization options, including accessories such as earrings, necklaces, rings, piercings, tattoos, hats, sunglasses, etc., in addition to clothing styles or colors.

[0027] A blank virtual model 105 of a face may be selected, generated, or pre-generated from a database. If, for example but not limited to, a blank virtual model is selected from a database, one or more of the feature parameters such as head shape, head length 115, head width, and jaw shape 116 may be set to initial values. The blank virtual model 105 may have other common feature parameter values. For example but not limited to, the values ​​of the feature parameters characterizing the facial feature points of the blank model may be average values ​​of the adult population or generated according to the golden ratio. The feature parameters determined from the image 104 may then be applied to the blank virtual model 105 to create a customizable virtual model. To facilitate further customization, the avatar customization system may provide a user interface UI with sliders 106 that allow the user to adjust the feature parameter values ​​of the customized virtual model according to the user's preferences. The starting point of one or more of the sliders may correspond to the feature parameter values ​​determined from the image. This will allow the user to more easily customize the virtual model according to his or her own preferences. Once the feature parameters are applied to the model and (optionally) the user modifies the customized virtual model, the virtual model may be finalized to have the feature parameters determined from the image 104 and / or the feature parameters modified by the user to generate the final model 112. The finalization process may apply the virtual model to an avatar in an application or generate an image with the three-dimensional final model as its subject.

[0028] Figure 2 2 is a flow chart depicting a method for image-based avatar customization using facial images according to aspects of the present disclosure. As shown, an image of a face is initially provided to the system, as indicated at 202. The image may be digitized and facial features may be identified from digital data corresponding to the image at 202, such as, but not limited to, the presence of face shape, head size, eye color, hair color, or hairstyle may be detected. From the identification of facial features, feature parameter values ​​are extracted at 203. As discussed above, the feature parameters may include facial feature points, such as eye width, eye shape, eye spacing, eye size, eye color, nose width, nose length, mouth shape, mouth position, mouth width, beard shape, beard position, beard width, beard length, hair shape, hair size, hairline position, hair color, hair texture, head shape, head length, head width, jaw shape, skin color, or any combination thereof or the like.

[0029] In this implementation, the system may access a database of pre-generated facial models. The model may have a variety of different preset facial features. The preset facial features may be facial features that are difficult to simply modify without changing the underlying structure of the model. For example, but not limited to, the preset facial features of the model may include face shape, jaw shape, head size, head width, head length, head shape, or any combination thereof. The pre-generated facial model may be selected from the database based on a subset of feature parameters and / or feature parameter values, as indicated at 204. In some optional implementations, the user may be prompted to confirm the selection of the pre-generated facial model or select a different model 207 that matches their preferences. This may allow the user to roughly modify the facial appearance of their avatar before fine-tuning other facial features in subsequent steps. It should be understood that in some alternative implementations, only a single facial model is available, and therefore that model will be selected as the base facial model. In yet other implementations, the base facial model may be generated using at least one feature parameter and / or feature parameter value determined from the image data.

[0030] Once the facial model is selected or generated, the model is modified based on facial feature parameters and / or facial feature parameter values, as indicated at 205. Each facial feature of the facial model may have a preset feature parameter associated therewith. Facial feature parameters and / or facial feature parameter values ​​obtained from an image of the face may be applied to the preset feature parameters to modify the appearance of the facial model. In addition, certain modifications may be made to the facial model by adding features to the model based on certain feature parameters determined from the image. By way of example (but not limitation), the added features may include a beard, a mustache, a hairstyle, one or more tattoos, one or more piercings, one or more accessories (e.g., glasses or sunglasses), one or more scars, one or more blemishes, one or more eyebrows, or any combination thereof. The feature parameters and / or feature parameter values ​​may be used to determine the location, size, shape, style, or any combination thereof of the added features. In some optional implementations, the user may be provided with an opportunity to change the feature parameters and / or feature parameter values ​​of the modified facial model, as indicated at 208. Sliders for each of the facial feature parameters may be provided to the user, for example, via a graphical user interface GUI, with the values ​​of the feature parameters determined from an image of the user as a starting point. The graphical user interface may also present an image corresponding to the facial model, which may change based on changes made by the user to the values ​​of the facial feature parameters using the sliders. This may allow the user to more easily understand the starting point for customization of their avatar's facial model and provide a simple anchor point for facial customization.

[0031] After the facial model is modified, the model may be completed or finalized, as indicated at 206. Finalizing the facial model may include, for example, but not limited to, applying the modified facial model to a body model or avatar, generating an image frame containing the modified facial model in a scene, or the like.

[0032] It should be understood that each feature parameter can have one or more corresponding feature parameter values ​​associated therewith to characterize the described feature. For example, if the subject in the image has blonde hair, then possible feature parameters for the subject's hair include, among others, hair color. In this case, the corresponding feature parameter value would be "blonde". In contrast, if the subject has no hair, then there is no feature parameter corresponding to hair color. Furthermore, as discussed above, the system can incorporate specific features into the model based on the presence of corresponding specific feature parameters determined from the image analysis. Additionally, the determination of the feature parameters can also include determining one or more of the corresponding feature parameter values ​​associated with the determined feature parameters. In some implementations, certain feature parameters may be necessary for the model. In this case, the feature parameter values ​​can be determined because the presence of the parameters themselves is predetermined.

[0033] Figure 3is a pictorial diagram of a method for image-based avatar customization using a body image according to aspects of the present disclosure. In this implementation, a user may take an image of their body 301. As shown, the user is using an imaging device 302 and a mirror 303 to create an image of their body. In some implementations, an image of the body may be simply provided to the system. Feature parameters and / or feature parameter values ​​are identified from the image 312 of the body. Here, an image of the body is taken using a mirror, and the imaging device is included in the image. In some implementations, the size of the imaging device may be used as an anchor point for determining a scale or feature parameter value in the image. Alternatively, another anchor point in the image may be selected for this purpose. For example, the user may be prompted to hold up a credit card or a piece of letter paper or other object of known size. The imaging device may be, for example, but not limited to, a mobile phone, a camera, a tablet PC with a camera, a webcam, a digital scanner, or a similar device. The imaging device may provide the image in digital form, or the avatar customization system may include a digital scanner appropriately configured for this purpose. Feature parameters and / or feature parameter values ​​may be identified from the digital image data by one or more neural networks trained with a machine learning algorithm and configured to determine body feature parameters and / or body feature parameter values ​​from an image of a body. Additionally, an image of a body may be analyzed by one or more neural networks trained with a machine learning algorithm and configured to detect one or more bodies present in an image. In some implementations, one or more additional neural networks may be used to determine facial feature parameters and / or facial feature parameter values, for example, as described above with respect to Figure 1 and Figure 2 discussed.

[0034] Here, the feature parameters are body feature parameters, and may include, by way of example but not limitation, height 304, shoulder width 305, leg length 306, arm length 309, arm width 307, hand width 308, etc. In addition, as discussed above, if an image of the face is not available, facial feature parameters may be extracted from an image of the body.

[0035] After the body feature parameters and / or body feature parameter values ​​are extracted, the body feature parameters and / or body feature parameter values ​​may be applied to the body model 310 to generate a modified body model. A graphical user interface GUI having a slider or knob 311 may be provided to the user, the slider or knob being configured to change the feature parameter values ​​of the modified model. The graphical user interface may also present an image corresponding to the body model, which may change according to the user's change of the value of the body feature parameter using the slider or knob 311. This allows the user to further customize the appearance of the body after modification. In addition, in some implementations, there may be a database of multiple unmodified body models, and the body feature parameter values ​​may be used to select the body model that best fits the parameters. For example, but not limited to, there may be a database with multiple unmodified models with different body types, such as mesomorph, ectomorph, and endomorph body models, the body feature parameters may be first analyzed to determine the body type according to the parameter values, and then the feature parameter values ​​may be applied to a body model having a body type that matches the body type identified according to the feature parameter values. In addition, certain modifications may be made to the body model by adding features to the model based on certain feature parameters determined from the image. By way of example (but not limitation), added features may include body hair, one or more tattoos, one or more scars, one or more freckles, one or more birthmarks, or any combination thereof.

[0036] Figure 4 4 is a flow chart depicting a method for image-based avatar customization using body images according to aspects of the present disclosure. First, an image 401 of a person's body is provided to an image-based customization system. The image may be an image of the user's body, or an image of another person's body. Image 401 may be, for example, but not limited to, an image of the person taken using a mirror, or an image of the person taken by another person, or an image of the person taken using a camera timer. Image 401 may be obtained in digital form, such as using a digital camera, or may be converted to such digital form, such as using a digital scanner.

[0037] At 402, feature parameters and / or feature parameter values ​​are extracted from digital image data corresponding to image 401 using one or more neural networks trained with a machine learning algorithm and configured to determine feature parameters and / or feature parameter values ​​from an image of a body. In some implementations, facial feature parameters and / or facial feature parameter values ​​may be extracted using one or more neural networks trained with a machine learning algorithm configured to determine facial feature parameters and / or facial feature parameter values ​​as discussed above. Additionally, object feature parameters and / or object feature parameter values ​​may be extracted from one or more objects in an image of a body using a neural network trained with a machine learning algorithm and configured to determine object feature parameters and / or object feature parameter values ​​from an image. In some implementations, an image of a body may be provided to one or more recognition neural networks trained with a machine learning algorithm to identify a body, face, and / or object that may be within the image, and then provided to one or more body feature parameter determination neural networks. The recognition neural network may determine the presence and location of a body, face, and / or object present in the image, which may be used to determine which portions of the image are provided to a body feature parameter determination network, a facial feature determination network, and / or an object feature determination network.

[0038] In the illustrated implementation, the body model is modified at 403 using the body feature parameters and / or the body feature parameter values ​​to generate a modified virtual body model of the avatar. Elements of the body model may be bound to the feature parameters, and when the feature parameter values ​​are adjusted, the corresponding elements of the body model are also adjusted. For example, but not limited to, a body feature parameter (such as arm length) may be bound to the length of the depicted arm element of the model. In some implementations, the body feature parameters may be bound to elements of the model that are not related to the body feature parameters. For example, a monster avatar may be created based on the body in the image by applying the body feature parameters to unrelated elements of the model, such as applying the arm length to the leg elements of the model or applying the chest length to the neck length, etc. In some other alternative implementations, completely different models (such as, but not limited to, models of dogs or cats) may apply the body feature parameters to elements, such as applying the arm length to the front leg length and applying the leg length to the back leg length, etc. In yet other implementations, a database of pre-generated virtual body models may be used together with the body feature parameters. For example, but not limited to, body feature parameters may be used to select a pre-generated virtual body model from a database, and in some implementations, proportions derived from the body features may be used to determine a body type, which may be used to select a virtual body model from a database.

[0039] The user may (optionally) further customize elements of the model after modifying the model using the body characteristic parameters 405. A graphical user interface may be provided to the user having sliders for each of the body characteristic parameters associated with the elements of the body model. The values ​​of the body characteristic parameters of the modified virtual body model may be used as starting values ​​for the sliders. The graphical user interface may also present an image corresponding to the virtual body model, which may change according to changes made by the user to the values ​​of the body characteristic parameters using the sliders.

[0040] After modifying the virtual body model, it is finalized, as shown at 404. Finalizing the virtual body model may include, for example, but not limited to, applying the modified facial model to the virtual body model or avatar, generating an image frame containing the modified virtual body model in the scene, or the like.

[0041] Figure 5is a drawing of a method for image-based virtual object customization using an object image according to aspects of the present disclosure. As shown, an image 501 of an object 502 may be provided to a system for image-based virtual object customization. The object 502 here is a vase, and the image is generated by an image capture device such as a mobile phone or a tablet computer having a camera. The image 501 may be obtained in digital form or converted to digital form for image analysis. One or more object recognition neural networks trained by a machine learning algorithm for identifying objects appearing in an image may be applied to the image (or digital data corresponding to the image) to determine the identity of the object 502 in the image 501. One or more object recognition neural networks may be trained from a database of labeled objects, and the one or more object neural networks generate labels for the object corresponding to one or more objects in the object database. In the illustrated implementation, the database includes three vase models 505, 506, 507. The one or more object recognition neural networks may determine that the object 502 in the image corresponds to the label 503 (large vase 002), and therefore select the model 506 corresponding to the large vase 002 in the database. In some implementations, one or more additional object parameter determination neural networks trained with a machine learning algorithm and configured to determine object feature parameters and / or object feature parameter values ​​from an image can be used with the image to generate object feature parameters and / or object feature parameter values. The object feature parameters can include object width, object height, object length, and / or similar parameters. In some implementations, the object feature parameters can be specifically used for objects identified within the image. By way of example (but not limitation), a trained neural network configured to determine one or more object feature parameters and / or object feature parameter values ​​can be trained to identify object feature parameters and / or object feature parameter values ​​for each object in a database. In addition, the object feature parameters can be used to modify the model to generate a modified model. The user can optionally customize the object feature parameters of the modified model. A graphical user interface with a slider or similar control can be provided to the user, which allows customization of the object feature parameter values, and the identified object feature parameter values ​​can be used as a starting setting. This can allow the user to have an easy starting point for customizing the appearance of the object. The graphical user interface can also present an image corresponding to the selected model 506, which can change according to the user's change of the value of the object feature parameter using the slider.

[0042] In addition, one or more textures, surface designs, and / or images can be captured from the surface of the object (as shown at 504) to generate a model surface decal. In some implementations, the output from one or more object recognition neural networks can be used to determine the location of the object. The portion of the image containing the object can then be copied and used as the model surface decal. In addition, in some implementations, the model surface decal can be modified to improve its appearance, for example, large black areas corresponding to shadows in the image can be removed.

[0043] Finally, the model surface decal can be applied 509 to the model 508 and the model can be finalized. To apply the decal to the model, the decal can be distorted to wrap the decal around the model surface.

[0044] Decals can be applied to models using, for example, conventional three-dimensional (3D) computer graphics. Specifically, a reprojection technique can be used to convert a three-dimensional model into one or more two-dimensional (2D) images in a vector format. The computer graphics system can then use rasterization to convert the two-dimensional image vector format, which can be converted into pixels for output on a video display or printer. Each pixel can be characterized by a position (e.g., in terms of vertical and horizontal coordinates) and a value corresponding to the intensity of the different colors that make up the pixel. Vector graphics represent images by using geometric objects such as curves and polygons. Some simple 3D rendering engines transform the object surface into a triangle mesh, and then rasterize the triangles in the depth order in the 3D scene.

[0045] Scanline algorithms are commonly used to rasterize polygons. Scanline algorithms overlay a grid of evenly spaced horizontal lines on a polygon. On each line, where there are consecutive pairs of polygon intersections, a horizontal line of pixels is drawn to the output device. Together, these lines cover the entire area of ​​the polygon with pixels on the output device. Decals can take the form of textures, which computer graphics can "draw" onto polygons. In this case, each pixel value drawn by the output device is determined by one or more pixels sampled from the texture. As used herein, a bitmap generally refers to a data file or structure that represents a roughly rectangular grid of pixels or color points on a computer monitor, paper, or other display device. The color of each pixel is defined individually. For example, a color pixel can be defined by three bytes - one byte each for red, green, and blue. A bitmap usually corresponds bit by bit to an image displayed on the screen, and its format may be the same as the format stored in the video memory of the display, or it may be a device-independent bitmap. A bitmap is characterized by the width and height of the image (in pixels) and the number of bits per pixel, which determines the number of colors it can represent.

[0046] The process of transferring a texture bitmap to a surface usually involves the use of texture MIP mapping (also known as mipmaps). These mipmaps are pre-computed, optimized collections of bitmap images that accompany the main texture, designed to increase rendering speed and reduce artifacts. They are widely used in 3D computer games, flight simulators, and other 3D imaging systems. The technique is called mipmapping. The "MIP" letters in the name are an acronym for the Latin phrase multumin parvo, which means "a lot in a small space."

[0047] Each bitmap image in a mipmap set is a version of the main texture, but with a reduced level of detail. Although the main texture is still used when the view is large enough to render full detail, when the texture is viewed from a distance or at a small size, the graphics device (usually called a renderer) that renders the final image switches to the appropriate mipmap image (or actually interpolates between the two nearest images). Because the number of texture pixels ("texels") processed can be much smaller than with a simple texture, rendering speed is increased. Because the mipmap images are actually already anti-aliased, artifacts can be reduced, alleviating some of the burden on real-time renderers.

[0048] If a texture has a base size of 256x 256 pixels (textures are usually square, and the side length must be equal to a power of 2), then the associated mipmap set can contain a series of 8 images, each half the size of the previous one: 128×128 pixels, 64×64, 32×32, 16×16, 8×8, 4×4, 2×2, 1×1 (a single pixel). For example, if the scene renders this texture in a 40×40 pixel space, an interpolation of the 64×64 and 32×32 mipmaps will be used. The simplest way to generate these textures is by successive averaging, but more complex algorithms (perhaps based on signal processing and Fourier transforms) are also possible. The storage space required to store all these mipmaps of the texture increases by one third, because the sum of the areas 1 / 4+1 / 16+1 / 256+... converges to 1 / 3. (This assumes that no compression is used.)

[0049] Blending between mipmap levels usually involves some form of texture filtering. As used herein, texture filtering refers to the method used to map texels (pixels of a texture) to points on a 3D object. A simple texture filtering algorithm can take a point on an object and find the texel closest to that location. The resulting point will then get its color from that one texel. This simple technique is sometimes called nearest neighbor filtering. More complex techniques combine more than one texel per point. Common algorithms include bilinear filtering and trilinear filtering with mipmaps. Anisotropic filtering and higher-order methods (such as quadratic or cubic filtering) can be used for even higher quality images.

[0050] Figure 6 601 is a flow chart illustrating a method for image-based virtual object customization using an object image according to aspects of the present disclosure. In the illustrated implementation, an image includes an object 601 as the subject of the image. The image is provided to a neural network, which is trained with a machine learning algorithm, and the machine learning algorithm is configured to recognize objects in the image. The neural network can recognize one or more objects 602 within the image. In addition, object feature parameters and / or object feature parameter values ​​can be determined from the image using one or more neural networks, which are trained with a machine learning algorithm to determine the object parameters and / or object feature parameter values.

[0051] One or more textures, designs, or surface images may then be unwrapped from the object 603. To unwrap one or more textures, designs, or images from the surface of the object, a portion of the image containing the object may be extracted, and background or other objects within the portion of the image may be removed to generate a model surface decal, e.g., in the form of one or more textures. The identity of the object identified at 602 may be used to further refine the model, e.g., the model shape identified from the object may be used to more accurately select one or more textures, designs, or surface images on the surface of the object.

[0052] Based on the identity of the object, a virtual model 604 can be selected from a database of virtual models of the object. One or more neural networks can be trained based on the database of virtual models, and in this way, the identity of the object can correspond to the identity of one of the objects in the database. Therefore, a virtual model with an identity corresponding to the identity of the object in the database can be selected. User input 607 can optionally be used to select different models from the database. Alternatively, in some implementations, user input can be optionally used to customize the virtual model. Object feature parameters can be used, for example but not limited to, first to modify the model, and then the user can be provided with the option of further customizing the model based on the modified feature parameters. Sliders or knobs can be used to customize the object feature parameters, and the determined object feature parameter values ​​are used as the initial values ​​of the knobs or sliders before user customization. The graphical user interface can also present an image corresponding to the virtual body model, which can change according to the user's change of the value of the object feature parameter using the slider.

[0053] After selecting and optionally customizing a virtual model, a model surface decal may be applied to the surface of the virtual model 605, for example, using textures as discussed above. The model surface decal may be modified to fit the model based on, for example, object feature parameters.

[0054] Once the virtual object model has had the model surface decals digitally applied to the model surface, it is finalized, as shown at 606. Finalizing the virtual object model may include, for example, but not limited to, applying the virtual object model to a virtual body model or avatar, generating an image frame containing the virtual object model in a scene, or the like. For example, but not limited to, the virtual object model may be Figure 5 The vase shown, and the virtual object model can be placed in the scene with the body model holding the vase.

[0055] Figure 7 is a pictorial diagram of another method for image-based virtual object customization using object images according to aspects of the present disclosure. In the illustrated implementation, one or more images 702 of an object 701 are taken. Here, the object 701 is a pyramid.

[0056] One or more neural networks trained with a machine learning algorithm and configured to recognize primitive shapes of objects appearing within one or more images can be used to recognize primitive shapes in one or more images 703. The primitive shapes here are recognized as pyramids. One or more neural networks can be used to recognize prototypes using a single image or multiple images. Once a primitive shape is recognized, it can be generated or selected from a database of pre-generated prototypes.

[0057] In addition, multiple images of the object can be used to generate a model surface decal. Images taken from a first angle 704, a second angle 705, and a third angle 706 of the object are shown here. Multi-angle images of the object can be used to assemble a complex model surface decal and determine object feature parameters. One or more neural networks trained with a machine learning algorithm to determine object feature parameters and / or object feature parameter values ​​from multiple images can be used to generate feature parameters and / or object feature parameters that can be applied to the original shape.

[0058] Once the model surface decals and object feature parameters and / or object feature parameter values ​​are determined, they can be applied to the prototype to create a modified prototype 707, thereby generating a virtual object. Each prototype may include elements bound to different feature parameters. For example, but not limited to, in the example of the pyramid, the elements of the bottom length, bottom width, and height can each be an element of the virtual model bound to the corresponding feature parameters of the object bottom length, object bottom width, and object height. After the virtual object is generated, it can be finalized. Finalizing the virtual object can include, for example, but not limited to, applying the virtual object model to a virtual body model or avatar, generating an image frame containing the virtual object model in a scene, or similar actions. For example, but not limited to, the virtual object can be a pyramid building, and the virtual object can be placed in a scene with a body model next to the pyramid building.

[0059] Figure 8 801 is a flow chart illustrating another method for image-based virtual object customization using an object image according to aspects of the present disclosure. Here, a plurality of images having at least one object as a subject are provided to the system. A prototype is identified from one or more of the plurality of images 802 using one or more neural networks, the one or more neural networks being trained with a machine learning algorithm, the machine learning algorithm being configured to identify the primitive shapes of the objects in the one or more images. In some implementations, before identifying the primitive shapes of the objects, the one or more neural networks trained with a machine learning algorithm to recognize objects appearing in the image may select portions of the image to provide to one or more primitive shape determination networks. Once the primitive shape is identified, it may be generated as a virtual model, or selected from a database or pre-generated prototype models. 803 User input 807 may be used to refine the selected base prototype or select a better prototype to fit the object in the image.

[0060] After a base prototype is selected, a surface decal made from multiple views of the object may be applied to the prototype's surface 804. The surface decal may be created by selecting portions of an image having the prototype and stitching together the portions of the image of the object's surface.

[0061] Once the surface decal is applied to the prototype, the prototype can be modified 805 to better fit the surface decal. For example, but not limited to, the size of the prototype can be changed so that the surface decal covers the entire surface of the prototype. The user can be provided with the ability to further customize the prototype so that the prototype better fits the object shape.

[0062] Finally, the prototype may be finalized using the surface decals and the modified original shape at 806. Finalizing the modified prototype may include, for example, but not limited to, applying the modified prototype to a virtual body model or avatar, generating an image frame containing a virtual object model in a scene, or the like. For example, but not limited to, the modified prototype may be Figure 7 The pyramid shown, and the virtual prototype can be placed in the scene with the body model standing next to it.

[0063] General Neural Network Training

[0064] According to various aspects of the present disclosure, the image-based customization system can use machine learning and neural networks (NN). For example, the recognition and parameter determination steps discussed above can use machine learning as discussed below. The machine learning algorithm can use a training data set, which can include an image with a face as the main body, and the image has a label for facial feature parameters (such as facial feature points) or facial features (such as eye color, hair color, beard color, etc.). The label for the facial feature parameter can include a label for the facial feature itself, and in some implementations can include one or more labels for facial feature parameter values ​​associated with the facial feature. In some implementations, the training data set can include an image with a body as the main body and a label for body feature parameters (such as body measurements) and other body features (such as skin color, scar location, or hair location). The label for the body feature parameter can include a label for the body feature itself, and in some implementations can include one or more labels for body feature parameter values ​​associated with the body feature. In yet other implementations, the training data set can have an image that includes one or more objects as the main body and has labels for object identity and / or object parameters (such as length, width, and height). The labels for object feature parameters may include labels for the object features themselves, and in some implementations may include one or more labels for object feature parameter values ​​associated with the object features. In other implementations, the training data may include labels for objects and shape prototypes corresponding to the object shapes. In further implementations, the training set may include any combination of images with labels, bodies, faces, prototypes, and objects. During training, the labels of the training set may be masked.

[0065] The NN may include one or more of several different types of neural networks and may have many different layers. By way of example and not limitation, the neural network may be composed of one or more convolutional neural networks (CNNs), recurrent neural networks (RNNs), and / or dynamic neural networks (DNNs). The motion decision neural network may be trained using the general training methods disclosed herein.

[0066] By way of example, and not limitation, Fig. 9A The basic form of an RNN that can be used, for example, in a trained model is depicted. In the example shown, the RNN has a layer of nodes 920, each of which is characterized by an activation function S, an input weight U, a recurrent hidden node transition weight W, and an output transition weight V. The activation function S can be any nonlinear function known in the art and is not limited to (hyperbolic tangent (tanh) function. For example, the activation function S may be a Sigmoid or ReLu function. Unlike other types of neural networks, the RNN has a set of activation functions and weights across the layer. As shown in FIG. Fig. 9B As shown, the RNN can be considered as a series of nodes 920 with the same activation function over time T and T + 1. Therefore, the RNN maintains historical information by feeding the results from the previous time T to the current time T + 1.

[0067] In some implementations, a convolutional RNN can be used. Another type of RNN that can be used is a long short-term memory (LSTM) neural network, which adds a memory block to an RNN node with an input gate activation function, an output gate activation function, and a forget gate activation function, thereby generating a gated memory that allows the network to retain some information for a longer period of time, as described in Hochreiter and Schmidhuber, "Long Short-term memory" Neural Computation 9(8): 1735-1780 (1997), which is incorporated herein by reference.

[0068] Fig. 9C An example layout of a convolutional neural network (such as a CRNN) that can be used, such as in a trained model, in accordance with aspects of the present disclosure is depicted. In this depiction, a convolutional neural network is generated for an input 932 having a size of 4 units in height and 4 units in width giving a total area of ​​16 units. The depicted convolutional neural network has a filter 933 of size 2 units in height and 2 units in width with a skip value of 1 and a channel 936 of size 9. For clarity, the 932 images are shown in Figure 10. Fig. 9COnly the connection 934 between the first column of channels and their filter windows is depicted. However, aspects of the present disclosure are not limited to such implementations. According to aspects of the present disclosure, the convolutional neural network can have any number of additional neural network node layers 931 and can include layer types of any size such as additional convolutional layers, fully connected layers, pooling layers, maximum pooling layers, local contrast normalization layers, etc.

[0069] As in Fig.9D As seen in FIG. 1 , training a neural network (NN) begins by initializing the weights of the NN at 941. Generally, the initial weights should be randomly distributed. For example, a NN with a tanh activation function should have weights distributed in and A random value between , where n is the number of inputs to the node.

[0070] After initialization, the activation function and optimizer are defined. Then, at 942, a feature vector or input data set is provided to the NN. Each of the different feature vectors can be generated by the NN from an input with a known label. Similarly, a feature vector corresponding to an input with a known label or classification can be provided to the NN. Then, at 943, the NN predicts the label or classification of the feature or input. At 944, the predicted label or classification is compared with the known label or classification (also referred to as the ground truth), and the loss function measures the total error between the prediction and the ground truth on all training samples. As an example and not limitation, the loss function can be a cross entropy loss function, a quadratic cost function, a triple contrast function, an exponential cost function, etc. A plurality of different loss functions can be used according to the purpose. As an example and not limitation, for training a classifier, a cross entropy loss function can be used, while for learning pre-trained embeddings, a triple contrast function can be used. Then, the result of the loss function is used and the NN is optimized and trained using a known method for training a neural network (such as back propagation with adaptive gradient descent, etc.), as indicated at 945. In each training epoch, the optimizer attempts to select model parameters (ie, weights) that minimize the training loss function (ie, total error). The data is divided into training, validation, and test samples.

[0071] During training, the optimizer minimizes the loss function on the training samples. After each training epoch, the model is evaluated on the validation samples by calculating the validation loss and accuracy. If there is no significant change, the training can be stopped and the resulting trained model can be used to predict the labels of the test data.

[0072] Thus, a neural network can be trained to recognize and classify inputs with known labels or classifications. Similarly, the described methods can be used to train a NN to generate feature vectors from inputs with known labels or classifications. Although the above discussion is related to RNNs and CRNNS, these discussions can be applied to NNs that do not include loops or hidden layers.

[0073] system

[0074] Fig.10 1 is a system block diagram of a system for image-based virtual model customization according to aspects of the present disclosure. By way of example and not limitation, according to aspects of the present disclosure, the system 1000 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a gaming console, and the like.

[0075] The system 1000 generally includes a central processor unit (CPU) 1003 and a memory 1004. The system 1000 may also include well-known support functions 1006, which may communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, a power supply (P / S) 1011, a clock (CLK) 1012, and a cache 1013.

[0076] The system 1000 may include a display device for presenting rendered graphics to a user. In an alternative implementation, the display device is a separate component that works in conjunction with the system 1000. The display device may be in the form of a flat panel display, a head mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other devices that can display visible text, numbers, graphic symbols, or images. In addition, the system 1000 may optionally include an image generator 1024. The image generator 1024 may generate one or more images 1009, which may be stored in the memory 1004. The image generator 1024 may be, for example, but not limited to, a camera, a video camera, a camera phone, a smart phone, a tablet personal computer with a camera, a webcam, or the like.

[0077] System 1000 includes a mass storage device 1015, such as a disk drive, CD-ROM drive, flash memory, solid state drive (SSD), tape drive, etc., to provide non-volatile storage for programs and / or data. System 1000 may also optionally include a user interface unit 1016 to facilitate interaction between system 1000 and a user. User interface 1016 may include a keyboard, mouse, joystick, light pen, game controller, or other devices that can be used in conjunction with a graphical user interface (GUI). System 1000 may also include a network interface 1014 to enable the device to communicate with other devices via a network 1020. Network 1020 may be, for example, a local area network (LAN), a wide area network (such as the Internet), a personal area network (such as a Bluetooth network), or other types of networks. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

[0078] The CPUs 1003 may each include one or more processor cores, such as a single core, dual cores, quad cores, octa cores, or more. In some implementations, the CPUs 1003 may include a GPU core or multiple cores of the same accelerated processing unit (APU). The memory 1004 may be in the form of an integrated circuit that provides addressable memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.). The main memory 1004 may include application data 1023 used by the CPU 1003 when processing. The main memory 1004 may also include an image 1009 received from the image generator 1024. A trained neural network (NN) 1010 may be loaded into the memory 1004 for determining feature parameters and body, face, or object recognition, such as with respect to Figure 1-Figure 8 As discussed. In addition, the memory 1004 may include a machine learning algorithm 1021 for training or adjusting the NN 1010. A database 1022 may be included in the memory 1004. Depending on the desired implementation, the database may contain virtual models of faces, bodies, objects, shape prototypes, etc. The memory may also contain a modified model 1008 generated from the database and feature parameters. The modified model may be further customized using input from a user received through the user interface 1016. The user interface 1016 may include a slider or knob for further customizing the modified model 1008 before finalization, and the feature parameter value may be the initial value of the slider to provide a consistent reference starting point for model modification.

[0079] According to aspects of the present disclosure, the CPU 1003 may execute the following Figure 2 , Figure 4 , Figure 6 and Figure 8The methods discussed herein for customizing a virtual model based on an image may be loaded into the memory 1004 as an application 1023. The processor may generate a modified model as an executable Figure 2 , Figure 4 , Figure 6 and Figure 8 Results of the method described in .

[0080] The mass storage 1015 may contain applications or programs 1017 that are loaded into the main memory 1004 when processing begins on the application 1023. In addition, the mass storage 1015 may contain data 1018 used by the processor during processing of the application 1023, the NN 1010, the machine learning algorithm 1021, and the population of the database 1022.

[0081] As used herein and generally understood by those skilled in the art, an application specific integrated circuit (ASIC) is an integrated circuit that is customized for a specific purpose and is not intended for general use.

[0082] As used herein and generally understood by those skilled in the art, a field programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or designer after manufacture (hence "field programmable"). FPGA configuration is typically specified using a hardware description language (HDL) similar to that used for ASICs.

[0083] As used herein and generally understood by those skilled in the art, a system on a chip or system on a chip (SoC or SOC) is an integrated circuit (IC) that integrates all the components of a computer or other electronic system into a single chip. It can contain digital, analog, mixed signal and usually radio frequency functions - all on a single chip substrate. Typical applications are in the field of embedded systems.

[0084] A typical SoC includes the following hardware components:

[0085] One or more processor cores (e.g., microcontroller, microprocessor, or digital signal processor (DSP) cores.

[0086] Memory blocks such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0087] A timing source, such as an oscillator or a phase-locked loop.

[0088] Peripherals such as counter-timers, real-time timers, or power-on reset generators.

[0089] External interfaces, for example, industry standards such as Universal Serial Bus (USB), FireWire, Ethernet, Universal Asynchronous Receiver / Transmitter (USART), Serial Peripheral Interface (SPI) bus.

[0090] Analog interfaces, including analog-to-digital converters (ADCs) and digital-to-analog converters (DACs).

[0091] Voltage regulators and power management circuits.

[0092] These components are connected by a proprietary or industry standard bus. A direct memory access (DMA) controller routes data directly between the external interface and the memory, bypassing the processor core and thereby increasing the data throughput of the SoC.

[0093] A typical SoC includes both the hardware components described above and executable instructions (eg, software or firmware) that control the processor core, peripherals, and interfaces.

[0094] Aspects of the present disclosure allow for great flexibility and ease of use in customizing avatars and digital assets. Users can more easily customize avatars and digital assets generated from digital images than creating custom avatars from scratch using generic templates. Digital cameras are widely available and easy to use. They are a common feature of many smartphones, laptops, and are often used as peripherals for game consoles. In addition, digital cameras and digital scanners can easily convert regular photos or other images into digital formats. By utilizing digital image creation, aspects of the present disclosure provide great flexibility and creative opportunities for customizing avatars and digital assets.

[0095] Although the above is a complete description of the preferred embodiment of the present invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the present invention should not be determined with reference to the above description, but should be determined with reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, Indefinite article "a" or "a kind" It refers to the quantity of one or more of the item following the article unless explicitly stated otherwise. The appended claims should not be read as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A method for image-based customization, comprising: a) extracting one or more characteristic parameters of a subject in a digital image using one or more neural networks, wherein the one or more neural networks are trained using a machine learning algorithm, and the machine learning algorithm is configured to determine the characteristic parameters of the subject; as well as b) applying the one or more characteristic parameters to the virtual model of the subject. The method of claim 1 , wherein the digital image corresponds to a still image. The method of claim 1 , wherein the digital image corresponds to a video frame. The method of claim 1 , wherein the digital image is a composite image generated from vertex data.

5. The method of claim 1, wherein the digital image is a composite image stitched together from two or more different images containing the subject. 6 . The method of claim 1 , wherein the subject in the digital image is a face of a user, and the one or more feature parameters of the subject include facial feature points.

7. The method of claim 1, wherein the one or more characteristic parameters include eye color, hair color, or hairstyle.

8. The method of claim 1, wherein the virtual model of the subject comprises a human face.

9. The method of claim 1, wherein the subject is a human body, and extracting one or more characteristic parameters comprises identifying a user's body and determining the user's body characteristic parameters.

10. The method according to claim 9, wherein the user's physical characteristic parameters include height, shoulder width, hip width, leg shape, arm length, leg length, arm shape or chest width.

11. The method of claim 9, wherein extracting one or more feature parameters further comprises recognizing a face of the user and determining facial feature parameters of the user. 12 . The method of claim 1 , wherein applying the one or more feature parameters to the model further comprises adjusting sliders of features of the model to match the feature parameters to generate an adjusted model.

13. The method of claim 12, wherein the adjusted model is a starting point for a user to adjust features of the model.

14. The method of claim 1, wherein applying the one or more feature parameters to a model comprises selecting the model from a database of potential models based on at least one of the feature parameters.

15. The method of claim 1, further comprising identifying the subject in the digital image using a neural network trained with a machine learning algorithm and configured to identify objects appearing within an image.

16. The method of claim 15, wherein identifying the subject in the digital image further comprises determining a primitive shape of the subject.

17. The method of claim 16, wherein the model is a primitive shape, and applying the one or more feature parameters to the model further comprises selecting the primitive shape of the model from a database of three-dimensional primitive shapes.

18. The method of claim 17, wherein applying the one or more feature parameters to the model further comprises modifying the model of the original shape according to at least one feature parameter to generate a modified prototype model.

19. The method of claim 18, wherein modifying the model of the original shape further comprises adjusting the modified prototype model using user adjustable feature parameter sliders.

20. The method of claim 1, wherein extracting the one or more characteristic parameters further comprises generating one or more textures, images, or designs from a surface of the subject.

21. The method of claim 20, wherein applying the one or more feature parameters to the model further comprises applying the one or more textures, images, or designs acquired from the surface of the subject to the model.

22. The method of claim 21, wherein applying the one or more textures, images, or designs to the surface of the model further comprises using one or more user adjustable sliders to fit the one or more textures, images, or designs onto the surface of the model.

23. A system for image-based customization, comprising: processor; a memory coupled to the processor; Non-transitory processor-executable instructions included in the memory, which, when executed by the processor, cause the processor to perform a method for image-based customization, comprising: a) extracting one or more characteristic parameters of a subject in a digital image using one or more neural networks, wherein the one or more neural networks are trained using a machine learning algorithm, and the machine learning algorithm is configured to determine the characteristic parameters of the subject; as well as b) applying the one or more characteristic parameters to a virtual model of the subject.

24. A computer-readable medium comprising non-transitory instructions, wherein the non-transitory instructions are configured to, when executed by a computer, cause the computer to perform a method for image-based customization, the method comprising: a) extracting one or more characteristic parameters of a subject in a digital image using one or more neural networks, wherein the one or more neural networks are trained using a machine learning algorithm, and the machine learning algorithm is configured to determine the characteristic parameters of the subject; as well as b) applying the characteristic parameters to a virtual model of the subject.

Images