Generating a 3d object mesh from a 2d image based on a template

Abstract

The present application relates to a method, system and computer readable medium for generating game text labels. In some implementations, a computer-implemented method for generating a three-dimensional mesh of an object using a trained machine learning model includes providing a two-dimensional image of the object as input to the trained machine learning model, obtaining a template three-dimensional mesh, and generating, using the trained machine learning model, a 3D mesh of the object based on the two-dimensional image and the template three-dimensional mesh, wherein the 3D mesh of the object is obtained by re-meshing the template three-dimensional mesh and is usable for mapping textures or generating 3D animations of the object.

Description

[0001] Cross-reference of related patent applications

[0002] This application claims priority benefit from U.S. Provisional Application No. 63 / 015,391, entitled “A Unified Framework for Mesh Reconstruction and Texture,” filed April 24, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to computer-based evaluation and computer-based games, and more specifically to methods, systems, and computer-readable media for generating three-dimensional (3D) object meshes from corresponding two-dimensional (2D) images in a virtual environment. Background Technology

[0004] Traditional standardized cognitive assessments primarily evaluate content mastery or domain knowledge, processing speed, and memory. The College Board, founded in 1923, established a set of college admission standards through the Scholastic Aptitude Test (SAT). In 1959, the American College Test (ACT) was introduced as an alternative to the SAT. Both the ACT and SAT focus on standardized content in mathematics, writing, science, and other subject-specific areas to create objective metrics. While widely used in the United States, these assessments reveal little about an individual's specific cognitive abilities or anticipated college performance.

[0005] To address the methodological and content shortcomings of traditional standardized college admissions tests, employers have adopted other conventional cognitive or intelligence tests in an effort to gather more predictive insights into applicants' cognitive abilities. These assessments focus on content mastery or domain knowledge, processing speed, memory, and the skills required for a 21st-century workforce. Assessments may include visual representations of real-life objects such as vehicles, airplanes, and biological assets to establish a baseline scenario for the assessee.

[0006] Some online platforms (such as gaming platforms and media communication platforms) allow users to connect and interact with each other via the internet (e.g., in games), create games, and share information. Users of online gaming platforms can participate in multiplayer game environments or virtual environments (such as 3D environments), design custom game environments, design characters and mechanics, exchange virtual items / objects with other users, communicate with other users using audio or text messages, and so on. Users can interact with objects in the virtual environment and can simulate objects within the virtual environment. Object simulation in the virtual environment can utilize various object representations, which can then be displayed on one or more user devices. Based on this, several implementation schemes are proposed. Summary of the Invention

[0007] A system includes one or more computers configured to perform specific operations or actions, utilizing software, firmware, hardware, or a combination thereof running on the system, and causing or making the system perform actions during runtime. One or more computer programs may be configured to perform specific operations or actions, utilizing instructions that, when executed by a data processing device, cause the device to perform actions. One possible implementation of this application includes a computer-implemented method for generating a 3D mesh of an object using a trained machine learning model. This computer-implemented method includes providing a two-dimensional 2D image of the object as input to the trained machine learning model; obtaining a template 3D 3D mesh; and generating a 3D mesh for the object using the trained machine learning model and based on the 2D image and the template 3D mesh, wherein the 3D mesh for the object can be used to map textures or generate 3D animations of the object. Other embodiments of this implementation include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0008] The implementation includes one or more of the following features. The computer implementation method includes displaying a user interface including the 3D mesh for the object. The 3D mesh for the object is obtained by re-meshing the template 3D mesh using the trained machine learning model. The computer implementation method further includes determining the category of the object based on the 2D image using image matching before obtaining the template 3D mesh. The computer implementation method further includes mapping the texture to the 3D mesh. The template 3D mesh is a skeletal binding including a plurality of vertices of the 3D mesh, and each of the plurality of vertices is connected to at least one other vertex of the plurality of vertices, and further includes: obtaining one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices of the plurality of vertices; and generating at least one object animation for the object based on the 3D mesh and a given template animation among the one or more template animations associated with the template 3D mesh, wherein the at least one object animation includes the motion of the same two or more vertices as the given template animation among the one or more template animations. The machine learning model includes a mesh encoder, an image encoder, and multiple mesh decoders, and further includes: receiving multiple target meshes; and training the multiple mesh decoders, wherein each mesh decoder corresponds to a given template mesh, wherein the training includes: encoding, using the mesh encoder, the target meshes are converted into shape space vectors; decoding, using the mesh decoders and based on the given template network, the shape space vectors are converted into meshes after mesh re-division; determining a first value of a first error function based on a comparison between the target meshes and the meshes after mesh re-division; and updating one or more parameters of the mesh decoders based on the first value. The machine learning model includes a mesh encoder, an image encoder, and multiple mesh decoders. It also includes training the image encoder, wherein the training includes: receiving multiple 2D training images and corresponding target meshes, wherein each image depicts a given object of a corresponding category and the category is associated with a corresponding template mesh; generating, using the image encoder, corresponding shape space vectors for each of the multiple 2D training images; comparing each generated shape space vector with a corresponding shape space vector for the corresponding target mesh, wherein the corresponding shape space vector associated with the target mesh is encoded using the mesh encoder; determining a second value of a second error function based on the comparison between the generated shape space vector and the corresponding shape space vector; and updating one or more parameters of the image encoder based on the second value. The mapping is used to apply the texture to the 3D mesh of the object. The UV regressor is trained using a loss function including a descriptor loss based on chamfer distance. The template 3D mesh is a homeomorphic template mesh.Implementation may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0009] One possible implementation of this application includes a system. The system further includes a memory storing instructions thereon; and a processing device coupled to the memory, the processing device being configured to access the memory, wherein the instructions, when executed by the processing device, cause the processing device to perform operations including: providing a two-dimensional 2D image of the object as input to a trained machine learning model; obtaining a template 3D mesh; and generating a 3D mesh for the object, utilizing the trained machine learning model and based on the 2D image and the template 3D mesh, wherein the 3D mesh for the object can be used to map textures or generate 3D animations of the object. Other embodiments of this implementation include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0010] The implementation includes one or more of the following features. The system wherein the 3D mesh for the object is obtained by re-meshing the template 3D mesh using the trained machine learning model. The template 3D mesh is a skeletal binding comprising a plurality of vertices of the 3D mesh, and each of the plurality of vertices is connected to at least one other vertex among the plurality of vertices. The operation further includes: obtaining one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices among the plurality of vertices; and generating at least one object animation for the object based on the 3D mesh and a given template animation among the one or more template animations associated with the template 3D mesh, wherein the at least one object animation includes the motion of two or more vertices identical to the given template animation among the one or more template animations. The template 3D mesh is a homeomorphic template mesh. The machine learning model includes a mesh encoder, an image encoder, and multiple mesh decoders. The operation further includes: receiving multiple target meshes; and training the multiple mesh decoders, each mesh decoder corresponding to a given template mesh, wherein the training includes: encoding, using the mesh encoder, the target meshes into shape space vectors; decoding, using the mesh decoders and based on the given template mesh, the shape space vectors into meshes after re-partitioning; determining a first value of a first error function based on a comparison between the target meshes and the meshes after re-partitioning; and updating one or more parameters of the mesh decoders based on the first value; and training the image encoder, including: receiving multiple 2D training images and corresponding target meshes, wherein each image depicts a given object of a corresponding category and the category is associated with a corresponding template mesh; generating, using the image encoder, corresponding shape space vectors for each of the multiple 2D training images; comparing each generated shape space vector with a corresponding shape space vector for the corresponding target mesh, wherein the corresponding shape space vector associated with the target mesh is encoded using the mesh encoder; determining a second value of a second error function based on a comparison between the generated shape space vectors and the corresponding shape space vectors; and updating one or more parameters of the image encoder based on the second value. Implementation may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0011] One possible implementation of this application includes a non-transitory computer-readable storage medium. The computer-readable storage medium stores computer instructions that, when executed on a processing device, cause the processing device to perform operations including: providing a two-dimensional 2D image of the object as input to a trained machine learning model; obtaining a template 3D mesh; and generating a 3D mesh for the object using the trained machine learning model and based on the 2D image and the template 3D mesh, wherein the 3D mesh for the object can be used to map textures or generate 3D animations of the object. Other embodiments of this implementation include a corresponding computer system, device, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0012] The implementation includes one or more of the following features. The non-transitory computer-readable storage medium wherein the template 3D mesh is a homeomorphic template mesh. The template 3D mesh is a skeletal binding comprising a plurality of vertices of the 3D mesh, and each of the plurality of vertices is connected to at least one other vertex of the plurality of vertices. The operation further includes: obtaining one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices of the plurality of vertices; and generating at least one object animation for the object based on the 3D mesh and a given template animation among the one or more template animations associated with the template 3D mesh, wherein the at least one object animation includes the motion of two or more vertices identical to the given template animation among the one or more template animations. The operation further includes: processing, using a UV regressor, the 2D image of the object and the 3D mesh to generate a mapping from the vertices of the 3D mesh of the object to the 2D image, wherein the mapping is used to apply the texture to the 3D mesh of the object. Implementations may include hardware, methods, or processes, or computer software on a computer-accessible medium.

[0013] One possible implementation of this application includes a method for training a machine learning model comprising a mesh encoder, an image encoder, and a plurality of mesh decoders to generate a 3D mesh from a 2D image of an object. The method includes: receiving a plurality of target meshes; and training the plurality of mesh decoders, wherein each mesh decoder corresponds to a given template mesh, wherein the training includes: encoding, using the mesh encoder, the target meshes into shape space vectors; decoding, using the mesh decoders and based on the given template network corresponding to the mesh decoders, the shape space vectors into a mesh re-divided mesh; determining a first value of a first error function based on a comparison between the target meshes and the mesh re-divided meshes; and updating one or more parameters of the mesh decoders based on the first value. Other embodiments of this implementation include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0014] The implementation includes one or more of the following features. Each of the plurality of mesh decoders in the method is pre-trained to approximate an identity function on the corresponding template mesh. The first error function is a fitness function used to indicate the distance between the re-divided mesh and the target mesh. Implementations may include hardware, methods, or processes, or computer software on a computer-accessible medium.

[0015] One possible implementation of this application includes a method for training a machine learning model comprising a mesh encoder, an image encoder, and a plurality of mesh decoders to generate a 3D mesh from a 2D image of an object. The method includes: receiving a plurality of 2D training images and corresponding target meshes, wherein each image depicts a given object of a corresponding category and the category is associated with a corresponding template mesh; generating, using the image encoder, corresponding shape space vectors for each of the plurality of 2D training images; comparing each generated shape space vector with a corresponding shape space vector for the corresponding target mesh, wherein the corresponding shape space vector associated with the target mesh is encoded using the mesh encoder; determining a second value of a second error function based on the comparison between the generated shape space vector and the corresponding shape space vector; and updating one or more parameters of the image encoder based on the second value. Other embodiments of this implementation include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0016] The implementation includes one or more of the following features. In the method, the second error function is a fitness function used to indicate the distance between the generated shape space vector and the corresponding shape space vector. Implementations may include hardware, methods, or processes, or computer software on a computer-accessible medium. Attached Figure Description

[0017] Figure 1 A schematic diagram of an example network environment for computer-based evaluation provided in an embodiment of this application is shown.

[0018] Figure 2 A schematic diagram of an example system architecture for generating 3D meshes of objects from 2D images, provided in an embodiment of this application, is shown.

[0019] Figure 3 An example method for generating a 3D mesh of objects from a 2D image using a trained machine learning (ML) model, as provided in an embodiment of this application, is illustrated.

[0020] Figure 4 A flowchart illustrating an example method for training an image encoder provided in an embodiment of this application is shown.

[0021] Figure 5A This illustration shows an example process of training a mesh decoder to generate a re-divided mesh, as provided in an embodiment of this application.

[0022] Figure 5B This illustration shows an example process of training an image encoder to generate shape space vectors according to an embodiment of this application.

[0023] Figure 6 An example of a 3D mesh generated from a corresponding 2D image, provided in an embodiment of this application, is shown.

[0024] Figure 7 An example of using segmented 2D images for classification, provided in an embodiment of this application, is shown.

[0025] Figure 8 An example of a 3D mesh of an object bound to a skeleton, as provided in an embodiment of this application, is shown.

[0026] Figure 9 An example evaluation framework provided by an embodiment of this application is shown.

[0027] Figure 10 A schematic diagram of an example computing device provided in an embodiment of this application is shown.

[0028] Figure 11 A schematic diagram of an example computing device provided in an embodiment of this application is shown. Detailed Implementation

[0029] In the following detailed description, reference is made to the accompanying drawings, which form part of this application. In the drawings, similar symbols generally identify similar components unless the context otherwise requires. The illustrative implementations described in the detailed description, drawings, and claims are not intended to be limiting. Other implementations and modifications may be used without departing from the spirit or scope of the subject matter described herein. Various aspects of this application, as generally described herein and illustrated in the drawings, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which should be understood as part of the scope of this application.

[0030] References to "some embodiments," "one implementation," "an example," etc., in the specification indicate that the described implementation may include a given feature, structure, or characteristic, but not every implementation necessarily includes the given feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same implementation. Additionally, when a given feature, structure, or characteristic is described in connection with an implementation, that feature, structure, or characteristic may be associated with other implementations, whether explicitly described or not.

[0031] In some respects, simulation-based assessments offer systems and methods that focus on evaluating how a person thinks, rather than what they know. Scenarios or tasks can be embedded in simulation-based assessments, abstracting the context of a given environment (e.g., a work environment) while preserving the user's opportunity to describe the problem-solving abilities required for the task. Through the scenarios that occur in simulation-based assessments, details of the user's cognitive processes can be observed, not just their final choices. Simulation-based assessments may consequently underestimate the high importance placed on memory and processing speed by traditional cognitive assessments such as AT and ACT.

[0032] In some embodiments, because skills like “critical thinking” can occur in many environments requiring different modes and intensities, simulation-based assessments can bring a dynamic, real-time adaptive, and / or highly realistic environment to life, reflecting real-world scenarios. These scenarios provide candidates with rich narratives that can elicit complex chains of reasoning and robust mental models. Creating realistic scenarios may require abstracting daily work activities and contextual elements containing real-world conditions, assessing efforts to predict future outcomes. To break down daily activities into components that build the task, granular descriptors called “skeletons” can be used as the atomic units behind task design. Skeletons can include verbs, objects, and adjectives or adjective phrases.

[0033] In some embodiments, designing a scenario may require assembling a series of skeletons that are theoretically plausible, meaning their sequence provides insight into the structure of interest and is plausible beneath the abstract environment in which the scenario takes place, referred to as the “skin.” The appearance of the scenario may determine the possible scenario structure and potential setting-induced biases that could confuse candidates’ true skill levels. When choosing a skin, a sufficiently dexterous and complex skin can be selected to represent a wide range of environments in which problem-solving skills are simulated real-world scenarios. To ensure that the vertices of the skeleton and the underlying structure represent the entire problem-solving space, as they exist in the work, small fragments of real-world scenarios can be mapped into the initial task design. Contextual characteristics—details of contextual constraints, objectives, timeframes, and information availability—can be diagnosed to ensure that the candidate’s structure is structurally consistent with real-world situations under the conditions for performing problem-solving skills.

[0034] In some embodiments, simulation-based assessments include one or more scenarios embedded within an abstract natural world environment. The natural environment can limit bias and provide an accessible setting for a global population, regardless of background and prior knowledge. All candidates, regardless of geographic location, field of study, gender, or ethnicity, possess some basic understanding of plants, animals, and weather—concepts that are useful components in designing complex scenarios. Assessments may require users to interact with a range of challenges within the natural environment, involving terrain, vegetation, and wildlife. For example, in one scenario, a user might need to identify an impending environmental threat to an ecosystem given certain evidence.

[0035] In some embodiments, narrative elements presented in a scenario cannot contradict reality, as doing so would penalize candidates' reasonable thought patterns about how the world works, and conversely, would misinterpret confusion with natural world narratives as an inability to perform skills of interest. One or more artificial rules and logics regarding how to perform simulated work are established to avoid any serious violations of common expectations. For example, giraffes live in savannas, not mountains, and penguins eat small fish, not deer. In some embodiments, in addition to having a meaningful narrative and detail in the simulated environment, conceptual parity can be achieved between the building blocks of the evaluation scenario and the real-world scenario. For example, the concept of "profit" can be translated into essentially similar abstract concepts, but is logically consistent as a component of a natural world scenario. "Profit" is a composite abstraction, so in a simulated natural world scenario, for example, a simulated scenario involving the spread of disease, it might be the concept of "spread," which is also a composite concept with a potential mathematical expression defining it.

[0036] In some embodiments, variations in views and scenes across different versions of a scene may be data-driven. Different views and scenes can be generated for different versions of a scene using logic associated with potential animal, plant, and terrain features in a way that aligns with human expectations. For example, in the generated scene, animals should swarm, animals should fly, and animals should mingle and meander, navigating the terrain as they would in real life. Things that are less obvious in our everyday perception of nature, plants, rocks, or terrain slopes may also follow real-world rules, thus keeping the scene's background within the context of the evaluation.

[0037] Online gaming platforms (also known as "user-generated content platforms" or "user-generated content systems") offer various ways for users to interact. For example, users on an online gaming platform might work together towards a common goal, share various virtual game items, send each other electronic messages, and so on. Users on an online gaming platform can join the game as virtual characters, playing specific roles within the game. For instance, a virtual character might be part of a team or multiplayer game environment, where each character is assigned a specific role and possesses corresponding parameters such as clothing, armor, weapons, and skills. In another example, virtual characters might be combined with computer-generated characters, for example, when a single player is part of the game.

[0038] Online gaming platforms can also support the use of virtual objects that simulate physical objects (rigid bodies) within a virtual environment. For example, online gaming platforms allow users to design and present various types of virtual objects, such as vehicles, weapons and weapon accessories, toys, structures, etc. Within a virtual environment supported by the online gaming platform, one or more users can view these objects. For instance, the movement of such objects within the virtual environment might be displayed on the user's device.

[0039] Online gaming platforms can also allow users (developers) to create new objects and / or mechanisms. For example, users of an online gaming platform can create, design, and / or customize new objects such as vehicles, tools, and toys, and make them available to other users.

[0040] Online gaming platforms (also known as gaming platforms) can provide tools to support the simulation of objects and mechanisms in virtual environments. In some implementations, rendering engines, physics engines, and / or physics solvers can be used to simulate objects.

[0041] Some objects and / or parameters in a virtual environment can make simulating the complexity of those objects and the environment computationally expensive, in order to meet the time requirements for real-time simulation and rendering of those objects. For example, the representation of fictional objects may have to be performed within a time period that meets the given frame refresh rate requirements of the display device. Failure to meet these time requirements can lead to unstable updates, resulting in a poor visual experience for the user.

[0042] The goal of game platform owners or administrators is to provide a better user experience. One technical challenge faced by online game platform operators is to accurately, timely, and computationally efficiently generate object representations within a virtual environment.

[0043] The following describes the generation of 3D meshes based on template-based 3D meshes using homeomorphic template meshes as a single parameter within a machine learning (ML) model framework. The geometry of the template, as a priori, can alleviate the learning problem and facilitate training the machine learning model even with limited training data. According to the technical solution described in this application, through training, the ML model implicitly learns the geometric correlations of various parts of an object. In the case of a template bound to a skeleton, the bound template can inherit the skeleton binding, thereby achieving efficient transfer of animations based on the skeleton binding to various objects.

[0044] Providing different views and scene variations for different versions of a scene, and providing real assets created from scratch, is very time-consuming and / or very expensive to purchase. In some embodiments, to provide a scalable method for generating different views and scenes for different versions of a scene, the system uses 2D-to-3D reconstruction to generate assets, or takes a single image as input and outputs a representation of the object of interest in three-dimensional space.

[0045] For example, 3D surface generation methods such as AtlasNet can be used, which consist of a combination of learnable parameterizations. These learnable parameterizations can transform a set of two-dimensional squares into a surface. The parameters of the transformation can come from learned weights from a neural network and learned representations of the shape. For example, the learned parameter transformations can be locally mapped anywhere on a surface, naturally adapting to its underlying complexity, sampling at any desired resolution, and allowing texture mapping onto the generated surface.

[0046] 3D reconstruction involves meshes of objects, such as fish and mammals, presenting a challenge. To address these drawbacks, template meshes can be used as parameterized domains, such as homeomorphic template meshes. A homeomorphic function is a one-to-one mapping between sets such that the function and its inverse are continuous and possess geometry within the topology that can be transformed from one mesh to another through flexible mesh remapping. The geometry of the template mesh can serve as a prerequisite for significantly simplifying the learning process, especially when training data is limited. For example, the network can implicitly learn geometric correlations, such as aligning the fins of a fish in the template mesh with the fins of a fish in the input image. If the template mesh is animated using skeletal rigging, it can inherit that animation even after mesh remapping. Skeletal rigging is a technique that uses a series of interconnected digital bones to represent a 3D model in animation. Specifically, skeletal rigging refers to the process of creating the skeletal structure of a 3D model. For example, a 3D model that remapses a deer mesh into a horse might inherit the deer's animation.

[0047] By generating objects (assets) for simulation-based assessments, such as bio-assets, the costs associated with traditional art and animation teams can be reduced. Furthermore, since asset generation can take time, there is no delay if the assets are pre-computed. In some embodiments, the generated assets can be augmented offline by the artist to ensure they look realistic. In some embodiments, assets can be generated from stock images using an asset generation pipeline, since the same image has neither multiple perspectives nor perspective between images. Figure 1 Consistency. In some embodiments, multiple input images can be used to utilize information with full field-of-view coverage.

[0048] In some respects, the systems and methods described provide one or more models for generating three-dimensional reconstructions from two-dimensional images of objects.

[0049] Figure 1 A schematic diagram of an example network environment for computer-based evaluation provided in an embodiment of this application is shown. Figure 1 An example network environment 100 according to some implementations of this application is shown. Network environment 100 (also referred to herein as the "system") includes an online evaluation platform 102, a client device 110, and a network 122. The online evaluation platform 102 may include an evaluation engine 104, one or more evaluations 105, an accessibility engine 107, and a data storage 108. The client device 110 may include a virtual evaluation 112, an accessibility application 113, and a display screen 114 for interacting with the online evaluation platform 102.

[0050] The following provides a description of network environment 100. In some implementations, network environment 100 may include the same, fewer, more, or different elements, and may be configured in the same or different ways, such as... Figure 1 As shown.

[0051] In some implementations, network 122 may include public networks (such as the Internet), private networks (such as local area networks (LANs) or wide area networks (WANs), wired networks (such as Ethernet), wireless networks (such as 802.11 networks, Wi-Fi networks, or wireless local area networks (WLANs), cellular networks (such as LTE networks), routers, hubs, switches, server computers, or combinations thereof.

[0052] In some implementations, data storage 108 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or other type of component or device capable of storing data. Data storage 108 may also include multiple storage components (e.g., multiple drives or multiple databases), and may also span multiple computing devices (e.g., multiple server computers).

[0053] In some implementations, the online evaluation platform 102 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, virtual server, etc.). In some implementations, a server may be included in the online evaluation platform 102, or it may be a standalone system, or part of another system or platform.

[0054] In some implementations, the online assessment platform 102 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage (such as hard disks, memory, databases), networks, software components, and / or hardware components that can be used to perform operations on the online assessment platform 102 and provide users with access to the online assessment platform 102. The online assessment platform 102 may also include a website (e.g., one or more web pages) or application backend software that can be used to provide users with access to content provided by the online assessment platform 102. For example, a user (or proctor) can access the online assessment platform 102 separately using the accessibility application 113 on client device 110.

[0055] In some implementations, the online assessment platform 102 may provide connectivity between one or more assessment providers and / or employers, allowing supervisors (e.g., assessment managers) to communicate with other supervisors via the online assessment platform 102. This communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., synchronous and / or asynchronous text-based communication). In some implementations of this disclosure, "user" may refer to a single individual. However, other implementations disclosed include "user" (e.g., test or assessment user), which is an entity controlled by a group of users or a group of assessed job and communication skills. For example, in some cases, a group of individual users united as a group being assessed may be considered a "user."

[0056] In some implementations, the online assessment platform 102 may include digital assets and digital assessment generation terms. For example, the platform may provide an administrator interface allowing for design, modification, personalized customization, and other modification features. In some implementations, assessments may include two-dimensional (2D), three-dimensional (3D), virtual reality (VR), or augmented reality (AR) games. In some implementations, assessment creators and / or proctors can search assessments, combine portions of assessments, customize assessments for a given activity (e.g., group assessments), and other features provided by the assessment platform 102.

[0057] In some implementations, the online evaluation platform 102 or client device 110 may include an evaluation engine 104 or a virtual evaluation 112. In some implementations, the evaluation engine 104 may be used for the development or execution of the evaluation 105. For example, the evaluation engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, a haptic engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or video support for film technologies. Components of the evaluation engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) to aid in the calculation and presentation of the evaluation.

[0058] The online evaluation platform 102 using evaluation engine 104 can execute some or all of the evaluation engine functions (e.g., generating physics commands, rendering commands, etc.), or it can offload some or all of the evaluation engine functions to the evaluation engine 104 on the client device 110 (not specified). In some implementations, there may be different ratios between the evaluation engine functions executed on the online evaluation platform 102 and the evaluation engine functions executed on the client device 110 for each evaluation 105.

[0059] In some implementations, evaluation instructions may refer to instructions that allow the client device 110 to render gameplay, graphics, and other features of the evaluation, such as natural world rendering representing logic problems. Instructions may include one or more user inputs (e.g., physical object location), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.). These instructions may be audibly prompted by an evaluation supervisor, presented aloud by a speech synthesizer, physically represented by haptic feedback (e.g., vibrations at boundaries, asymmetry, etc.), or a combination thereof.

[0060] In some implementations, each client device 110 may include a computing device such as a personal computer (PC), a mobile device (e.g., a laptop, mobile phone, smartphone, tablet, or netbook), a network-connected television, a game console, etc. In some implementations, client device 110 may also be referred to as a "user device." In some implementations, one or more client devices 110 can connect to the online evaluation platform 102 at any given time. It should be noted that the number of client devices 110 is illustrative and not a limitation. In some implementations, any number of client devices 110 can be used.

[0061] In some implementations, each client device 110 may include an instance of a virtual assessment 112. The virtual assessment 112 may represent an educational assessment exam, a vocational assessment exam, or any suitable assessment, whether standardized or specifically tailored to a given individual or group of individuals.

[0062] Figure 2 A schematic diagram of an example system architecture for generating 3D meshes of objects from 2D images in an online game environment, provided by an embodiment of this application, is shown. Figure 2 Similar notation is used to identify similar elements as in other attached figures. A letter following a reference number, such as "210," indicates that the Weibo post specifically refers to the element with the given reference number. Reference numbers without a following letter in the text, such as "210," refer to any or all numbers containing that reference number (e.g., "210" in the text refers to reference numbers "210a," "210b," and / or "210n" in the attached figures).

[0063] System architecture 200 (also referred to herein as the "system") includes an online game server 202, a data storage device 220, client devices 210a, 210b, and 210n (collectively referred to as "client devices 210"), and developer devices 230a and 230n (collectively referred to as "developer devices 230"). The game server 202, data storage device 220, client devices 220, and developer devices 230 are coupled via network 122. In some implementations, client devices 210 and developer devices 230 may refer to the same or similar types of devices.

[0064] Online game server 202 may include a game engine 204, one or more games 206, and a graphics engine 208. In some implementations, the graphics engine 208 may be a system, application, or module that allows the online game server 202 to provide graphics and animation capabilities. In some implementations, the graphics engine 208 may perform one or more of the operations described below and interact with... Figure 3 or Figure 4 The flowchart shown or Figure 5A or Figure 5B The process is related. Client device 210 may include game application 212 and input / output (I / O) interface 214 (e.g., input / output device). Input / output device may include one or more microphones, speakers, headphones, display devices, mice, keyboards, game controllers, touch screens, virtual reality consoles, etc.

[0065] Developer equipment 230 may include a game application 232 and an input / output (I / O) interface 234 (e.g., an input / output device). The input / output device may include one or more microphones, speakers, headphones, display devices, mice, keyboards, game controllers, touchscreens, virtual reality consoles, etc.

[0066] The following provides a description of system architecture 200. In different implementations, system architecture 200 may include the same, fewer, more, or different elements, configured in the same or different ways, such as... Figure 2 As shown.

[0067] In some implementations, network 222 may include public networks (such as the Internet), private networks (such as local area networks (LANs) or wide area networks (WANs), wired networks (such as Ethernet), wireless networks (such as 802.11 networks, Wi-Fi networks, or wireless LANs (WLANs), cellular networks (such as 5G networks, LTE networks, etc.), routers, hubs, switches, server computers, or combinations thereof.

[0068] In some implementations, data storage 220 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or other type of component or device capable of storing data. Data storage 220 may also include multiple storage components (e.g., multiple drives or multiple databases), and these components may span multiple computing devices (e.g., multiple server computers). In some implementations, data storage 220 may incorporate cloud-based storage.

[0069] In some implementations, the online game server 202 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, etc.). In some implementations, the online game server 202 may be a standalone system, may include multiple servers, or may be part of another system or server.

[0070] In some implementations, the online game server 202 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage (such as hard drives, memory, databases), networks, software components, and / or hardware components that can be used to perform operations on the online game server 202 and provide users with access to the online game server 202. The online game server 202 may also include website (e.g., web pages) or application backend software that can be used to provide users with access to content provided by the online game server 202. For example, a user may access the online game server 202 using a game application 212 on a client device 210.

[0071] In some implementations, play session data is generated via an online game server 202, game application 212, and / or game application 232, and stored in data storage 220. With the player's permission, play session data may include relevant metadata such as game identifiers; device data associated with the player; player demographics; play session identifiers; chat logs; session start time, session end time, and session duration for each player; the relative positions of participant characters within the virtual game environment; in-game purchases made by one or more players; accessories used by the player; and others.

[0072] In some implementations, chat logs are generated by game application 212 and / or game application 232 and stored in data storage 220. These chat logs may include chat content and related metadata, such as the text content of the chat, where each message has a corresponding sender and recipient; message formatting (e.g., bold, italic, etc.); message timestamps; the relative positions of participant characters in the virtual game environment; attachments used by participants, etc. In some implementations, the chat logs may contain multilingual content, with different language information from different gameplay rounds stored in data storage 220.

[0073] In some implementations, chat logs can be stored as conversations between participants, based on timestamps. In some implementations, chat logs can be stored based on the skeletal binding address of the message. In some implementations, a "user" can be represented as a single individual. However, other implementations disclose that a "user" (e.g., the creator of a user) is an entity controlled by a group of users or an automated source. For example, in a user-generated content system, a group of individual users as a community or group association can be considered a "user".

[0074] In some implementations, the online game server 202 may be a virtual game server. For example, the game server may provide single-player or multiplayer gaming to a community of users accessing or interacting with the game via client device 210 over network 222. In some implementations, the game (also referred to as a "video game," "online game," or "virtual game") may be a two-dimensional (2D) game, a three-dimensional (3D) game (such as a 3D user-generated game), a virtual reality (VR) game, or an augmented reality (AR) game. In some implementations, users can participate in the game together with other users. In some implementations, a game can be played in real-time with other users in the game.

[0075] In some implementations, gameplay may refer to the interaction (e.g., 206) between one or more players using a client device (e.g., 210) in the game, or the interactive behavior on the display or other output device (e.g., 214) of the client device 210.

[0076] In some implementations, game 206 may contain an electronic file that can be executed or loaded using software, firmware, or hardware to present game content (such as digital media projects) to an entity. In some implementations, game application 212 can be executed and game 206 presented via a connected game engine 204. In some implementations, game 206 may have a common set of rules or common objectives, and the environment of game 206 may also share a common set of rules or common objectives. In some implementations, different games may have different rules or objectives.

[0077] In some implementations, a game may have one or more environments (also referred to here as "game environments" or "virtual environments"), where multiple environments may be linked. An example of an environment might be a three-dimensional environment. One or more environments in game 206 may be collectively referred to as a "world," a "game world," a "virtual world," or a "universe." An example of a world might be a 3D world in game 206. For instance, a user might construct a virtual environment that is linked to another virtual environment created by another user. A character in a virtual game can cross virtual boundaries to enter adjacent virtual environments.

[0078] We can observe that the graphics used in 3D environments or 3D worlds employ a three-dimensional representation of the geometric data representing the game content (or at least render the game content as 3D content, regardless of whether a 3D representation of the geometric data is used). 2D environments or 2D worlds use two-dimensional graphics to represent the geometric data of the game content.

[0079] In some implementations, an online game server 202 may host one or more games 206 and allow users to interact with the games 206 using client devices 210 of the game application 212. Users of the online game server 202 may play, create, interact with, or build games 206, communicate with other users, and / or create and build objects of game 206 (e.g., referred to herein as “projects”, “game objects”, or “virtual game projects”).

[0080] For example, when generating user-generated virtual items, users can create character decorations for characters, create one or more virtual environments for interactive games, or construct structures within game 206, and so on. In some implementations, users can buy, sell, or trade virtual game objects, such as in-platform currency (like virtual currency), with other users on online game server 202. In some implementations, online game server 202 may transfer game content to game applications (such as 212). In some implementations, game content (also referred to herein as "content") may refer to any data or software instructions (such as game objects, games, user information, videos, images, commands, media items, etc.) related to online game server 202 or game applications. In some implementations, game objects (e.g., also referred to herein as "items" or "objects" or "virtual objects" or "virtual game items") may refer to objects used, created, shared, or otherwise described in game application 206 on online game server 202 or game application 212 on client device 210. For example, game objects may include a part, model, character, accessory, tool, weapon, clothing, building, vehicle, currency, flora, fauna, and the above components (such as windows of a building), etc.

[0081] In some implementations, game 206 may be associated with a given user or a given group of users (e.g., a private game) or widely available to users accessing online game server 202 (e.g., a public game). In some implementations, online game server 202 associates one or more games 206 with a specific user or group of users, and online game server 202 may use user account information (e.g., user account identifiers such as username and password) to associate a specific user with game 206.

[0082] In some implementations, the online game server 202 or client device 210 may contain a game engine 204 or a game application 212. In some implementations, the game engine 204 may be used for the development or execution of game 206. For example, the game engine 204 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graphics capabilities, or video support for movies. Components of the game engine 204 may generate commands that aid in the calculation and rendering of the game (e.g., rendering commands, collision commands, physics commands, etc.). In some implementations, the game application 212 of client devices 210 / 216 may, respectively, operate independently, cooperate with the game engine 204 of the online game server 202, or a combination of both.

[0083] In some implementations, both the online game server 202 and the client device 210 can execute a game engine (204 and 212, respectively). The online game server 202, using game engine 204, can execute some or all of the game engine functions (such as generating physics commands, rendering commands, etc.), or it can offload some or all of the game engine functions to the game engine 204 on the client device 210. In some implementations, the ratio between the game engine functions executed on the online game server 202 and the game engine functions executed on the client device 210 may differ for each game 206. For example, the game engine 204 of the online game server 202 might be used to generate physics commands in the event of a conflict between at least two game objects, while additional game engine functions (e.g., generating rendering commands) might be offloaded to the client device 210. In some implementations, the ratio of game engine functions executed on the online game server 202 and the client device 210 may change (e.g., dynamically) depending on gameplay conditions. For example, if the number of users participating in a given game 206 exceeds a threshold, the online game server 202 may execute one or more game engine functions that were previously performed by the client device 210.

[0084] For example, a user might play game 206 on client device 210 and send control commands (e.g., user input, such as right, left, up, down, user selection, or character position and speed information) to online game server 202. Upon receiving control commands from client device 210, online game server 202 can send gameplay commands (e.g., position and speed information or commands for characters participating in group gameplay, such as rendering commands, collision commands, etc.) to client device 210 based on the control commands. For example, online game server 202 can perform one or more logical operations on the control commands (e.g., using game engine 204) to generate gameplay commands for client device 210. In other cases, online game server 202 might pass one or more control commands from one client device 210 to other client devices participating in game 206 (e.g., from client device 210a to client device 210b). Client device 210 can use the gameplay commands and present the gameplay for display on its monitor.

[0085] In some implementations, control instructions may refer to actions performed by a user character in the game. For example, control instructions may include user input controlling in-game actions, such as right, left, up, down, user selection, gyroscope position and direction data, force sensor data, etc. Control instructions may include character position and speed information. In some implementations, control instructions are sent directly to the online game server 202. In another implementation, control instructions may be sent from client device 210 to another client device (e.g., from client device 210b to client device 210n), where the other client device uses a local game engine 204 to generate gameplay instructions. Control instructions may include instructions to play voice communication information or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.), such as voice communication or other sounds generated using the audio spatialization technology described in this application.

[0086] In some implementations, gameplay instructions may refer to instructions that cause the client device 210 to display gameplay, such as in multiplayer games. Gameplay instructions may include one or more user inputs (such as control instructions), character position and velocity information, or commands (such as physics commands, rendering commands, collision commands, etc.).

[0087] In some implementations, a character (or game object) is composed of components, which the user can select one or more of which are automatically linked together to aid in editing. In other implementations, a character is implemented as a 3D model, including a surface representation (also called skin or mesh) for drawing the character and a set of interconnected, layered bones (also called skeleton or skeletal rigging). The skeletal rigging can be used to animate and simulate the movement and motion of the character. The 3D model can be represented as a data structure whose one or more parameters can be modified to change various attributes of the character, such as size (height, width, perimeter, etc.); body shape; movement style; number / type of body parts; proportions (e.g., shoulder-to-hip ratio); head size; etc. One or more characters (also referred to herein as "avatars" or "models") may be associated with a user, who can control the character to facilitate user interaction with the game.206

[0088] In some implementations, a character may include body parts (such as hair, arms, legs, etc.) and accessories (such as t-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts include head type, body part types (arms, legs, torso, and hands), face type, hair type, and skin type, etc. In some implementations, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0089] In some implementations, a component, such as a body part, may be a primitive geometry, such as a block, cylinder, sphere, etc., or other primitive shapes, such as wedges, tori, tubes, grooves, etc. In some implementations, components may be biological assets, such as mammals, fish, or birds. In some implementations, the creator module can publish a user's character for other users on the online game server 202 to view or use. In some implementations, the creation, modification, or customization of characters, other game objects, game 206, or game environment may be performed by the user using I / O interfaces (such as developer interfaces), with or without scripts, or with or without application programming interfaces (APIs). It can be noted that, for illustrative purposes, characters are described as having a humanoid form. It can also be further noted that characters can take any form, such as vehicles, animals, inanimate objects, or other creative forms.

[0090] In some implementations, the online game server 202 may store user-created characters in data storage 220. In some implementations, the online game server 202 maintains a character directory and a game directory that may be presented to the user. In some implementations, the game directory includes game images stored on the online game server 202. Furthermore, the user can select a character (e.g., a character created by the user or another user) from the character directory to participate in the selected game. The character directory includes character images stored on the online game server 202. In some implementations, one or more characters in the character directory may have been created or customized by the user. In some implementations, the selected character may have character settings that define one or more components of that character.

[0091] In some implementations, client device 210 may include computing devices such as personal computers (PCs), mobile devices (such as laptops, mobile phones, smartphones, tablets, or netbooks), network-connected televisions, game consoles, etc. In some implementations, client device 210 may also be referred to as a "user device." In some implementations, one or more client devices 210 can connect to the online game server 202 at any given time. Note that the number of client devices 210 is for illustrative purposes. In some implementations, any number of client devices 210 can be used.

[0092] In some implementations, each client device 210 may include an instance of the game application 212. In one implementation, the game application 212 may allow a user to use and interact with the online game server 202, such as controlling virtual characters in a virtual game hosted by the online game server 202, or viewing or uploading content such as games 206, images, video projects, web pages, documents, etc. In one example, the game application may be a web application (e.g., an application that runs with a web browser) that can access, retrieve, render, or navigate content provided by a web server (e.g., virtual characters in a virtual environment, etc.). In another example, the game application may be a native application (e.g., a mobile application, application, or game program) that is installed and executed locally on the client device 210 and allows the user to interact with the online game server 202. The game application may present, display, or render content (e.g., web pages, media viewers) to the user. In implementations, the game application may also include an embedded media player (e.g., a Flash player) embedded in a web page.

[0093] According to some aspects of this application, the game application may be an online game server application that allows users to build, create, edit, and upload content to an online game server 202, as well as interact with the online game server 202 (e.g., play a game 206 hosted by the online game server 202). Similarly, the game application may be provided by the online game server 202 to the client device 210. In another example, the game application may be an application downloaded from the server.

[0094] In some implementations, each developer device 230 may include an instance of the game application 232. In one implementation, the game application 232 may allow the developer user to use and interact with the online game server 202, such as controlling virtual characters in a virtual game hosted by the online game server 202, or viewing or uploading content, such as games 206, images, video projects, web pages, documents, etc. In one example, the game application may be a web application (e.g., an application that runs with a web browser) that can access, retrieve, render, or navigate content provided by a web server (e.g., virtual characters in a virtual environment). In another example, the game application may be a native application (e.g., a mobile application, application, or game program) that is installed and executed locally on the client device 230 and allows the user to interact with the online game server 202. The game application may present, display, or render content (e.g., web pages, media viewers) to the user. In implementations, the game application may also include an embedded media player (e.g., a Flash player) embedded in a web page.

[0095] According to some aspects of this application, the game application 232 may be an online game server application for users to build, create, edit, upload content to the online game server 202, and interact with the online game server 202 (e.g., provide and / or play games 206 hosted by the online game server 202). Similarly, the game application may be provided to client device 230 by the online game server 202. In another example, the game application 232 may be an application downloaded from the server. The game application 232 may be configured to interact with the online game server 202 and gain access to user credentials, user currency, etc., of one or more games 206 developed, hosted, or provided by the game developer.

[0096] Generally, the functions performed by the online game server 202 as described in one implementation can also be performed by the client device 210 or the server, if appropriate, in another implementation. Furthermore, functions belonging to a given component can be performed by different or multiple components operating together. The online game server 202 can also be accessed as a service provided to other systems or devices via a suitable application programming interface (API), and is therefore not limited to use on a website.

[0097] Figure 3 An example method 300 for generating a 3D mesh of objects from a 2D image using a trained machine learning (ML) model, as provided in an embodiment of this application, is illustrated. In some implementations, method 300 may be implemented, for example, in reference... Figure 1 and Figure 2The online evaluation platform 102 or game server 202 are implemented on the ground, respectively. In some implementations, some or all of the methods 300 can be implemented on one or more client devices 110 and / or 210, such as... Figure 1 and Figure 2 As shown, this can be implemented on one or more developer devices 230, or on one or more server devices 202, and / or on a combination of developer devices, server devices, and client devices. In the described examples, the system implementation includes one or more digital processors or processing circuitry (“processors”), and one or more storage devices (e.g., data memory 220, data memory 108, or other memory). In some implementations, different components of one or more servers and / or clients may execute different blocks or other portions of method 300. In some examples, the first device is described as executing blocks of method 300. Some implementations may have one or more blocks of method 300 executed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0098] Method 300 may begin with block 310. In block 310, a two-dimensional (2D) image of the object is provided as input to a trained machine learning model. The 2D image may be retrieved from a data storage device or provided by the user as user input. The object may be a representation of an object that is part of a virtual environment, such as a biological object, like an animal, fish, mammal, bird, etc., or an artificial object, such as a car, airplane, etc., contained in an evaluation or game environment. The 2D image may include an image of the object that has not previously been provided as input to the ML model, or it may be an image of the object that has previously been provided as input to the ML model. In some implementations, the 2D image may be a photograph of a real object. Block 310 may be followed by block 320.

[0099] At block 320, a template 3D mesh associated with the object's category is obtained. In some implementations, the object's category can be determined based on image analysis of the provided 2D image. As previously mentioned, objects and / or characters in a virtual environment can be implemented as 3D models, potentially including a surface representation for drawing the object / character (also known as skin or mesh) and a set of interconnected, layered bones (also known as skeleton or skeletal rigging). The skeletal rigging can be used by the object to create character animations and simulated movements. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes (attributes) of the object and / or character, such as size (height, width, perimeter, etc.); body shape; movement style; number / type of body parts; proportions (e.g., shoulder-to-hip ratio); head size; and others.

[0100] In some implementations, the template 3D mesh is a homeomorphic template mesh. The template 3D mesh can be selected from a set of previously generated and stored template meshes, and may include user-generated and automatically generated meshes. In some implementations, multiple template meshes can be displayed to the user, and one template mesh can be selected based on input received from the user. In some implementations, multiple template meshes can be selected for the performance of method 300. Block 320 may be followed by block 330.

[0101] In block 330, a 3D mesh of the object is generated using the trained machine learning model based on the 2D image and the template 3D mesh. In some implementations, the trained machine learning model may include an artificial neural network (ANN), such as a feedforward neural network like a multilayer perceptron (MLP). In some implementations, the trained machine learning model may include a mesh encoder, an image encoder, and multiple mesh decoders, trained to generate 3D meshes of the object based on a 2D image of the object.

[0102] Machine learning models are trained using various objects and their corresponding geometry, meshes, and images. For further details on training machine learning models, please refer to [link / reference]. Figure 4 and Figures 5A to 5B More detailed instructions are provided.

[0103] In some implementations, an image encoder encodes the provided 2D image into a shape space vector. The shape space vector can be an n-dimensional vector, a reduced-dimensional representation of the provided 2D image. A mesh decoder, associated with the object's category, decodes the shape space vector into a 3D mesh of the object. Mesh remapping of the template 3D mesh, i.e., mesh adjustment (movement) of the vertex template based on the shape space vector using a homeomorphic transformation to preserve the topological properties of the template 3D mesh, is performed. In some implementations, the 3D mesh of the object generated by remapping the template 3D mesh can be displayed via a user interface.

[0104] In some implementations, the category of the object can be determined using image matching based on the 2D image before obtaining the template 3D mesh. In some implementations, image segmentation can be performed to determine the category of the object. In some implementations, the category of the object can be specified based on user input.

[0105] In some implementations, one or more textures from the 2D image can be mapped to the 3D mesh of the object, depending on the texture information in the 2D image. In some implementations, semantic segmentation can be performed, and texture information from different parts of the 2D image can be mapped to corresponding parts of the 3D mesh of the object.

[0106] In some implementations, the template 3D mesh is associated with a skeletal rigging, which includes multiple vertices (joints) of the mesh arranged in space. The skeletal rigging may comprise a set of interconnected hierarchical components (called bones, which together form a skeleton or skeletal rigging). In such an implementation, each of the multiple vertices is connected to at least one other vertex of the multiple joints.

[0107] One or more animations associated with the template 3D mesh can be predefined, such that each template animation includes the motion of two or more of a plurality of vertices to simulate the motion of the underlying object. Template animations from a 3D mesh generated in the object can be reused using a homeomorphic template 3D mesh, which is obtained by re-dividing the template 3D mesh.

[0108] In some implementations, at least one object animation can be generated for the object, which is based on the 3D mesh for the object and one or more template animations associated with the template 3D mesh, including the movement of the same two or more vertices in the corresponding template animations of one or more (predefined) template animations.

[0109] In some implementations, a UV regressor can be used to apply textures to the re-meshed mesh mapped to the object. In other implementations, a UV regressor can be used to generate a mapping from the 3D mesh vertices of the object to the 2D image vertices, where the mapping is used to apply textures to the 3D mesh vertices of the object.

[0110] UV mapping is a 3D modeling process that projects a 2D image onto the surface of a 3D model to create a texture. The letters "U" and "V" represent the axes of the 2D texture, while "X", "Y", and "Z" represent the axes of objects in the 3D model space. Vertices are assigned UV coordinates by projecting pixels from a depth image onto vertices in the 3D model. These coordinates indicate the pixel position corresponding to a vertex in the texture image. For example, if the size of the texture image is (256, 512), and a vertex is projected onto a pixel at (112, 234), the UV coordinates of that vertex are (112 / 256, 234 / 512) = (0.4375, 0.457).

[0111] In some implementations, the machine learning model can utilize the object and a 2D image of the corresponding mesh-re-divided mesh to generate a UV mapping from the vertices of the mesh-re-divided mesh to a second 2D image of the object. The ML model can be a trained statistical model, fine-tuned to perform fine-grained semantic segmentation on a predefined set of descriptors (e.g., a dataset containing animals, mammals, or fish). These descriptors can be used to guide regression through the UV space. For example, the trained statistical model might be a trained Deeplab model or other suitable model. In some embodiments, the UV regressor might attempt to "cast a net" across a given image in UV space. The ML model can learn how to project the mesh onto the image, rather than distorting the image to fit the mesh. The input image might be the same as the image used for 3D reconstruction, so this regressor can run as part of an automated 3D reconstruction pipeline. In some embodiments, any image with a sufficiently large visible portion of an animal might be available, so multiple textures can be applied to the same mesh. The input could be a new 2D image a and a source mesh B. B can be segmented using the same descriptors used when training the Deeplab model to generate the image. The output might be a UV mapping from the vertices of B to a. B may not need a predefined UV mapping, but if it does, the UVs generated by the UV regressor can be inherited into the old UV mapping.

[0112] In some embodiments, training the UV regressor may include training a network, such as AtlasNet, to learn a mapping from a mesh surface to an image domain using descriptor loss. The input image may be processed by the descriptor network to obtain a descriptor for each pixel. At each iteration, points for each descriptor class may be sampled from both the mesh surface and the target image. The network can be used to map the sampled mesh points to the UV space of the image. Regression may be performed using an AdamOptimizer (e.g., with a learning rate of 0.0001) to minimize the descriptor loss. The network may be pre-trained using a mean squared loss and UV coordinates generated from a side-view projection of the mesh. The UV regressor may be trained using a loss function that includes a descriptor loss based on chamfer distance.

[0113] The network can continue to be trained using descriptor loss, defined as follows:

[0114] Descriptor loss: Let Q j Let R be a set of N sample points from the image used for descriptor category j, and let R be... j It is a set of M sample points from the grid used for descriptor category j. The loss is the chamfer distance (Qj) of j in J. j R j The sum of distances to j, where J is the set of all descriptor categories, and the chamfer distance is the sum of the distances from each point in the grid to its nearest neighbor.

[0115] Just as a mapping from a template mesh surface to a target mesh surface can be generated, a mapping from a template mesh surface to a 2D image can be generated to achieve texturing. In some implementations / examples, the decoder for texturing may include five fully connected layers with output sizes of 1024, 512, 256, 128, and 2, respectively. Each layer may use a rectified linear unit (ReLU) activation function, but the last layer may be unactivated.

[0116] In some embodiments, the richest texture information is found in the target image, which contains as much of the object as possible (e.g., a frontal view of an animal has less information to texture the mesh than a side view). Better results may be obtained when the animal in the image appears at an approximate side view perspective than other perspectives. For example, the model can be started by pre-training using mean squared loss and UV coordinates generated from the side view projection of the mesh.

[0117] However, it may be unlikely to obtain a perfect side-view target image because, in each iteration, a mapping to the target image is learned from the entire surface of the mesh, where one side will match the image better than the other. Therefore, when attempting textured meshes, the texture on both sides of the symmetry plane may be inconsistent and unrealistic. To address this, at the end of the regression, the descriptor loss for each side of the mesh is calculated separately, and the side with the smallest value is used as the basis for the entire mesh's reflectance UV map. In some embodiments, the AdamOptimizer can be trained with the following learning parameters: learning rate = 0.0001; Beta1 = 0.9; Beta2 = 0.999; total iterations = 1000.

[0118] Figure 4 A flowchart illustrating an example method for training an image encoder provided in an embodiment of this application is shown.

[0119] In some implementations, method 400 can be found in the reference. Figure 2 The method 400 is implemented on server 202. In some implementations, some or all of method 400 may be implemented on one or more client devices 210, such as... Figure 2As shown, the implementation is carried out on one or more developer devices 230, or on one or more server devices 202, and / or on a combination of developer devices, server devices, and client devices. In the example, the implemented system includes one or more digital processors or processing circuitry (“processors”), and one or more storage devices (e.g., data storage 220 or other storage devices). In some implementations, different components of one or more servers and / or clients may execute different blocks or other portions of method 400. In some examples, the first device is described as executing blocks of method 400. Some implementations may have one or more blocks of method 400 executed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0120] In some implementations, method 400 or a portion thereof may be started automatically by the system. In some implementations, the implementation system is the primary device. For example, the method (or a portion thereof) may be executed periodically, or executed based on one or more given events or conditions, such as the accuracy of 3D mesh generation of an object falling below a threshold, the addition of a new category of object (or multiple objects) to the evaluation platform or game platform, the elapsed time since the last performance of method 400, and / or one or more other events, the method to be read may be specified in the settings.

[0121] In this illustrative example, the ML model is an artificial neural network that includes a mesh encoder, an image encoder, and multiple mesh decoders for generating a three-dimensional mesh from a two-dimensional image of an object.

[0122] Method 400 can begin with block 410. In block 410, multiple target meshes are received. Block 420 may follow block 410.

[0123] In block 420, a mesh encoder and multiple mesh decoders are trained. In some implementations, two-stage training can be performed. Each mesh decoder in the multiple target meshes corresponds to a given template mesh. In some implementations, two-stage training can be performed and repeated on the available training target meshes. In some implementations, updating the mesh encoder includes retraining the mesh encoder based on a first value of a first error function, and updating the image encoder includes retraining the image encoder based on a second error function.

[0124] In the first stage, a mesh encoder is used to encode a given target mesh into a first shape space vector. A corresponding mesh decoder, corresponding to the type and / or category of the given target mesh, is used to decode the first shape space vector into a re-meshed mesh.

[0125] The first value of the first error function is determined by comparing the target mesh and the mesh after mesh repartitioning, and one or more parameters of the mesh decoder are updated based on this first value. The machine learning model is trained based on the first value, which is a measure of the distance from the template mesh bound to the skeleton to the target mesh. This process can be repeated until the mesh decoder of the machine learning model is trained on the complete set of target meshes.

[0126] In the second phase, an image encoder is trained. Multiple two-dimensional (2D) images of objects are received, each image corresponding to an object of a specific category, where each category is associated with a corresponding template mesh. In some implementations, the 2D images may include object images previously used to train the mesh decoder in the first phase of training described above and / or object images not previously used for training in the first phase.

[0127] An image encoder is used to generate a shape space vector for each of the multiple 2D images contained in the 2D image training set of the object. A mesh encoder trained in the first stage is used to generate a shape space vector for each target mesh as the training set.

[0128] The shape space vector generated by the image encoder is compared with the corresponding shape space vector associated with the target mesh in the corresponding object, where the shape space vector associated with the target mesh is encoded using a mesh encoder. One or more parameters of the image encoder are updated based on the comparison.

[0129] The second value of the second error function is determined by comparing the target mesh and the mesh after re-meshing, and one or more parameters of the image encoder are updated based on the second value. In some implementations, the second error function is a fitness function that represents the distance between the shape space vector generated by the mesh encoder and the shape space vector generated using the image encoder.

[0130] This process can be repeated until the image encoder of the machine learning model is trained on a complete set of target grids. In some implementations, one or more of multiple grid decoders can be pre-trained to approximate the identity function on the template grid.

[0131] Figure 5A This illustration shows an example process of training a mesh decoder to generate a re-divided mesh, as provided in an embodiment of this application. Figure 5B This illustration shows an example process of training an image encoder to generate shape space vectors according to an embodiment of this application.

[0132] In some embodiments, the described system and method provide a 3D reconstruction model, including an encoder-decoder architecture, for example, based on AtlasNet or other suitable methods. Training can be performed in two consecutive phases: a first phase (500) trains a mesh encoder (504) to map the point cloud in the target mesh (502) to a latent space represented by shape space vectors (506); and a second phase (550) trains an image encoder (554) to map the training image (552) to the same latent space (556).

[0133] In some implementations, the decoder can be optimized to map the encoded latent vectors and random points on the template surface to the target surface. In some implementations, the decoder may consist of five fully connected layers with output sizes of 1024, 512, 256, 128, and 3, respectively. Each layer may include a Rectified Linear Unit (ReLU) activation function, but the last layer may not have an activation. The chamfer distance can be used as a loss for both stages and can be optimized using the AdamOptimizer or other suitable methods. During the first stage of pre-training, there may be an additional mean squared error loss term on the template vertices.

[0134] In the first phase of training, the goal might be to learn a "shape space" representing the different possible geometries of the training data. For example, this might involve learning to encode the geometry as n-dimensional shape space vectors. Biological assets can receive a template mesh corresponding to a specific animal, such as a fish, a mammal, etc. For example, a template mesh corresponding to a fish might be received.

[0135] Using a first statistical model, the template mesh can be encoded into a first shape space vector. For example, the first statistical model might be a mesh encoder that takes the target mesh as input and generates an n-dimensional shape space vector as output.

[0136] Using a second statistical model, the first shape space vector can be decoded into a re-mesh correspondence of the biological asset. For example, the second statistical model could be a mesh decoder, pre-trained to approximate an identity function on a template mesh. The mesh decoder takes the shape space vector as input and generates a re-mesh as output that matches the geometry represented by the shape space vector. There may be multiple mesh decoders, one for each template mesh, serving as input to the mesh encoder.

[0137] A first value of a first error function can be determined based on the comparison (506) between the template mesh and the mesh after re-meshing. For example, the first error function might be a fitness function that measures the distance (distance) between the mesh after re-meshing and the target mesh provided as input to the mesh encoder.

[0138] Feedback 514 can be used to update the first statistical model based on a first value of the first error function being higher than a first threshold. For example, the first threshold might indicate an acceptable error level regarding the closeness of the re-grid to the target grid. The grid encoder can be retrained based on the value of the first error function.

[0139] In some embodiments, PointNet or other suitable methods can be used for the encoder in the first stage of training. In some embodiments, the encoder can encode the point cloud into a latent space. First, the model can be pre-trained on the dataset so that the decoder approximates the identity function on the template. The effect of doing so is to guide the parameter space to mesh repartitions suitable for various purposes, such as mapping textures, animation, etc. The model is then trained for 2000 iterations to reconstruct the target point cloud. In some implementations, the following learning parameters can be used: point network learning rate = 0.001; Beta1 = 0.9; Beta2 = 0.999; total iterations = 2000; batch size = 8.

[0140] In the second phase of training, the goal might be to learn how to map images to shape space vectors corresponding to the object, such as an animal. Training can be performed using a set of images mapped to a template mesh provided as input to the mesh encoder.

[0141] After updating the first statistical model, a two-dimensional image of the object may be received. For example, after training a mesh encoder, a two-dimensional image corresponding to the target mesh, provided as input to the mesh encoder, may be received.

[0142] The updated first statistical model can be used to map the target mesh to the object and encode it as an updated first shape space vector. For example, after the mesh encoder is trained, using the trained mesh encoder, the target mesh may be encoded by the system as an n-dimensional shape space vector mapping to the target mesh.

[0143] Using a third statistical model, a two-dimensional image can be encoded into a second shape space vector. For example, the third statistical model could be an image encoder that takes an image of an object as input and generates an n-dimensional shape space vector as output.

[0144] The second value of the second error function can be determined based on a comparison of the updated first shape space vector and the second shape space vector (564). For example, the second error function could be a fitness function that measures how close the shape space vector generated by the image encoder is to the shape space vector generated by the mesh encoder.

[0145] Using feedback 566, the third statistical model can be updated based on a second value of the second error function that is higher than a second threshold. For example, the second threshold could indicate an acceptable level of error in the closeness between the shape space vector generated by the image encoder and the shape space vector generated by the mesh encoder. The image encoder can be retrained based on the value of the second error function.

[0146] In some embodiments, during the second phase of training, the encoder may use an object recognition model, such as VGG, or other suitable methods.

[0147] In some embodiments, the encoder can encode images into a latent space. During training, optimization can only update the last two fully connected layers of the encoder, and the 3D decoder produced by the first stage may be frozen. In some implementations, the following learning parameters can be used: decoder learning rate = 0.001; Vgg learning rate = 1e-5; Beta1 = 0.9; Beta2 = 0.999; total iterations = 10,000; batch size = 16.

[0148] After training, the trained model can be used to generate a 3D reconstruction of a received 2D image object. After updating a third statistical model (e.g., an image encoder), a new 2D image of the object may be received.

[0149] Using a third, updated statistical model, the second two-dimensional image can be encoded into a third shape space vector. For example, this system can be used to encode a trained two-dimensional image encoder into an n-dimensional shape space vector correspondence to display a new image of a fish.

[0150] This system can be used for decoding, using a second statistical model, to transform a third shape space vector into a second mesh re-divided mesh corresponding to a biological asset. For example, the system can be used to decode, using a mesh decoder, an n-dimensional shape space vector corresponding to a new, unseen image of a fish into a corresponding mesh re-divided mesh, thereby generating a 3D reconstruction of the fish.

[0151] In some embodiments, the system can be used to generate a multi-template model that uses K template meshes (and K corresponding parameterizations) simultaneously. This can be helpful for learning because it allows the network to potentially map templates to mismatched targets, and it allows users to access several possible 3D reconstructions so they can choose one that suits their purpose.

[0152] Figure 6 An example of a 3D mesh generated from a corresponding 2D image, provided in an embodiment of this application, is shown.

[0153] This illustrative example depicts 2D images of a wolf and a bear, and the corresponding 3D meshes generated using the method described above. Figure 6 As shown, the 3D mesh is a mesh re-divided from a template mesh of a certain category. Figure 6 As shown, texture attributes are also transferred to provide a more accurate representation of the object.

[0154] Figure 7 An example of using segmented 2D images for classification, provided in an embodiment of this application, is shown.

[0155] Fine-grained semantic segmentation can be performed on a predefined set of descriptors to determine a given part of an object and classify the object, determine the category type, etc. In some implementations, the standard Deeplab model can be used to perform semantic segmentation. The output categories can be modified appropriately to match the categories of objects and / or biological assets. For example, the number of output classes of a standard Deeplab model can be changed to the number of matching descriptors (e.g., 8 for mammals, 3 for fish) and trained with a learning rate of 1E-4. In this illustrative example, Figure 7 The left side describes the input image, and the right side shows the result of the trained ML model.

[0156] In some implementations, the descriptor set defined for the mammalian instance set of the object class may include eyes, head, legs, snout, ears, tail, horns, and body, while the descriptor set defined for the fish instance set of the object class may include eyes, body, and fins.

[0157] Figure 8 An example of a 3D mesh of an object bound to a skeleton, as provided in an embodiment of this application, is shown.

[0158] In this illustrative example, the object is a biological asset (bear). Figure 8 This describes a mesh superimposed on a 2D image. For example... Figure 9 As shown, the re-mesh includes multiple connected joints, which can be used to animate 3D objects in a virtual environment, such as depicting moving cars, swimming fish, running animals, etc.

[0159] Figure 9 An example evaluation framework provided by an embodiment of this application is shown.

[0160] In some respects, an example implementation of the system and method is as follows: Figure 9 As shown. In particular, Figure 9An example implementation of a system for calculating evaluation scores according to some embodiments of this application is shown. Specifically, system 900 may include one or more processors 910 operable to generate evaluation scores (e.g., element 940) for a user. This information may be stored in memory or persistently stored in a storage medium. In some embodiments, processor 910 may receive user telemetry data 920, including mouse movements, clicks, selections, timestamps, and other suitable telemetry data. In some embodiments, processor 910 may, according to at least some of the described systems and methods, use an evaluation framework 930 (e.g., associated programs or software) to receive and / or generate one or more evaluation scores for each user's telemetry data. Processor 910 may be configured to perform at least some of the described systems and methods to generate evaluation score 940 based on telemetry data 920 and / or evaluation framework 930.

[0161] Figure 10 A schematic diagram of an example computing device provided in an embodiment of this application is shown.

[0162] In some respects, Figure 10 An illustrative embodiment of a computing device 1000 is shown, which can be used in conjunction with any of the embodiments provided in this application. In particular, Figure 10 An example computer system performing simulation-based evaluations according to some embodiments of this application is shown. Computing device 1000 may include one or more processors 1010 and one or more manufactured products containing non-transitory computer-readable storage media (e.g., memory 1020 and one or more non-volatile storage media 1030). Processor 1010 may control the writing of data to and reading data from memory 1020 and non-volatile storage device 1030 in any suitable manner. To perform any of the functions described herein, processor 1010 may execute one or more processor-executable instructions stored in one or more non-transitory computer-readable storage media (e.g., memory 1030) for execution by processor 1010.

[0163] In some embodiments, at least some of the described systems and methods are implemented in a cloud-based architecture. A cloud-based architecture can provide on-demand access to a shared pool of configurable computing resources (e.g., processors, graphics processors, memory, disk storage, network bandwidth, and other suitable resources). The processors in a cloud-based architecture may be able to operate to receive or acquire user telemetry data, including mouse movements, clicks, selections, timestamps, and other suitable telemetry data, and calculate evaluation scores for the user. The memory in a cloud-based architecture may store user telemetry data and evaluation frameworks (e.g., related programs or software) that can be operated by the processor to calculate evaluation scores. In some embodiments, the cloud-based architecture may provide a graphics processor for calculating evaluation scores, which is faster and more efficient than traditional processors.

[0164] Figure 11 The illustration shows an example computing device 1100 provided in an embodiment of this application, which can be used to implement one or more features described below. In one example, device 1100 can be used to implement a computer device (e.g., Figure 2 The computing device 1100 may be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 1100 may be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, mobile phone, smartphone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some implementations, the device 1100 includes a processor 1102, memory 1104, input / output (I / O) interface 1106, and audio / video input / output device 1114.

[0165] Processor 1102 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 1100. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functions, or other systems. Processing is not limited to a given geographical location or time constraints. For example, a processor may perform its functions in a "real-time," "offline," or "batch processing mode." Different (or identical) processing systems may perform portions of processing at different times and in different locations. A computer may be any processor that communicates with memory.

[0166] Memory 1104 is typically provided in device 1100 for access by processor 1102 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions executed by the processor and may be separate from or integrated with processor 1102. Memory 1104 may store software that runs on server device 1100 by processor 1102, including operating system 1108, one or more application programs 1110, such as an audio spatialization application, and application data 1112. In some implementations, application program 1110 may contain instructions that enable processor 1102 to perform the functions (or control functions) described herein, for example, referenced in [reference]. Figures 3 to 4 as well as Figure 5A and Figure 5B The described method may be partially or entirely.

[0167] For example, application 1110 may include an audio spatialization module 1112, as described in this application, which can provide audio spatialization in an online game server (e.g., 102). Software elements in memory 1104 may also be stored in any other suitable storage location or on a computer-readable medium. Furthermore, memory 1104 (and / or other connected storage devices) may store instructions and data used in the functions described in this application. Memory 1104 and any other type of storage (disk, optical disk, magnetic tape, or other tangible media) can be considered as "memory" or "storage device".

[0168] I / O interface 1106 can provide interface functionality between server device 1100 and other systems and devices. For example, network communication devices, storage devices (such as memory and / or data storage 120), and input / output devices can communicate through interface 1106. In some implementations, the I / O interface can connect to interface devices, including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0169] Audio / video input / output device 1114 may include user input devices (such as a mouse, etc.) that can be used to receive user input, display devices (such as a screen, monitor, etc.) and / or combined input and display devices that can be used to provide graphical and / or visual output.

[0170] For ease of explanation, Figure 11Each of the following is represented by a separate block: processor 1102, memory 1104, I / O interface 1106, operating system 1108, and game application 1110. These blocks may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software engines. In other implementations, device 1100 may not have all of the components shown, and / or may have other elements, including other types of elements, in place of or supplementing the elements shown herein. While online game server 202 is described as performing the operations described in some implementations of this protocol, any suitable component or combination of components of online game server 202 or similar systems, or any suitable processor associated with that system, may perform the described operations.

[0171] The user equipment can also implement and / or be used with the features described in this application. Example user equipment may be a computer device, including components similar to device 1100, such as processor 1102, memory 1104, and I / O interface 1106. An operating system, software, and applications suitable for the client device may be provided in memory and used by the processor. The client device's I / O interface may be connected to network communication devices, or to input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device in audio / video input / output device 1114 may be connected to (or included in) device 1100 to display the image preprocessing and post-processing described in this application, wherein the display device may include any suitable display device, such as an LCD, LED or plasma display screen, CRT, television, monitor, touchscreen, 3D display screen, projector, or other visual display device. Some implementations can provide an audio output device, such as speech output or text synthesis.

[0172] One or more methods described in this application (e.g., methods 300 and 400) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (such as microprocessors or other processing circuits) and can be stored on a computer program product, including non-transitory computer-readable media (such as storage media), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memories, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, skeletonized ID disks, optical discs, solid-state storage drives, etc. Program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or in a combination of hardware and software. Example hardware can be a programmable processor (such as a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), and so on. One or more methods can be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0173] One or more methods described in this application can run in a standalone program that can run on any type of computing device, including programs running on web browsers, mobile applications (“apps”) running on mobile computing devices (e.g., mobile phones, smartphones, tablets, wearable devices (watches, armbands, jewelry, headwear, goggles, glasses, etc.), laptops, etc. In one example, a client / server architecture can be used, whereby the mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computation can be performed within a mobile application (and / or other applications) on the mobile computing device. In yet another example, computation can be partitioned between the mobile computing device and one or more server devices.

[0174] Although the given implementations have been described, they are illustrative only and not limiting. The concepts illustrated in the examples can be applied to other examples and implementations.

[0175] The functional blocks, operations, features, methods, devices, and systems described in this application can be integrated or divided into different combinations of systems, devices, and functional blocks well known to those skilled in the art. Routines of a given implementation can be implemented using any suitable programming language and programming technique. Different programming techniques can be employed, such as procedural or object-oriented. Routines can be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be shown in a specific order, the order may vary in different given implementations. In some implementations, multiple steps or operations shown sequentially in this specification may be performed simultaneously.

Claims

1. A computer-implemented method for generating a 3D mesh of an object using a trained machine learning model, characterized in that, The method includes: A 2D image of the object is provided as input to the trained machine learning model, wherein the object is associated with a category; Determining the category of an object based on a 2D image of the object, wherein determining the category of an object based on a 2D image of the object includes: Semantic segmentation of a 2D image of the object is performed based on a set of predefined descriptors to determine a given portion of the object, wherein the number of output classes for semantic segmentation matches the number of the predefined descriptors; and The category of the object is determined based on the given portion of the object; Obtain a template 3D mesh associated with the category; and Using the trained machine learning model and based on the 2D image of the object and the template 3D mesh, a 3D mesh of the object is generated. The 3D mesh of the object is obtained by re-meshing the template 3D mesh using the trained machine learning model. The 3D mesh of the object can be used to map the texture of the object or generate 3D animations of the object. The re-meshing is a homeomorphic transformation. The template 3D mesh is a skeletal binding comprising multiple vertices of the template 3D mesh, and each of the multiple vertices is connected to at least one other vertex among the multiple vertices. The method further includes: Obtain one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices of the plurality of vertices; and Based on the 3D mesh of the object and a given template animation from one or more template animations associated with the template 3D mesh, generate at least one object animation for the object, wherein the at least one object animation inherits from the given template animation.

2. The computer implementation method according to claim 1, characterized in that, It also includes a user interface that displays a 3D mesh containing the object.

3. The computer implementation method according to claim 1, characterized in that, It also includes determining the category of the object using image matching based on the 2D image of the object before obtaining the template 3D mesh.

4. The computer implementation method according to claim 1, characterized in that, It also includes mapping the texture to a 3D mesh of the object.

5. The computer implementation method according to claim 1, characterized in that, The machine learning model includes a grid encoder and multiple grid decoders, and the method further includes: Receive multiple target grids; and Training the plurality of mesh decoders, wherein each mesh decoder corresponds to a given template 3D mesh, wherein the training includes: The target mesh is encoded into a shape space vector using the mesh encoder. Using the mesh decoder and the 3D mesh based on the given template, the shape space vector is decoded into a mesh after mesh re-division; Based on the comparison between the target mesh and the mesh after re-division, a first value of the first error function is determined; and Based on the first value, update one or more parameters of the grid decoder.

6. The computer implementation method according to claim 1, characterized in that, The machine learning model includes a grid encoder and an image encoder, and the method further includes training the image encoder, wherein the training includes: Receive multiple 2D training images and corresponding target meshes, wherein each image depicts a given object of a corresponding category and the category corresponds to a specific template 3D mesh; Using the image encoder, a corresponding shape space vector is generated for each of the plurality of 2D training images; Each generated shape space vector is compared with a corresponding shape space vector for the corresponding target mesh, wherein the corresponding shape space vector associated with the corresponding target mesh is encoded using the mesh encoder; The second value of the second error function is determined based on a comparison between the generated shape space vector and the corresponding shape space vector; and Based on the second value, one or more parameters of the image encoder are updated.

7. The computer implementation method according to claim 1, characterized in that, Also includes: Using a UV regressor, the 2D image of the object and the 3D mesh of the object are processed to generate a mapping from the vertices of the 3D mesh of the object to the 2D image of the object, wherein the mapping is used to apply the texture to the 3D mesh of the object.

8. The computer implementation method according to claim 7, characterized in that, The UV regressor is trained using a loss function that includes descriptor loss based on chamfer distance.

9. The computer implementation method of claim 7, wherein processing a 2D image of the object and a 3D mesh of the object to generate a mapping from the vertices of the 3D mesh of the object to the 2D image of the object comprises: Determine the descriptor loss for each side of the 3D mesh of the object; Determine the specific edges of the 3D mesh of the object associated with the minimum descriptor loss; as well as The specific edges of the object's 3D mesh are used as the basis for the object's 3D mesh reflection UV map.

10. A computer system, characterized in that, include: It contains a memory that stores instructions; and A processing device coupled to the memory, the processing device being configured to access the memory, wherein the operations that the instructions cause the processing device to perform when executed by the processing device include: Provide a 2D image of an object as input to a trained machine learning model, wherein the object is associated with a category; Determining the category of an object based on a 2D image of the object, wherein determining the category of an object based on a 2D image of the object includes: Semantic segmentation of a 2D image of the object is performed based on a set of predefined descriptors to determine a given portion of the object, wherein the number of output classes for semantic segmentation matches the number of the predefined descriptors; and The category of the object is determined based on the given portion of the object; Obtain a template 3D mesh associated with the category; and Using the trained machine learning model and based on the 2D image of the object and the template 3D mesh, a 3D mesh of the object is generated. The 3D mesh of the object is obtained by re-meshing the template 3D mesh using the trained machine learning model. The 3D mesh of the object can be used to map the texture of the object or generate 3D animations of the object. The re-meshing is a homeomorphic transformation. The template 3D mesh is a skeletal binding comprising multiple vertices of the template 3D mesh, and each of the multiple vertices is connected to at least one other vertex among the multiple vertices. The operation further includes: Obtain one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices of the plurality of vertices; and Based on the 3D mesh of the object and a given template animation from one or more template animations associated with the template 3D mesh, generate at least one object animation for the object, wherein the at least one object animation inherits from the given template animation.

11. The computer system according to claim 10, characterized in that, The machine learning model includes a grid encoder, an image encoder, and multiple grid decoders, and the operation further includes: Receive multiple target grids; and Training the plurality of mesh decoders, wherein each mesh decoder corresponds to a given template 3D mesh, wherein the training includes: The target mesh is encoded into a shape space vector using the mesh encoder. Using the mesh decoder and the 3D mesh based on the given template, the shape space vector is decoded into a mesh after mesh re-division; Based on the comparison between the target mesh and the mesh after re-division, a first value of the first error function is determined; and Based on the first value, update one or more parameters of the grid decoder; and Training the image encoder includes: Receive multiple 2D training images and corresponding target meshes, wherein each image depicts a given object of a corresponding category and the category corresponds to a specific template 3D mesh; Using the image encoder, a corresponding shape space vector is generated for each of the plurality of 2D training images; Each generated shape space vector is compared with a corresponding shape space vector for the corresponding target mesh, wherein the corresponding shape space vector associated with the corresponding target mesh is encoded using the mesh encoder; The second value of the second error function is determined based on a comparison between the generated shape space vector and the corresponding shape space vector; and Based on the second value, one or more parameters of the image encoder are updated.

12. A non-transitory computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a processing device, cause the processing device to perform the following operations: Provide a 2D image of an object as input to a trained machine learning model, wherein the object is associated with a category; Determining the category of an object based on a 2D image of the object, wherein determining the category of an object based on a 2D image of the object includes: Semantic segmentation of a 2D image of the object is performed based on a set of predefined descriptors to determine a given portion of the object, wherein the number of output classes for semantic segmentation matches the number of the predefined descriptors; and The category of the object is determined based on the given portion of the object; Obtain a template 3D mesh associated with the category; and Using the trained machine learning model and based on the 2D image of the object and the template 3D mesh, a 3D mesh of the object is generated. The 3D mesh of the object is obtained by re-meshing the template 3D mesh using the trained machine learning model. The 3D mesh of the object can be used to map the texture of the object or generate 3D animations of the object. The re-meshing is a homeomorphic transformation. The template 3D mesh is a skeletal binding comprising multiple vertices of the template 3D mesh, and each of the multiple vertices is connected to at least one other vertex among the multiple vertices. The operation further includes: Obtain one or more template animations associated with the template 3D mesh, wherein each template animation defines the motion of two or more vertices of the plurality of vertices; and Based on the 3D mesh of the object and a given template animation from one or more template animations associated with the template 3D mesh, generate at least one object animation for the object, wherein the at least one object animation inherits from the given template animation.

13. The non-transitory computer-readable storage medium according to claim 12, characterized in that, The operation also includes: Using a UV regressor, the 2D image of the object and the 3D mesh of the object are processed to generate a mapping from the vertices of the 3D mesh of the object to the 2D image of the object, wherein the mapping is used to apply the texture to the 3D mesh of the object.

Images