Tokenizing a scene graph using one-hot word vectors and metadata

Abstract

A token engine can receive a data file that describes a three-dimensional (3D) virtual environment using tags for attributes in the 3D virtual environment. The token engine generates a set of one-hot token vectors from the tags in the data file and generates a set of metadata vectors from metadata in the data file, where one or more of the set of metadata vectors correspond to one or more of the set of one-hot token vectors. The token engine combines the set of one-hot token vectors and the set of metadata vectors. The token engine provides the set of combined one-hot token vectors and metadata vectors as input to a deep learning model.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 610,868, filed December 15, 2023, entitled “TOKENIZING A SCENE GRAPH USING ONE-HOT TOKEN VECTORS AND METADATA”, the entire contents of which are hereby incorporated by reference. Technical Field

[0002] This disclosure generally relates to communications and computer graphics, and more specifically, but not exclusively, to methods, systems, and computer-readable media for lexicalizing data files. Background Technology

[0003] A virtual environment is a simulated 3D environment generated from graphical data. A virtual environment can be described using various attributes, including objects with different positions, sizes, colors, etc. Virtual environments can be stored as data files, such as extensible markup language (XML) files.

[0004] The background description provided herein is for the purpose of presenting the context of this disclosure. The work of the currently named inventors described in this background section, and other aspects described that may not constitute prior art at the time of submission, are not, by way of express or implied, an admission that they are prior art to this disclosure. Summary of the Invention

[0005] The embodiments generally relate to a system and method for lexicalizing a data file using one-hot vectors and metadata. A computer-implemented method includes receiving a data file describing a three-dimensional (3D) virtual environment, wherein the data file includes labels for attributes in the 3D virtual environment. The method further includes generating a set of one-hot lexical vectors from the labels in the data file. The method further includes generating a set of metadata vectors from the metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more one-hot lexical vectors in the set of one-hot lexical vectors. The method further includes combining the set of one-hot lexical vectors and the set of metadata vectors. The method further includes providing a combined set of one-hot lexical vectors and metadata vectors as input to a deep learning model.

[0006] In some embodiments, the set of metadata vectors includes regions reserved for one or more floating-point vectors. In some embodiments, the one or more floating-point vectors are associated with a predetermined size suitable for data associated with one or more of the group selected from: coordinate systems, initial size, current size, mesh identifiers, lighting, image textures, mesh components, audio, web services, and combinations thereof. In some embodiments, metadata vectors associated with one-hot meta-vectors lacking metadata in the set of metadata vectors include metadata features set to zero to indicate the absence of metadata. In some embodiments, the deep learning model outputs one or more of the group selected from: recommendations for content generated in the 3D virtual environment, identifiers of service terms violations, scene enhancements in the 3D virtual environment, optimal performance settings, streaming priorities for meshes in the 3D virtual environment, and combinations thereof. In some embodiments, the deep learning model is trained using training data comprising multiple one-hot meta-vectors fused into multiple metadata vectors. In some embodiments, the data file is of the group selected from: XML, JSON, YAML, and universal scene description (USD). In some embodiments, the tag in the data file includes a tag for the name, and the method further includes removing the tag for the name and the name from the data file before generating the set of one-hot meta-vectors. In some embodiments, the set of one-hot meta-vectors and the set of metadata vectors are combined using a fusion operation.

[0007] According to one aspect, a system includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations including: receiving a data file describing a 3D virtual environment, wherein the data file includes labels for attributes in the 3D virtual environment; generating a set of one-hot word vectors from the labels in the data file; generating a set of metadata vectors from metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more one-hot word vectors in the set of one-hot word vectors; combining the set of one-hot word vectors and the set of metadata vectors; and providing a set of combined one-hot word vectors and metadata vectors as input to a deep learning model.

[0008] In some embodiments, the set of metadata vectors includes regions reserved for one or more floating-point vectors. In some embodiments, the one or more floating-point vectors are associated with a predetermined size suitable for data associated with one or more of the group selected from: coordinate systems, initial sizes, current sizes, grid identifiers, lights, image textures, grid components, audio, web services, and combinations thereof. In some embodiments, metadata vectors associated with one-hot word vectors lacking metadata in the set of metadata vectors include metadata features set to zero to indicate the absence of metadata. In some embodiments, the deep learning model is selected from the group selected from: large language models, natural language processing models, and combinations thereof. In some embodiments, the deep learning model is trained using training data comprising multiple one-hot word vectors fused into multiple metadata vectors.

[0009] According to one aspect, a non-transitory computer-readable medium having instructions that, when executed by one or more processors at a user device, cause the one or more processors to perform operations including: receiving a data file describing a 3D virtual environment, wherein the data file includes labels for attributes in the 3D virtual environment; generating a set of one-hot term vectors from the labels in the data file; generating a set of metadata vectors from metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more one-hot term vectors in the set of one-hot term vectors; and providing a combined set of one-hot term vectors and metadata vectors as input to a deep learning model.

[0010] In some embodiments, the set of metadata vectors includes regions reserved for one or more floating-point vectors. In some embodiments, the one or more floating-point vectors are associated with a predetermined size suitable for data associated with one or more of the group selected from: coordinate systems, initial sizes, current sizes, grid identifiers, lighting, image textures, grid components, audio, web services, and combinations thereof. In some embodiments, the metadata vectors associated with one-hot terminator vectors lacking metadata in the set of metadata vectors include metadata features set to zero to indicate the absence of metadata. In some embodiments, the deep learning model is selected from the group selected from: large language models, natural language processing models, and combinations thereof.

[0011] According to another aspect, parts, features, and implementation details of systems, methods, and non-transitory computer-readable media may be combined to form additional aspects (including omitting and / or modifying some or some of the components or features in the individual components or features) including additional components or features and / or other modifications; and all such modifications are within the scope of this disclosure. Attached Figure Description

[0012] Figure 1 Example three-dimensional virtual environments are illustrated according to some embodiments described herein.

[0013] Figure 2 This is a block diagram of an example network environment based on some embodiments described herein.

[0014] Figure 3 These are illustrative diagrams illustrating conventional and novel methods for generating lexical units according to some embodiments described herein.

[0015] Figure 4A These are examples of mesh components described using Extensible Markup Language data according to some embodiments described herein.

[0016] Figure 4B How to perform according to some embodiments described herein Figure 4A An example of reducing the number of lexical units by lexicalizing XML data to include one-hot lexical vectors and metadata vectors.

[0017] Figure 5 This is an example method for combining a set of one-hot word vectors and a set of metadata vectors according to some embodiments described herein.

[0018] Figure 6 This is an example flowchart of an example method for generating a set of combined one-hot word vectors and metadata vectors according to some embodiments described herein.

[0019] Figure 7 This is a block diagram illustrating an example computing device according to some embodiments described herein. Detailed Implementation

[0020] A scene graph is a spatial representation of a graphical scene, organized as a collection of nodes representing different entities and entity attributes in a graph or tree structure. Entities can include three-dimensional (3D) mesh components, lights, sound emitters, scripts, web services, etc., used to create interactive, immersive, and connected 3D experiences.

[0021] In some embodiments, the computing device converts the scene graph into an Extensible Markup Language (XML) file, and the XML file can be lexicalized using byte pair encoding (BPE). BPE works by creating tokens from the text in the XML file, where each token corresponds to one or more characters in the text. Lexicalized text combined with machine learning models has a wide range of applications.

[0022] BPE currently has two limitations. First, BPE supports text data but not other data modalities such as images, audio, and 3D meshes. Second, BPE is inefficient at lexicalizing XML files. This problem is exacerbated when the XML file contains a large number of floating-point numbers.

[0023] Figure 1 An example of a 3D graphics scene 100 is shown. The 3D graphics scene 100 includes a 3D mesh, an audio emitter, and textures. In an XML file, non-textual data (such as the 3D mesh, audio data, and textures) is represented by a Uniform Resource Language (URL). Client devices can render the 3D graphics scene 100 by retrieving the non-textual data using the URL associated with each entity.

[0024] right Figure 1 Lexicalizing a serialized XML file of a 3D graphics scene 100 might yield approximately 28,000 lexical units, a number that could be overwhelming for use by machine learning models. For example, large language models (LLMs) typically support a maximum of about 32,000 lexical units. Even... Figure 1 Even simple scenarios within a virtual environment have scene graphs, which may be unavailable for most use cases leveraging LLM or other machine learning techniques when performing lexicalization using conventional methods. For use cases related to virtual experiences (e.g., any type of immersive space where avatars can view 3D virtual objects and move within the space), the actual scene may be more... Figure 1 The simple examples are far more complex and therefore involve a much larger scene graph. Therefore, current lexicalization techniques are insufficient and unsuitable for such applications. Additionally, according to conventional techniques, non-textual data is represented as a lexicalized version of the corresponding URL, resulting in the loss of any modality-specific information.

[0025] In some embodiments, the techniques described herein advantageously address the problems of creating an extremely large number of lexical units and manipulating multimodal data by using a custom lexicalization method that represents 3D scenes with a significantly fewer number of lexical units (e.g., an order of magnitude fewer than conventional techniques). The lexicalization method includes generating one-hot lexical vectors for labels in a data file and generating corresponding metadata vectors for metadata such as floating-point numbers. This enables multimodal input and reduces the number of lexical units required to represent 3D scenes. Therefore, using… Figure 1 The number of tokens generated during the lexicalization of the serialized XML file of the graphical scene is approximately 28,000, while the number of tokens generated using the lexicalization technique described below is reduced to 519.

[0026] Figure 2 An example network environment 200 according to some embodiments of the present disclosure is illustrated. Figure 2 The same reference numerals are used to identify the same elements as in other figures. A letter following the reference numeral (such as "210a") indicates that the text specifically refers to the element with that particular reference numeral. Reference numerals without a subsequent letter (such as "210") refer to any or all elements in the figure that bear that reference numeral (e.g., "210" in the text refers to reference numerals "210a", "210b", and / or "210n" in the figure).

[0027] The network environment 200 (also referred to herein as the “platform”) includes an online virtual experience server 202, a data storage device 208, client devices 210 (or multiple client devices), and a machine learning server 238, all of which are connected via a network 222.

[0028] The online virtual experience server 202 may include a virtual experience engine 204, one or more virtual experiences 105, and a lexicalization engine 230, etc. In some embodiments, the online virtual experience server 202 may be configured to provide virtual experiences 205 to one or more client devices 110 and to provide lexicalization functionality via the lexicalization engine 230.

[0029] Data storage device 208 is shown coupled to online virtual experience server 202, but in some embodiments, it may also be provided as part of online virtual experience server 202. In some embodiments, the data storage device may be configured to store advertising data, user data, engagement data, and / or other contextual data associated with lexicalization engine 230.

[0030] Client device 210 (e.g., 210a, 210b, 210n) may include virtual experience application 212 (e.g., 212a, 212b, 212n) and I / O interface 214 (e.g., 214a, 214b, 214n) to interact with online virtual experience server 202 and view a graphical user interface (GUI), for example, through a computer monitor or display (not illustrated). In some embodiments, client device 210 may be configured to execute and display a virtual experience, which may include a virtual user engagement portal as described herein.

[0031] A network environment 200 is provided for illustration. In some embodiments, the network environment 100 may include interfaces with... Figure 2 The same, fewer, more, or different components are configured in the same or different ways as shown.

[0032] In some embodiments, network 222 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a long term evolution (LTE) network), a router, a hub, a switch, a server computer, or a combination thereof.

[0033] In some embodiments, the data storage device 208 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or another type of component or device capable of storing data. The data storage device 208 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0034] In some embodiments, the online virtual experience server 202 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, virtual server, etc.). In some embodiments, the server may be included in the online virtual experience server 202, and may be a standalone system or part of another system or platform. In some embodiments, the online virtual experience server 202 may be a single server, or any combination of multiple servers, load balancers, network devices, and other components. The online virtual experience server 202 may also be implemented on a physical server, but in some embodiments, virtualization technology may be utilized. Other variations of the online virtual experience server 202 are also applicable.

[0035] In some embodiments, the online virtual experience server 202 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage devices (e.g., hard disks, storage devices, databases), networks, software components, and / or hardware components that can be used to perform operations on the online virtual experience server 202 and provide access to the online virtual experience server 202 to users (e.g., to users 214 via client device 210).

[0036] The online virtual experience server 202 may also include a website (e.g., one or more web pages) or application backend software that can be used to provide users with access to the content offered by the online virtual experience server 202. For example, a user (or developer) may access the online virtual experience server 202 using the virtual experience application 212 on the client device 210.

[0037] In some embodiments, the online virtual experience server 202 may include digital asset and digital virtual experience generation capabilities. For example, the platform may provide an administrator interface for design, modification, personalized customization, and other modifications. In some embodiments, virtual experiences may include, for example, two-dimensional (2D) games, three-dimensional (3D) games, virtual reality (VR) games, or augmented reality (AR) games. In some embodiments, virtual experience creators and / or developers may search for virtual experiences, combine portions of virtual experiences, customize virtual experiences for specific activities (e.g., group virtual experiences), and other features provided by the virtual experience server 202.

[0038] In some embodiments, the online virtual experience server 202 or client device 210 may include a virtual experience engine 204 or a virtual experience application 212. In some embodiments, the virtual experience engine 204 may be used for the development or execution of a virtual experience 205. For example, the virtual experience engine 204 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting functionality, a haptic engine, an artificial intelligence engine, network functionality, streaming functionality, memory management functionality, threading functionality, scene graph functionality, or video support for movies, and other features. Components of the virtual experience engine 204 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) to help calculate and render the virtual experience.

[0039] The online virtual experience server 202 using virtual experience engine 204 can execute some or all of the virtual experience engine functions (e.g., generating physics commands, rendering commands, etc.), or offload some or all of the virtual experience engine functions to the virtual experience engine 204 on the client device 210 (not illustrated). In some embodiments, the ratio between the virtual experience engine functions executed on the online virtual experience server 202 and the virtual experience engine 204 functions executed on the client device 210 for each virtual experience 205 can be different.

[0040] Lexicalization engine 230 may receive, for example, a data file describing a 3D virtual environment from virtual experience engine 204. The data file uses tags for attributes within the 3D virtual environment. Lexicalization engine 230 can generate a set of one-hot lexical vectors from the tags in the data file. Lexicalization engine 230 can also generate a set of metadata vectors from the metadata in the data file, wherein one or more metadata vectors in this set correspond to one or more one-hot lexical vectors in this set of one-hot lexical vectors. Lexicalization engine 230 can combine this set of one-hot lexical vectors and the set of metadata vectors. Lexicalization engine 230 can provide a combined set of one-hot lexical vectors and metadata vectors as input to machine learning server 238.

[0041] Machine learning server 238 hosts machine learning model 240, such as a deep learning model. In some embodiments, machine learning model 240 is trained on training data including one-hot word vectors and metadata vectors. In some embodiments, machine learning model 240 is a large language model (LLM) that receives prompts from, for example, a virtual experience engine 204 and provides responses. Machine learning model 240 may receive prompts for scene graphs that match features described in the prompts. Machine learning model 240 identifies one or more scene graphs that match the prompts and provides responses to virtual experience engine 204. Virtual experience engine 204 can use the scene graphs to generate a virtual experience 205.

[0042] In some embodiments, virtual experience instructions may be instructions that enable client device 210 to render gameplay, graphics, and other features of virtual experience 205. Instructions may include one or more of the following: user input (e.g., physical object location), character location and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0043] In some embodiments, client devices 210 may each include computing devices (such as personal computers (PCs)), mobile devices (e.g., laptops, mobile phones, smartphones, tablets, or netbooks), network-connected televisions, game consoles, etc. In some embodiments, client devices 210 may also be referred to as "user devices." In some embodiments, one or more client devices 210 may connect to the online virtual experience server 202 at any given time. It should be noted that the number of client devices 210 is provided as an example and not as a limitation. In some embodiments, any number of client devices 210 may be used.

[0044] In some embodiments, each client device 210 may include an instance of a virtual experience application 212. The virtual experience application 212 may be rendered for interaction at the client device 210. During user interaction within a virtual experience or another GUI of the online platform 200, the user can create avatars including different body parts from different libraries.

[0045] The virtual experience engine 204 can generate an instance that hosts a specific location for the virtual experience 205. The virtual experience application 212 on the client device 210 joins the virtual experience 205 by maintaining a connection with the online virtual experience server 202 for the specific virtual experience 205.

[0046] Figure 3 This is an example illustration of a conventional method 300 for generating lexical vectors and another method 350 for generating lexical vectors. According to the conventional method 300, a data file is lexicalized to create one-hot lexical vectors 305. The data file can be XML data, and one-hot lexical vectors 305 can be created using BPE. For example, in an XML structure... <item class=""MeshPart”">In XML, the structure generates seven tokens. Specifically, these seven tokens are <, Item, class, =", Mesh, Part, and ">. Although the following examples are described with reference to XML data, these techniques can also be used with other data file types, such as JSON, YAML, or Common Scenario Description (USD).

[0047] One-hot lexical vectors are 1×N matrices (i.e., vectors) used to distinguish each word in the vocabulary from every other word in that vocabulary by using 0 in all cells, except for one cell containing a 1 to uniquely identify the word (one-hot means that any vector has only one cell with a value of 1). The 0s and 1s are indexed such that the position of the 1 maps to the key identifying the corresponding word. For example, in a set of BaseParts including TrussPart, SpecialMesh, BlockMesh, WedgePart, MeshPart, and Part, the one-hot vector of the BaseParts can be represented as 0, 0, 0, 0, 1, 0 to represent six distinct BaseParts. MeshParts are indexed with a 1 at position 4 (i.e., 1 is the fifth position in the vector when the index starts from 0). Therefore, using one-hot lexical vectors to map MeshParts to single lexical units is preferable to using BPEs to create seven lexical units. <item class=""MeshPart”">.

[0048] In the simple example above, lemmatization is slightly reduced. However, for a scenario involving many different objects with different attributes, the results are cumulative. Furthermore, attributes can be represented by many more variations. For example, when describing materials in a scene, there might be 45 different materials, such as plastic, wood, marble, slate, limestone, etc. In this example, wood might be located at position 4 out of 45, where that position indicates 45 different types of materials.

[0049] The scene is represented by a set of one-hot word vectors 305. These one-hot word vectors 305 can be combined to form a word sequence 310. In this example, the word sequence 310 comprises four one-hot word vectors 305. The word sequence 310 can be represented in a matrix of one-hot word vectors 305. The word sequence 310 (e.g., as a matrix) can be provided as input to a deep learning model 315.

[0050] Method 350 similarly lexicalizes the data file to create one-hot lexical vectors 355. The one-hot lexical vectors 355 can be combined to form a lexical sequence 360. The lexical sequence 360 ​​can be a matrix of lexical vectors. The lexical sequence 360 ​​is combined with metadata features 365. In some embodiments, the one-hot lexical vectors are appended with metadata features 365.

[0051] The matrix of word vectors is used as follows Figure 3 The fusion operation 370 illustrated here combines the matrix of metadata features 365. The fusion operation can take various forms, including single matrix multiplication and the output from a deep neural network. Deep neural networks can include different layers, such as convolutions and Transformers.

[0052] In some embodiments, some lexical vectors in the lexical vectors do not have metadata. In these cases, the row in the metadata feature corresponding to the lexical can be set to zero. For example, the combined one-hot lexical vector without metadata and the metadata vector can be 0, 0, 1, 0, 0, 0, 0, 0 for the one-hot lexical vector and 0, 0, 0 for the metadata vector.

[0053] The fusion operation 370 can output a fused matrix. This fused matrix can be provided as input to a deep learning model 375. The deep learning model 375 can be a machine learning model using the fused matrix for various applications. For example, the deep learning model 375 can include Large Language Model (LLM), Natural Language Processing (NLP) models, etc.

[0054] Figure 4A This is an example of a mesh component described in XML data 400 according to some embodiments described herein. The mesh component represents a 3D mesh, such as a mesh for an object used as part of a virtual environment. The XML data 400 for the mesh component includes a coordinate system, initial size, material, mesh identifier, name, and current size (referred to as "size" in the XML data 400).

[0055] A coordinate system describes the 3D position and orientation of an object within a virtual environment. A coordinate system consists of a point in space and a set of three mutually perpendicular directions called axes (i.e., the x-axis, y-axis, and z-axis). A CFrame is an attribute that encodes the position and orientation of the coordinate system relative to the virtual environment's coordinate system. In some embodiments, a CFrame is described using 12 numbers, where the first three numbers describe the position of the CFrame's origin with reference to the x-axis, y-axis, and z-axis, respectively; the next three numbers describe the direction vector of the x-axis; the next three numbers describe the direction vector of the y-axis; and the last three numbers describe the direction vector of the z-axis. In this example, the CFrame is described by (29.4411926, 16.5932083, 25.9831161), (R00, R01, R02), (R10, R11, R12), and (R20, R21, R22).

[0056] The XML data 400 also includes initial size, material, mesh identifier (meshID), name, and size. Initial size defines the initial mesh size. Material defines what type of material the mesh should be considered, which may be important for physical or audio functionality. In some embodiments, meshID refers to a URL (or other type of pointer) to a 3D mesh file. Name refers to a custom name that can be freely assigned. Size can be the current size of the mesh component in the scene.

[0057] Use the BPE lexicizer to Figure 4A The XML data described above, when lexicalized (400 bytes), will produce 358 tokens. As described above, when using BPE for lexicalization, only... <item class="MeshPart">It requires seven lexical units.

[0058] Figure 4B How to perform according to some embodiments described herein Figure 4A Example 450 shows how XML data 400 is lexicalized to include one-hot lexical vectors 455 and metadata vectors 460, thereby reducing the number of lexical units. Tags MeshPart 457, CoordinateFrame 459, Size 461, and MeshID 463 are converted into their corresponding one-hot lexical vectors 455.

[0059] The lexical engine 230 identifies the parts of XML data to be represented by lexical tokens and the parts designated as metadata. Any type of data structure can be designated as metadata, but the lexical engine 230 is best suited to designate floating-point numbers and embeddings as metadata because it is difficult to efficiently represent floating-point numbers using a text-based lexicalizer.

[0060] In some embodiments, the lexicalization engine 230 generates one-hot lexical units for tags, making <item class="MeshPart">Mapped to one lexical instead of the seven lexical units of the conventional technique, and <coordinateframe name="CFrame">Instead of the nine lemmas used in conventional techniques, it maps to a single lexical unit. In some embodiments, the lexicalization engine 230 removes the name field (e.g., ...). Figure 4A In<stringname="Name"> Leaves), because the name field does not need to represent the 3D scene, nor does it need to have contextual meaning. Due to the tokenization of tags in XML data 400 into individual tokens, the number of tokens (for...) Figure 4A The number of XML terms was reduced from 358 when using the BPE morphistor to 186.

[0061] MeshPart 457 represents an item category and is not associated with metadata. Therefore, the metadata vector 460 for CFrame 462, the vector3 field 464, and the meshID field 466 is set to zero. CoordinateFrame 459 has a metadata vector 460 for CFrame 462 with a size of 12, which is a floating-point vector including floating-point values. Size 461 is associated with a size of 3 for the metadata vector 460 for vector3 464. The Mesh ID field 463 is associated with a size of 1024, allowing the mesh-specific embedding to fit within the vector. In some embodiments, the mesh-specific embedding is generated using principal component analysis (PCA), a linear dimensionality reduction technique used to transform a set of data points into vectors. Other embedding methods include geometric embedding or image-based embedding.

[0062] Other types of tags in the data file can be used and processed in different ways. For example, such as Figure 4A The material tag illustrated can be associated with metadata, or it can be associated with a custom lexicogram representing different types of materials (e.g., 45 different types). The lexicography engine 230 may or may not lexicize the name field.

[0063] Figure 5 This is an example method 500 for combining a set of one-hot lexical vectors 505 and a set of metadata vectors 515 according to some embodiments described herein. Lexicalization engine 230 generates one-hot lexical vectors 505 and combines the one-hot lexical vectors to form a lexical sequence 510.

[0064] Lexicalization engine 230 generates metadata vector 515. Metadata vector 515 may include coordinate system 520, current size 525, initial size 530, and grid ID 535. Lexicalization engine 230 may reserve a 12-bit floating-point vector for coordinate system 520, injecting a 12-bit numerical coordinate system into this vector. Similarly, lexicalization engine 230 may reserve three floating-point vectors for the current size 525 and three floating-point vectors for the initial size 530. Lexicalization engine 230 can generate a custom lexical to represent grid ID 535. In this example, PCA feature extraction is used to generate a 1024-dimensional floating-point vector for grid ID 535, but other techniques are also possible.

[0065] The table below shows the names of different types of tokens, the corresponding structure names in XML, and the size of the metadata vectors generated from the metadata.

[0066] Table 1.0

[0067] The techniques described in this article can also be used with other types of metadata. For example, metadata can include audio, image textures, mesh components, lighting, web services, etc.

[0068] The lexical sequence 510 is combined with the metadata vector 515. For example, the matrix of lexical vectors can be combined with the metadata feature matrix using a fusion operation 540. Although the metadata vector 515 is exemplified as including a coordinate system field 520, a current size field 525, an initial size field 530, and a grid ID field 535, some of these fields can be omitted, other fields can be added, and the order of the fields can be different. In some embodiments, some lexical vectors in the lexical vector do not have metadata. In these cases, the row in the metadata feature corresponding to the lexical can be set to zero.

[0069] The fusion operation 540 can output a fused matrix. The fused matrix can be provided as input to a deep learning model 545. The deep learning model 545 can be a machine learning model using the fused matrix for various applications.

[0070] The deep learning model 545 can output recommendations for content (e.g., virtual experiences, movies, games, etc.) based on a 3D scene represented by a fused matrix provided as input. The deep learning model 545 can output recommendations for advertisements placed within the scene graph, where the recommendations include the advertisement's location and type within the 3D scene. When a virtual experience is at least partially rendered at a remote server but displayed on one or more client devices, ensuring that high-priority (those with the greatest impact on user experience) grids are streamed with higher priority than other grids within the 3D scene can help improve the responsiveness of the virtual experience on the virtual experience platform. The deep learning model 545 can also identify appropriate performance settings for rendering the scene based on the fused matrix as input.

[0071] Deep learning model 545 can predict enhancements, expansions, or modifications to a 3D scene represented by a scene graph. Deep learning model 545 can perform scene enhancements, such as generating virtual objects for placement in the 3D scene, identifying virtual objects suitable for placement in the 3D scene (e.g., 3D models or other assets), and can help developers of virtual experiences accelerate their workflow in designing 3D scenes. In some implementations, text prompts (e.g., written by the developer) and a fused matrix can be additionally provided to deep learning model 545. For example, the developer can include text such as "Please suggest a clock that can be placed along the far wall in this scene." In response, deep learning model 545 can analyze the prompt and the fused matrix, for example, to identify the far wall, determine its size, identify scene characteristics (e.g., a medieval dungeon), and automatically identify virtual assets that can be placed along the far wall, as well as any appropriate modifications to the clock before placing it in the scene (e.g., scaling, rotation, color change, or other transformations). In another example, deep learning model 545 can receive prompts for a scene graph that matches the characteristics described in the prompt. For example, the prompt could include a request for a scene image of a village with a wooden church. Machine learning model 240 identifies one or more scene images that match "wooden" and "church".

[0072] In some embodiments, deep learning model 545 can identify service terms violations. For example, machine learning model 240 can periodically examine the live experience and determine if the scene graph exhibits patterns that are known service terms violations. Because the experience dynamically loads data, static analysis of game files is insufficient to identify service terms violations. In some embodiments, deep learning model 545 is trained to operate on game states to improve the accuracy of service terms violation detection.

[0073] Figure 6 This is an example flowchart of an example method 600 for generating a set of combined one-hot word vectors and metadata vectors. In some embodiments, all or part of method 600 is generated by... Figure 2 The word meta-engine 230 stored on the online virtual experience server 202 or the virtual experience application 212 located on the client device 210 is executed.

[0074] Method 600 may begin at box 602. At box 602, a data file describing the 3D virtual environment using tags for attributes within the 3D virtual environment is received. The data file may be XML, JSON, YAML, USD, or other binary representations. Box 602 may be followed by box 604.

[0075] At box 604, a set of one-hot word vectors is generated from the labels in the data file. For example, <item class="MeshPart">Mapped to single-hot words, <coordinateframe name="CFrame">Mapping to one-hot terms, etc. The data file may include labels for names. In such cases, the labels for names and the corresponding names can be removed from the data file before generating the set of one-hot term vectors. Box 604 can be followed by box 606.

[0076] At box 606, a set of metadata vectors is generated from the metadata in the data file, wherein one or more metadata vectors in this set correspond to one or more one-hot term vectors in this set of one-hot term vectors. This set of metadata vectors may include regions reserved for one or more floating-point vectors. The one or more floating-point vectors may be associated with a predetermined size suitable for data associated with one or more of the following: coordinate system, initial size, current size, mesh identifier, lighting, image texture, mesh components, audio, and / or web services.

[0077] In some embodiments, one-hot term vectors may not be associated with metadata. Therefore, the metadata vector associated with a one-hot term vector lacking metadata in this set of metadata vectors includes a single-line metadata feature set to zero to indicate the absence of metadata. Box 606 may be followed by box 608.

[0078] At box 608, the set of one-hot word vectors and the set of metadata vectors are combined. The set of one-hot word vectors can be combined with the set of metadata vectors using a fusion operation or by appending the metadata vectors to the corresponding one-hot word vectors. The fusion operation can take various forms, including single matrix multiplication and outputs from deep neural networks comprising different layers (such as convolutions and Transformers). Box 608 can be followed by box 610.

[0079] At box 610, a set of combined one-hot word vectors and metadata vectors is provided as input to a deep learning model. The deep learning model can be a Large Language Model (LLM), a Natural Language Processing model, and / or any other sequence-based deep learning model. The deep learning model can be trained using training data comprising multiple one-hot word vectors fused into multiple metadata vectors. In some embodiments, the deep learning model output is selected from one or more of the following: recommendations for content generated in a 3D virtual environment, identifiers of service term violations, scene enhancements in a 3D virtual environment, optimal performance settings, streaming priority of meshes in a 3D virtual environment, and combinations thereof.

[0080] Example computing device In the following text, refer to Figure 7 A more detailed description is provided of various computing devices that can be used to implement Figure 2 The various devices and / or components illustrated herein.

[0081] Figure 7 This is a block diagram of an example computing device 700 according to some embodiments, which can be used to implement one or more features described herein. In one example, device 700 may be used to implement a computer device (e.g., Figure 2 (202, 210), and performs the appropriate operations as described herein. The computing device 700 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 700 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smartphone, tablet computer, television, TV set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the device 700 includes a processor 702, a memory 704, an input / output (I / O) interface 706, and an audio / video input / output device 714 (e.g., display screen, touchscreen, display goggles or glasses, audio speaker, headphones, microphone, etc.).

[0082] Processor 702 can be one or more processors and / or processing circuitry to execute program code and control the basic operations of device 700. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functionality, or other systems. Processing is not limited to a specific geographical location or time. For example, a processor may perform its functions in "real-time," "offline," or "batch mode." Parts of the processing may be executed by different (or the same) processing systems at different times and locations. A computer can be any processor that communicates with memory.

[0083] Memory 704 is typically disposed in device 700 for access by processor 702 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. This processor-readable storage medium is suitable for storing instructions for execution by the processor and is separate from and / or integrated with the processor 702. Memory 704 may store software operated by processor 702 on server device 700, including operating system 708, software application 710, and associated data 712. In some embodiments, application 710 may include instructions that enable processor 702 to perform the functions described herein. Software application 710 may include some or all of the functionality required to generate lexical units. In some embodiments, one or more portions of the software application 710 may be implemented in dedicated hardware, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), a machine learning processor, etc. In some embodiments, one or more portions of the software application 710 may be implemented in a general-purpose processor, such as a central processing unit (CPU) or a graphics processing unit (GPU). In various embodiments, the software application 710 may be implemented using a suitable combination of dedicated processing hardware and / or general-purpose processing hardware.

[0084] For example, software application 710 stored in memory 704 may include instructions for retrieving user data, displaying / presenting avatars, and / or other functionalities or software (such as modeling component 130, VE engine 104, and / or VE application 112). Any software in memory 704 may alternatively be stored in any other suitable storage location or on a computer-readable medium. Furthermore, memory 704 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 704 and any other type of storage device (disk, optical disk, magnetic tape, or other tangible media) may be considered a "storage device" or "storage apparatus".

[0085] I / O interface 706 provides functionality that enables server device 700 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage device 106), and input / output devices can communicate via interface 706. In some embodiments, the I / O interface can connect to interface devices including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0086] For ease of illustration, Figure 7 A block is shown for each of processor 702, memory 704, I / O interface 706, software blocks 708 and 710, and database 712. These blocks may represent one or more processors or processing circuitry, operating systems, memory, I / O interfaces, applications, and / or software modules. In other embodiments, device 700 may not have all of the components shown and / or may have other elements, including elements that replace those shown herein or other types of elements besides those shown herein. While online server 202 is described as performing the operations described in some embodiments herein, any suitable component or combination of components of online server 202 or similar systems, or any suitable one or more processors associated with such systems, may perform the described operations.

[0087] User equipment may also implement and / or be used with the features described herein. Example user equipment may be a computer device including components similar to those of device 700 (e.g., processor 702, memory 704, and I / O interface 706). Operating systems, software, and applications suitable for the client device may be provided in memory and used by the processor. The I / O interface for the client device may be connected to network communication devices and input / output devices, such as microphones for capturing sound, cameras for capturing images or video, audio speaker devices for outputting sound, display devices for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 714 may be connected to device 700 (or included in the device) to display pre-processed and post-processed images as described herein, wherein such display devices may include any suitable display device, such as LCD, LED, or plasma displays, CRT, television, monitor, touchscreen, 3D display, projector, or other visual display devices. Some embodiments may provide audio output devices, such as speech output for reading text aloud or synthesized speech.

[0088] The methods, blocks, and / or operations described herein may be performed in a different order than those shown or described, and / or, where appropriate, performed concurrently (partially or completely) with other blocks or operations. Some blocks or operations may be performed on a portion of the data and then performed again later (e.g., on another portion of the data). In various embodiments, not all described blocks and operations need to be performed. In some embodiments, blocks and operations may be performed multiple times in a method in different orders and / or at different times.

[0089] In some embodiments, some or all of the methods may be implemented on a system such as one or more client devices. In some embodiments, one or more methods described herein may be implemented, for example, on a server system, and / or on both a server system and a client system. In some embodiments, different components of one or more servers and / or clients may perform different blocks, operations, or other parts of the methods.

[0090] One or more methods described herein (e.g., method 600) can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium), such non-transitory computer-readable medium being, for example, magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memory, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, hard disks, optical disks, solid-state storage drives, etc. The program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0091] One or more methods described herein can operate in the following ways: a standalone program that can run on any type of computing device; a program that runs on a web browser; or a mobile application ("app") that executes on a mobile computing device (e.g., a mobile phone, smartphone, tablet, wearable device (watch, armband, jewelry, headwear, goggles, glasses, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, whereby the mobile computing device (as a client device) transmits user input data to a server device and receives real-time feedback data from the server for output (e.g., for display). In another example, computation can be split between the mobile computing device and one or more server devices.

[0092] In some embodiments discussed herein, where user data (e.g., user demographics, user behavior data, user contextual data, user advertising settings, etc.) may be available or used, users are provided with options to control whether and how such information is collected, stored, or used. That is, the embodiments discussed herein collect, store, and / or use user information only when explicit user authorization is received and applicable regulations are met.

[0093] Users can control whether a program or feature collects user information about that specific user or other users associated with that program or feature. Options are presented to each user whose information is to be collected (e.g., via a user interface), allowing the user to control the information collection related to that user, thereby granting permission or authorization for whether information is collected and which parts of the information are collected. Furthermore, some data can be modified in one or more ways before storage or use, thereby deleting personally identifiable information. As an example, a user's identity can be modified (e.g., by replacing names with pseudonyms, numbers, etc.) to make it impossible to determine any personally identifiable information. In another example, a user's geographic location can be generalized to a larger area (e.g., city, postal code, state, country, etc.).

[0094] While the description has been given with reference to specific embodiments thereof, these specific embodiments are merely illustrative and not limiting. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0095] Note that the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. Routines of a particular embodiment can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. The routine can be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be presented in a specific order, this order can be varied in different particular embodiments. In some embodiments, multiple steps or operations shown as being performed sequentially in this specification may be performed simultaneously.< / coordinateframe> < / item> < / coordinateframe> < / item> < / item> < / item> < / item>

Claims

1. A computer-implemented method, the computer-implemented method comprising: Receive a data file describing a three-dimensional (3D) virtual environment, wherein the data file includes tags for attributes in the 3D virtual environment; Generate a set of unique-hot word vectors from the tags in the data file; A set of metadata vectors is generated from the metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more unique hot word vectors in the set of unique hot word vectors; Combine the set of unique hot word vectors and the set of metadata vectors; as well as Provide a set of combined one-hot word vectors and metadata vectors as input to the deep learning model.

2. The method of claim 1, wherein the set of metadata vectors includes a region reserved for one or more floating-point vectors.

3. The method of claim 2, wherein the one or more floating-point vectors are associated with a predetermined size, the predetermined size being suitable for data associated with one or more of the group consisting of: coordinate system, initial size, current size, grid identifier, light, image texture, grid component, audio, network service, and combinations thereof.

4. The method of claim 1, wherein the metadata vector associated with the one-hot meta-vector lacking metadata in the set of metadata vectors includes a metadata feature set to zero to indicate the absence of metadata.

5. The method of claim 1, wherein the deep learning model output is selected from one or more of the group consisting of: recommendations for content generated in the 3D virtual environment, identifiers of violations of terms of service, scene enhancements in the 3D virtual environment, optimal performance settings, streaming priorities for grids in the 3D virtual environment, and combinations thereof.

6. The method according to claim 1, wherein the deep learning model is trained using training data, the training data including multiple one-hot word vectors, the multiple one-hot word vectors being fused into multiple metadata vectors.

7. The method of claim 1, wherein the file type of the data file is selected from the group consisting of: Extensible Markup Language (XML), JSON, YAML, and Universal Scenario Description (USD).

8. The method of claim 1, wherein the tags in the data file include tags for names, and the method further comprises: Before generating the set of unique hot word vectors, the tag for the name and the name are removed from the data file.

9. The method of claim 1, wherein combining the set of unique hot word vectors and the set of metadata vectors is performed using a fusion operation.

10. A system comprising: processor; and A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations including: Receive a data file describing a three-dimensional (3D) virtual environment, wherein the data file includes tags for attributes in the 3D virtual environment; Generate a set of unique-hot word vectors from the tags in the data file; A set of metadata vectors is generated from the metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more unique hot word vectors in the set of unique hot word vectors; Combine the set of unique hot word vectors and the set of metadata vectors; as well as Provide a set of combined one-hot word vectors and metadata vectors as input to the deep learning model.

11. The system of claim 10, wherein the set of metadata vectors includes regions reserved for one or more floating-point vectors.

12. The system of claim 11, wherein the one or more floating-point vectors are associated with a predetermined size, the predetermined size being suitable for data associated with one or more of the group consisting of: coordinate system, initial size, current size, grid identifier, light, image texture, grid component, audio, network service, and combinations thereof.

13. The system of claim 10, wherein the metadata vector associated with the one-hot meta-vector lacking metadata in the set of metadata vectors includes a metadata feature set to zero to indicate the absence of metadata.

14. The system of claim 10, wherein the deep learning model is selected from the group consisting of large language models, natural language processing models, and combinations thereof.

15. The system of claim 10, wherein the deep learning model is trained using training data, the training data comprising multiple one-hot word vectors, the multiple one-hot word vectors being fused into multiple metadata vectors.

16. A non-transitory computer-readable medium having instructions that, when executed by one or more processors at a user equipment, cause the one or more processors to perform operations, the operations including: Receive a data file describing a three-dimensional (3D) virtual environment, wherein the data file includes tags for attributes in the 3D virtual environment; Generate a set of unique-hot word vectors from the tags in the data file; A set of metadata vectors is generated from the metadata in the data file, wherein one or more metadata vectors in the set of metadata vectors correspond to one or more unique hot word vectors in the set of unique hot word vectors; Combine the set of unique hot word vectors and the set of metadata vectors; as well as Provide a set of combined one-hot word vectors and metadata vectors as input to the deep learning model.

17. The non-transitory computer-readable medium of claim 16, wherein the set of metadata vectors includes regions reserved for one or more floating-point vectors.

18. The non-transitory computer-readable medium of claim 17, wherein the one or more floating-point vectors are associated with a predetermined size suitable for data associated with one or more of the group consisting of: coordinate systems, initial sizes, current sizes, grid identifiers, lights, image textures, grid components, audio, network services, and combinations thereof.

19. The non-transitory computer-readable medium of claim 16, wherein the metadata vector associated with a one-hot term vector lacking metadata in the set of metadata vectors includes a metadata feature set to zero to indicate the absence of metadata.

20. The non-transitory computer-readable medium of claim 16, wherein the deep learning model is selected from the group consisting of large language models, natural language processing models, and combinations thereof.

Images