SYSTEM AND METHOD FOR IDENTIFYING MATERIAL TYPE OF OBJECTS IN A REAL WORLD ENVIRONMENT - Patent application

Abstract

A method and system for identifying the material type of an object in a real-world environment is described. One of the methods includes receiving a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments, the method further includes receiving a plurality of input data sets for a plurality of material types of the plurality of objects, training an artificial intelligence (AI) model based on the plurality of audio data sets and the plurality of input data sets, and applying the AI ​​model to the audio data sets captured from the real-world environment to identify material types of objects in the real-world environment and enhance immersion in augmented reality (AR) or virtual reality (VR) video games and applications.

Description

[Technical field]

[0001] The present disclosure relates to systems and methods for identifying material types of objects in a real-world environment. [Background technology]

[0002] In a multiplayer game, there are multiple game players. Each player wears a head-mounted display (HMD) to play the game or view the environment generated by the execution of an application. While playing the game or running the application, some objects are displayed on the HMD. However, sometimes the players cannot recognize these objects in the environment.

[0003] It is against this background that embodiments of the present invention arise. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide systems and methods for identifying material types of objects in a real-world environment.

[0005] In an embodiment, the material of an object in a real-world environment has properties that can produce sounds when interfered with, such as compression, contact, and movement. For example, when a user sits on a chair, the chair responds with a noise based on the physical properties of the chair's seat material. Illustratively, the seat of the chair is made of plastic, and when a user sits on it, it makes a squishing sound reminiscent of being deflated. By identifying the properties of an object, it is possible to leverage that information to augment a virtual space, such as a virtual reality (VR) environment or an augmented reality (AR) environment, to mimic the object being interfered with. For example, if a user sits on a soft chair in a real-world environment, a similar soft chair with similar properties reminiscent of or related to the sound detected when the user sits on the chair may be depicted in the AR view or VR space. As another example, if a user sits on a bus seat made of cheap plastic and foam, a virtual bus seat with similar properties reminiscent of or related to the sound detected when the user sits on it may be depicted in the AR view or VR space. To leverage the properties, a VR recreation of a certain type of chair may be created and a virtual user may be shown sitting on the same type of chair. Thus, a mapping is performed from the real-world environment to the virtual space, and audio cues from the real-world environment are used to mimic and recreate the surfaces, sounds, and properties of objects in the real-world environment when a similar virtual object is placed in the virtual space. Physics associated with sounds made by real-world objects may be recreated by virtual objects in the virtual space. Thus, the physics recreated in the virtual space may show the virtual seat of a virtual chair denting when a virtual user or character in a game sits in the virtual chair.

[0006] In embodiments, when a particular material or object is transformed in the real world, lighting, sound, and other environmental characteristics are detected to determine how that material or object will react in the real world, and the reaction of the real-world object is then utilized in the virtual space to display a virtually changing object that is consistent with the transformations and changes occurring in the real-world environment.

[0007] In one embodiment, a method for identifying a material type of an object in a real-world environment is described. The method includes receiving a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments. The method further includes receiving a plurality of input data sets related to a plurality of material types of the plurality of objects, training and / or inferring an artificial intelligence (AI) model based on the plurality of audio data sets and the plurality of input data sets, and applying the AI ​​model to the audio data sets captured from the real-world environment to identify a material type of the object in the real-world environment. By way of example, the input data may include audio data, image data, additional input data such as Light Detection and Ranging (LiDAR) data, or Inertial Measurement Unit (IMU) data, or a combination of the audio data, image data, LiDAR data, and the additional input data.

[0008] In an embodiment, a server for identifying a material type of an object in a real-world environment is described. The server includes a processor that receives a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments. The processor further receives a plurality of input data sets related to a plurality of material types of the plurality of objects. The processor also trains and / or infers an AI model based on the plurality of audio data sets and the plurality of input data sets. By way of example, the input data includes additional input data such as audio data, image data, LiDAR data, or IMU data, or a combination of audio data, image data, LiDAR data, and additional input data. The processor applies the AI ​​model to the audio data set captured from the real-world environment to identify a material type of the object in the real-world environment. The server includes a memory device connected to the processor.

[0009] In one embodiment, a system for identifying a material type of an object in a real-world environment is described. The system includes a plurality of client devices. The plurality of client devices generate a plurality of audio data sets based on sounds received from a plurality of objects in the plurality of environments. The client devices also receive a plurality of input data sets related to a plurality of material types of the plurality of objects. By way of example, the input data includes additional input data such as audio data, image data, LiDAR data, or IMU data, or a combination of the audio data, image data, LiDAR data, and the additional input data. The system further includes a server connected to the plurality of client devices via a computer network. The server receives the plurality of audio data sets from the plurality of client devices via the computer network, receives the plurality of input data sets from the plurality of client devices via the computer network, and performs training and / or inference on an AI model based on the plurality of audio data sets and the plurality of input data sets. The server applies the AI ​​model to the audio data sets captured from the real-world environment to identify a material type of the object in the real-world environment.

[0010] Some advantages of the systems and methods described herein include providing a way for a visually impaired person to be guided where to sit. For example, a visually impaired person wears glasses. The glasses output a sound to the visually impaired person that indicates a material type of a seat in a real-world environment. If a first seat is made of a hard material compared to a second seat made of a soft cushion material, the glasses inform the visually impaired person. The visually impaired person may then sit on the second seat. Further, in an example, audio data is received by an AI model regarding sounds emanating from the seat when other people sit or stand up. The AI ​​model may be trained based on the audio data. The AI ​​model is then applied to determine whether the visually impaired person is attempting to sit on a hard material or a soft cushion material.

[0011] Further advantages of the systems and methods described herein include providing tools for creating a metaverse that appears real to users. For example, the systems and methods create a virtual seating surface that has characteristics such as material type and graphic and physical parameters of a seating surface in a real-world environment. By using an AI model to identify the material type, the virtual seating surface is displayed realistically. By way of example, a physics engine may apply physics related to the material type, enabling realistic virtual gameplay related to virtual object interactions with the material type.

[0012] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments described in the present disclosure.

[0013] The various embodiments of the present disclosure are best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram of an embodiment of a system illustrating a method for replicating properties of a real-world object in a virtual world. [Figure 2A] FIG. 2 is a diagram of an embodiment of the system showing one of multiple environments of the system of FIG. 1. [Figure 2B] FIG. 1 is a diagram of an embodiment of a system illustrating a method for capturing audio and image data for training an artificial intelligence (AI) model. [Figure 2C] FIG. 2B is an embodiment of a list of material types for the seats and their covers in the system of FIG. 2A. [Figure 2D] FIG. 2C is an embodiment of a list of material types for the seats and their covers in the system of FIG. 2B. [Figure 3A]FIG. 1 is a diagram of an embodiment of a system showing training an AI model based on multiple audio datasets or a combination of multiple image and audio datasets. [Figure 3B] FIG. 4 is an embodiment of a table illustrating an association between a set including an audio data set and a plurality of audio parameters and a set including a seat material type and a seat cover material type in the system of FIGS. 2A and 2B. [Figure 3C] FIG. 13 is an embodiment of a table illustrating an association of a set including an audio dataset, an audio parameter, an image dataset, a number of physical parameters, and a number of graphical parameters with a set including a seat material type and a seat cover material type in the system. [Figure 4A] FIG. 1 is a diagram of an embodiment of a system illustrating a method for determining the probability that the seat material is of a particular type and the probability that the seat cover material is of a particular type. [Figure 4B] FIG. 1 is a diagram of an embodiment of a system showing an augmented reality (AR) video game played using the methods described herein. [Diagram 5] FIG. 1 is a diagram of an embodiment of a system showing communication between the glasses and a server system over a computer network. [Figure 6] FIG. 1 is a diagram of an embodiment of glasses showing a sound output system and a microphone. [Figure 7] FIG. 1 illustrates an embodiment of an input controller showing haptic feedback. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Systems and methods for identifying material types of objects in a real-world environment are described. It should be noted that various embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order to not unnecessarily obscure various embodiments of the present disclosure.

[0016] FIG. 1 is a diagram of an embodiment of a system 100 illustrating a method for reproducing properties of a real-world object in a virtual world. The system includes an image capture system 102, a sound capture system 104, and a server system 106. An example of the image capture system 102 includes one or more cameras, such as a depth camera and a digital camera. Illustratively, the one or more cameras are located in a head-mounted display (HMD) or AR glasses, or a display device, or a game console, or are standalone cameras. An example of the sound capture system 104 includes one or more microphones. Illustratively, the one or more microphones are part of an HMD, or AR glasses, or a display device, or a game console, or are standalone microphones. An example of the server system 106 includes multiple servers in a data center, or multiple servers of virtual machines, or a combination of processors and memory devices.

[0017] Examples of processors used herein include application specific integrated circuits (ASICs), programmable logic devices (PLDs), central processing units (CPUs), and combinations thereof. Examples of memory devices used herein include read only memory (ROM) devices, random access memories (RAMs), and combinations thereof. Illustratively, the memory devices are flash memory devices, or redundant array of independent disks (RAID).

[0018] The server system 106 includes an inference training engine 108, which includes a material identification system 110. As an example, an engine as used herein is a computer program executed by one or more of the processors of the server system 106. A computer program is an example of software. As another example, an engine as used herein includes an ASIC, or a PLD, or a combination thereof. It is noted that ASICs, PLDs, and processors are examples of hardware. Examples of the material identification system 110 include hardware, or software, or a combination thereof.

[0019] The server system 106 includes a physical phenomenon imparting system 112, a sound imparting system 114, and a graphic imparting system 116. Examples of any of the physical phenomenon imparting system 112, the sound imparting system 114, and the graphic imparting system 116 include hardware, software, or a combination thereof.

[0020] The image capture system 102 and the sound capture system 104 are connected to an inference training engine 108. The material identification system 110 is in turn connected to a physics implication system 112, a sound implication system 114, and a graphics implication system 116.

[0021] The system 100 further includes a plurality of environments 118 and an environment 120. The environments 118 and 120 are real-world environments. By way of example, the real-world environments exist outside of a virtual reality (VR) environment or an augmented reality (AR) environment. By way of example, the real-world environments cannot be created by a processor.

[0022] Image data is captured, e.g., generated, from the environment 118 by the image capture system 102, and audio data is captured from the environment 118 by the sound capture system 104. The inference training engine 108 is trained according to the audio data or a combination of the audio data and the image data.

[0023] The sound capture system 104 captures audio data from the environment 120. The audio data captured from the environment 120 is transmitted from the sound capture system 104 to the material identification system 110. The material identification system 110, trained based on the audio data captured from the environment 118 or a combination of audio data and image data captured from the environment 118, identifies one or more materials of one or more real-world objects in the environment 120, outputs one or more identities of the one or more materials, and provides the one or more identities to the physics implication system 112, the sound implication system 114, and the graphics implication system 116. Examples of real-world objects include a chair seat, a chair seat cushion, a sofa seat, a sofa seat cushion, a chair back cushion, a sofa back cushion, a chair arm cushion, a sofa arm cushion, a dining table with a wooden table top, and a dining table with a glass table top.

[0024] The physics imparting system 112 imparts physics parameters, such as movement according to laws of physics or changes in position and orientation according to laws, to one or more virtual objects in a virtual environment displayed on a display device disposed in the environment 120. An example of a virtual environment is a virtual scene, such as a VR scene or an AR scene. An example of a display device is an HMD, an AR glasses, a computer monitor, a television, etc. The laws of physics are imparted to one or more virtual objects based on one or more types of one or more materials in one or more real-world objects.

[0025] Additionally, the sound imparting system 114 imparts one or more sound parameters, such as one or more combinations of amplitude and frequency, to one or more virtual objects in the virtual environment. For example, a first sound is output when a first virtual object is displayed in the virtual environment with a physics phenomenon imparted to the first virtual object, and a second sound is output when a second virtual object is displayed in the virtual environment with a physics phenomenon imparted to the second virtual object. In an example, the first sound is output by one or more speakers and the second sound is output by one or more speakers of a display device disposed in the environment 120.

[0026] Further, the graphics imparting system 116 imparts one or more sets of graphics parameters, such as intensity, color, and texture, to one or more virtual objects displayed within the virtual environment. In an example, when a first virtual object is displayed within the virtual environment with the physical parameters imparted to the first virtual object, the first virtual object is controlled by one or more processors, such as a CPU, or a graphics processing unit (GPU), or a combination thereof, of a display device disposed within the environment 120 to have a first set of graphics parameters. In a further example, when a second virtual object is displayed within the virtual environment with the physical parameters imparted to the second virtual object, the second virtual object is controlled by one or more processors of a display device disposed within the environment 120 to have a second set of graphics parameters.

[0027] In one embodiment, the inference training engine 108 is the same as the material identification system 110 .

[0028] In one embodiment, the system 100 excludes the image capture system 102 .

[0029] In one embodiment, the terms capture and generate are used interchangeably herein.

[0030] 2A is a diagram of an embodiment of system 200 showing one of environments 118. Examples of system 200 include a room in a house, or a room in a cabin, or a room in a building. System 200 is an example of one of environments 118. System 200 includes multiple real-world objects, such as an office chair 202, a sofa 204, a curtain behind sofa 204, a doorway to the left of sofa 204, a hallway to kitchen 206, a desk 208, a computer monitor 210, a keyboard, a computer mouse, a handheld controller 212, and a game console 214. Sofa 204 has a seating surface 204A, and user 2 sits on seating surface 204A. By way of example, seating surface 204A is made of a material such as cloth, leather, corduroy, linen, or a combination thereof. The kitchen occupies another room of system 200 and has a dishwasher. Above the computer monitor 210 is a camera 216, which is an example of an image capture system 102 (FIG. 1).

[0031] The handheld controller 212 is connected to the game console 214, which is connected to the display device 210 and the server system 106 via a computer network. Examples of computer networks include the Internet, an intranet, and combinations thereof. The camera 216 is connected to the game console 214, or to the glasses 218, or to both the glasses 218 and the game console 214. For example, the camera 216 is connected to the glasses 218 via a wireless connection, such as a Bluetooth connection, or a wired connection. The game console 214 accesses a game from the server system 106 and provides virtual environment data to the display device 210, where a virtual scene 220 is displayed. The virtual scene 220 includes a plurality of virtual objects, such as virtual characters, and a virtual background, such as virtual trees and a virtual mountain range. The glasses 218 are connected to the server system 106 via a computer network. For example, the glasses 218 are connected to the server system 106 via the game console 214 and the computer network. As another example, the glasses 218 are connected directly to the server system 106 via a computer network without using the game console 214. Examples of glasses include HMDs and AR glasses.

[0032] User 1 is seated in an office chair 202 and holds a handheld controller 212 to play the game. As an example, a game engine of the game is executed by one or more processors of the server system 106 (FIG. 1) to generate a virtual scene 220 of the game. The office chair 202 has a seat 202A, which is manufactured from a material such as leather and is covered with a plastic cover. User 1 is also wearing glasses 218, which include a microphone M1 and a camera C1. The microphone M1 is an example of a sound capture system 104 (FIG. 1). The camera C1 is attached to the bottom surface of the glasses 218 and positioned below the bottom surface. For example, when user 1 wears the glasses 218, the lens of the camera C1 is oriented downward toward the floor of the system 200. For example, the field of view of the camera C1 is directed toward the floor of the room where the user 1 is located. As another example, the field of view of the camera C1 is wide enough to capture images of the movement of the seats 202A and 204A. User 2 is also listening to music on his smartphone. Camera C1 is an example of image capture system 102 (FIG. 1).

[0033] User 1 accesses the game from server system 106 via a computer network and plays the game with virtual scene 220 displayed on display device 210. For example, user 1 selects one or more buttons on handheld controller 212 to provide authentication information, such as a username and password. Handheld controller 212 transmits the authentication information to game console 214, which forwards the authentication information over the computer network to server system 106. Server system 106 determines whether the authentication information is authentic and, if so, provides user account 1 and access to a game engine running on server system 106. When the game engine is executed by one or more processors of server system 106, image frames of the game are generated and encoded, and the encoded image frames are output. The encoded image frames are transmitted to game console 214. Game console 214 decodes the encoded image frames and provides the image frames to display device 210 to display virtual scene 220 of the game, thereby enabling user 1 to play the game. While User 1 is playing the game, a virtual scene 220 of the game is displayed on the display device 210 .

[0034] The user 1 activates the glasses 218 before or during playing the game. After the user 1 activates the glasses 218, the microphone M1 captures the audio data set 1a related to the seat 202A. For example, the user 1 sits on the office chair 202 before playing the game, and the sound generated by the user's sitting action is detected by the microphone M1, and the audio data set 1a is captured, output, or generated. For example, the microphone M1 captures the creaking sound of the seat 202A when the seat 202A is compressed, or the sound of air blowing out when the seat 202A is compressed, to generate the audio data set 1a. For example, when the user 1 sits on the seat 202A, the seat 202A is compressed. For another example, the microphone M1 captures the creaking sound or the creaking sound of the seat 202A when the seat 202A is decompressed, to generate the audio data set 1a. In the example, when the user 1 stands up from the seat 202A, the seat 202A is decompressed. As another example, when the user 1 jumps on the seat 202A during gameplay and the seat 202A is compressed and decompressed multiple times, the microphone M1 captures multiple creaking sounds of the seat 202A. In the example, the sound is detected and the audio dataset 1a is captured. As yet another example, the microphone M1 captures the sound of a dishwasher running in the kitchen together with the sound generated by the movement of the seat 202A to output the audio dataset 1a. The audio dataset 1a is further described below with reference to FIG. 3A.

[0035] After the user 1 activates the glasses 218, the microphone M1 also captures the audio data set 1b associated with the seat 204A. As an example, the microphone M1 detects a sound generated simultaneously when the user 2 sits on or stands up from the seat 204A. In an example, the user 2 sits on or stands up from the seat 204A before or during the game played by the user 1. Also, in an example, the sound is detected and the audio data set 1b is captured, for example, generated. As yet another example, the microphone M1 captures the sound of a dishwasher operating in the kitchen together with the sound generated by the movement of the seat 204A, and outputs the audio data set 1b. The audio data set 1b is further described below with reference to FIG. 3A.

[0036] The camera C1 or the camera 216 of the combination of the camera C1 and the camera 216 detects the movement of the seat 202A and other objects in the system 200 and captures an image data set 1a related to the seat 202A. By way of example, when the user 1 sits on the seat 202A, or jumps on the seat 202A, or stands up from the seat 202A, one or more of the camera C1 and the camera 216 captures image data of the seat 202A.

[0037] Additionally, camera C1 or camera 216 of the combination of camera C1 and camera 216 detects the movement of seat 204A and other objects in system 200 to capture or output image data set 1b related to seat 204A. Illustratively, during a period when user 2 sits on seat 204A, stands up from seat 204A, or jumps on seat 204A, one or more of camera C1 and camera 216 capture image data of seat 204A.

[0038] To train the AI ​​model, the camera 216 transmits the image datasets 1a and 1b to the server system 106 via a computer network. For example, the camera 216 transmits the image datasets 1a and 1b to the glasses 218 via a wireless connection, and the glasses 218 forward the image datasets 1a and 1b to the server system 106. As another example, the camera 216 transmits the image datasets 1a and 1b to the game console 214 and the server system 106 via a computer network.

[0039] To train an artificial intelligence (AI) model, the glasses 218 transfer the image datasets 1a and 1b and the audio datasets 1a and 1b to the server system 106 ( FIG. 1 ) via a computer network. For example, the glasses 218 transmit the image datasets 1a and 1b and the audio datasets 1a and 1b to the server system 106 via the game console 214 and the computer network. As another example, the glasses 218 transmit the image datasets 1a and 1b and the audio datasets 1a and 1b to the server system 106 via a computer network without using the game console 214.

[0040] In one embodiment, instead of playing a game, User 1 accesses an application from server system 106 (FIG. 1) and the virtual environment of the application is displayed on display device 210. Examples of applications include a video conferencing application, or a chat application, or a social networking application.

[0041] In an embodiment, the glasses 218 do not include a camera C1 and the image data sets 1a and 1b are not generated.

[0042] In embodiments, the AI ​​model is sometimes referred to herein as a machine learning model.

[0043] In an embodiment, the seat 202A is made from plastic, or cloth, or vinyl, or mesh, or synthetic leather, or polyurethane, or memory foam, or other materials.

[0044] In one embodiment, the back of the office chair 202 is made from the same material as the seat 202A or is made from a different material than the seat 202A.

[0045] In an embodiment, the office chair 202 includes a headrest, which is made from the same material as the seat 202A or is made from a different material than the seat 202A.

[0046] In one embodiment, office chair 202 includes two armrests, each made from the same material as seat 202A or made from a different material than seat 202A.

[0047] In an embodiment, camera C1 is mounted at another location on glasses 218. For example, camera C1 is fixed to the side of glasses 218, with the lens of camera C1 oriented downward toward the floor of system 200.

[0048] In one embodiment, one or more additional cameras are attached to the glasses 218. For example, a second camera is attached to the left edge of the glasses 218. In an example, the second camera also has a lens oriented downward toward the floor of the system 200. Additionally, in an example, a camera C1 is attached to the right edge of the glasses 218.

[0049] In one embodiment, in addition to User 1, User 2 wears glasses such as AR glasses or an HMD. The glasses capture an audio dataset from detected sounds from the system 200 and / or capture an image dataset of seat movements in the system 200 and transmit the audio dataset and / or the image dataset to the server system 106 via a computer network. The glasses worn by User 2 are connected to the server system 106 via the computer network.

[0050] In an embodiment, user 1 accesses user account 1 after logging in to user account 1 using glasses 218. For example, glasses 218 are connected to an input controller via a wired or wireless connection. User 1 selects one or more buttons on the input controller to provide authentication information. Upon receiving the authentication information, the input controller generates an input signal according to the authentication information and transmits the input signal to glasses 218 via a wired or wireless connection. Glasses 218 transmit the authentication information to server system 106 via a computer network or via both game console 214 and a computer network. If server system 106 determines that the authentication information is authentic, it allows user 1 to log in to user account 1.

[0051] In one embodiment, both seating surface 202A and seating surface 204B are made from the same material.

[0052] In an embodiment, the covers of seat surface 202A and seat surface 204B are made from the same material.

[0053] In one embodiment, the audio data sets are sometimes referred to herein as audio frames, and the image data sets are sometimes referred to herein as image frames.

[0054] In an embodiment, a seat in a real-world environment is an example of an object or object type in a real-world environment. For example, the identity of seat 202A indicates that its object type in system 200 is seat 202A, and the identity of seat 204A indicates that its object type in system 200 is seat 204A.

[0055] FIG. 2B is a diagram of an embodiment of a system 250 illustrating a method for capturing audio and image data for training an AI model. The system 250 includes a real-world environment in a vehicle, such as a bus or van. The system 250 is an example of one of the environments 118 (FIG. 1). The system 250 includes a plurality of bus chairs 252A, 252B, and 252C. The bus chairs 252A and 252B are arranged in a row, and the bus chair 252C is arranged behind the bus chair 252B. A user 3 sits on a seat 254B of the bus chair 252B, and a user 4 sits on a seat 254C of the bus chair 252C. Also, a user 5 stands in the vehicle, and a user 6 sits on a seat 254A of the bus chair 252A. As an example, each of the seats 254A-254C is manufactured from a material such as wool, plastic, or cloth. As another example, each of the seating surfaces 254A-254C is made from a material such as cloth and covered with a material such as a vinyl or plastic cover. As yet another example, each of the seating surfaces 254A-254C is made from a material different from the material forming the seating surface 202A (FIG. 2A). As yet another example, each of the seating surfaces 254A-254C is manufactured from a material different from the material forming the seating surface 204A (FIG. 2A).

[0056] The user 3 is wearing glasses 256, which include a camera C2 and a microphone M2. The glasses 256 are connected to the server system 106 via a computer network. For example, the glasses 256 are directly connected to the server system 106 via the computer network without using a game console (not shown).

[0057] Microphone M2 is an example of a sound capture system 104 (FIG. 1). Camera C2 is attached to and positioned below the bottom surface of glasses 256. For example, when user 3 wears glasses 256, the lens of camera C2 is oriented downward toward the floor of system 250. For example, the field of view of camera C2 faces the floor of a vehicle in which user 3 is traveling. As another example, the field of view of camera C2 is wide enough to capture images of the movement of seats 254A and 254B. System 250 includes rear window 258 and multiple side windows 260A and 260B. Camera C2 is an example of an image capture system 102 (FIG. 1).

[0058] The microphone M2 detects sounds generated in the real-world environment of the vehicle and outputs the audio data set 2. For example, the microphone M2 detects the sound of the user 3 sitting on the seat 254B and the sound of the seat 254B moving up and down while the vehicle is traveling on the road, and outputs the audio data set 2. As another example, the microphone M2 detects the sound of the user 3 standing up from the seat 254B before or while the vehicle is traveling on the road, and outputs the audio data set 2. As yet another example, the microphone M2 detects the sound of the user 6 sitting on the seat 254A and the sound of the seat 254A moving up and down while the vehicle is traveling on the road, and outputs the audio data set 2. As another example, the microphone M2 detects the sound of the user 6 standing up from the seat 254A before or while the vehicle is traveling on the road, and outputs the audio data set 2. As yet another example, microphone M2 detects the sound of user 4 sitting on seat 254C and the sound of seat 254C moving up and down while the vehicle is traveling on a road, and outputs audio data set 2. As another example, M2 detects the sound of user 4 standing up from seat 254C before or while the vehicle is traveling on a road, and outputs audio data set 2. Audio data set 2 is described further below.

[0059] Also, the camera C2 of the glasses 256 detects the movement of one or more of the seats 254A to 254C to generate the image data set 2. For example, when the user 3 sits on the seat 254B, or jumps on the seat 254B, or moves on the seat 254B due to the movement of the vehicle, or stands up from the seat 254B, the camera C2 detects the movement of the seat 254B and outputs the image data set 2. Furthermore, as another example, when the user 6 sits on the seat 254A, or jumps on the seat 254A, or moves on the seat 254A due to the movement of the vehicle, or stands up from the seat 254A, the camera C2 detects the movement of the seat 254A and outputs the image data set 2. As yet another example, when the user 4 sits on the seat 254C, or jumps on the seat 254C, or moves on the seat 254C due to the movement of the vehicle, or stands up from the seat 254C, the camera C2 detects the movement of the seat 254C and outputs the image dataset 2. In the example, when the seat 254C is within the field of view of the camera C2, the camera C2 detects the movement of the seat 254C. For example, when the user 3 stands up from the seat 254B and leans on the seat 254C to talk to the user 4, and then faces the bath chair 252C, the camera C2 detects the movement of the seat 254C. The audio dataset 2 and the image dataset 2 are transmitted from the glasses 256 to the server system 106 via a computer network for training the AI ​​model.

[0060] It should be noted that a user sitting on the seat, a user jumping on the seat, a user moving on the seat, a user touching the seat, a user using their hand to decompress or compress the seat, and a user standing up from the seat are examples of interactions between the user and the seat.

[0061] In one embodiment, one or more additional cameras are attached to the glasses 256. For example, a second camera is attached to the left edge of the glasses 256. In an example, the second camera also has a lens oriented downward toward the floor of the system 250. Additionally, in an example, a camera C2 is attached to the right edge of the glasses 256.

[0062] In one embodiment, in addition to user 3, one or more of users 4-6 each wear one or more glasses, such as AR glasses or an HMD, each capturing an audio dataset from detected sounds from system 250 and / or an image dataset of seat movements within system 250, and transmitting the audio dataset and / or the image dataset to server system 106 via a computer network.

[0063] In an embodiment, the glasses 256 do not include a camera C2 and no image dataset 2 is generated.

[0064] In an embodiment, the user 3 accesses the user account 3 after logging in to the user account 3 using the glasses 256. For example, the glasses 256 are connected to an input controller via a wired or wireless connection. The user 3 selects one or more buttons on the input controller to provide authentication information. Upon receiving the authentication information, the input controller generates an input signal based on the authentication information and transmits the input signal to the glasses 256 via a wired or wireless connection. The glasses 256 transmit the authentication information to the server system 106 via a computer network. If the server system 106 determines that the authentication information received from the glasses 256 is authentic, it allows the user 3 to log in to the user account 3.

[0065] In one embodiment, the seats 254A-254C are made from the same material.

[0066] In an embodiment, seating surface 254A is made from a different material than the material of seating surfaces 254B or 254C.

[0067] In an embodiment, the covers of the seats 254A-254C are made from the same material.

[0068] In an embodiment, the cover of seating surface 254A is made from a different material than the cover material of seating surfaces 254B or 254C.

[0069] In one embodiment, where the user 3 is in a room with a game console, the glasses 256 are connected to the server system 106 via the game console and a computer network.

[0070] In an embodiment where the user 3 is in a room with a game console, the glasses 256 are connected to the server system 106 via a computer network without the use of a game console.

[0071] 2C is an illustration of an embodiment of a list 270 of material types of seat surfaces and their covers in the system 200 (FIG. 2A). The glasses 218 (FIG. 2A) display the list 270 on one or more display screens of the glasses 218. For example, in response to User 1 logging into User Account 1, the CPU of the glasses 218 controls the GPU of the glasses 218 to display the list 270 on one or more display screens of the glasses 218. In an example, the CPU of the glasses 218 receives the list 270 from one or more processors of the server system 106 (FIG. 1) via a computer network and controls the GPU of the glasses 218 to display the list 270 on the glasses 218. In an example, the list 270 is generated by one or more processors of the server system 106.

[0072] List 270 includes a blank field to accept a material type of seat 202A (FIG. 2A) that user 1 interacts with, and another blank field to accept a material type of seat 204A (FIG. 2A) that user 2 interacts with. List 270 further includes a blank field to accept a type of material used to cover seat 202A, and another blank field to accept a type of material used to cover seat 204A. List 270 also includes blank fields to accept identities such as names or types of real-world objects, such as seat 202A of chair 202, seat 204A of chair 204, and covers for seats 202A and 204A.

[0073] User 1 logs into user account 1 and accesses list 270. User 1 selects one or more buttons on the input controller connected to glasses 218 to provide identities such as names of real-world objects, such as seat 202A of chair 202, seat 204A of chair 204, and covers for seats 202A and 204A. User 1 also selects one or more buttons on the input controller connected to glasses 218 to provide material types for seats 202A and 204A, the type of material used to cover seat 202A, and the type of material used to cover seat 204A. For example, user 1 selects one or more buttons on the input controller connected to glasses 218 to spell out the material type, such as plastic, or leather, or vinyl, used for seat 202A, and selects one or more buttons to spell out the type of material used for the cover for seat 202A. As another example, user 1 selects one or more buttons on an input controller connected to glasses 218 to spell out that seat 202A is an office chair seat and seat 204A is a sofa seat.

[0074] Upon receiving the identity of a real-world object, such as the seat 202A of the chair 202 and the seat 204A of the chair 204, and receiving the material type of the seats 202A and 204A and the material type of the covers of the seats 202A and 204A, the glasses 218 transmit the identity of the real-world object, the material type of the seats 202A and 204A, and the material type of the covers of the seats 202A and 204A to the server system 106 via a computer network, or to the game console 214 and the server system 106 via a computer network, for AI model training. As an example, the glasses 218 include a CPU, which receives the identity of the seat 202A, which is the seat of an office chair, and assigns the alphanumeric character 1a to the identity of the seat 202A. The alphanumeric character 1a assigned to the identity of the seat 202A is sometimes referred to herein as the identity I1a. As another example, the CPU of the glasses 218 receives the identity of the seat 204A, which is a sofa seat, and assigns an alphanumeric character, such as 1b, to the identity of the seat 204A. The alphanumeric character 1b assigned to the identity of the seat 204A is sometimes referred to herein as the identity I1b.

[0075] In one embodiment, instead of using an input controller connected to the glasses 218 to select the material type of the seats 202A and 204A and the material type of the covers of the seats 202A and 204A, the selection is made using eye gestures. For example, the glasses 218 include an internal camera directed at the eyes of the user 1. The user 1 makes eye gestures to identify the identity of the seat 202A of the chair 202, the seat 204A of the sofa 204, the material type of the seat 202A, and the material type of the cover of the seat 202A, which are detected by the internal camera. By way of example, the user 1 makes eye gestures to select from the list 270 the identity of the seat 202A of the chair 202, the material type of the seat 202A, and the material type of the cover of the seat 202A. The internal camera captures image data including the eye gesture. The CPU of the glasses 218 receives the identities of the seat surfaces 202A and 204A selected using the eye gestures, the material types of the seat surfaces 202A and 204A, and the material types of the covers of the seat surfaces 202A and 204A. The CPU transmits a list 270 including the identities of the seat surfaces 202A and 204A, the material types of the seat surfaces 202A and 204A, and the material types of the covers of the seat surfaces 202A and 204A, to the server system 106 via the computer network, or both, via the game console 214 and the computer network, for training the AI ​​model.

[0076] In an embodiment, image data including the eye gesture is transmitted over a computer network from the glasses 218 to the server system 106. One or more processors in the server system 106 analyze the image data to identify the eye gesture and obtain the identity of the seat 202A of the chair 202, the identity of the seat 204A of the sofa 204, the material types of the seats 202A and 204A, and the material types of the covers of the seats 202A and 204A.

[0077] In an embodiment, the handheld controller 212 is used in place of the input controller used with the glasses 218 .

[0078] In one embodiment, list 270 is pre-populated with the identity of seat 202A, which is the seat of chair 202, and the identity of seat 204A, which is the seat of couch 204. For example, when user 1 logs into user account 1 and camera C1 captures image data sets 1a and 1b, one or more processors of server system 106 generate list 270 and transmit list 270 over the computer network to glasses 218. The one or more processors of server system 106 determine from image data sets 1a and 1b that system 200 includes office chair 202, couch 204, seats 202A and 204A, and covers for seats 202A and 204A. Illustratively, the one or more processors compare a pre-stored shape of a pre-stored object, such as an office chair, or a sofa, or a seat, or a seat cover, to a shape of an image of an object, such as office chair 202, sofa 204, seat 202A, seat 204A, seat 202A cover, and seat 204A cover. Illustratively, a comparison is made to determine whether there is a similarity between the two shapes, and if there is, the object is identified as the pre-stored object. An example of similarity between the two shapes is when the two shapes are the same. Another example of similarity between the two shapes is when a large portion of the pre-stored shape matches a large portion of the shape of the image of the object.

[0079] 2D is an illustration of an embodiment of a list 280 of material types of seat surfaces and their covers in system 250 (FIG. 2B). Glasses 256 (FIG. 2B) display list 280 on one or more display screens of glasses 256. For example, in response to user 3 logging into user account 3, a CPU of glasses 256 controls a GPU of glasses 256 to display list 280 on one or more display screens of glasses 256. In an example, a CPU of glasses 256 receives list 280 from one or more processors of server system 106 (FIG. 1) via a computer network and controls a GPU of glasses 256 to display list 280 on glasses 256. In an example, list 280 is generated by one or more processors of server system 106.

[0080] List 280 includes a blank for accepting a material type of seat 254B (FIG. 2B) with which user 3 interacts, another blank for accepting a material type of seat 254A (FIG. 2B) with which user 6 interacts, and yet another blank for accepting a material type of seat 254C with which user 4 interacts. List 280 further includes a blank for accepting a type of material used to cover seat 254B, another blank for accepting a type of material used to cover seat 254A, and yet another blank for accepting a type of material used to cover seat 254C. List 280 also includes blanks for accepting identities such as names or types of real-world objects, such as seat 254B of bath chair 252B, seat 254A of bath chair 252A, and seat 254C of bath chair 252C, as well as covers for seats 254A-254C.

[0081] User 3 logs into user account 3 and accesses list 280. User 3 selects one or more buttons on the input controller connected to glasses 256 to provide identities such as names of real-world objects, such as seat 254B of bath chair 252B, seat 254A of bath chair 252A, seat 254C of bath chair 252C, and covers of seats 254A-254C. User 3 also selects one or more buttons on the input controller connected to glasses 256 to provide material types of seats 254A-254C and types of material used to cover seats 254A-254C. For example, user 3 selects one or more buttons on the input controller connected to glasses 256 to spell out a material type, such as plastic, or leather, or vinyl, used for seat 254B, and selects one or more buttons to spell out a material type used for the cover of seat 254B. As another example, user 3 selects one or more buttons on an input controller connected to glasses 256 to spell out that seat 254B is a bus chair seat.

[0082] When the glasses 256 receives the identities of real-world objects such as the seat 254B of the bath chair 252B, the seat 254A of the bath chair 252A, and the seat 254C of the bath chair 252C, and receives the material types of the seats 254A-254C and the material types of the covers of the seats 254A-254C, the glasses 256 transmits the identities of the real-world objects, the material types of the seats 254A-254C, and the material types of the covers of the seats 254A-254C to the server system 106 via the computer network for AI model training. As an example, the glasses 256 includes a CPU, and the CPU receives the identity of the seat 254B of the bath chair and assigns the alphanumeric character 2 to the identity of the seat 254B. The alphanumeric character 2 assigned to the identity of the seat 254B is sometimes referred to as the identity I2 in this specification.

[0083] In one embodiment, instead of using an input controller connected to the glasses 256 to select the material type of the seats 254A-254C and the material type of the covers of the seats 254A-254C, the selection is made using eye gestures. For example, the glasses 256 include an internal camera directed at the eyes of the user 3. The user 3 makes an eye gesture to identify the identity of the seat 254B of the bath chair 252B, the material type of the seat 254B, and the material type of the cover of the seat 254B, which are detected by the internal camera. By way of example, the user 3 makes an eye gesture to select from the list 280 the identity of the seat 254B of the bath chair 252B, the material type of the seat 254B, and the material type of the cover of the seat 254B. The internal camera captures image data including the eye gesture. The CPU of the glasses 256 receives the identities of the seats 254A-254C selected using the eye gestures, the material types of the seats 254A-254C, and the material types of the covers of the seats 254A-254C. The CPU transmits a list 280 including the identities of the seats 254A-254C, the material types of the seats 254A-254C, and the material types of the covers of the seats 254A-254C to the server system 106 via the computer network for training the AI ​​model.

[0084] In an embodiment, image data including the eye gesture is transmitted over a computer network from glasses 256 to server system 106. One or more processors in server system 106 analyze the image data to identify the eye gesture and obtain the identity of seat 254A of bath chair 252A, seat 254B of bath chair 252B, and seat 254C of bath chair 252C, the material types of seats 254A-254C, and the material types of the covers of seats 254A-254C.

[0085] In an embodiment, a handheld controller is used instead of the input controller used with the glasses 256 .

[0086] In one embodiment, the list 280 is pre-populated with the identity of the seat 254A as the seat of the bath chair 252A, the identity of the seat 254B as the seat of the bath chair 252B, and the identity of the seat 254C as the seat of the bath chair 252C. For example, when the user 3 logs into the user account 3 and the camera C2 captures the image data set 2, the one or more processors of the server system 106 generate the list 280 and transmit the list 280 to the glasses 256 via the computer network. The one or more processors of the server system 106 determine from the image data set 2 that the system 250 includes covers for the bath chairs 252A-252C, the seats 254A-254C, and the seats 254A-254C. Illustratively, the one or more processors compare a pre-stored shape of a pre-stored object, such as a bus chair, or a seat, or a cover for the seat, to the shape of an image of an object, such as bus chair 252B, seat 254B, a cover for seat 254B, etc. Illustratively, a comparison is made to determine whether there is a similarity between the two shapes, and if there is, the object is identified as the pre-stored object.

[0087] 3A is a diagram of an embodiment of a system 300 illustrating training of an AI model based on audio datasets 1a, 1b, and 2, or a combination of image datasets 1a, 1b, and 2 and audio datasets 1a, 1b, and 2. The system 300 includes a client device 1, a client device 2, a client device 3, and a server system 106. An example of the client device 1 includes glasses 218, or a combination of glasses 218 and a camera 216 (FIG. 2A), or a combination of glasses 218 and a game console 214 (FIG. 2A), or a combination of glasses 218, a camera 216, and a game console 214. An example of the client device 2 includes glasses 256 (FIG. 2B).

[0088] The server system 106 includes a game engine and an inference training engine 108, sometimes referred to herein as an AI processor system. The game engine is used to execute a game. For example, the game engine includes game code that implements the laws of physics to provide physical parameters in the game, or generates states of virtual objects in the game, or generates graphical parameters of the virtual objects. Also, the game code executes to apply the graphical parameters to one or more virtual objects of the game. The game engine is connected to the inference training engine 108.

[0089] The inference training engine 108 includes an AI processor and a memory device 302. The AI ​​processor is an example of any of the processors of the server system 106, and the memory device 302 is an example of any of the memory devices of the server system 106. The AI ​​processor is connected to the memory device 302. The input datasets 1a, 1b, and 2 are stored in the memory device 302 after being received from the glasses 218 and the glasses 256 (FIGS. 2A and 2B). The input dataset 1a includes an image dataset 1a or an audio dataset 1a, or a combination thereof. The input dataset 1b includes an image dataset 1b or an audio dataset 1b, or a combination thereof, and the input dataset 2 includes an image dataset 2 or an audio dataset 2, or a combination thereof. The AI ​​processor stores the image datasets 1a, 1b, and 2 and the audio datasets 1a, 1b, and 2 in the memory device 302. As an example, the image dataset 1b is received from glasses 218 or a combination of a camera 216 (FIG. 2A) and glasses 218.

[0090] The AI ​​processor includes a feature extractor, a classifier, and an AI model. For example, the AI ​​processor includes a first integrated circuit that applies a function of the feature extractor, a second integrated circuit that applies a function of the classifier, and a third integrated circuit that applies a function of the AI ​​model. As another example, the AI ​​processor executes a first computer program that applies a function of the feature extractor, a second computer program that applies a function of the classifier, and a third computer program that applies a function of the AI ​​model. The feature extractor is coupled to the classifier, and the classifier is coupled to the AI ​​model. The AI ​​model is an example of a material identification system 110 (FIG. 1).

[0091] The feature extractor extracts, such as identifies, audio parameters, such as one or more amplitudes and one or more frequencies, or a combination thereof, from the audio datasets 1a, 1b, and 2, and provides the audio parameters to the classifier. For example, the feature extractor identifies the magnitudes, or peak-to-peak amplitudes, or zero-to-peak amplitudes, and frequencies of the audio datasets 1a, 1b, and 2. By way of example, the feature extractor identifies the absolute maximum power of the audio dataset m, such as 1a, 1b, or 2, or the absolute minimum power of the audio dataset m, to identify the magnitude of the audio dataset m. In an example, the absolute power is the magnitude within the entire period during which the audio dataset m was generated. In another example, the feature extractor identifies the local maximum magnitude of the audio dataset m and the local minimum magnitude of the audio dataset m. In an example, the local magnitude is the magnitude within a predetermined period, the predetermined period being shorter than the entire period during which the audio dataset m was generated. In the example, multiple local maxima and multiple local minima are identified from the audio dataset m, and a feature extractor applies a best fit, or average, or median to the local maxima and minima to identify the maximum and minimum magnitudes.

[0092] As another example, the feature extractor identifies a first time when the audio data set m reaches a predetermined magnitude and a second time when the audio data set m reaches the same predetermined magnitude, and calculates the difference between the first time and the second time to identify a time interval. The feature extractor inverts the time interval to identify an absolute frequency of the audio data set m. In an example, the absolute frequency is a frequency within the entire period during which the audio data set m was generated. In an example, the feature extractor identifies the absolute frequency as the frequency of the audio data set m. As yet another example, the feature extractor identifies a local frequency of the audio data set m. In an example, the local frequency is a frequency within a predetermined period, which is shorter than the entire period during which the audio data set m was generated. In an example, a plurality of local frequencies are identified from the audio data set m, and the feature extractor applies a best fit, average, or median to the local frequencies to identify the frequency of the audio data set m. In an example, each local frequency is identified in the same manner as the absolute frequency is identified, except that the local frequency is identified for each predetermined period. As yet another example, the feature extractor identifies a frequency in the audio data set m that is a maximum frequency, the maximum frequency being the largest of all frequencies identified from the audio data set m within a predetermined time period. As yet another example, the feature extractor identifies a frequency in the audio data set m that is a minimum frequency, the minimum frequency being the smallest of all frequencies identified from the audio data set m within a predetermined time period.

[0093] The classifier receives the audio parameters from the feature extractor and classifies the audio parameters obtained from the audio datasets 1a, 1b, and 2. The audio parameters are classified and an association between the audio parameters and the material type of the seat and the material type of the seat cover used in the systems 200 and 250 is output, and the association is provided from the classifier to the AI ​​model for training the AI ​​model. For example, when one or more of the processors of the server system 106 determine that the feature extractor has generated the audio parameters from the audio datasets 1a and 1b, the processors generate a list 270 (FIG. 2C) and transmit the list 270 to the glasses 218 worn by the user 1 via a computer network. For example, the list 270 is transmitted from the server system 106 to the glasses 218 within a predetermined period of time after the audio parameters are generated from the audio datasets 1a and 1b. In an example, when the material types accepted in the list 270 are received from the glasses 218 via the computer network, the classifier associates, such as establishes a one-to-one correspondence, or link, or unique relationship, between the first set of material types of the seats and seat covers in the system 200 and the second set of audio parameters identified from the audio data sets 1a and 1b. As an example, it is noted that if the seats 202A and 204A are not provided with covers, the list 270 excludes the material types of the covers of the seats 202A and 204A. By way of example, the first set includes the material types of the seats 202A and 204A without including the material types of the covers of the seats 202A and 204A. As another example, it is further noted that the system 200 excludes the seat 204A. In an example, the first set includes the material type of the seat 202A and excludes the material type of the seat 204A. Also in the example, audio data set 1a is received without receiving audio data set 1b, and audio parameters are determined from audio data set 1a.It should be noted that in any of the previous examples in this paragraph, the material type of seat 202A of system 200 and the material type of the cover of seat 202A are examples of types 1ax and 1ay, and the material type of seat 204A of system 200 and the material type of the cover of seat 204A are examples of types 1bx and 1by. For example, the material type of seat 202A is an example of type 1ax, and the material type of the cover of seat 202A is an example of type 1ay. As another example, the material type of seat 204A is an example of type 1bx, and the material type of the cover of seat 204A is an example of type 1by. Each of types 1ax, 1ay, 1bx, and 1by is an example of an output parameter.

[0094] As another example, one or more of the processors of the server system 106 generate a list 280 (FIG. 2D) in response to determining that the feature extractor has generated the audio parameters from the audio data set 2, and transmit the list 280 via a computer network to the glasses 256 (FIG. 2B) worn by the user 3. For illustrative purposes, the list 280 is transmitted within a predetermined period of time after the audio parameters are generated from the audio data set 2. In an example, when the material types accepted in the list 280 are received from the glasses 256 via the computer network, the classifier associates, such as establishes a one-to-one correspondence, or link, or unique relationship, between the third set of material types of the seats and seat covers in the system 250 and the fourth set of audio parameters identified from the audio data set 2. As an example, it is noted that if the seats 254A, 254B, and 254C are not provided with covers, the list 280 excludes the material types of the covers of the seats 254A, 254B, and 254C. In the example, the third set includes the material types of seating surfaces 254A, 254B, and 254C without including the material type of the covers of seating surfaces 254A, 254B, and 254C. Note that in any of the previous examples in this paragraph, the material types of seating surfaces 254A, 254B, and 254C of system 250 and the material types of the covers of seating surfaces 254A, 254B, and 254C are examples of types 2x and 2y. For example, the material type of seating surfaces 254A, 254B, or 254C is an example of type 2x, and the material type of the covers of seating surfaces 254A, 254B, or 254C is an example of type 2y. Each type 2x and 2y is an example of an output parameter.

[0095] Based on the association between the input datasets 1a, 1b, and 2, such as the audio parameters and image datasets 1a, 1b, and 2 associated with the systems 200 and 250 (FIGS. 2A and 2B), the AI ​​model is trained. For example, an association between a first set of audio parameters generated from the audio datasets 1a and 1b and a second set of material types of the seats 202A and 204A and covers of the seats 202A and 204A is provided to the AI ​​model. As another example, an association between a first set of audio parameters generated from the audio datasets 1a and 1b and a second set of material types of the seats 202A and 204A is provided to the AI ​​model. As yet another example, an association between a first set of audio parameters generated from the audio dataset 1a and a second set of material types of the covers of the seats 202A and 204A is provided to the AI ​​model. As yet another example, an association between a first set of audio parameters generated from the audio dataset 1a and a second set of material types of the seats 202A and 204A is provided to the AI ​​model.

[0096] As another example, an association between a third set of audio parameters generated from audio dataset 2 and a fourth set of material types of seats 254A-254C and covers of seats 254A-254C is provided to the AI ​​model. As another example, an association between a third set of audio parameters generated from audio dataset 2 and a fourth set of material types of seats 254A-254C is provided to the AI ​​model.

[0097] In one embodiment, the image datasets 1a, 1b, and 2 are received by the server system 106 from the client devices 1 and 2 via a computer network. In an embodiment, the image datasets 1a, 1b, and 2 are used in conjunction with the audio datasets 1a, 1b, and 2 to facilitate training of an AI model. For example, the feature classifier identifies real-world objects in the systems 200 and 250 from the image datasets 1a, 1b, and 2, and further identifies physical and graphical parameters of the real-world objects. By way of example, the feature classifier compares the size and shape of an image of a first seat surface, such as seat surface 202A or 204A, received in a first image dataset, such as image dataset 1a or 1b, with pre-stored sizes and pre-stored shapes of pre-stored seat surfaces to identify or identify that the first seat surface is an office chair seat surface. By way of example, the pre-stored size, pre-stored shape, and pre-stored seat surface identification are stored in the memory device 302. By way of example, the pre-stored seat surface identification includes alphanumeric characters. Also, in the example, the feature classifier extracts or identifies physical parameters, such as a change in motion, i.e., position and orientation, of the first seating surface from the first image data set. In the example, the feature classifier identifies a change in motion of the first seating surface from a first position to a second position, a change in motion from a first orientation to a second orientation, or a combination thereof. In the example, the change in motion of the first seating surface occurs when the first seating surface is compressed or decompressed due to an interaction of the first user, such as user 1 or 2, with the first seating surface. Furthermore, in the example, the feature classifier extracts or identifies graphical parameters, such as intensity, or color, or shading, or texture, of the first seating surface from the first image data set when the position and orientation of the first seating surface change. In the example, to facilitate training of the AI ​​model, the feature extractor provides the classifier with the identity of the first seating surface in the system 200, the physical parameters of the first seating surface, and the graphical parameters of the first seating surface.In an example, the classifier makes an association, such as establishing a one-to-one or unique relationship or link, between a set including the identity of the first seat, the audio parameters determined based on a sound generated by an interaction of the first user with the first seat, the physical parameters of the first seat, and the graphical parameters of the first seat, and a set including the material type of the first seat and the material type of a cover of the first seat, and outputs the association. In an example, the association is provided from the classifier to an AI model for training the AI ​​model.

[0098] It should be noted that the physical parameters of the seat surface 202A from the first image data set are sometimes referred to herein as physical parameters PP1a, and the physical parameters of the seat surface 204A from the first image data set are sometimes referred to herein as physical parameters PP2a. It should also be noted that the graphic parameters of the seat surface 202A from the first image data set are sometimes referred to herein as graphic parameters GP1a, and the graphic parameters of the seat surface 204A from the first image data set are sometimes referred to herein as graphic parameters GP2a. By way of example, the graphic parameters GP1a include a first set of graphic parameters when the seat surface 202A is in a first position and orientation, and a second set of graphic parameters when the seat surface 202A is in a second position and orientation. By way of example, the first set includes a different amount of light, intensity, shading, color, texture, or combination thereof compared to the amount of light, intensity, shading, color, texture, or combination thereof of the second set.

[0099] As another example, the feature classifier compares the size and shape of the image of the second seat, such as seat 254A, or 254B, or 254C, received in a second image data set, such as image data set 2, with the pre-stored size and pre-stored shape of the pre-stored seat to identify the second seat as a bus chair seat. In an example, the pre-stored size, pre-stored shape, and pre-stored seat identification are stored in the memory device 302. In an example, the pre-stored seat identification includes alphanumeric characters. Also, in an example, the feature classifier extracts or identifies physical parameters, such as the movement, i.e., the change in position and orientation, of the second seat from the second image data set. In an example, the feature classifier identifies the movement of the second seat from a third position to a fourth position and from a third orientation to a fourth orientation. In the example, the change in the motion of the second seat occurs when the second seat is compressed or decompressed due to an interaction of a second user, such as each of users 3, 4, or 6, with the second seat. Further, in the example, the feature classifier extracts or identifies a graphical parameter, such as intensity, or color, or shading, or texture, of the second seat from the second image dataset when the position and orientation of the second seat changes due to an interaction by the second user. In the example, for training the AI ​​model, the feature extractor provides the identity of the second seat in the system 250, the physical parameters of the second seat, and the graphical parameters of the second seat to the AI ​​model. In an example, the classifier makes an association, such as establishing a one-to-one or unique relationship or link, between a set including the identity of the second seat, the audio parameters determined based on a sound generated by an interaction of the second user with the second seat, the physical parameters of the second seat, and the graphical parameters of the second seat, and a set including the material type of the second seat and the material type of a cover for the second seat, and outputs the association. In an example, the association is provided from the classifier to an AI model for training the AI ​​model.

[0100] It should be noted that the physical parameters of the seat 254A, or 254B, or 254C from the second image data set are sometimes referred to herein as physical parameters PP2. It should also be noted that the physical parameters of the seat 254A, or 254B, or 254C from the second image data set are sometimes referred to herein as graphic parameters GP2. It should also be noted that the identity of the seat 254A, or 254B, or 254C is sometimes referred to herein as identity I2.

[0101] In some embodiments, communication between the server system 106 and the client devices 1-3 may be facilitated using wireless technologies. Such technologies may include, for example, 5G wireless communication technologies. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which a provider's coverage area is divided into small geographical areas called cells. Analog signals representing sound and images are digitized at the client device, converted by an analog-to-digital converter, and transmitted as a bit stream to the cell. All 5G wireless devices within a cell communicate over radio waves with a local antenna array and a low-power automatic transceiver (transmitter and receiver) within the cell, over frequency channels assigned by the transceiver from a pool of frequencies reused in other cells. The local antenna array is connected to the cellular network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, a mobile device moving from one cell to another is automatically transferred to the new cell. It should be understood that 5G networks are merely an example type of communication network, and that embodiments of the present disclosure may utilize previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies following 5G.

[0102] In one embodiment, to train an AI model, one or more of processors 1-P (FIG. 5) identify errors between an output of the AI ​​model, such as material type, and an output parameter, such as an actual material type. One or more of processors 1-P back-propagate the errors to adjust the AI ​​model. In an embodiment, the output parameters are not used as inputs to the AI ​​model, but instead are used as inputs to identify errors in the AI ​​model. Also, in an embodiment, the errors are not used as inputs to the AI ​​model. Rather, the errors are used by one or more of processors 1-P to make corrections, such as changing weights in the AI ​​model.

[0103] 3B is a diagram of an embodiment of a table 350 showing associations 352a, 352b, and 354. The associations 352a, 352b, and 354 are formed between a set including audio data sets 1a, 1b, 2 and a plurality of audio parameters AP1a, AP1b, and AP2, and a set including identities I1a, I1b, and I2 in systems 200 and 250 (FIGS. 2A and 2B), material types 1ax, 1bx, and 2x of seats 202A, 204A, 254A, 254B, and 254C, and material types 1ay, 1by, and 2y of covers of seats 202A, 204A, 254A, 254B, and 254C. The audio parameter AP1a represents an audio parameter identified from audio data set 1a by a feature classifier. The audio data set 1a is generated based on sounds received from an interaction between user 1 and seat 202A (FIG. 2A). Additionally, audio parameters AP1b represent audio parameters determined from audio data set 1b by a feature classifier. The audio data set 1b is captured based on sounds received from an interaction between user 2 and seat 204A (FIG. 2A). The audio parameters AP2 represent audio parameters determined by a feature classifier based on audio data set 2. The audio data set 2 is generated based on sounds received from an interaction between user 3 and seat 254B (FIG. 2B), or user 6 and seat 254A (FIG. 2B), or user 4 and seat 254C (FIG. 2B), or a combination thereof.

[0104] Material type 1ax is the material type of seat 202A, and material type 1ay is the material type of the cover of seat 202A, and types 1ax and 1ay are received from user 1 via user account 1 as part of input data set 1a (FIG. 3A). For example, material types 1ax and 1ay are received from user 1 in list 270 (FIG. 2C). Similarly, material type 1bx is the material type of seat 204A, and material type 1by is the material type of the cover of seat 204A, and types 1bx and 1by are received from user 1 via user account 1 as part of input data set 1b (FIG. 3A). For example, material types 1bx and 1by are received from user 1 in list 270. Also, material type 2x is the material type of any of seats 254A, 254B, and 254C, and material type 2y is the material type of any of the covers of seats 254A, 254B, and 254C. Material types 2x and 2y are received as part of input data set 1b (FIG. 3A) from user 3 via user account 3. By way of example, material types 2x and 2y are received in list 280 (FIG. 2D) from user 3 via user account 3.

[0105] The association 352a is a unique relationship between a set including an audio data set 1a, an audio parameter AP1a, and a set including an identity I1a, and material types 1ax and 1ay. Similarly, the association 352b is a unique relationship between a set including an audio data set 1b, an audio parameter AP1b, and a set including an identity I1b, and material types 1bx and 1by. And the association 354 is a one-to-one relationship between a set including an audio data set 2, an audio parameter AP2, and a set including an identity I2, and material types 2x and 2y.

[0106] In one embodiment, table 350 excludes material types 1ay, 1by, and 2y for seat covers in systems 200 and 250. This occurs when the seats in systems 200 and 250 do not have covers or when material types 1ay, 1by, and 2y have not been received from users 1 and 3 via their respective user accounts 1 and 3.

[0107] 3C is an illustration of an embodiment of table 360 ​​showing associations 362a, 362b, and 364. Associations 362a, 362b, and 364 are formed between a set including audio data sets 1a, 1b, and 2, a plurality of audio parameters AP1a, AP1b, and AP2, image data sets 1a, 1b, and 2, physical parameters PP1a, PP1b, and PP2, and graphic parameters GP1a, GP1b, and GP2, and a set including identities I1a, I1b, and I2 in systems 200 and 250 (FIGS. 2A and 2B), material types 1ax, 1bx, and 2x of seats 202A, 204A, 254A, 254B, and 254C, and material types 1ay, 1by, and 2y of seat covers.

[0108] The association 362a is a unique relationship between a set including an audio data set 1a, an audio parameter AP1a, a physical parameter PP1a, and a graphic parameter GP1a, and a set including an identity I1a, and material types 1ax and 1ay. Similarly, the association 362b is a unique relationship between a set including an audio data set 1b, an audio parameter AP1b, a physical parameter PP1b, and a graphic parameter GP1b, and a set including an identity I1b, and material types 1bx and 1by. Also, the association 364 is a one-to-one relationship between a set including an audio data set 2, an audio parameter AP2, a physical parameter PP2, and a graphic parameter GP2, and a set including an identity I2, and material types 2x and 2y.

[0109] In one embodiment, table 360 ​​excludes material types 1ay, 1by, and 2y for covers for seats 202A, 204A, 254A, 254B, and 254C of systems 200 and 250. This occurs when the seats of systems 200 and 250 do not have covers or when material types 1ay, 1by, and 2y have not been received from users 1 and 3 via their respective user accounts 1 and 3.

[0110] FIG. 4A is a diagram of an embodiment of a system 400 illustrating a method for determining a probability N% that the material of a seat 402A is of type 1ax and a probability M% that the material of the cover of the seat 402A is of type 1ay, where M and N are real numbers. Types 1ax and 1ay are examples of output parameters. The system 400 is an example of an environment 120 (FIG. 1). The system 400 includes a chair 402 with a seat 402A, a user 7, an AI model, a table 404, a virtual chair 408 with a virtual seat 408A, and a virtual user 416. The user 7 is wearing glasses 410 with a microphone M3 and a camera C3. The glasses 410 are an example of a client device 3 (FIG. 3A).

[0111] The glasses 410 are connected to the server system 106 via a computer network. For example, the glasses 410 are connected to the server system 106 (FIG. 1) via a game console (not shown) and a computer network. As another example, the glasses 410 are connected directly to the server system 106 via a computer network without the use of a game console (not shown).

[0112] Microphone M3 is an example of a sound capture system 104 (FIG. 1). Camera C3 is an example of an image capture system 104 (FIG. 1). Camera C3 is attached to the bottom of the rim of glasses 410. For example, the field of view of camera C3 is oriented downward toward the floor of a real-world environment, such as a room, in which glasses 410 are located. Glasses 410 are connected to an input controller 414 via a wired or wireless connection.

[0113] The microphone M3 detects sounds emanating from the real-world environment in which the user 7 is located. For example, when the user 7 stands up or sits on the seat 402A, noise is generated, which is detected by the microphone M3 and an audio dataset p is captured (p is an integer). The audio dataset p is transmitted from the glasses 410 to the server system 106 (FIG. 1) via a computer network. A feature extractor in one or more processors in the server system 106 extracts audio parameters APp from the audio dataset p. For example, the feature extractor extracts audio parameters APp from the audio dataset p in the same way as the audio parameters AP1a are obtained from the audio dataset 1a or the audio parameters AP1b are obtained from the audio dataset 1b.

[0114] The one or more processors of the server system 106 store the audio parameters APp and the audio data set p in one or more memory devices of the server system 106. The audio parameters APp are provided to the AI ​​model from the feature extractor. Upon receiving the audio parameters APp, the AI ​​model determines a probability N% that the material of the seat 402A is type 1ax and a probability M% that the material of the cover of the seat 402A is type 1ay. For example, the AI ​​model determines that the audio parameters APp are within a predetermined range from the audio parameters AP1a and outside a predetermined range from the audio parameters AP1b or AP2, indicating that the audio data set p is more than 50% likely to have been generated based on sound reflected from the seat 402A made of the same material as the seat 202A (FIG. 2A) and that the audio data set p is more than 50% likely to have been generated based on sound reflected from the cover of the seat 402A made of the same material as the cover of the seat 202A (FIG. 2A). In the example, the probability that seat surface 402A is made of the same material as seat surface 202A (FIG. 2A) and the probability that the cover of seat surface 402A is made of the same material as the cover of seat surface 202A are examples of model outputs 412. The model output 412 is the output of the AI ​​model.

[0115] For example, when the AI ​​model determines that the maximum amplitude of the audio parameter APp is within a predetermined range from the maximum amplitude of the audio parameter AP1a and is outside a predetermined range from the maximum amplitude of the audio parameter AP1b, or is outside a predetermined range from the maximum amplitude of the audio parameter AP2, the AI ​​model indicates that the probability that the seat 402A is made of the same material as the seat 202A exceeds 50%, and that the cover of the seat 402A is made of the same material as the cover of the seat 202A exceeds 50%. As another example, when the AI ​​model determines that the frequency of the audio parameter APp is within a predetermined range from the frequency of the audio parameter AP1a and is outside a predetermined range from the frequency of the audio parameter AP1b, or is outside a predetermined range from the frequency of the audio parameter AP2, the AI ​​model indicates that the probability that the seat 402A is made of the same material as the seat 202A exceeds 50%, and that the cover of the seat 402A is made of the same material as the cover of the seat 202A exceeds 50%. As another example, when the AI ​​model determines that the maximum amplitude of audio parameter APp is within a first predetermined range from the maximum amplitude of audio parameter AP1a and outside the first predetermined range from the maximum amplitude of audio parameter AP1b, or outside the first predetermined range from the maximum amplitude of audio parameter AP2, and the frequency of audio parameter APp is within a second predetermined range from the frequency of audio parameter AP1a and outside the second predetermined range from the frequency of audio parameter AP1b, or outside the second predetermined range from the frequency of audio parameter AP2, this indicates that there is a greater than 50% probability that seat surface 402A is made of the same material as seat surface 202A, and that there is a greater than 50% probability that the cover of seat surface 402A is made of the same material as the cover of seat surface 202A.

[0116] Further, in the example, user 7 selects one or more buttons on input controller 414 to instruct glasses 410 to display a virtual image of virtual seat 408A similar to seat 402A and a virtual image of virtual user 416. By way of example, virtual user 416 is an in-game character or avatar controlled by user 7 during gameplay. Illustratively, virtual seat 408A resembles seat 402A when it has the same material, or the same physical parameters, the same audio parameters, or the same graphic parameters, or a combination thereof, as seat 402A.

[0117] In the example, upon receiving the instructions, the glasses 410 transmit the instructions over a computer network to one or more processors of the server system 106. In the example, upon receiving instructions from the glasses 410 to display a virtual image of a virtual seat 408A similar to seat 402A, the AI ​​model of the one or more processors of the server system 106 calculates or determines, or has already calculated or determined, a model output 412 indicating that there is an N% probability that seat 402A is of the same material as seat 202A and an M% probability that the covering of seat 402A is of the same material as the covering of seat 202A.

[0118] Further, in the example, the one or more processors of the server system 106 determine to output the audio parameters AP1a of the seat 202A on which the model output 412 was determined with the probability M% and N% after receiving the instruction from the glasses 410 to display the virtual image of the virtual seat 408A similar to the seat 402A or 202A. Further, in the example, the one or more processors of the server system 106 determine to display the graphic parameters GP1a and the physical parameters PP1a of the seat 202A, and determine to display the graphic parameters GP1a according to the physical parameters PP1a. For example, when the physical parameters PP1a indicate that the virtual seat 408A is to be displayed in a first position and orientation, a first graphic parameter set is assigned to the virtual seat 408A, and when the physical parameters PP1a indicate that the virtual seat 408A is to be displayed in a second position and orientation, a second graphic parameter set is assigned to the virtual seat 408A.

[0119] Also, in the example, when the one or more processors of the server system receive the instruction, they generate virtual image data for displaying an image of the virtual user 416. In the example, the one or more processors of the server system determine how the virtual seat 408A moves based on the movement of the virtual user 416. For example, when the virtual user 416 sits on the seat 408A, the virtual seat 408A is compressed according to the same physical parameters PP1a, the same audio parameters AP1a, and the same graphic parameters GP1a as when the user 1 sits on the seat 202A. As another example, when the virtual user 416 stands up from the seat 408A, the virtual seat 408A is decompressed according to the same physical parameters PP1a, the same audio parameters AP1a, and the same graphic parameters GP1a as when the user 1 stands up from the seat 202A. In an example, one or more processors of the server system 106 access the audio parameters AP1a, the graphics parameters GP1a, the physical parameters PP1a, and virtual image data for displaying an image of the virtual user 416 from one or more of the memory devices of the server system 106, and transmit the audio parameters AP1a, the graphics parameters GP1a, the physical parameters PP1a, and the virtual image data to the glasses 410 via a computer network.

[0120] Further, in the example, the CPU of the glasses 410 receives the audio parameter AP1a, the graphic parameter GP1a, the physical parameter PP1a, and virtual image data for displaying an image of the virtual user 416, and controls the GPU of the glasses 410 to display the virtual chair 408 having the virtual seat 408A in which the graphic parameter GP1a changes according to the change in the physical parameter PP1a. Also, in the example, the CPU of the glasses 410 controls the GPU of the glasses to display the virtual user 416 sitting on the virtual seat 408A or standing up from the virtual seat 408A. Also, in the example, the CPU of the glasses 410 controls the sound output system of the glasses 410 to output sound according to the audio parameter AP1a. For example, the CPU of the glasses 410 controls the GPU of the glasses 410 to display the virtual seat 408A being compressed or decompressed based on the physical parameter PP1a. For example, the GPU of the glasses 410 displays the virtual seat 408A having the intensity, color, and texture indicated in the graphic parameter GP1a. Furthermore, in the example, when the manner in which the virtual seat surface 408A is compressed and decompressed according to the physical parameter PP1a is displayed, the CPU controls the sound output system of the glasses 410 to output the sounds of compression and decompression.

[0121] In one embodiment, each input data set described herein includes light detection and ranging (LiDAR) data. For example, in the system 200, the LiDAR data 1a is captured using a LiDAR scanner, and the LiDAR data 1a is transmitted to the server system 106 in the same manner as the image data 1a is transmitted from the camera C1 to the server system 106. For example, the LiDAR scanner is implemented in the glasses 218, and the LiDAR data 1a includes a LiDAR image of the seat surface 202A. Also, in the system 250, the LiDAR data 1b is captured using a LiDAR scanner, and in the system 400, the LiDAR data 2 is captured using a LiDAR scanner. For example, as shown in FIG. 2B, the LiDAR scanner is implemented in the glasses 256, and the LiDAR data 1b includes a LiDAR image of one or more of the seat surfaces 254A, 254B, and 254C. 4A, the LiDAR scanner is implemented in the glasses 410, and the LiDAR data 2 includes a LiDAR image of the seat surface 402A. The LiDAR data 1a and 1b are used in the same way that the image data 1a and 1b are used to train the AI ​​model. The AI ​​model then processes the LiDAR data 2 to identify the probabilities N% and M%.

[0122] In an embodiment, each input data set described herein includes inertial measurement unit (IMU) data. For example, each pair of glasses described herein includes inertial sensors, such as magnetometers, gyroscopes, and accelerometers, that detect the movement of the head of a user wearing the glasses. By way of example, the movement includes the position and orientation of the user's head. The glasses 218 (FIG. 2A) detect the movement of the head of the user 1 while the user 1 walks on the floor material of the system 200 and output an IMU data set 1a. Additionally, the microphone M1 captures an audio data set 1a' indicative of the sound generated by the floor material of the system 200 as the user 1 walks on the material. Similarly, the glasses 256 (FIG. 2B) detect the head movement of the user 3 while the user 3 is walking on the floor material of the system 250 and output the IMU dataset 1b, and the microphone M2 captures the audio dataset 1b' indicative of the sound generated by the floor material of the system 250 when the user 3 is walking on the floor material of the system 250. In the same way that the input datasets 1a and 1b are used to train the AI ​​model, the audio datasets 1a' and 1b' and the IMU datasets 1a and 1b are used to train the AI ​​model. Also, the microphone M3 captures the audio dataset 2' while the user 7 is walking on the floor material of the system 400, and the inertial sensor detects the head movement of the user 7 while the user 7 is walking and output the IMU dataset 2. The audio dataset 2' indicative of the sound generated by the floor material of the system 400 when the user 7 is walking on the floor of the system 400. In the same way that input dataset 2 is processed by the AI ​​model to identify a probability N%, audio dataset 2' and IMU dataset 2 are processed by the AI ​​model to output a probability of the material type of the floor of system 400.

[0123] In one embodiment, the AI ​​model identifies model outputs 412 prior to receiving an instruction to display a virtual image of seat 408A similar to seat 402A.

[0124] In one embodiment, glasses 410 exclude camera C3.

[0125] In one embodiment, a handheld controller is used in place of the input controller 414. The handheld controller is connected to the glasses 410 via a wired or wireless connection.

[0126] In an embodiment, instead of receiving a selection of one or more buttons on an input controller connected to glasses worn by the user, the user performs a gesture. A camera in the glasses captures image data of the gesture. For example, the gesture is used to select a material type of a real-world object in the system, or a material type of a cover for the real-world object. The image data is analyzed by the glasses' CPU or one or more processors in the server system 106 to determine the material type of the real-world object or material type of the cover selected by the user.

[0127] FIG. 4B is a diagram of an embodiment of a system 450 illustrating an augmented reality (AR) video game using the methods described herein. When a handheld controller including a microphone is placed on a coffee table, the handheld controller helps provide a material estimation of the coffee table. A virtual character then pops out of the handheld controller onto the coffee table, virtually bumping into the coffee table and creating a realistic three-dimensional (3D) audio sound. In this example, the sound characteristics of the 3D audio resulting from the virtual object interacting with the coffee table can be dynamically adjusted using the estimated material, such as wood or glass. The actual audio data of the physical interaction of the handheld controller bumping into the coffee table is used to modify the sound resulting from the virtual character interacting with the coffee table, so that the user 7 (FIG. 4A) feels as if the virtual character is actually on the coffee table.

[0128] System 450 includes a view of a living room from the perspective of user 7 wearing glasses 410 (FIG. 4A). The living room is an example of system 450 and a real-world environment. The living room includes a coffee table 452, a real video game controller 454, a virtual robot 456, and a display device 458, all of which are in the view of user 7 through glasses 410. Virtual robot 456 is a character in a video game. By way of example, video game controller 454, or the combination of video game controller 454 and display device 458, are examples of client device 3 (FIG. 3A). Also by way of example, coffee table 452 has a table surface made of wood, or glass, or another material.

[0129] A user 7 logs into a user account 7 assigned to the user 7 by the server system 106 (FIG. 3A) using a video game controller 454. Once the user 7 logs into the user account 7, the glasses 410 access a game, such as an AR video game, from the server system 106. During game play, the video game controller 454 is placed on a coffee table 452 by the user 7, generating a sound, which is shown as an exemplary sound wave 460. For example, the sound wave 460 is generated by the interaction of the video game controller 454 with the coffee table 452. The sound wave 460 is captured by a microphone M3, or a combination of the microphones of the microphone M3 and the video game controller 454, to generate an audio data set q, where q is an integer. The audio data set q is transmitted from the microphone M3, or the microphone of the video game controller 454, or a combination thereof, to the server system 106 over the Internet. The server system 106 processes the audio data set q in the same manner as the audio data set p of Figure 4A is processed to determine a probability r% (where r is a positive real number) that the material type of the coffee table 452 is wood or glass. The audio data set q is also stored in one or more memory devices of the server system 106 by one or more processors of the server system 106.

[0130] Additionally, during game play, camera C3 of glasses 410 captures images of the real-world environment of system 450 and transmits the images over the Internet to server system 106. Server system 106 determines the location of objects, such as coffee table 452 and video game controller 454, in the real-world environment of system 450 from a reference point in system 450, such as glasses 410. Additionally, server system 106 determines the distance between glasses 410 and any point on the table surface of coffee table 452 from the image of the real-world environment of system 450.

[0131] During game play, the user 7 selects one or more buttons on the video game controller 454 after the audio data set q is processed by the server system 106 to determine the probability r %. Upon receiving the selection, the video game controller 454 generates an input signal, which is transmitted from the video game controller 454 to the server system 106 via the Internet. Upon receiving the input signal, the server system 106 executes game code of the game to generate data to display the virtual robot 456 jumping out of the video game controller 454 and landing on the coffee table 452 based on the position of the coffee table 452 and the position of the video game controller 454. The one or more processors of the server system 106 also generate audio data to output as a sound when the virtual robot 456 lands on the coffee table 452. For example, the audio data set q is accessed from one or more memory devices of the server system 106 to mimic the sound that would have been made if the user 7 had placed the video game controller 454 on the coffee table 452. In the example, an audio data set q obtained from the glasses 410 and generated based on the sound of the user 7 placing the video game controller 454 on the coffee table 452 is accessed from one or more memory devices of the server system 106. Further, in the example, the audio data set q is used to generate an audio data set that is output as a sound when the virtual robot 456 lands on the coffee table 452. Also, in the example, the one or more processors of the server system 106 adjust audio parameters of the audio data set q accessed from the one or more memory devices of the server system 106 based on the distance between the glasses 410 and the virtual robot 456 displayed on the glasses 410. Illustratively, the further the position where the virtual robot 456 is displayed on the table surface of the coffee table 452 is from the glasses 410, the one or more processors proportionally reduce the peak-to-peak amplitude of the audio data set q.Conversely, in the illustrated example, the closer the virtual robot 456 is displayed on the surface of the coffee table 452 relative to the glasses 410, the one or more processors proportionally increase the peak-to-peak amplitude of the audio data set q. The virtual robot 456 is a character controlled by the user 7 using a video game controller 454. The one or more processors of the server system 106 transmit data for displaying the virtual robot 456 landing on the coffee table 452 and audio data to be output as sound along with the display of the virtual robot 456 to the glasses 410 via the Internet.

[0132] Upon receiving the data for displaying the virtual robot, the GPU of the glasses 410 displays the virtual robot 456 on the table surface of the coffee table 452 in the view of the living room presented by the glasses 410. For example, the GPU of the glasses 410 displays the virtual robot 456 landing on the table surface of the coffee table 452. The CPU of the glasses 410 also receives audio data from the server system 106 via the Internet. When the virtual robot 456 lands on the coffee table 452, the CPU of the glasses 410 controls the sound output system of the glasses 410 to generate a sound shown as a sound wave 462, which is output from the speaker of the glasses 410. The sound is output based on the audio data received from the server system 106. Thus, a simulation is generated in the glasses 410 showing the sound wave 462 being generated when the virtual robot 456 lands on the coffee table 452. It should be appreciated that the current technology described herein provides a way for the virtual robot 456 to respond to the user 7 in a perceptually natural manner, as if the virtual robot 456 were interacting with the coffee table 452. Examples of interactions include outputting realistically suitable 3D sounds from the sound output system of the glasses 410 so that the sounds appear to be coming from the same location on the physical coffee table 452 as the visual representation of the virtual robot 456 presented by the glasses 410, and further adjusting the virtual physics attributes applied to the virtual robot 456 so that the landing motion and animation of the virtual robot 456 appears realistic and natural, and is significantly different from the landing motion and animation of the virtual robot 456 that may land and bounce on a different physical surface, such as the seat surface 204A or 202A shown in FIG. 2A. Thus, it should be appreciated that the systems and methods described herein may be used to modify a video game application as shown in FIG. 4B to provide more realistic and immersive AR entertainment.

[0133] In one embodiment, when it is determined that the virtual robot 456 interacts with the coffee table 452, haptic feedback is provided to the user 7 via the video game controller 454. For example, when it is determined that the virtual robot 456 interacts with the coffee table 452, one or more of the processors 1-P (FIG. 5) generate haptic feedback data and transmit the haptic feedback data via the computer network 504 (FIG. 5) to a haptic feedback device, such as an eccentric rotating mass (ERM) actuator or a piezoelectric actuator, disposed within the video game controller 454. Additionally, the one or more of the processors 1-P transmit instructions to the video game controller 454 via the computer network 504 to output haptic feedback during a period when the virtual robot 456 is displayed interacting with the coffee table 452, such as landing on or moving on the coffee table 452. Upon receiving the haptic feedback data, the haptic feedback device vibrates to provide haptic feedback regarding the interaction to the user 7 holding the video game controller 454. A video game controller 454 or an input controller 414 are examples of body-connected devices.

[0134] In an embodiment, haptic feedback is provided to the user 7 via the glasses 410 or another body-connected device. For example, the glasses 410 include a haptic feedback device. In an example, the one or more processors 1-P generate haptic feedback data and transmit the haptic feedback data to the haptic feedback device of the glasses 410 or another body-connected device via the computer network 504. Further, in an example, the one or more processors 1-P transmit instructions to the glasses 410 or another body-connected device via the computer network 504 to output haptic feedback during a period in which the virtual robot 456 interacting with the coffee table 452 is displayed. In response to receiving the haptic feedback data, the haptic feedback device of the glasses 410 or another body-connected device vibrates to provide the user 7 with haptic feedback regarding the interaction.

[0135] FIG. 5 is a diagram of an embodiment of a system 500 showing communication between glasses 502 and a server system 106 via a computer network 504. The system 500 includes glasses 502, a computer network 504, a server system 106, and an input controller 501. Examples of input controllers used herein include handheld controllers such as game controllers. Illustratively, a game controller is an input device used in a video game. Illustratively, a user makes one or more selections on the game controller to control an object or character in the video game. The glasses 502 are examples of glasses 218 (FIG. 2A), glasses 256 (FIG. 2B), or glasses 410 (FIG. 4A). Examples of glasses 502 include AR glasses or HMDs. Examples of computer network 504 include a wide area network (WAN) such as the Internet, a local area network (LAN) such as an intranet, and combinations thereof.

[0136] The glasses 502 include a CPU 504, a GPU 506, a display screen 508, a camera 510, a video encoder 512, a network transport device 514, an audio encoder 516, a microphone 518, a communication device 507, a video decoder 528, and an audio decoder 530. Examples of the CPU 504 include a processor, an ASIC, and a PLD. Examples of the GPU 506 include a processor, an ASIC, and a PLD. Examples of the display screen 508 include a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. The camera 510 is an example of any of the cameras C1, C2, and C3. An example of a video encoder as used herein includes a circuit that applies a video conversion protocol, such as a video encoding protocol, to an image dataset to output an encoded image dataset, such as an encoded image frame. Illustratively, the video encoder 512 generates an I-frame, a P-frame, or a B-frame, which are examples of encoded image frames. Examples of video encoding protocols include H.262, H.263, and H.264.The microphone 518 is a device that converts sound energy into an audio data set.

[0137] An example of an audio encoder as used herein includes a circuit that compresses an audio data set into an encoded audio data set, such as an encoded audio frame. Illustratively, the audio encoder applies an audio encoding protocol, such as lossless compression or lossy compression, to encode the audio data set and generate an encoded audio frame. An example of lossy compression includes a modified discrete cosine transform (MDCT), which transforms a sampled waveform in the time domain into the frequency domain. Another example of lossy compression includes a linear predictive coding (LPC) protocol, which analyzes the audio data set.

[0138] An example of a network forwarding device as used herein includes a network interface controller, such as a network interface card (NIC). Another example of a network forwarding device includes a wireless access card (WAC). Microphone 518 is an example of any of microphones M1, M2, and M3 (FIGS. 2A, 2B, and 4).

[0139] The server system 106 includes a network forwarding device 516, an audio encoder 518, a video decoder 520, a processor system 522, an audio encoder 524, and a video encoder 526. The processor system 522 includes processors 1, 2, 3, ..., up to processor P, where P is an integer. The processor system 522 also includes memory devices 1, 2, 3, ..., up to memory device P. As an example, a combination of processor 1 and memory device 1 forms a first server, a combination of processor 2 and memory device 2 forms a second server, and so on, with the final combination of processor P and memory device P forming a Pth server. As another example, one or more of processors 1-P form the AI ​​processor of FIG. 3A. By way of example, one or more of the feature extractor, classifier, and AI model are implemented in or executed by one or more of processors 1-P. Memory device 302 (FIG. 3A) is an example of any of memory devices 1-P.

[0140] The processors 1-P are examples of the sound imparting system 114, the physical phenomenon imparting system 112, and the graphic imparting system 116 (FIG. 1). Each of the processors 1-P of the system 522 is connected to a corresponding memory device 1-P of the processor system 522. For example, the processor P is connected to the memory device P. An example of an audio decoder is a circuit that decompresses an encoded audio data set into an audio data set, such as an audio frame. For example, the audio decoder applies an audio decoding protocol, such as an audio decompression protocol, to decode the encoded audio frames into audio frames. An example of a video decoder as used herein is a circuit that performs a video decoding protocol, such as H.262, H.263, or H.264 decoding, or another type of video decompression, on an encoded image frame to output decoded data, such as an image frame.

[0141] The input controller 501 includes an input control 503 and a communication device 505. Examples of the input control 503 include one or more buttons, a touch pad, a touch screen, and a joystick, all of which allow a user to make a selection. Examples of communication devices of the device as used herein include circuitry that applies a transfer communication protocol, such as a wired communication protocol or a wireless communication protocol. Examples of wireless communication protocols include the Bluetooth® protocol, a short-range wireless communication protocol, a Wi-Fi® protocol, and a radio frequency (RF) communication protocol.

[0142] The CPU 504 is connected to other components of the glasses 502, such as a GPU 506, a display screen 508, a camera 510, a video encoder 512, a microphone 518, an audio encoder 516, a network transfer device 514, a communication device 507, a video decoder 528, and an audio decoder 530. The GPU 506 is connected to the display screen 508. The camera 510 is connected to the video encoder 512, which is connected to the network transfer device 514. The microphone 518 is connected to the audio encoder 516, which is connected to the network transfer device 514. The network transfer device 514 is connected to the computer network 504. The communication device 507 is connected to the network transfer device 514. The network transfer device 514 is connected to the video encoder 516, the video decoder 528, and the audio decoder 530.

[0143] Furthermore, the network transfer device 516 of the server system 106 is connected to an audio decoder 518, a video decoder 520, an audio encoder 524, and a video encoder 526. Furthermore, the audio decoder 518, the video decoder 520, the audio encoder 524, and the video encoder 526 of the server system 106 are connected to the processor system 522. For example, each of the one or more processors 1 to P is connected to the audio decoder 518, the video decoder 520, the audio encoder 524, and the video encoder 526.

[0144] When the microphone 518 detects sounds emanating from one or more real-world objects, such as seat 202A or 204A or 254A or 254B or 254C or 402A (FIGS. 2A, 2B, and 4) in the real-world environment in which the microphone 518 is located, the microphone 518 generates an audio dataset n, such as audio dataset 1a or 1b or 2 or p (FIGS. 3A and 4), where n is an integer, or a combination thereof. The microphone 518 transmits the audio dataset n to an audio encoder 516. The audio encoder 516 applies an audio encoding protocol to the audio dataset n to output an encoded audio dataset, such as an encoded audio frame. The audio encoder 516 transmits the encoded audio dataset to the network transport device 514.

[0145] The camera 510 also captures an image dataset n, e.g., image dataset 1a or 1b or 2, or a combination thereof, of one or more real-world objects, such as seat 202A or 204A or 254A or 254B or 254C or office chair 202, in the real-world environment in which the camera 510 is located. The image dataset n is transmitted from the camera 510 to a video encoder 512. The video encoder 512 applies a video encoding protocol to the image dataset n to output encoded image frames, such as a combination of I-frames, B-frames, and P-frames, and provides the encoded image frames to a network transport device 514.

[0146] Furthermore, when the user makes one or more selections in the input control 503, an input data set n, such as input data set 1a or 1b or 2 or a combination thereof, is generated by the input control 503. Examples of input data set n include list 270 or 280 (FIG. 2C or FIG. 2D). The input data set n is transmitted from the input control 503 to the communication device 505 of the input controller 501. The communication device 505 applies a transfer communication protocol to the input data set n to generate a transfer packet. The communication device 505 transmits the transfer packet to the communication device 507 of the glasses 502 via a connection, such as a wired or wireless connection between the communication device 505 and the communication device 507.

[0147] The communication device 507 applies a transport communication protocol to the transport packets to extract an input data set n. The communication device 507, under the control of the CPU 504, transmits the input data set n to the network transport device 514.

[0148] The network forwarding device 514 applies a network forwarding protocol, such as Transmission Control Protocol over Internet Protocol (TCP / IP), to embed the encoded image frames received from the video encoder 512, the encoded audio frames received from the audio encoder 516, and / or the input data set n received from the communication device 507 into output data packets. The network forwarding device 514 transmits the data packets over the computer network 504 to the network forwarding device 516 of the server system 106.

[0149] The network forwarding device 516 of the server system 106 applies a network forwarding protocol to the data packets received from the network forwarding device 514 to extract from the data packets the encoded audio frames, or the encoded image frames, or the input data set n, or a combination thereof. The network forwarding device 514 transmits the encoded audio frames to an audio decoder 518 and transmits the encoded image frames to a video decoder 520. The audio decoder 518 applies an audio decoding protocol to the encoded audio frames to identify an audio data set n from the encoded audio frames and transmits the audio data set n to a processor system 522. The video decoder 520 also applies a video decoding protocol to the encoded video frames to identify an image data set n and transmits the image data set n to a processor system 522. The input data set n is then transmitted from the network forwarding device 516 to the processor system 522.

[0150] The processor system 522 applies a feature extractor to extract audio parameters from the audio data set n and graphic and physical parameters from the image data set n. Furthermore, the processor system 522 applies a classifier to identify associations such as associations 352a or 352b or 354 or 362a or 362b or 364 (FIGS. 3B and 3C). An example of an association is a correspondence between a set of audio parameters identified from the audio data sets 1a, 1b, and 2 and a set of seat material types and seat cover material types received in the input data sets 1a, 1b, and 2. Another example of an association is a unique relationship between a set of audio parameters identified from the audio data sets 1a, 1b, and 2, graphic parameters identified from the image data sets 1a, 1b, and 2, and physical parameters identified from the image data sets 1a, 1b, and 2 and a set of seat material types and seat cover material types received in the input data sets 1a, 1b, and 2. The processor system 522 applies an AI model and trains the AI ​​model based on the association. Upon receiving the audio data set p, the AI ​​model trained based on the association analyzes the audio data set p and outputs the model output 412 (FIG. 4A).

[0151] In one embodiment, the input controller 501 is not used with the glasses 502. Rather, in an embodiment, the glasses 502 include an additional internal camera that is directed toward the eyes of a user wearing the glasses 502. The additional internal camera is connected to the CPU 504, the video encoder 512, and the network forwarding device 514. The additional internal camera captures image data based on the user's eye gestures. As an example, an eye gesture is made to select a seat material type and a seat cover material type in a real-world environment in which the glasses 502 are placed. As an example, an eye gesture is made to select a seat material type and a seat cover material type from the list 270 or 280. The image data captured by the additional internal camera is an example of input data 1a or 1b or 2 (FIG. 3A). Under the control of the CPU 504, the image data is sent from the additional internal camera to the video encoder 512. The video encoder 512 applies a video encoding protocol to the image data to output encoded image frames and sends the encoded image frames to the network forwarding device 514. The network transport device 514 applies a network communication protocol to the encoded audio frames to generate data packets and transmits the data packets over the computer network 504 to a network transport device 516 of the server system 106 .

[0152] Network transport device 516 also applies a network communication protocol to the data packets to extract encoded image frames and sends the encoded image frames to video decoder 520. Video decoder 520 applies a video decoding protocol to the encoded image frames to identify image data captured by the additional internal camera and sends the image data to processor system 522. One or more of processors 1-P analyze the image data captured by the additional internal camera to identify eye gestures made by the user and further identify a seat material type and a seat cover material type in the real-world environment in which glasses 502 are placed.

[0153] 6 is a diagram of an embodiment of glasses 502, showing a sound output system 602 and a microphone 518. The glasses 502 include a CPU 504, a sound output system 602, a microphone 518, and an audio memory device 606.

[0154] The microphone 518 includes a transducer, an acoustic energy to electrical energy converter (SE converter), an analog-to-digital converter (ADC), and a processor. The transducer is connected to the SE converter, and the SE converter is connected to the ADC. The processor of the microphone 518 is connected to the transducer, the SE converter, and the ADC. An example of a transducer is a diaphragm. An example of an SE converter is a capacitor or a series of capacitors. The sound output system 602 includes a digital-to-analog converter (DAC), an amplifier, and a speaker. The DAC is connected to the amplifier, and the amplifier is connected to the speaker. The CPU 504 is connected to the DAC of the sound output system 602, the ADC of the microphone 518, and the audio memory device 606.

[0155] The transducer detects sound emanating from and / or reflected from a real-world object in a real-world environment, such as system 200 or 250 or 400 (FIGS. 2A, 2B, and 4), and outputs vibrations. The vibrations are provided to an SE transducer, which modifies the electric field created in the SE transducer and outputs an audio analog signal, which is an electrical signal. The audio analog signal is provided to an ADC, which converts the audio analog signal from analog form to digital form and outputs audio data, such as audio data set 1a or 1b or 2 or p (FIGS. 3A and 4).

[0156] When it is determined to display the virtual seat 408A (FIG. 4A) of the virtual chair 408 and output sound in response to the movement of the virtual seat 408A, one or more of the processors 1-P of the server system 106 access, for example, read, the audio parameters AP1a, the graphic parameters GP1a, and the physical parameters PP1a from one or more of the memory devices 1-P. The one or more of the processors 1-P generate image frames from the graphic parameters GP1a and the physical parameters PP1a, generate audio frames from the audio parameters AP1a, and transmit the image frames and audio frames to the glasses 502 via the computer network 504 to display the virtual seat 408A and output sound based on the movement of the virtual seat 408A. For example, one or more of the processors 1-P receive an instruction from the input controller 501 via the glasses 502, the computer network 504, and the network transfer device 516 to display the virtual user 416 (FIG. 4A) sitting on the virtual seat 408A. In the example, once such a determination is made, one or more of the processors 1 to P generate a series of image frames having an output sequence of the physical parameters PP1a of the virtual seat 408A for displaying the virtual seat 408A rising from a decompressed position to a compressed position, and an output sequence of the graphic parameters GP1a of the virtual seat 408A synchronized with the output sequence of the physical parameters PP1a.

[0157] Illustratively, an instance of the virtual seat 408A displayed in a first image frame in the sequence of image frames has physical parameters PPi providing a decompressed position Dpi and a decompressed orientation DOi of the virtual seat 408A, and has graphic parameters GPi corresponding to the decompressed position Dpi and the decompressed orientation DOi of the virtual seat 408A. Illustratively, an instance of the virtual seat 408A displayed in a last image frame in the sequence of image frames has physical parameters PPf providing a compressed position CPf and a compressed orientation COf, and has final graphic parameters GPf corresponding to the compressed position CPf and the compressed orientation COf. Illustratively, all intermediate image frames between the first and last image frames have physical parameters corresponding to intermediate positions and intermediate orientations, and have intermediate graphic parameters. Also illustratively, each image frame, such as the first image frame, intermediate image frame, and last image frame, is time-stamped, and a display order of the image frames is provided to show the virtual seat 408A going from a decompressed state to a compressed state. In the example, the timestamps of the image frames are generated by one or more processors 1-P based on the generation times, such as by copying or synchronizing with the generation times of the images including the representation of the movement of the seat surface 202A from the decompressed position Dpi and decompressed orientation DOi to the compressed position CPf and compressed orientation COf. In the example, the generation times of the images including the representation of the movement of the seat surface 202A are generated by the camera 510 and received together with the images from the glasses 502 via the computer network 504 by the one or more processors 1-P.

[0158] Also in the example, once so determined, one or more of the processors 1-P generate a series of audio frames having an output sequence of audio parameters AP1a to output sounds as the virtual seat 408A rises. Continuing with the previous example, the first audio frame has first audio parameters to output a first sound at a time when an instance of the virtual seat 408A is displayed in a first image frame. In the example, the last audio frame has last audio parameters to output a last sound at a time when an instance of the virtual seat 408A is displayed in a last image frame. In the example, all intermediate audio frames between the first and last audio frames have audio parameters corresponding to intermediate positions and orientations of the virtual seat 408A. Also in the example, each audio frame, such as the first audio frame, intermediate audio frames, and last audio frame, is time-stamped to provide an order for outputting the audio frames as sounds at the same time as the virtual seat 408A is displayed going from a decompressed state to a compressed state. In the example, the time stamps of the audio frames are generated by one or more of the processors 1-P based on the generation times, such as by copying the generation times of the audio data generated in synchronization with the movement of the seat surface 202A from the decompression position Dpi and the decompression orientation DOi to the compression position CPf and the compression orientation COf, or in synchronization with the generation times. In the example, the generation times of the audio data generated in synchronization with the movement of the seat surface 202A are generated by a processor of the microphone 518 and received together with the audio data from the glasses 502 via the computer network 504 by the one or more of the processors 1-P.

[0159] Continuing with the example, one or more of the processors 1-P send audio frames generated from the audio parameters AP1a to the audio encoder 524. Also, in the example, one or more of the processors 1-P send image frames generated from the graphic parameters GP1a and based on the physical parameters PP1a to the video encoder 526. In the example, the audio encoder 524 applies an audio encoding protocol to the audio frames generated from the audio parameters AP1a to output encoded audio frames, and provides the encoded audio frames to the network forwarding device 516. Also, in the example, the video encoder 526 applies a video encoding protocol to the image frames generated from the graphic parameters GP1a to output encoded image frames, and sends the encoded image frames to the network forwarding device 516.

[0160] In the example, the network forwarding device 516 applies a network forwarding protocol to the encoded image frames received from the video encoder 526 and the encoded audio frames received from the audio encoder 524 to generate data packets. Also in the example, the network forwarding device 516 transmits the data packets to the network forwarding device 514 over the computer network 504. Continuing the example, the network forwarding device 514 receives the data packets and applies the network forwarding protocol to the data packets to extract the encoded image frames and the encoded audio frames. In the example, the network forwarding device 514 transmits the encoded image frames to the video decoder 528 of the glasses 502 and transmits the encoded audio frames to the audio decoder 530 of the glasses 502.

[0161] Also, in the example, the video decoder 528 applies a video decode protocol to the encoded image frame to output an image frame for displaying the virtual seat 408A going from a decompressed state to a compressed state, and provides the image frame to the CPU 504 of the glasses 502. In the example, the audio decoder 530 of the glasses 502 applies an audio decode protocol to the encoded audio frame to output an audio frame for outputting sound in synchronization with the display of the virtual seat 408A going from a decompressed state to a compressed state, and provides the audio frame to the CPU 504. In the example, the CPU 504 controls the GPU 506 and further controls the display screen 508 to display the virtual seat 408A of the virtual chair 408 changing from the initial decompressed position DPi and the initial decompressed orientation DOi to the final compressed position DPf and the final compressed orientation DOf. In the example, the change from the initial decompressed position DPi and the initial decompressed orientation DOi to the final compressed position DPf and the final compressed orientation DOf is an example of the physical parameter PP1a. In the example, the GPU 506 displays the virtual seat 408A with graphic parameters GP1a synchronized with the movement of the virtual seat 408A from the initial decompression position DPi and the initial decompression orientation DOi to the final compression position DPf and the final compression orientation DOf. Also, in the example, the CPU 504 controls the sound output system 602 of the glasses 502 to output sound in synchronization with the display of the virtual seat 408A being compressed from the initial decompression position DPi and the initial decompression orientation DOi to the final compression position DPf and the final compression orientation DOf. In the example, the sound is output in response to the audio frames received by the CPU 504.

[0162] FIG. 7 is a diagram of an embodiment of the input controller 501. The input controller 501 includes a controller 702, a driver system 704, a haptic feedback system 706, and a network forwarding device 708. An example of the controller 702 includes a combination of a processor and a memory device. Illustratively, the controller 702 includes a microprocessor or is a microcontroller. An example of a driver system includes one or more drivers, such as one or more transistors. An example of a haptic feedback system described herein includes one or more haptic feedback devices, such as actuators. An example of an actuator includes a motor. The controller 702 is connected to the network forwarding device 708 and the communication device 505. The controller 702 is also connected to the driver system 704, which is connected to the haptic feedback system 706. The network forwarding device 708 is connected to the computer network 504.

[0163] The controller 702 receives haptic feedback data from one or more of the processors 1-P (FIG. 5) via the network forwarding device 708 and the computer network 504. Upon receiving the haptic feedback data, the controller 702 transmits one or more control signals to the driver system 704 based on the haptic feedback data. In response to receiving the control signals, the driver system 704 generates one or more driver signals, such as current signals, and transmits the one or more driver signals to the haptic feedback system 706. Upon receiving the one or more driver signals, the haptic feedback system 706 vibrates to provide haptic feedback to the user 7 holding the input controller 501. For example, during the same time period that the GPU 506 (FIG. 5) displays the virtual robot 456 interacting with the coffee table 452, the controller 702 transmits the one or more control signals to the driver system 704. As another example, at the same time that control signals are sent from the GPU 506 to the display screen 508 (FIG. 5) to display the virtual robot 456 interacting with the coffee table 452, the controller 702 sends one or more control signals to the driver system 704.

[0164] In an embodiment, the glasses 502 (FIG. 5) or another body-connected device includes a driver system and a haptic feedback system. A CPU 504 (FIG. 5) is connected to the driver system of the glasses 502. The driver system is connected to the haptic feedback system of the glasses 502. The CPU 504 receives haptic feedback data from one or more of the processors 1-P (FIG. 5) via a network forwarding device 514 (FIG. 5) and a computer network 504. When the CPU 504 receives the haptic feedback data for the glasses 502, it transmits one or more control signals to the driver system of the glasses 502 based on the haptic feedback data. In response to receiving the control signals, the driver system generates one or more driver signals and transmits the one or more driver signals to the haptic feedback system of the glasses 502. When the haptic feedback system receives the one or more driver signals, it vibrates to provide haptic feedback to a user 7 wearing the glasses 502. For example, during the same time that the GPU 506 (FIG. 5) displays the virtual robot 456 interacting with the coffee table 452, the CPU 504 sends one or more control signals to the driver system of the glasses. As another example, at the same time that the GPU 506 sends control signals to the display screen 508 (FIG. 5) to display the virtual robot 456 interacting with the coffee table 452, the CPU 504 sends one or more control signals to the driver system.

[0165] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.

[0166] The embodiments described in this disclosure may be implemented in a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. In one implementation, the embodiments described in this disclosure are implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0167] With the above embodiments in mind, it should be understood that in one implementation, the embodiments described in this disclosure employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments described in this disclosure are useful machine operations. Some embodiments described in this disclosure also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. Specifically, in one embodiment, various general-purpose machines are used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus to perform the required operations.

[0168] In an embodiment, some embodiments described in this disclosure are embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that stores data that is subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), CD-recordable (CD-R), CD-rewriteable (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, etc. By way of example, the computer readable medium includes computer readable tangible media distributed across network-connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0169] Additionally, although some of the above embodiments are described with respect to a gaming environment, in some embodiments other environments, such as, for example, a video conferencing environment, are used in place of a game.

[0170] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be arranged to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various processing-related intervals, so long as the processing of the overlay operations is performed in the desired manner.

[0171] Although the foregoing embodiments described in this disclosure have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for identifying a material type of an object in a real-world environment, comprising: capturing an audio data set from the real-world environment including the object having the material type; applying an artificial intelligence (AI) model to the audio data set captured from the real-world environment to identify the material type of the object in the real-world environment; The method comprising:

2. receiving a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments; further receiving a plurality of output parameters for the plurality of objects in the plurality of environments; training the AI ​​model based on the plurality of audio data sets and the plurality of output parameters; The method of claim 1 further comprising:

3. The method of claim 2 , further comprising extracting a plurality of features from the plurality of audio data sets, the features comprising a plurality of amplitudes of the plurality of audio data sets and a plurality of frequencies of the plurality of audio data sets.

4. classifying the plurality of features and outputting an association between the plurality of features and the plurality of output parameters; training the AI ​​model includes providing the associations between the features and the output parameters to the AI ​​model; and applying the AI ​​model includes determining, by the AI ​​model, a probability that the audio data set captured from the real-world environment exhibits the material type of the object based on the association between the plurality of features and the plurality of output parameters. The method according to claim 3.

5. receiving a plurality of input data sets comprising a plurality of image data sets captured by a plurality of cameras in the plurality of environments, or Light Detection and Ranging (LiDAR) data captured from the plurality of environments, or Inertial Measurement Unit (IMU) data captured from the plurality of environments, or a combination thereof; said extracting the plurality of features comprises extracting the plurality of features from the plurality of input data sets; the plurality of features including a plurality of identities of the plurality of objects, a plurality of physical parameters defining a movement of the plurality of objects, and a plurality of graphical parameters of the plurality of objects; The method according to claim 3.

6. The plurality of physical parameters includes a first physical parameter and a second physical parameter, and the plurality of graphic parameters includes a first graphic parameter and a second graphic parameter; The method further includes classifying the plurality of features and outputting an association between the plurality of features and the plurality of output parameters; training the AI ​​model includes providing the associations between the features and the output parameters to the AI ​​model. The method according to claim 5.

7. 7. The method of claim 6, wherein identifying the material type of the object in the real-world environment includes identifying a probability that the object has the material type, the probability being identified based on the association between the plurality of features and the plurality of output parameters.

8. generating virtual object data for displaying a virtual object interacting with the material type in a virtual environment; generating audio data to be output by the virtual object when the virtual object interacts with the material type; generating haptic feedback data output by a body-connected device when the virtual object interacts with the material type; Further comprising: the virtual object data represents the material type of the object in the real-world environment, a movement of the object in the real-world environment from the first physical parameter to the second physical parameter, and a change of the object in the real-world environment from the first graphical parameter to the second graphical parameter. The method of claim 7.

9. the output parameters include data identifying the materials of the objects in the environments; The method further includes simulating a virtual interaction between a virtual object of a video game and the object based on the material type of the object. The method of claim 2.

10. receiving a plurality of input data sets; the multiple input datasets include multiple image datasets captured by multiple cameras in the multiple environments; the plurality of audio datasets and the plurality of image datasets are captured as the plurality of objects in the plurality of environments interact with a plurality of users; The method of claim 2.

11. 1. A server for identifying a material type of an object in a real-world environment, comprising: capturing an audio data set from the real-world environment including the object having the material type; applying an artificial intelligence (AI) model to the audio data set captured from the real-world environment to identify the material type of the object in the real-world environment; A processor configured to execute a memory device coupled to the processor; The server.

12. The processor, receiving a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments; further receiving a plurality of output parameters for the plurality of objects in the plurality of environments; training the AI ​​model based on the plurality of audio data sets and the plurality of output parameters; The server of claim 11 configured to execute:

13. 13. The server of claim 12, wherein the processor is configured to extract a plurality of features from the plurality of audio data sets, the features comprising a plurality of amplitudes of the plurality of audio data sets and a plurality of frequencies of the plurality of audio data sets.

14. 14. The server of claim 13, wherein the processor is configured to train the AI ​​model based on the plurality of audio data sets and the plurality of output parameters.

15. The processor, configured to classify the plurality of features and output an association between the plurality of features and the plurality of output parameters; To train the AI ​​model, the processor is configured to provide the associations between the features and the output parameters to the AI ​​model; and to apply the AI ​​model, the processor is configured to use the AI ​​model to determine a probability that the audio data set captured from the real-world environment is indicative of the material type of the object based on the association between the features and the output parameters. The server of claim 14.

16. The processor is configured to receive a plurality of input data sets; the multiple input data sets include multiple image data sets captured by multiple cameras in the multiple environments, or Light Detection and Ranging (LiDAR) data captured from the multiple environments, or Inertial Measurement Unit (IMU) data captured from the multiple environments, or any combination thereof; To extract the features, the processor is configured to obtain the features from the input data sets; the plurality of features including a plurality of identities of the plurality of objects, a plurality of physical parameters defining a movement of the plurality of objects, and a plurality of graphical parameters of the plurality of objects; The server of claim 13.

17. The plurality of physical parameters includes a first physical parameter and a second physical parameter, and the plurality of graphic parameters includes a first graphic parameter and a second graphic parameter; the processor is configured to classify the plurality of features and output an association between the plurality of features and the plurality of output parameters; the processor is configured to train the AI ​​model based on the plurality of audio data sets and the plurality of output parameters; To train the AI ​​model, the processor is configured to provide the associations between the features and the output parameters to the AI ​​model.

17. The server of claim 16.

18. To identify the material type of the object in the real-world environment, the processor is configured to identify a probability that the object has the material type; the probability is determined based on the association between the plurality of features and the plurality of output parameters.

20. The server of claim 17.

19. The processor, generating virtual object data for displaying a virtual object interacting with the material type in a virtual environment; generating audio data to be output by the virtual object when the virtual object interacts with the material type; generating haptic feedback data output by a body-connected device when the virtual object interacts with the material type; configured to run the virtual object data represents the material type of the object in the real-world environment, a movement of the object in the real-world environment from the first physical parameter to the second physical parameter, and a change of the object in the real-world environment from the first graphical parameter to the second graphical parameter.

20. The server of claim 18.

20. 1. A system for identifying a material type of an object in a real-world environment, comprising: a client device configured to capture an audio dataset from the real-world environment; a server connected to the client device via a computer network; The server comprises: receiving the audio data set from the client device via the computer network; applying an artificial intelligence (AI) model to the audio data set captured from the real-world environment to identify the material type of the object in the real-world environment; The system is configured to execute the following:

21. The plurality of client devices further comprises: generating a plurality of audio data sets based on sounds received from a plurality of objects in a plurality of environments; receiving a plurality of output parameters for a plurality of material types of the plurality of objects; configured to run The server is connected to the plurality of client devices via the computer network, the server comprising: receiving the plurality of audio data sets from the plurality of client devices over the computer network; receiving the output parameters from the client devices over the computer network; training the AI ​​model based on the plurality of audio data sets and the plurality of output parameters; extracting a plurality of features from the plurality of audio data sets, the features including a plurality of amplitudes of the plurality of audio data sets and a plurality of frequencies of the plurality of audio data sets; 21. The system of claim 20 configured to execute:

22. the server is configured to classify the plurality of features and output an association between the plurality of features and the plurality of output parameters; To train the AI ​​model, the server is configured to provide the association of the features and the output parameters to the AI ​​model; and to apply the AI ​​model, the server is configured to use the AI ​​model to determine a probability that the audio data set captured from the real-world environment is indicative of the material type of the object based on the association between the features and the output parameters.

22. The system of claim 21.

Images