Using machine learning to identify controller actions from video frames
By analyzing computer game video frames using machine learning models, controller operation information is generated and presented, solving the problem of lacking controller action information in game videos and improving the game learning effect for players.
Patent Information
- Application Number
- CN202280032614.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-05-04
- Filing Date
- 2022-04-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-04-29
AI Technical Summary
In the existing technology, controller action information is not captured when computer game videos are recorded, making it difficult for players to learn the timing of controller button presses from game videos.
Machine learning models, particularly recurrent neural networks (such as long short-term memory networks) and convolutional neural networks, are used to analyze video frame sequences to generate controller operation information, which is then displayed on the screen.
By analyzing video frame sequences to identify and present controller operation information, it helps players better learn game controls and enhances the gaming experience.
Smart Images

Figure CN117242498B_ABST
Abstract
Description
Technical Field
[0001] This application relates to unconventional solutions that must be rooted in computer technology and produce specific technological improvements. Background Technology
[0002] As understood in this article, videos of previously played computer games can be shared over computer networks to guide viewers on how to succeed in the game, such as by completing levels. As further understood in this article, such game videos may not include information about which controller buttons were pressed at what time during gameplay, because controller actions during gameplay may not have been captured when the game video was recorded. Summary of the Invention
[0003] As this article understands, information such as when to press which controller buttons can be valuable for players learning to play computer games, making gameplay more enjoyable for many types of players, from beginners to speedrunners. This article presents machine learning techniques to generate controller action information by analyzing a series of video frames without requiring additional controller data.
[0004] Therefore, an apparatus includes at least one computer memory, which is not a transient signal, and subsequently includes instructions executable by at least one processor to perform the following operations: receiving a recorded computer simulation comprising a sequence of video frames. The instructions are executable to process the video frame sequence in a machine learning (ML) model and to receive from the ML model identification of computer simulation controller (CSC) operations associated with the generation of the recorded computer simulation. Additionally, the instructions are executable to present the recorded computer simulation, along with at least one indication of at least one of the CSC operations received from the ML model, on at least one audio-visual (AV) display.
[0005] In an example implementation, the ML model includes at least one recurrent neural network (RNN), such as at least one long short-term memory (LSTM) network. Convolutional neural networks (CNNs) may also be used.
[0006] The device may include the processor, which may be embodied in an AV display or in a computer-simulated source, such as a computer simulation console and / or a server communicating with the AV display via a wide area computer network.
[0007] In another aspect, an apparatus includes at least one display configured to present video of at least one recorded computer simulation generated under the control of at least one computer simulation controller. However, the recorded computer simulation does not include information regarding the operation of the computer simulation controller during the generation of the video of the at least one recorded computer simulation. The apparatus therefore includes at least one processor configured with instructions to identify from the video information regarding the operation of the computer simulation controller during the generation of the video of the at least one recorded computer simulation. The instructions are executable to provide the at least one display with the information regarding the operation of the computer simulation controller during the generation of the video of the at least one recorded computer simulation, so that the information is presented simultaneously with the generation of the video of the at least one recorded computer simulation.
[0008] In an example implementation, the instructions may be executable to use at least one machine learning (ML) model to identify information from the video regarding the operation of the computer simulation controller during the generation of the at least one recorded computer simulation video.
[0009] In another aspect, a method includes inputting at least a training set into at least one machine learning (ML) model. The training set includes sequences of video frames from a plurality of recorded computer simulations and information associated with the video frame sequences regarding computer simulation controller (CSC) operations performed during the generation of the video frame sequences. The method then includes inputting at least a first recorded computer simulation into the ML model, the first recorded computer simulation not including information regarding the CSC operations performed during the generation of the first recorded computer simulation. The method includes presenting the first recorded computer simulation together with audible and / or visual information received from the ML model regarding the CSC operations performed during the generation of the first recorded computer simulation.
[0010] The details of both the structure and operation of this application can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description
[0011] Figure 1 This is a block diagram of an example system that includes examples based on the principles of the present invention;
[0012] Figure 2 An example game controller according to the principles of the present invention is shown, wherein example key operations of the example game controller can be learned from recorded computer simulation (such as computer games) videos;
[0013] Figure 3 An example of viewing a recorded computer simulation (such as a computer game) according to the principles of the present invention is shown;
[0014] Figure 4 An example flowchart is shown in the form of an example logic for training a machine learning (ML) model to obtain information from recorded computer simulation videos about which controller actions to perform and when to perform them.
[0015] Figure 5 A sequence of video frames accompanied by information about controller operation is illustrated schematically; this information can be used to train an ML model.
[0016] Figure 6 The diagram schematically illustrates a sequence of video frames that do not contain information about controller operation, as well as alternative video paths that may appear depending on controller operation, indicating the identification of controller operation.
[0017] Figure 7 Example logic for identifying controller operations from a sequence of recorded video frames from a recorded computer simulation is shown in an example flowchart format; and
[0018] Figure 8 An example output from an ML model, based on the principles of the present invention, is shown. Detailed Implementation
[0019] The principles of this invention can be applied to machine learning models, including deep learning models. Machine learning models use a variety of algorithms trained in various ways, including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, autonomous learning, and other forms of learning. Examples of such algorithms (which can be implemented using computer circuits) include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) that may be well-suited for learning information from a series of images, and a class of RNNs called long short-term memory (LSTM) networks. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models.
[0020] As understood in this article, performing machine learning involves accessing a model and then training the model with training data so that the model can process other data to make predictions. A neural network may include an input layer, an output layer, and multiple intermediate hidden layers that are configured and weighted to infer an appropriate output.
[0021] This disclosure generally relates to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server components and client components that are network-connected, enabling the exchange of data between the client components and the server components. Client components may include one or more computing devices, including game consoles (such as Sony...). This includes game consoles made by Microsoft, Nintendo, or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (e.g., smart TVs, internet-enabled TVs); portable computers (such as laptops and tablets); and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use operating systems such as Linux, Microsoft, or Unix, or operating systems made by Apple, Inc. or Google. These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, the operating environment according to the principles of the present invention can be used to execute one or more computer game programs.
[0022] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (such as Sony). Instantiation of personal computers, etc.
[0023] Information can be exchanged between clients and servers over a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements methods for providing secure communities (such as online social networking sites) to network members.
[0024] The processor can be a single-chip or multi-chip processor, which can perform logic by means of various lines (such as address lines, data lines and control lines) as well as registers and shift registers.
[0025] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or depicted in the figures may be combined, interchanged, or excluded from other embodiments.
[0026] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes: a system having only A; a system having only B; a system having only C; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having both A, B and C, etc.
[0027] Now for specific reference Figure 1 An example system 10 is illustrated, which may include one or more of the example devices mentioned above and further described below according to the principles of the invention. The first device among the example devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized Internet-enabled (“smart”) phone, tablet computer, laptop computer, HMD, wearable computerized device, computerized Internet-enabled music player, computerized Internet-enabled headset, computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., to communicate with other CE devices to implement the principles of the invention, to perform the logic described herein, and to perform any other functions and / or operations described herein).
[0028] Therefore, in order to implement this principle, AVD 12 can be... Figure 1Some or all of the components shown may be constructed. For example, AVD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include one or more speakers 16 for outputting audio according to the principles of the invention, and includes at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. A graphics processor may also be included. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that the processor 24 controls the AVD 12 to implement the principles of the invention, including controlling other elements of the AVD 12 described herein, such as controlling the display 14 to present images on the display and to receive input from the display. Furthermore, it should be noted that the network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.
[0029] In addition to the foregoing, AVD 12 may also include one or more input ports 26, such as a High Definition Multimedia Interface (HDMI) port or a USB port that physically connects to another CE device and / or a headphone port that connects to AVD 12 to present audio from AVD 12 to a user via headphones. For example, input port 26 may be wired or wirelessly connected to a wired or satellite source 26a of audio / video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44.
[0030] AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. In some cases, the one or more computer memories are embodied as stand-alone devices within the AVD's housing, or as personal video recording devices (PVRs) or video disk players for playing back AV programs, either inside or outside the AVD's housing, or as removable memory media. Furthermore, in some embodiments, AVD 12 may include a location or positioning receiver, such as, but not limited to, a cellular phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular phone base station and provide said information to processor 24 and / or in conjunction with processor 24 to determine the altitude of AVD 12. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of AVD 12 in three dimensions.
[0031] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into AVD 12 and capable of being controlled by processor 24 to acquire pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 to communicate with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.
[0032] In addition, the AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetometers), infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, and gesture sensors (e.g., for sensing gesture commands) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 40 that provides input to the processor 24 for receiving OTATV broadcasts. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, such as a kinetic energy harvester that converts kinetic energy into electrical energy to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array 46 may also be included.
[0033] Still referencing Figure 1In addition to AVD 12, System 10 may also include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send audio and video of a computer game to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by the player. In the example shown, only two CE devices are shown; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.
[0034] Referring now to the aforementioned at least one server 52, said at least one server includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 58, said at least one network interface, under the control of the server processor 54, allowing communication via network 22 with... Figure 1 It communicates with other devices and can actually facilitate communication between the server and client devices according to the principles of the invention. It should be noted that network interface 58 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).
[0035] Therefore, in some implementations, server 52 may be an internet server or an entire server "farm," and in example implementations for, for example, online gaming applications, the server may include and perform "cloud" functionality, enabling devices of system 10 to access the "cloud" environment via server 52. Alternatively, server 52 may be one or more game consoles or... Figure 1 The other devices shown are implemented in the same room or on other computers nearby.
[0036] The components in the subsequent drawings may include Figure 1 Some or all of the components shown.
[0037] Figure 2 A non-limiting example of a game controller, according to the principles of the present invention, can be used to control a computer simulation (such as a computer game) during gameplay, wherein video (along with audio / haptic feedback, etc.) of the game associated with the game is displayed on a display such as display 14 (in the case of haptic feedback, generated for haptic detection using, for example, the controller) and is also recorded, for example... Figure 1 The computer memory 28 and / or shown Figure 1 The server memory 56 shown is used for playback.
[0038] It should be understood that the game controller may be combined with one or more of the above components to communicate with a computer-simulated source (such as CE device 48 embodied as a computer game console and / or server 52) to control the computer game presented on the display 14.
[0039] Figure 2 A controller 200 is shown, comprising a lightweight handheld housing having a rounded, generally cylindrical left handle 202 and a right handle 204, each handle defining a top surface on which four operable keys are disposed. For example, four directional keys 206 may be arranged in a cross pattern on the top of the left handle 202. The keys 206 can be used to move an object in the corresponding direction on the display.
[0040] Additional L1 and L2 buttons 208 can be positioned directly in front of the left handle 202. A bridge connects handles 202 and 204, and a select button 210 can be mounted on the bridge together with the start button 212.
[0041] The four buttons on the right controller 204 may include a triangle button 214, a square button 216, an "O" button 218, and an "X" button 220, each of which can be configured to perform a specific function according to the game designer's wishes. Additional R1 and R2 buttons 222 may be located on the front of the right controller 204.
[0042] Furthermore, between handles 202 and 204, the left joystick 224 may be positioned just inside the left handle 202. The left joystick 224 may include a retractable top 226. Similarly, the right joystick 228 may be positioned just inside the right handle 204. The right joystick 228 may include a retractable top 230.
[0043] Figure 3 Device 300 is shown that records computer-simulated (such as computer games) video and, when needed, audio and haptic feedback associated with the video. Device 300 may be implemented by, for example, a device on which a game is played to generate video, a computer game console, a computer game server located remotely from the player, or a combination thereof.
[0044] The recorded computer simulation (such as a computer game) is provided to the source 302 of the recorded computer game. The source 302 provides the recorded video (and, if necessary, the recorded audio and other sensory outputs of the game, referred to herein as audio-visual) to the viewer / learner's computer device 304.
[0045] Because the actions of a computer game controller inputted to control the game during recording may not be recorded when recording a computer game AV, a machine learning (ML) engine 306 is provided to execute according to the principles herein to replicate controller actions from the computer game audio-visual file, so as to present the instructions of said controller actions along with the recorded game AV on the viewer / learner computer 304. The ML model 306 may include at least one recurrent neural network (RNN), such as at least one long short-term memory (LSTM) network. A convolutional neural network (CNN) may also be used. The ML model 306 may be executed by any processor or combination thereof disclosed herein, such as the processor in the viewer / learner computer 304, the recorded game source 302 (such as a remote server or local game console), etc. Note that in some embodiments, elements 300, 302, and 304 may be implemented by the same device. For example, a user may attempt to learn from game recordings of different users who previously played the game on the same console.
[0046] Figure 4 The training method is shown. Figure 3 The logic of ML model 306 is described in box 400. The training dataset is then fed into the ML model. The training set may include video and / or audio associated with multiple computer simulations, such as computer games.
[0047] More specifically, the training set may include sequences of video frames from multiple recorded computer simulations and / or, if necessary, audio associated with the video. Furthermore, the training set includes real-world data associated with the video frame sequences regarding the computer simulation controller (CSC) operations performed during the generation of the video frame sequences. CSC operations may be due to, for example… Figure 2 The manipulation of any one or more controls shown in the example is generated. From the training set and using appropriate learning techniques (e.g., supervised, unsupervised, semi-supervised, etc.), at box 402, ML model 306 learns CSC operations from the video frame sequence.
[0048] Figure 5 A sequence 500 of video frames 502 in a training set used to train an ML model is shown. Metadata 504 is associated with sequence 500 (frame-by-frame, if necessary) indicating when controller actions occurred during the recording of the training sequence. Note that in some implementations, the metadata does not need to be frame-by-frame. The metadata is only updated when the CSC operation changes compared to the previous frame. Furthermore, the CSC operation, along with timestamps that can be used to synchronize video data with CSC data, can be stored entirely in a separate file. Metadata 504 may include which specific control surfaces were manipulated (e.g., in...). Figure 2The controller shown is on a controller (such as the controller shown) and an indication of when and which control surfaces were manipulated during the generation of the recorded sequence 500.
[0049] Training sets can be created in various ways. Real-world button manipulation data can be collected during gameplay and correlated with the generated video in time alignment for use as part of the training set. Training sets can also include pre-recorded game video with controller overlays, generated while playing the original game and presented along with the video. The controller action data in the overlays is already time-aligned with the video, as it is typically presented on the video. Pixel values in the video can be examined for each frame to determine which buttons were pressed, as indicated in the overlays, to generate label data. This also provides timing data, as each frame can be associated with a timestamp. The overlay feature can be enabled when starting to play the training set video.
[0050] Real-time controller actions can be streamed from the controller to a device that records game video under the controller's control to associate the real-time controller actions with the video, thereby establishing elements of a training set.
[0051] After training, the ML model can be used to receive video frame sequences of 600 ( Figure 6 (Optional audio is not shown for clarity.) The first recorded computer simulation does not include information about CSC operations performed during the generation of the first recorded computer simulation. Depending on how the video frame sequence evolves, the ML model is based on data from... Figure 5 The training aims to identify what CSC operations are performed during the generation of the first recorded simulation and when those operations are performed. For example, as in... Figure 6 Between sequence 1 and possible sequence 2A, no CSC controller operation may occur. On the other hand, between sequence 1 and possible sequence 2B (a sequence different from possible sequence 2A), a CSC operation may be performed to shift the entire sequence from sequence 1-sequence 2A to sequence 1-sequence 2B, and the ML model makes the identification of such CSC operations 602.
[0052] Figure 7 The above principle is illustrated in an example flowchart format. Figure 7 The logic can be executed by any one or more processors embodied in the viewer / learner AV display 304 and / or the source 300 of the generated computer simulation and / or the source 302 of the recorded computer simulation, either or both of the sources 300 and 302 can be instantiated as a computer simulation console or a server communicating with the AV display via a wide area computer network (such as the Internet).
[0053] It should be understood that, in addition to the following, preprocessing and reduction boxes for reducing the time required for ML models to identify CSC operations, and postprocessing and synchronization boxes for filtering out unsupported CSC operations and adjusting waiting times, may also be provided.
[0054] Starting at frame 700, recorded computer simulations, such as recorded computer games, are received. These simulations include a sequence of video frames and / or an accompanying audio track. Typically, the recorded simulations do not include information about the operation of the computer simulation controller during the generation of the recorded simulation. The frame sequence may, for example, be a short video clip of a game from an internet platform.
[0055] Move to box 702, through Figure 3 The ML model 306 in the process processes the recorded computer simulation, such as a sequence of video frames in a recorded simulation, the ML model being a reference. Figure 4 Training is performed as disclosed. The ML model identifies and generates computer simulation controller (CSC) operations associated with the recorded computer simulation and, based on its training, outputs instructions for the CSC operations, which are received and associated with the recorded computer simulation. CSC operations identified by the ML model may include, for example, manipulation... Figure 2 Which control elements are shown in the example and when are they manipulated?
[0056] Move to box 704 and present the recorded computer simulation along with at least one instruction for at least one CSC operation received from the ML model in a format such as Figure 3 The viewer / learner's computer, such as a 304, is on at least one audio-visual (AV) display.
[0057] Figure 8 As shown. It can be found in, for example... Figure 1 The recorded video game 800 is displayed on a monitor such as display 14 shown. In the illustration, game 800 includes a character 802 firing a weapon 804 at a flying object 806. Line 808 indicates that object 806 has been hit.
[0058] Indicator 810 is displayed on the monitor, indicating what CSC action occurred in the game (“red key pressed”) and when the action occurred, in the example shown, “now.” It should be understood that indicator 810 may indicate a past CSC action that led to the explosion and the time of that action, as well as an anticipated upcoming CSC action. Indicator 810 may be visually presented as shown and / or audibly presented on a speaker associated with the monitor. Thus, the recorded computer simulation is presented together with information received from the ML model regarding the CSC actions performed during the generation of the recorded computer simulation.
[0059] The above logic can be provided as a plugin along with computer game controllers or other features to enable players to download game videos and obtain information about the sequence of controller actions that generated the videos.
[0060] It should be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements can be used to achieve the subject matter claimed herein.
Claims
1. A device for guiding a viewer, the device comprising: At least one computer memory, wherein the at least one computer memory is not a transient signal and includes instructions executable by at least one processor to perform the following operations: Receive a recorded computer simulation including a sequence of video frames, the recorded computer simulation being generated under the control of at least one computer simulation controller (CSC) and not including information about the operation of at least one control element of the CSC during the generation of the recorded computer simulation; The video frame sequence is processed in a machine learning (ML) model, wherein the ML model is trained to generate information about the operation by analyzing the recorded video frame sequence, excluding information about the operation on the control element. Receive from the ML model the identification of the operation associated with generating the recorded computer simulation; as well as The recorded computer simulation, together with at least one instruction of at least one of the operations received from the ML model, is presented on at least one audio-visual (AV) display.
2. The apparatus of claim 1, wherein the ML model comprises at least one recurrent neural network (RNN).
3. The apparatus of claim 2, wherein the RNN comprises at least one Long Short-Term Memory (LSTM) network.
4. The apparatus of claim 1, wherein the ML model comprises at least one convolutional neural network (CNN).
5. The apparatus of claim 1, wherein the at least one processor is embodied in the AV display.
6. The apparatus of claim 1, wherein the apparatus includes the at least one processor, wherein the at least one processor is embodied in the source of the recorded computer simulation.
7. The apparatus of claim 6, wherein the source comprises at least one computer simulation console.
8. The apparatus of claim 6, wherein the source comprises at least one server communicating with the AV display via a wide area computer network.
9. The apparatus of claim 1, wherein the operation of the at least one control element of the CSC includes recognizing at least one manipulation button on the CSC.
10. A device for guiding a viewer, the device comprising: At least one display, the at least one display being configured to present at least one recorded computer simulation, the at least one recorded computer simulation being generated under the control of at least one computer simulation controller and not including information regarding the operation of at least one control element on the computer simulation controller during the generation of the at least one recorded computer simulation; as well as At least one processor, the at least one processor being configured with instructions to: Information about the operation of at least one control element on the computer simulation controller during the generation of the at least one recorded computer simulation is identified from the at least one recorded computer simulation using at least one machine learning (ML) model, wherein the ML model is trained to generate the information about the operation by analyzing a sequence of recorded video frames that does not include information about the operation on the control element. and The information about the operation during the generation of the at least one recorded computer simulation is provided to the at least one display so that the information is presented together with the presentation of the at least one recorded computer simulation.
11. The device of claim 10, wherein the at least one processor is embodied in the at least one display.
12. The device of claim 10, wherein the at least one processor is embodied in a computer simulation console.
13. The device of claim 10, wherein the at least one processor is embodied in a server communicating with the at least one display via a wide area network.
14. The device of claim 10, wherein the ML model comprises at least one recurrent neural network (RNN).
15. The device of claim 14, wherein the RNN comprises at least one Long Short-Term Memory (LSTM) network.
16. The device of claim 10, wherein the operation of the at least one control element of the computer simulation controller includes recognizing at least one manipulation button on the computer simulation controller.
17. A method for instructing a viewer, the method comprising: At least one machine learning (ML) model is trained using at least one training set to generate information about the operations by analyzing a sequence of recorded video frames that does not include information about the operations on the control elements. The training set includes a plurality of recorded computer-simulated video frame sequences and information associated with the video frame sequences about the operations of at least one control element on the computer simulation controller (CSC) performed during the generation of the video frame sequences. Input at least a first recorded computer simulation into a trained ML model, wherein the first recorded computer simulation does not include information about the operations performed during the generation of the first recorded computer simulation; as well as The first recorded computer simulation is presented together with information output from the trained ML model regarding the operations performed during the generation of the first recorded computer simulation.
18. The method of claim 17, the method comprising audibly presenting information received from the ML model regarding the operations performed during the generation of the first recorded computer simulation, while visually presenting the first recorded computer simulation.
19. The method of claim 17, the method comprising visually presenting information received from the ML model regarding the operations performed during the generation of the first recorded computer simulation, while visually presenting the first recorded computer simulation.
20. The method of claim 17, the method comprising associating the first recorded computer simulation with information regarding the operations performed during the generation of the first recorded computer simulation at a server, the server providing the first recorded computer simulation to a display presenting the first recorded computer simulation.
21. The method of claim 17, the method comprising associating the first recorded computer simulation with information about the operations performed during the generation of the first recorded computer simulation at a local source, the local source providing the first recorded computer simulation to a display presenting the first recorded computer simulation.
22. The method of claim 17, wherein the operation of the at least one control element of the CSC includes identifying at least one manipulation button on the CSC.
Citation Information
Patent Citations
Method, device and storage medium for controlling vehicle in use
CN108635861A
System and method for generating an input signal
US20190381401A1
Video game overlay
WO2021066849A1
