Ai on-board assistant

Abstract

To provide a system, method, and storage medium for assisting virtual interaction on an entertainment device.SOLUTION: A method by a system for providing on-board assistance personalized on the basis of a user profile receives and stores the user profile including user data associated with the use of an entertainment device, and detects an indication of new use by a user of the entertainment device on the basis of the user profile. An activity associated with the new user of the entertainment device is determined. The method also provides assistance for the activity associated with the new user of the entertainment device based on the user profile and updates the user profile on the basis of the provided assistance.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The system of the present invention generally relates to providing virtual assistance to a user. More specifically, the system of the present invention relates to assisting a user based on analysis of the user's profile information.

Background Art

[0002] Many different types of individuals with various experiences can be involved in the use of entertainment systems. Video game consoles, handheld devices, computers, and virtual reality systems, as well as the related digital content accessed and played by them, can be continuously updated or refreshed with new versions offering new experiences, as well as challenges for learning and adaptation. For some individuals, learning to play a new video game on new hardware can be a frustrating process not only because they are not familiar with a particular game title, but also because they are not familiar with how to use the hardware to perform intended movements and other game interactions. Even experienced and skilled players can experience some difficulties or awkwardness when transitioning to new or unfamiliar systems, devices, and game titles. The power of muscle memory and habits associated with old or familiar systems and titles may not always transfer smoothly, and such users may ultimately cause various input errors leading to unintended in-game movements and interactions.

[0003] Entertainment systems and new media content typically only offer users a general introduction, such as a quick setup guide for a new console or an introductory tutorial for a video game. Depending on the user's experience level and the scope of the material, such material can be overwhelming in terms of volume and complexity for users who rarely interact with such devices or content, thereby only serving to further confuse, frustrate, and / or overwhelm the user.

[0004] Therefore, in this field of technology, there is a need to provide improved systems and methods for personalized onboard assistance that can guide users through new entertainment systems and various forms of content on such systems, in terms of familiarizing users with, improving comfort, and simplifying their entry into new experiences. [Overview of the Initiative]

[0005] A method for assisting virtual interactions on an entertainment device is disclosed. The method includes storing a user profile that includes user data associated with the use of the entertainment device. The method also includes detecting assistance triggers made by the user of the entertainment device. The method also includes determining the activity associated with the assistance trigger. Assistance content related to the activity may be identified as being associated with the assistance trigger, and the assistance content may be personalized based on the user profile. The user profile may further be updated based on the assistance content provided.

[0006] A system for assisting virtual interaction on an entertainment device is disclosed. This system includes memory and a processor that executes instructions stored in memory. The execution of instructions by the processor stores a user profile containing user data associated with the use of the entertainment device. The execution of instructions by the processor also detects assistance triggers from the user of the entertainment device. The execution of instructions by the processor also determines the activity associated with the assistance trigger. Assistance content related to the activity may be identified as being associated with the assistance trigger, and the assistance content may be personalized based on the user profile. The user profile may further be updated based on the provided assistance content.

[0007] A non-temporary, computer-readable storage medium is described that embodies a processor-executable program for performing a method to assist virtual interaction on a disclosed entertainment device. The method includes storing a user profile containing user data associated with the use of the entertainment device. The method also includes detecting assistance triggers by the user of the entertainment device. The method also includes determining the activity associated with the assistance trigger. Assistance content relating to the activity may be identified as being associated with the assistance trigger, and the assistance content may be personalized based on the user profile. The user profile may further be updated based on the provided assistance content. [Brief explanation of the drawing]

[0008] [Figure 1] This illustrates an exemplary network environment in which a system for providing personalized onboard assistance based on user profiles may be implemented. [Figure 2] This flowchart illustrates an example of how to provide personalized onboard assistance based on a user profile. [Figure 3] This shows an exemplary controller input guide that may be provided as part of personalized onboard assistance. [Figure 4] This document illustrates an exemplary embodiment of a system for providing personalized onboard assistance based on the user profile within the in-game environment. [Figure 5a] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 5b] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 5c] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 5d] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 5e] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 5f] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to the display and addition of peers as social connections. [Figure 6a]The following are illustrative screenshots that may be provided during the use of a system to offer personalized onboard assistance in connection with the purchase of media content from an online store. [Figure 6b] The following are illustrative screenshots that may be provided during the use of a system to offer personalized onboard assistance in connection with the purchase of media content from an online store. [Figure 7a] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to recommending new media content based on the media content in the user's library. [Figure 7b] The following are illustrative screenshots that may be provided during the use of the system to provide personalized onboard assistance in relation to recommending new media content based on the media content in the user's library. [Figure 8] The following are illustrative screenshots that may be provided during the use of a system to provide personalized onboard assistance in relation to recommending new media content based on the purchase history of users' peers. [Figure 9] This is a block diagram of an exemplary electronic entertainment system. [Modes for carrying out the invention]

[0009] A method and system for providing personalized onboard assistance is disclosed. A user profile containing user data associated with the use of an entertainment device is stored. Assistance triggers by the user of the entertainment device are detected. The activity associated with the assistance trigger is determined. Assistance content related to the activity may be identified as being associated with the assistance trigger, and the assistance content may be personalized based on the user profile. The user profile may further be updated based on the assistance content provided.

[0010] Figure 1 shows an exemplary network environment 100 in which a system for providing personalized onboard assistance based on user profiles may be implemented. The network environment 100 may include one or more interactive content servers 110 that provide streaming content (interactive video, video games, etc.), one or more platform servers 120, one or more user devices 130, and one or more databases 140.

[0011] An interactive content server 110 may maintain, stream, and host interactive media that can be streamed or downloaded on a user device 130 via a communication network. Such an interactive content server 110 may be implemented in the cloud (e.g., one or more cloud servers). Each piece of media may contain one or more sets of object data that can be made available for user participation (e.g., viewing or interacting with content activity). Data about the objects shown in the media may be stored by the media streaming server 110, the platform server 120, and / or the user device 130.

[0012] The platform server 120 may be capable of communicating with different interactive content servers 110, databases 140, and user devices 130. Such a platform server 120 may be implemented on one or more cloud servers. A streaming server 110 may communicate with multiple platform servers 120, while a media streaming server 110 may be implemented on one or more platform servers 120. The platform server 120 may also execute commands, for example, to receive user requests to stream streaming media (games, activities, videos, podcasts, user-generated content (UGC), and publisher content, etc.). Furthermore, the platform server 120 may execute commands, for example, to stream streaming media content titles.

[0013] A set of streaming media and at least one associated object data may be provided by an Application Programming Interface (API) 160, which enables various types of media streaming servers 110 to communicate with different platform servers 120 and different user devices 130. The API 160 may be specific to the computer programming language, operating system, protocol, etc., of the media streaming server 110 that provides the streaming media content title, the platform server 120 that provides the media and at least one associated object data, and the user device 130 that receives the object data. Similarly, in a network environment 100 that includes multiple different types of media streaming servers 110 (or platform servers 120 or user devices 130), there may be a corresponding number of APIs 160.

[0014] The user device 130 may include multiple different types of computing devices. For example, the user device 130 may include any number of different game consoles, mobile devices, laptops, and desktops. In another example, the user device 130 may be implemented in the cloud. Such a user device 130 may also be configured to access data from other storage media, such as memory cards or disk drives, which may be appropriate in the case of downloaded services. Such a device 130 may include, but is not limited to, standard hardware computing components such as network interfaces and media interfaces, non-temporary computer-readable storage (memory), and a processor for executing instructions that can be stored in memory. The user device 130 may include various hardware sensors for detecting user interaction, such as cameras, microphones, haptic feedback input mechanisms, and gyroscopes. The hardware sensors of the user device may be used to capture user responses and feedback, such as gestures, speech, and facial expressions. These user devices 130 may also be launched using various different operating systems, such as iOS or Android®. The user device 130 may also launch various applications or computer languages, such as C++ or JavaScript®. A user device may include one or more devices associated with a user, or user devices that can be displayed on one or more screens.

[0015] The database 140 can be stored in the platform server 120, in the media streaming server 110, in the same server, in different servers, in a single server, across different servers, or in any of the user devices 130. Such a database 140 can store a set of streaming media and / or related object data. Such streaming media can represent one or more objects or activities in which a user can participate or interact. One or more user profiles can also be stored in the database 140. Each user profile can include information about the user such as the user's progress in activities and / or media content titles, user ID, related game characters, team mates, opponents, in-game levels achieved, skill levels, scores, game metrics, user devices, media titles, game settings, etc.

[0016] The personalized on-board assistant system 170 can be stored in and executed from the database 140, the user device 130, the platform server 120, or the interactive content server 110. Each user profile can track the use of the personalized on-board assistant system 170 and store in the database 140 the user's on-board progress and preferences regarding the information delivered by the personalized on-board assistant system 170. The personalized on-board assistant system 170 can be involved by the user on the user device 130 that utilizes a digital display such as a TV, projector, monitor, smartphone, virtual reality (VR) head-mounted display, or handheld device, and can incorporate one or more physical input devices such as a touch screen, remote, controller, joystick, or mouse and keyboard. <>

[0017] <> In some embodiments, the personalized onboard assistant system 170 may also refer to user profiles of other users connected to or identified as similar to the current user. For example, a user's contacts or friends (identified by explicit instructions from the user or based on tracked chats or other communications) may have more developed user profiles if they are more experienced, provide more data, or are otherwise associated with more data from one or more sources. Using the profiles of such contacts or friends, the system may predict the user's likely preferences or tendencies. Furthermore, if other users exhibit similar gameplay characteristics (e.g., the same gameplay title, the same level score, or other metrics), the system may use their profiles to predict the likely trajectory of the current user's gameplay. The personalized onboard assistant system 170 may also continuously update user profiles and improve predictions and insights over time as new data is received that confirms or indicates that such predictions are incorrect. Different patterns may emerge with respect to a user's gameplay in relation to different types of game titles and in-game conditions, and these may be detected by the personalized onboard assistant system 170. Such patterns can be used to update user profiles and further personalize onboard assistance to the user. Thus, a personalized onboard assistance system 170 may have an updated user profile for use when customizing in-game assistance in a way that is relevant and useful to the user.

[0018] Therefore, personalized onboarding for a user can be presented in a different manner than personalized onboarding for another user. Further, since users may exhibit different habits and tendencies for each title, personalized onboarding for users can be presented differently for each title. For example, a user may have one friendship with a friend regarding one game genre or title (e.g., a first-person shooting game), and a different friendship with a different friend regarding a different game genre or title (e.g., a music game or a dance game). In some embodiments, the personalized onboarding can indicate a player's input (combos), and can be provided by a virtual character in a virtual environment that performs movements and operations within the game. Also, the character can function as a tour guide, tutor, guide, or other type of assistant within the virtual environment. Therefore, the former personalized onboarding can utilize different audio-visuals than the latter personalized onboarding.

[0019] Figure 2 is a flowchart 200 showing an exemplary method for providing personalized onboarding assistance based on a user profile. The steps identified in Figure 2 are exemplary and not limiting, and may include various alternative, equivalent, or derivative forms including the order of execution of the steps. The steps of the process of Figure 2 and any alternative similar processes can be embodied in hardware or software including a computer-readable storage medium containing instructions executable by something such as a processor within a computing device. The exemplary process shown in Figure 2 can be repeated during use of an entertainment system.

[0020] In step 210, user profile data is received by the personalized onboard assistant system 170. The user profile data may include information about the user's activity on the user device 130, such as the user's history of using the entertainment system profile, games played, or other user activity. The user profile data may also include legacy system data, such as users who have previously interacted with the entertainment system associated with the current user device 130. The user profile data may also include information about the user's prior media content and activities, such as records of media content the user has previously engaged with, the user's global purchase history of media content from online stores, the user's peers' purchase history of media content, preferences related to receiving assistance from the personalized onboard assistant system 170, and records of input received from the user of the entertainment device. The user profile can track the user's interactions and progress with the personalized onboard assistant system 170 after the user first engages with the entertainment system or media title. When a new user profile is created, the user profile data may not include the user's history of using the entertainment system, or specific media content, or specific activities.

[0021] In step 220, assistance triggers may be detected by a personalized onboard assistant system 170. Assistance triggers may include the user who first engages with an entertainment system, a specific media content file, or a specific activity, for which the user profile does not have a record of past interactions with the system, media, or activity. Each entertainment system, media content file, or activity in which the user engages may be tracked separately by the user profile.

[0022] Other types of assistance triggers may indicate unfamiliarity with system hardware, content titles, or other aspects of the virtual environment. For example, such assistance triggers may include repeated key presses, a series of variations of the same combo key presses, verbal indicators, lack of progress over time, wandering in-game, and other indicators. Detection of assistance triggers may also be based on a clear request for help or assistance from the user, whether it is a verbal request (e.g., "Help!" or "This is difficult!") or a written request (e.g., a chat message to a friend), or an item selected from an options menu.

[0023] In one embodiment, a user profile may not include data regarding the usage history of an entertainment system or associated legacy entertainment system. A user without such data may be detected as a new user of the entire system. A new user of an entertainment system may be determined by the user creating a new user profile. For example, a user may interact with Sony PlayStation 5® and create a new user profile without using an existing PlayStation Network profile, associated entertainment system, or available media content data. In this case, the user may be detected as a new user who is initially involved with the system. In another example, a user profile may include data associated with an associated legacy entertainment system and may not include profile data associated with the current entertainment system. For example, a user may have previously been involved with Sony PlayStation 4® and may have an associated PlayStation Network profile and data. Such a user may interact with Sony PlayStation 5® using an existing profile and be detected as a user with experience of the Sony PlayStation 4® system, but may be determined to be a new user of the Sony PlayStation 5® system.

[0024] In another embodiment, the user profile may not contain data associated with a specific media content file. The user may first engage with a media content file from any source, such as content streamed or downloaded from the interactive content server 110, or from a physical media device such as a USB hard disk, solid-state drive (SSD), hard disk drive (HDD), Blu-ray® disc, DVD-ROM, other optical disc, or plug-and-play storage device. The first interaction with a new media content file may be recorded and stored in the user profile. For example, the user may stream a television program from a streaming service on a cloud server or from the interactive content server 110. Upon detecting the first user input request to view the program, the personalized onboard assistant system 170 may detect an assistance trigger associated with the media content title. The user profile may be updated to reflect the interaction with the media content, and information and metadata associated with the content, such as title, genre, and duration, may be stored.

[0025] In different embodiments, a user profile may not contain data associated with specific activities related to the entertainment system. Activities related to the entertainment system may include, but are not limited to, interactions such as searching for and adding peers on a network as social contacts, accessing and searching online stores for media content to purchase, playing video games, streaming media content, and customizing the user profile by adding interests or descriptions about the user.

[0026] In step 230, the personalized onboard assistant system 170 may determine the activity associated with the assistance trigger. When data does not exist in the user profile for the activity, the personalized onboard assistant system 170 may determine an activity that the user may attempt to initiate based on the media title and the user profiles of other users interacting with the media title. The initial engagement with the activity may be recorded in the user profile and received by the personalized onboard assistant system 170, which includes data identifying the activity associated with the assistance trigger. The personalized onboard assistant system 170 may identify that the user is currently interacting with a particular media content title at a specific location (e.g., level, layer, section) within the relevant virtual environment. Using such interaction data, the system may filter the available activities and identify an activity that the user may be struggling with. In some embodiments, the personalized onboard assistant system 170 may present the filtered set of activities to the user as an overlay menu, thereby enabling the user to access different assistance content from within the virtual environment.

[0027] In one embodiment, a user may require assistance browsing an online store. The user may have access to the online store and their user profile may not contain any historical data associated with online store usage. The user may browse multiple pages of media content in the online store without selecting any particular content to see more details about the content or to purchase the content. Furthermore, the user may continue to return to the online store's home screen and attempt to browse different categories or sections of the online store without selecting anything. In this case, the user may be unfamiliar with the sections of content in the online store and may be unable to identify the content they are looking for. The personalized onboard assistant system 170 may determine that the user is shopping in the online store for the first time and may require assistance to identify the content.

[0028] In another embodiment, a user may require assistance in adding peers as social contacts. The user may have access to a list of peers added as their social contacts on the network, or a "friends" list. The user may navigate to the friends list and perform activities such as searching for peers or receiving results from search text entered by the user. The personalized onboard assistant system 170 may determine that assistance is needed in correctly entering profile names when the user searches for friends, or that assistance is needed in understanding the workflow of the user searching for peers and adding them as friends.

[0029] In different embodiments, the user may require assistance in interacting with the game. The user may obtain the game from an online store and initially access the game through the entertainment system. The user may not understand how to launch the game application or may not know where the downloaded game is located, and may navigate to different sections of the entertainment system or repeatedly return to the home screen without launching the game. The personalized onboard assistant system 170 may determine that a user navigating the entertainment system without launching an application may require assistance in identifying a recently purchased game application. Alternatively, the user may identify and launch the game application, but may remain idle on the game application's title screen. The personalized onboard assistant system 170 may determine that the user may require various different assistance in playing the game or becoming familiar with the entertainment system input devices.

[0030] In step 240, the personalized onboard assistant system 170 may provide assistance content to the user based on a selected workflow associated with an assistance trigger detected in the entertainment system. Activity assistance content may include various on-screen displays, audiovisual content, text annotations, video clips, etc. Each screen display may be configured and presented according to the user profile. For example, a novice user requiring more assistance may be presented with a different type and amount of information than a more experienced user who only needs usage tips to navigate a new system or title. Conversely, an expert user may be interested in detailed information on complex combos and game theory, which could overwhelm a new user. The selection of the type and amount of information to present to the current user, as well as how such information is presented within the current environment, may be based on the user profile. Similarly, users with different social circles and habits may be provided with different advice and guidance on how to connect, compete, spectate, or otherwise interact with other users in the virtual environment.

[0031] Following the activity determined in step 230, the personalized onboard assistant system 170 may provide the user with on-screen prompts to initiate assistance related to the entertainment system, media content, or assistance triggers detected in the activity. Prompts displayed to the user may require user input to initiate assistance content, suspend upcoming assistance, or terminate currently provided assistance.

[0032] In step 250, the user profile may be updated based on the assistance content provided by the personalized onboard assistant system 170. Based on the type of assistance provided in step 240 and the results of providing the assistance content in step 240, the user profile may be updated with respect to the user's preferences for completing activities and receiving assistance for activities. The user's preferences for receiving assistance for activities may include successfully completing the activity with the assistance, suspending the assistance, or exiting the assistance mode in which the assistance content is presented. Successfully completing the activity after the assistance content has been provided may indicate that the assistance content was useful, and the personalized onboard assistant system 170 may continue to provide assistance content in the next sequence of the workflow associated with the activity. Suspending the assistance mode may be temporary in response to user input and may include pausing the assistance mode associated with a particular activity. The personalized onboard assistant system 170 may provide different assistance content for the same activity when the suspension of the activity's assistance mode is indicated by user input. For example, different assistance content may be provided for the same activity, but at different initial points, different presentation methods of the assistance content may be used, such as visual highlighting or audio narration, as opposed to text assistance content, or additional explanations may be provided at each step. User preferences related to the assistance content may further include settings regarding the termination of the assistance mode, such as indefinitely canceling the assistance mode associated with the activity. The personalized onboard assistant system 170 may cease providing assistance content for terminated activities, but may provide different assistance content for new activities.Completing, suspending, or ending assistance mode can be recorded and remembered in the user profile.

[0033] For example, an on-screen display may be provided to the user, including a text overlay that guides the user to enter a series of commands to begin playback of a media content file. Following the successful execution of the on-screen display and media content playback, the personalized onboard assistant system 170 may update the user profile to reflect the successful completion of the activity. Taking advantage of the successful completion of the activity related to the assistance content provided by the personalized onboard assistant system 170, the user profile may be updated regarding restricting or excluding further assistance content (e.g., or subsequently providing more advanced assistance content) provided to the user for the entertainment system, the associated media content file, or the associated activity. Furthermore, the user may choose to suspend any future assistance modes related to the playback of the media content file, and the request to suspend may be recorded in the user profile. The user may also choose to exit the guided prompts provided by the personalized onboard assistant system 170 before completing the associated steps to perform playback of the media content file.

[0034] Steps 210–250 may be repeated by the personalized onboard assistant system 170 each time the user interacts with the entertainment system. The personalized onboard assistant system 170 may automatically terminate the provision of assistance content if it detects that user input has been terminated or assistance related to the activity has been suspended. Furthermore, if the personalized onboard assistant system 170 detects user frustration with the entertainment system, media content files, or activity, it may resume the assistance mode associated with the terminated or suspended activity. Various hardware sensors may detect user frustration in various ways. For example, a camera may record facial expressions, hand gestures, and body language associated with negative emotions. In another example, a microphone may record user speech or words spoken at an increased volume associated with negative emotions. In yet another example, a haptic feedback device or gyroscope embedded in the controller may receive rapid changes in input associated with hitting, throwing, shaking, or other disruptive movements, and rapid changes in input associated with negative emotions.

[0035] User frustration through input can also be detected by the personalized onboard assistant system 170 based on user input related to software interactions with entertainment systems, media content files, or activities. For example, a user may receive assistance from the personalized onboard assistant system 170, choose to terminate the assistance, and then immediately close a game application. Closing an application immediately after terminating assistance from the personalized onboard assistant system 170 may be identified as user frustration with the game. In another example, a user may navigate an online store or social contact list, receive and terminate assistance related to the online store or social contact list, and repeatedly backtrack to menus within the store or social contact list. User input received to repeatedly access the same menu may be identified by the personalized onboard assistant system 170 as user frustration with identifying activities related to the online store or contact list.

[0036] In some embodiments, any combination of hardware sensor data and input data related to the entertainment system software may be tracked by a personalized onboard assistant system 170 to identify user frustration. For example, while playing a game, a user may not complete an in-game activity. During the period when the user is not completing an in-game activity, the user may be recorded by a camera and recognized as looking down at a controller or input device. Simultaneously with the camera recording the user, the personalized onboard assistant system 170 may detect that the user is not entering any commands into the controller or input device. Such a combination of camera data and input data may indicate that the user is frustrated, confused about how to operate the device or perform an activity, and may require assistance related to the game or input device.

[0037] If user frustration is detected, the personalized onboard assistant system 170 may override the user profile preferences regarding assistance content that the user has previously terminated. The personalized onboard assistant system 170 may display assistance content that has additional steps and more detailed information than the assistance content that the user has previously terminated. For example, the personalized onboard assistant system 170 may provide the user with a tutorial to add a peer as a social contact or to add a friend. The initial tutorial provided by the personalized onboard assistant system 170 may include on-screen text and highlights to the navigation items necessary to add a friend. After the user terminates the tutorial assistance and further user frustration is detected, the personalized onboard assistant system 170 may resume the tutorial to add a peer as a friend. The new assistance displayed by the personalized onboard assistant system 170 may provide on-screen displays of the controller inputs necessary to navigate the menu for adding a peer. Furthermore, the personalized onboard assistant system 170 allows users to complete workflows more easily by disabling other menu options besides the relevant menu options for the workflow's guided assistance, adding peers as friends, and eliminating potential errors such as selecting the wrong menu.

[0038] Figure 3 shows an exemplary controller input guide 300 that may be provided as part of personalized onboard assistance. Assistance triggers may include indicators that the user is unfamiliar with the input device, such as a game controller (e.g., repeatedly pressing keys). The personalized onboard assistance system 170 may provide an on-screen display of the controller and controller functions 300 on the user device 130 to help the user become familiar with the controller's button layout, use, and configuration. Assistance to the user may include images or videos of the controller 310 that may involve the user in operating an entertainment system with text or audio explanations of each input function. Indicators of the controller's input functions, such as button locations and button names 320 and button labels 330, may be displayed. The input indicators and explanatory text provided by the personalized onboard assistance system 170 may be highlighted or emphasized to draw the user's attention to specific areas of the controller during the explanation, such as highlighting buttons in a different color that is brighter or more conspicuous compared to the background of the controller or display, while unrelated areas are darkened or grayed out.

[0039] In one embodiment, the personalized onboard assistant system 170 may guide the user through various input mechanisms and functions on the controller for the entertainment system. For example, basic controls for navigating the entertainment system menu may be provided. For instance, the personalized onboard assistant system 170 may instruct the user to press up, down, left, or right on the directional pad (d-pad) 340 to move the on-screen cursor, press the X button 350 to select an option, or press the circle button 360 to cancel or return to the previous menu.

[0040] In another embodiment, the personalized onboard assistant system 170 may guide the user with specific input mechanisms and functions associated with media content files or activities. For example, the personalized onboard assistant system 170 may display an on-screen controller input guide when the user first interacts with a new media content file. Such an on-screen controller input guide may include input functions for the current gameplay and the player character of an overlaid game. If the system detects user frustration with the media content, the same on-screen controller input guide may be displayed.

[0041] Figure 4 shows an exemplary embodiment of a system for providing personalized onboard assistance based on a user profile within a game environment. In addition to on-screen text and images, the personalized onboard assistant system 170 may provide interactive tutorials and training to familiarize the user with input devices, media content, or entertainment system activities. In one embodiment, the user may operate the controller 410 to manipulate the camera angle and move an avatar in interactive media content such as a game. The personalized onboard assistant system 170 may provide visual representations to the user operating the controller 420, which performs gameplay functions simultaneously with active gameplay or training. For example, tilting the left analog stick to the right moves the avatar to the right in the game world, and tilting the right analog stick to the left changes the camera angle to the left, making the user move around the avatar. The visual representation 420 may first simulate the necessary inputs by the user 430 on-screen and display examples of the avatar and camera operating separately and in parallel. Next, the visual representation 420 can track user inputs to display to the user when training is performed correctly by changing the on-screen display of controller inputs based on actual user inputs to the controller 410.

[0042] Figures 5a to 5f show exemplary screenshots that may be provided during the use of a system for providing personalized onboard assistance in relation to the display and addition of peers as social connections. The personalized onboard assistant system 170 may suggest adding the user's peers on the entertainment device as social connections. Various indicators may notify the personalized onboard assistant system 170 of a suggestion to add the user's peers as social connections. These peers may include, for example, peers on different social networks that are pre-connected to the user profile by linked accounts, recent interactions by the user with game peers, peers with similar media content interactions and purchase history, and peers of peers that the user has previously added as social connections. The personalized onboard assistant system 170 may guide the user through each step of the process of searching for and adding peers on the entertainment system.

[0043] Figure 5a shows the home screen dashboard 500 of the entertainment system on the user device 130. The personalized onboard assistant system 170 may overlay input figures 510, such as controllers associated with the navigation of the entertainment system. Furthermore, the personalized onboard assistant system 170 may provide tutorial text 520 linked to the controller input figure 510, visually guiding the user to begin the process of adding peers.

[0044] In Figure 5b, the user may follow prompts provided by the personalized onboard assistant system 170, provide the necessary inputs, view different displays on the dashboard 500, and then proceed to the next step of adding a peer as a contact. The personalized onboard assistant system 170 may highlight the next step in the process to draw the user's attention to the navigation item 530 associated with adding a peer. For example, the personalized onboard assistant system 170 may provide a callout highlight 540 by drawing a line from the navigation item 530 to an image of the same navigation item with increased size and resolution. Furthermore, the personalized onboard assistant system 170 may adjust the position and size of the controller input diagram 510 to avoid obstructing the navigation item on the dashboard. The controller input diagram 510 may be updated with a pointer to the inputs required to select the navigation item and updated tutorial text 520 that reflects the instructions for selecting and confirming the navigation item.

[0045] In Figure 5c, the social submenu 550 may expand and overlay the dashboard 500 when the user selects a navigation item, as instructed by the personalized onboard assistant system 170 in Figure 5b. Similar to the assistance provided for selecting a navigation item 530, the personalized onboard assistant system 170 may re-highlight any newly appearing navigation items 535 from the expanded submenu 550. The controller input diagram 510 may be updated with changes to the input required for selecting a navigation item, and the tutorial text 520 may be updated to reflect instructions for selecting the newly appearing navigation item 535 from the submenu 550. For example, the tutorial text 520 and the controller input diagram 510 may be updated in real time as the user navigates the dashboard 500 and submenu 550. For instance, the tutorial text 520 may initially appear in Figure 5c and instruct the user with guidance such as, "Press the D-pad up, or tilt the left analog stick up to go to the [Show All Friends] option." When a user enters a command into the controller, the tutorial text may update automatically and instantly to reflect the cursor position and the accuracy of the next command to be issued. When the correct option for selection is highlighted, the tutorial text 520 may update, instructing the user to select "Show All Friends" by pressing X, highlighting the desired navigation item, and further guiding the user to the highlighted button for selection in the controller input diagram 510.

[0046] In Figure 5d, the selection of a newly appearing navigation item 535 may update the display and provide a new display related to the selected submenu item, such as the "Friends" dashboard 560. In one embodiment, the personalized onboard assistant system 170 may provide assistance content regarding the search for the user's peers to add as "Friends." In this case as well, the personalized onboard assistant system 170 may provide a callout highlight 540 by drawing a line from the navigation item 530 to an image of the same navigation item with increased size and resolution. Furthermore, the personalized onboard assistant system 170 may adjust the position and size of the controller input figure 510 to avoid interfering with the navigation item. The personalized onboard assistant system 170 may adjust the position of the controller input figure 510 to be closer to the required navigation item, further drawing the user's attention to the correct position on the screen. The tutorial text 520 may be updated by the personalized onboard assistant system 170 to provide appropriate instructions for highlighting and selecting the "Search" navigation item.

[0047] In Figure 5e, the "Friends" dashboard 560 may be updated to display a "Search" function 570 associated with locating the location of user peers to add as "Friends". The personalized onboard assistant system 170 may highlight the "Search" function 570 in a callout 540. Furthermore, the personalized onboard assistant system 170 may update the tutorial text 520 with instructions guiding the user to enter text such as a peer profile name, network ID, or other unique identifier associated with the user profile. Various embodiments for entering text may be provided to the user. In one embodiment, the user may select the "Search" function 570 and enter text using an on-screen keyboard, a physical keyboard attached to the controller user device 130, or a standalone physical keyboard connected to the entertainment system as a second user device 130, separate from the controller input device. In a different embodiment, the user may enter text by speaking to a microphone sensor located on the controller or entertainment system. Upon receiving text input, the personalized onboard assistant system 170 may update the tutorial text 520 to provide assistance guiding the user to confirm the initiation of the search process based on the received text.

[0048] In Figure 5f, the peer user profile 580 is displayed based on the search results queried by the personalized onboard assistant system 170 using the search text entered by the user in Figure 5e. The peer user profile 580 may display peer-related information, such as a list of the peer's activities and media content, the peer's peers, and activities and media content common to both the user and the peer. The peer user profile information may be reviewed by the user to identify that the peer to be added as a “friend” has been identified. The personalized onboard assistant system 170 may highlight the “Add Friend” function 590 with a callout 540. Furthermore, the personalized onboard assistant system 170 may update the controller input figure 510 and tutorial text 520 with instructions guiding the user to confirm the request to add the peer as a “friend”. Once the user has entered to add the peer as a “friend”, the personalized onboard assistant system 170 may detect that the activity is complete and terminate the assistance. Furthermore, the personalized onboard assistant system 170 can continue to provide users with different assistance even after the initial assistance for adding a peer is complete.

[0049] Figures 6a and 6b show exemplary screenshots that may be provided during the use of a system for providing personalized onboard assistance in connection with the purchase of media content from an online store. The personalized onboard assistant system 170 may provide the user with guided assistance related to accessing and searching the online store in a process similar to the steps included in Figures 5a and 5f. In Figure 6a, the user may select new media content from the online store based on search results queried by the personalized onboard assistant system 170 and be displayed a media content purchase page 600. The personalized onboard assistant system 170 may track user input on the media content purchase page 600 and provide recommendations prior to the final purchase made by the user. In the displayed embodiment, the user may initially select a first version of media content item 610 with the intention of purchasing a version such as the "standard edition" of a video game. The personalized onboard assistant system 170 may provide assistance to the user by highlighting a second version of media content item 620 and displaying recommendation text 630 suggesting that the user purchase the second version of media content item 620. The recommendation text 630 may include various information related to the version of a media content item based on data available to the personalized onboard assistant system 170 via the interactive content server 110, platform server 120, database 140, or any cloud server connected to the entertainment system network. In the displayed embodiment, the personalized onboard assistant system 170 may receive data indicating that the second version 620 is a more popular choice among network users than the first version 610.Recommended text 630 may incorporate data points to explain version recommendations based on popularity, such as "70% of players who purchased the standard version upgraded to the ultimate version."

[0050] In Figure 6b, the personalized onboard assistant system 170 may provide further recommendation assistance by displaying a shortcut input guide to the user for selecting a recommended version of a media content item. The shortcut inputs for purchasing the recommended media content provided by the personalized onboard assistant system 170 may skip steps in the traditional purchase process that the user may have to perform when selecting a media content item, reducing the time spent by the user and decreasing the number of inputs required to move to the selection range. When a recommendation (e.g., the recommendation shown in Figure 6a) is displayed to the user on the media content purchase page 600, the personalized onboard assistant system 170 may update the display to include input assistance for completing the purchase of a second version 620 of the recommendation. The personalized onboard assistant system 170 may provide a controller input diagram 510 that displays the button layout and inputs required to select the second version 620. Furthermore, the personalized onboard assistant system 170 may provide updated recommendation text 630 that explains the inputs required to select the recommended version.

[0051] Figures 7a and 7b show exemplary screenshots that may be provided during the use of a system for providing personalized onboard assistance in relation to recommending new media content based on media content in the user's library. The personalized onboard assistant system 170 may provide feedback and recommendations to the user based on user profile information, including a history of engagement with media content and the user's activities in their library. In Figure 7a, the user may browse a library of media content with which they have previously interacted, such as a list of games, movies, TV shows, podcasts, and publisher content, or any combination thereof. In the displayed embodiment, the user may browse a library of video games 700 and user history associated with each game, such as a "trophy" list, which may include a measure of the user's progress toward completing game achievements. In one example, the personalized onboard assistant system 170 may determine that recommendations should be provided to the user based on a history of engagement and progress in games. A feedback notification 710 may be displayed by the personalized onboard assistant system 170, informing the user of their history of engagement and progress in a particular game 720 and praising the user for achievements achieved in the game. Providing the user with feedback on their engagement with media content in their user library may be displayed by the personalized onboard assistant system 170 as a preliminary step intended to provide the user with context based on recommendations.

[0052] In Figure 7b, the personalized onboard assistant system 170 may provide recommendations for the user to engage with new media content items based on the user's engagement with similar media content in the user library. Media content in the user library may be similar to new media content according to the personalized onboard assistant system 170, based on a comparison of various factors such as media metadata, global purchase history, the user's peers' purchase history, and the aforementioned user and user peer engagement history. When the personalized onboard assistant system 170 detects a new media content item that is similar to the user's media content library, it may provide assistance in guiding the user to engage with the new media content item. In the displayed embodiment, a particular game 720 in the library of video games 700 may be similar to new media content items according to the personalized onboard assistant system 170. When the user detects a new media content item that is available, the personalized onboard assistant system 170 may provide recommendation text 730, including prompts or input shortcuts, to view more information about the new media content. Furthermore, the personalized onboard assistant system 170 may provide controller input diagrams 510 in conjunction with recommendation text 730 that guides the user to button inputs where the user may be able to see more information about recommended media content.

[0053] Figure 8 shows an exemplary screenshot that may be provided during the use of a system to provide personalized onboard assistance in relation to recommending new media content based on the purchase history of the user's peers. The personalized onboard assistant system 170 may provide recommendations to the user based on the media content purchase history of the user's peers while the user is engaged with the entertainment system, such as searching an online store, browsing media content in an online store, or viewing a specific media content item in an online store. Such recommendation assistance provided to the user by the personalized onboard assistant system 170 may include displaying message text that incorporates data available in the user profiles of the user's peers on the network. In the displayed embodiment, the user may select a game title while browsing or searching an online store and view the game's purchase page 800. The personalized onboard assistant system 170 may detect peers of users who have similarly purchased the same media content and display a peer recommendation message 810 that incorporates data from the user's peers' purchase profiles, such as a message like "80% of your friends pre-ordered this." Furthermore, the personalized onboard assistant system 170 can work in conjunction with peer recommendation messages 810 to provide controller input diagram 510, guiding the user to input shortcuts for purchasing recommended media content.

[0054] The detailed description in Figures 3 to 8 above represents exemplary embodiments of a personalized onboard assistant system 170 that provides assistance to the user for various activities that the user may interact with on the entertainment system and that may be initiated under various circumstances. The personalized onboard assistant system 170 may provide similar assistance in various additional user interactions with the entertainment system, and therefore Figures 3 to 8 should not be considered exhaustive or restrictive of the functionality of the personalized onboard assistant system 170. The disclosed embodiments have been selected to best represent the technology, practical embodiments, and applications of the system described.

[0055] Figure 9 is a block diagram of an exemplary electronic entertainment system 900. The entertainment system 900 in Figure 9 includes a main memory 905, a central processing unit (CPU) 910, a vector unit 915, a graphics processing unit 920, an input / output (I / O) processor 925, an I / O processor memory 930, a controller interface 935, a memory card 940, a Universal Serial Bus (USB) interface 945, and an IEEE interface 950. The entertainment system 900 further includes an operating system read-only memory (OS ROM) 955, an audio processing unit 960, an optical disc control unit 970, and a hard disk drive 965, which are connected to the I / O processor 925 via a bus 975.

[0056] The entertainment system 900 may be an electronic game console. Alternatively, the entertainment system 900 may be implemented as a general-purpose computer, set-top box, handheld game device, tablet computing device, or mobile computing device or cell phone. The entertainment system may include more or fewer operating components depending on the specific form factor, purpose, or design.

[0057] The CPU 910, vector unit 915, graphics processing unit 920, and I / O processor 925 in Figure 9 communicate via the system bus 985. Furthermore, the CPU 910 in Figure 6 communicates with the main memory 905 via the dedicated bus 980, while the vector unit 915 and graphics processing unit 920 may communicate via the dedicated bus 990. The CPU 910 in Figure 9 executes programs stored in the OS ROM 955 and the main memory 905. The main memory 905 in Figure 9 may contain pre-stored programs and programs transferred via the I / O processor 925 from a CD-ROM, DVD-ROM, or other optical disc (not shown) using the optical disc control unit 970. The I / O processor 925 in Figure 9 may also enable the introduction of content transferred wirelessly or via other communication networks (e.g., 4G, LTE, 3G, etc.). The I / O processor 925 in Figure 9 primarily controls data exchange between various devices of the entertainment system 900, including the CPU 910, vector unit 915, graphics processing unit 920, and controller interface 935.

[0058] The graphics processing unit 920 in Figure 9 executes graphics instructions received from the CPU 910 and the vector unit 915 to generate an image for display on a display device (not shown). For example, the vector unit 915 in Figure 9 may convert an object from 3D coordinates to 2D coordinates and send the 2D coordinates to the graphics processing unit 920. Furthermore, the audio processing unit 960 executes instructions to generate an audio signal to be output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 900 via a USB interface 945 and an IEEE interface 950 such as a wireless transceiver, and these interfaces may also be embedded in the system 900 or as part of several other components such as a processor.

[0059] The user of the entertainment system 900 in Figure 9 provides commands to the CPU 910 via the controller interface 935. For example, the user may instruct the CPU 910 to store specific game information on the memory card 940 or other non-temporary computer-readable storage medium, or to instruct a character in the game to perform a specific action.

[0060] This system can be implemented in applications that may be operable on various end-user devices. For example, end-user devices may be personal computers, home entertainment systems (e.g., Sony PlayStation 2® or Sony PlayStation 3®, Sony PlayStation 4®, or Sony PlayStation 5®), portable game devices (e.g., Sony PSP® or Sony Vita®), or lower-end but different manufacturers' home entertainment systems. The methods of the present invention described herein are fully intended to be operable on a variety of devices. This system may also be implemented in a cross-title neutral manner, and embodiments of the system of the present invention may be used across various titles from various publishers.

[0061] The system of the present invention can be implemented in applications that may be operable using various devices. Non-temporary computer-readable storage media refer to any one or more media involved in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, but are not limited to optical or magnetic disks and non-volatile and volatile media such as dynamic memory. Common forms of non-temporary computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, and any other memory chips or cartridges.

[0062] Various forms of transmission media can be involved in conveying one or more sequences of one or more instructions to the CPU for execution. A bus transmits data to system RAM, and the CPU retrieves and executes instructions from system RAM. Instructions received by system RAM can, optionally, be stored on a fixed disk either before or after execution by the CPU. Various forms of storage can be implemented, as can the network interfaces and network topologies required to implement the storage.

[0063] The above detailed description of the Art is presented for illustrative and explanatory purposes only. The Art is not intended to be exhaustive or to limit the Art to the detailed forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments described have been selected to best illustrate the principles of the Art, its practical applications, and to enable those skilled in the art to utilize the Art in various embodiments and in various modifications suitable for specific uses conceivable. The scope of the Art is intended to be defined by the claims.

Claims

1. 1. A method for personalizing assistance in a virtual environment, comprising: storing a user profile in a memory, the user profile including data regarding interactions within one or more virtual environments associated with one or more media titles streamed by the entertainment device; Detecting an assistance trigger associated with the entertainment device configured to communicate over a communications network, the assistance trigger being detected during one or more user interactions with a current virtual environment associated with an identified one of the media titles streamed by the entertainment device; identifying an activity associated with the assistance trigger based on the user's interaction with the current virtual environment associated with the identified media title when the assistance trigger is detected; customizing assistance content associated with the identified activity based on the user profile, the customized assistance content being associated with a virtual character; providing the customized assistance content to the entertainment device over the communications network for presentation by the virtual character within the current virtual environment associated with the identified media title streamed by the entertainment device; A method comprising:

2. The method of claim 1 , further comprising updating the user profile based on the customized assistance content.

3. The method of claim 2 , wherein customization of the assistance content in response to a subsequent assistance trigger is based on the updated user profile.

4. The method of claim 1 , wherein detecting the assistance trigger is based on data from the user profile indicative of prior engagement with either the entertainment device or the identified media title.

5. The method of claim 1 , wherein customizing the assistance content comprises filtering a set of content associated with the identified activity based on the user profile.

6. 2. The method of claim 1 , wherein customizing the assistance content includes configuring a set of content associated with the identified activity based on the user profile, and the customized assistance content is configured for presentation within the virtual environment of the identified media title.

7. The method of claim 1 , wherein the customized assistance content comprises a visual simulation of one or more controller button presses associated with successful completion of the activity.

8. The method of claim 1 , further comprising suspending presentation of the customized assistance content and providing different assistance content in response to a user request.

9. The method of claim 1 , further comprising terminating the presentation of the customized assistance content in response to a user request, and storing the user request in the user profile.

10. detecting another assistance trigger associated with the identified activity; resuming presentation of the customized assistance content based on the detected other assistance trigger; and The method of claim 8 further comprising:

11. The method of claim 1 , wherein the user profile comprises a record of one or more inputs received from a user via the entertainment device.

12. The method of claim 1 , wherein the assistance trigger comprises one or more indicators of user frustration.

13. The method of claim 1 , wherein the assistance trigger comprises accessing an online store for a media title from the entertainment device.

14. The method of claim 1 , wherein the assistance trigger comprises searching for one or more media titles in an online store.

15. The method of claim 1 , wherein the assistance trigger includes searching, the searching including searching for new media titles not present in the user profile.

16. 16. The method of claim 15, wherein customizing the assistance content is further based on a version of the new media title based on a global purchasing history and a purchasing history of one or more of the user's peers.

17. 10. The method of claim 1, further comprising identifying a skill level based on the interaction with the identified media title, and wherein customizing the assistance content is further based on the identified skill level.

18. 1. A system for personalizing assistance in a virtual environment, comprising: a memory for storing a user profile, the user profile including data regarding interactions within one or more virtual environments associated with one or more media titles; A processor that executes instructions stored in a memory, the processor executing the instructions to: Detecting an assistance trigger associated with the user of an entertainment device configured to communicate over a communications network, the assistance trigger being detected during one or more user interactions with a current virtual environment associated with an identified one of the media titles streamed by the entertainment device; identifying an activity associated with the assistance trigger based on the user's interaction with the current virtual environment associated with the identified media title when the assistance trigger is detected; customizing assistance content associated with the identified activity based on the user profile, the customized assistance content being associated with a virtual character; and a communications network communicating over the communications network to provide the customized assistance content to the entertainment device for presentation by the virtual character within the current virtual environment associated with the identified media title streamed by the entertainment device; Including, the system.

19. 1. A non-transitory computer readable storage medium embodying a program executable by a processor to perform a method for personalizing assistance in a virtual environment, the method comprising: storing a user profile in a memory, the user profile including data regarding interactions within one or more virtual environments associated with one or more media titles streamed by the entertainment device; Detecting an assistance trigger associated with the entertainment device configured to communicate over a communications network, the assistance trigger being detected during one or more user interactions with a current virtual environment associated with an identified one of the media titles streamed by the entertainment device; identifying an activity associated with the assistance trigger based on the user's interaction with the current virtual environment associated with the identified media title when the assistance trigger is detected; customizing assistance content associated with the identified activity based on the user profile, the customized assistance content being associated with a virtual character; providing the customized assistance content to the entertainment device over the communications network for presentation by the virtual character within the current virtual environment associated with the identified media title streamed by the entertainment device; A non-transitory computer readable storage medium comprising:

Images