Integrated headphone controller storage and connection

Abstract

A controller for providing input to an interactive application includes a first surface having a pair of slots for storing earplugs; and a plurality of control buttons and a touch input surface, the plurality of control buttons and the touch input surface being set on one or more surfaces including the first surface. A processor is configured to pair the controller with a computing device executing the interactive application, and is coupled to each of the slots to pair the earplugs received therein with the controller, and share a pairing between the controller and the computing device with the earplugs received in the pair of slots.

Description

Technical Field

[0001] The present disclosure relates to a controller for providing input to an interactive application, and more particularly, to a controller that provides a storage, charging, and connection case for in-ear headphones. Background Art

[0002] In recent years, video games have become more and more popular. As video games become more and more popular, various input / output devices have been developed to help users interact with video games (i.e., interactive applications). For example, game controllers are developed to assist users in providing inputs to video games and receiving feedback responses. Head-mounted displays with display screens have been developed and provided to allow users to have immersive, private, and non-intrusive gaming experiences. Earphones (in-ear headphones and external headphones) have been developed and provided to allow users to listen to the audio of video games and interact with other users inside and outside the video games, etc. For example, in-ear headphones (i.e., earplugs) can be wired or wireless. Wireless earplugs provide a small and portable appearance size to listen to in-game audio and external audio during playing video games. The earplugs are stored in a separate box and charged via the electrical coupling provided in the box. In order to allow the use of various input / output devices to provide input and receive output, various devices must be paired with a gaming device.

[0003] As more and more input / output devices are available to users, users need to bear the burden of managing the pairing, storage and charging of these devices. Managing various input / output devices is a time-consuming and complicated process. For example, in the current setting, the user needs to manage two separate pairing processes (i.e., pairing both the controller and the earbuds) to successfully pair the earbuds and the controller to the gaming device. The gaming device is used to execute a video game using user input provided by the input / output device and provide game content to be rendered on the display screen of each user's client device. In addition to performing two separate pairing processes, the user must also track two separate battery life states to ensure that the earbuds, earbud boxes, and controllers are all charged and ready for the game. In addition, the location of the earbuds and controllers must always be tracked. As portability and playing games anytime, anywhere become more and more mainstream and important, the audio experience becomes equally important.

[0004] It is against this background that embodiments of the present disclosure are presented. Summary of the invention

[0005] Implementations of the present disclosure relate to devices, systems, and methods for storing in-ear headphones (i.e., earbuds), charging them, and pairing them with controllers for providing input to video games. The controller is paired with a gaming device, such as a game console or a gaming server that executes an interactive application (such as a video game). The controller is designed to act as both an input device and a headset storage, charging, and connection box. Using the controller as a headset box and earbud charger eliminates the need to manage two separate devices. This simplifies the overall user experience because the user only needs to use one less device to manage communication with the gaming device. Using the controller for audio storage (i.e., earbud storage) and integrating the earbuds into the controller itself provides a proprietary low-latency audio connection to the gaming device, a charging space within the controller, and provides the ability to switch to the controller's built-in battery or power source when the earbud's internal power source (e.g., battery) runs out of power.

[0006] In addition, using the controller as the earbud box, communication between the two devices (earbuds and controller) is greatly simplified, providing greater opportunities for the earbuds and controller to work together. Since two different audio sources are available at the controller, the user has more flexibility to direct specific audio through each microphone source. In addition to simplifying the pairing, storage, and charging of the earbuds, the microphone / speaker of the controller can also be used with the earbuds to triangulate the audio source so that the audio can be processed to provide better audio quality. For example, triangulation of audio sources can be used to accurately identify and eliminate extraneous noise, and to selectively filter out certain ambient sounds and enhance certain other ambient sounds.

[0007] In one implementation, a controller for providing input to an interactive application is disclosed. The controller includes a first surface having a pair of slots set thereon. Each slot in the pair of slots is configured to store an earplug. Each slot is coupled to a power source of the controller to charge the earplug when the earplug is received in each slot in the pair of slots. The controller also includes a plurality of control buttons and one or more touch input surfaces set on one or more surfaces (including the first surface). Each of the plurality of control buttons and each of the one or more touch input surfaces is configured to provide input for performing an action in an interactive application. A processor is configured to pair the controller with a computing device executing the interactive application. The processor is also coupled to each of the slots so as to pair the earplug received therein with the controller and share the pairing between the controller and the computing device with the earplug received in the pair of slots.

[0008] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present disclosure may be best understood by reference to the following description taken in conjunction with the accompanying drawings.

[0010] Figure 1 A simplified block diagram of a gaming system having a controller with earbud storage for storing and charging earbuds received therein and sharing pairing of the controller with a computing device for communicating with the computing device is shown in accordance with one implementation of the present disclosure.

[0011] Figure 2A A simplified block diagram of an earbud setup module for charging the earbuds, pairing the earbuds with a controller, and communicating with a computing device, such as a gaming device, is shown according to one implementation of the present disclosure.

[0012] Figure 2B In one implementation, Figure 2A A simplified block diagram of the various modules contained in the content stream configuration module.

[0013] Figure 3A A view of the back side of a controller according to one implementation of the present disclosure is shown, on which a box for receiving and storing earplugs is provided.

[0014] Figure 3B The front side of a controller according to an implementation of the present disclosure is shown, on which a plurality of buttons and an input interface are set.

[0015] Figure 3C An alternative configuration of earbud storage provided on the back of a controller according to one implementation of the present disclosure is shown. Figure 3A The configuration shown in is different.

[0016] Figure 3D Another alternative configuration of earbud storage provided on the back of the controller according to one implementation of the present disclosure is shown. Figure 3A and Figure 3C The configuration shown in is different.

[0017] Figure 3E shows different configurations of earbud storage provided on the back of a controller according to one implementation of the present disclosure, the configuration being similar to Figure 3A , Figure 3C and Figure 3D The configuration shown in is different. DETAILED DESCRIPTION

[0018] In the following description, a number of specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the present disclosure may be practiced without some or all of these specific details. In other cases, well-known process steps are not described in detail in order to avoid obscuring the present disclosure.

[0019] As portability and playing games anytime, anywhere become more and more mainstream and important, the audio experience of using a microphone paired with a computing device (e.g., a game console or a game server) is also becoming more and more important. A computing device for executing an interactive application (such as a video game) is associated with one or more input devices to allow a user to provide input to the video game. Some input devices that can be used to provide input (e.g., control input) include a keyboard, a mouse, a game controller (referred to as "controller"), a tablet computing device or an analog controller with a touch screen, a mobile computing device, etc. In addition to the above-mentioned input devices for providing control input, a microphone of a controller, a microphone associated with a display screen / computing device, or a microphone associated with a pair of earplugs can also be used to provide voice input (e.g., commands). The game state of the video game is updated using input, and game-related content corresponding to the game state is generated and provided for rendering. The game-related content of the video game includes game content and social interaction content, such as chat content. Game content also includes video content, tactile content, audio content, etc. The video content is forwarded to a display screen associated with a computing device or a head-mounted display, or forwarded to a television monitor or display surface for rendering. The audio content is forwarded to a controller for rendering via a speaker of the controller, or to a pair of earbuds communicatively paired with the computing device for rendering via speakers of the earbuds.

[0020] In various implementations described herein, a pair of earbuds are stored integrally in an earbud case disposed on the surface of a controller. This allows the earbuds to: (a) share a power source for charging the controller; (b) communicate with the computing device using a communication connection established from pairing the controller with the computing device; (c) provide a dedicated low-latency audio connection between the earbuds and the computing device; (d) provide greater flexibility to configure the earbuds to selectively receive different audio content; (e) simplify pairing the earbuds with the computing device; (f) simplify gaming-centric headset storage by reducing the number of devices to track; and (g) simplify communication between the microphone / speaker of the earbuds and the controller.

[0021] With the present disclosure generally understood, specific details of the disclosure will now be described with reference to the various figures.

[0022] Figure 1A simplified block diagram of a system for executing an interactive application, such as a video game, and providing content of the interactive application to one or more client devices for rendering is shown in one implementation. Figure 1 The illustrated system shows a gaming system in which the controller 110 is used to provide user input to an interactive application (e.g., a video game) executed on the computing device 100. To understand the advantages of the present disclosure described herein, Figure 1 The system shown is intentionally kept simple and basic. As shown, the system includes a computing device 100 (represented as a game console), a display screen 115, a controller 110, and a pair of earbuds 120. Figure 1 In an implementation of FIG. 1 , computing device 100 is shown as being co-located with user 10 providing input via controller 110. Computing device 100 need not be co-located but may be remotely located. Computing device 100 is communicatively connected to controller 110 by performing a pairing process in order to receive user input provided via controller 110 and return video game content for rendering on display screen 115. Throughout this application, a controller may alternatively be referred to as a "client device."

[0023] The computing device 100 is not limited to a standalone game console, but can be a connected device (such as a thin client), or a local server device communicatively connected to a remote server (such as a cloud server 400 of a game cloud), or can be a remotely located cloud server as part of a game cloud. In some implementations where the computing device 100 is connected to the cloud server 400, a first portion of the video game can be executed on the computing device 100, and a second portion of the video game can be executed on the cloud server 400. In these implementations, some inputs from the controller 110 to the video game and some game content generated for the video game are processed at the computing device 100, and the remainder of the inputs and game content are processed at the cloud server 400. In this case, the game state of the video game is synchronized between the computing device 100 and the cloud server 400. In an implementation where the computing device 100 is a thin client communicatively connected to the cloud server, the computing device 100 only acts as an intermediary to transmit game inputs from the controller 110 to the cloud server 400, and forward game content generated at the cloud server 400 to the display screen 115 associated with the computing device 100 for rendering. In these implementations, all processing occurs on the cloud server 400 .

[0024] The game content of the video game returned by the computing device 100 is rendered on the display screen 115. The display screen can be a television monitor as shown, or a screen associated with the computing device (not shown), or a screen of a head mounted display (not shown), or an external display surface (not shown) on which the content can be rendered.

[0025] Controller 110 is shown as including a pair of handles that can be operated with both hands of a user. The types of controllers are not limited to the above types, but may include a single handle that can be operated with one hand of a user or a controller without any handles, such as a tablet computing device or a game pad. Controller 110 includes a plurality of control buttons and / or one or more touch screen interfaces (e.g., touch input interfaces) that set one or more surfaces to provide user input to a video game (i.e., an interactive application). Figure 1 A view of the back side 110a and the front side 110b of the controller 110 is shown. The back side 110a of the controller 110 shows a first surface (i.e., the back surface) on which a pair of slots 130 are set for accommodating a pair of earplugs 120. The front side 110b shows a second surface (i.e., the front surface) of the controller 110, on which a plurality of control buttons and a touch screen interface are disposed. The surface of the controller 110 on which the control buttons and one or more touch screen interfaces are set is not limited to the front surface, but may include a top surface, a bottom surface, a side surface, and a back surface. User input is used to perform operations in an interactive application. The tablet computing device represents the controller and includes a touch screen interface on which the control buttons of a physical controller are simulated. The controller 110 is communicatively paired with the computing device 100 so that user input provided through the control buttons and / or the touch screen interface can be detected and interpreted by the computing device 100 to affect the game state of the video game. In some implementations, pairing can be high-speed pairing to avoid latency, so that user input provided to the computing device 100 through the control buttons and / or touch screen interface and game content from the computing device 100 to the display screen 115 are transmitted in near real time. The power supply is used to charge the controller 110. The power supply can be an internal power supply (such as a battery 112) or an external power supply 111. The controller 110 also includes a microphone / speaker (not shown) to receive voice commands from the user and render any audio generated during the play of the video game. The voice commands from the user can be used as user input for the video game, or as a chat or social media input on the chat interface to be posted on a social media application interface rendered with the game content during the play of the video game. Similarly, the audio generated during the game play can include game audio and chat / social media audio provided by the user via the chat / social media application interface.

[0026] In addition to the microphone / speaker of the controller 110, a pair of earbuds 120 may be included to provide audio commands and / or listen to audio generated during the playing of a video game. The earbuds 120 are stored in an earbud box set on one surface of the controller 110. In some implementations, the earbud box is defined as a pair of slots on the back of the controller 110, and the earbuds 120 are received in the corresponding slots 130 to fully integrate the earbuds 120 with the controller 110.

[0027] The controller in which the earbud storage is integrated is different from the traditional controller. In the traditional controller, there is no storage space for the earbuds set on any surface. Instead, the earbuds are stored separately in the earbud box, resulting in the user having to keep track of the location of multiple devices such as the earbuds, the earbud box, and the controller. In addition, since the earbuds and the controller are separate devices, the earbuds and the controller are separately connected to different or the same power source for charging. Therefore, the user must manage the battery life status of each of these devices separately to ensure that both devices are fully and timely charged and ready to play games when needed. In addition, the earbuds and the controller are each separately paired with the computing device to communicate with the computing device, and the earbuds are also paired with the controller to allow communication and synchronization between the controller and the earbuds. Therefore, in addition to tracking the battery status of the separate devices and each of these devices, the user must also manage different pairing processes. The user must also ensure that the communication between the devices is properly managed.

[0028] Therefore, in order to avoid repeated connection, storage and pairing, and to simplify the tracking of devices and the communication between devices, the earplug 120 is integrated into the controller 110. A pair of slots 130 are set on the first surface of the controller 110, and a plurality of control buttons and / or one or more touch input surfaces are set on the second surface. In one implementation, the first surface is the back 110a of the controller 110, and the second surface is the front 110b of the controller 110. Additional control buttons can be set on other surfaces of the controller 110, and the other surfaces include the top surface, the bottom surface, or one or two side surfaces of the controller 110. The integration of the earplug 120 allows the microphone / speaker of the earplug 120 and the controller 110 to work together, and simplifies the pairing process, device tracking, and management of battery life status. The integration also allows more flexibility to direct different audio content generated during the execution of the interactive application to different devices, thereby enhancing the overall user experience.

[0029] The pair of slots 130 are not limited to being set on the back side 110a of the controller 110, but can also be set on other surfaces of the controller, including the front side 110b, any one or both side surfaces, the top surface, or the bottom surface. In some implementations, each of the pair of slots 130 is configured to include a spring-fitting mechanism to receive and secure each earplug 120 into the corresponding slot 130. The spring-fitting mechanism is configured to allow the earplug 120 to be pressed into the appropriate position within the slot 130 so as to be securely received in the slot 130. For example, the spring-fitting mechanism is designed to generate a mechanical click in response to the earplug 120 being pressed into the alignment slot for the first time to indicate that the earplug 120 is correctly received in the slot 130 (i.e., snapped into place). Subsequent pressing of the earplug 120 will cause the earplug 120 to be released from the slot 130 (i.e., popped out of the slot). Other ways can also be used to indicate that the earplug 120 is correctly received in the slot 130. In alternative implementations, the slots 130 may be configured to be formed using a conformal material or include one or more magnets or employ any other material or mechanism to securely receive the earbuds 120 into the corresponding slots 130. Each slot 130 is sized and shaped to accommodate an earbud 120 such that the earbud 120, when received in the slot 130, is flush with the surface of the controller 110 (e.g., the back surface 110a) on which the slot 130 is defined.

[0030] To facilitate storage and charging of the earbuds 120 (when received in the slots 130), in some implementations, each slot 130 includes one or more slot contacts ( Figure 3A The slot contacts 132 shown in the figure are set at the bottom and are coupled to the power source of the controller 110 (e.g., the internal power source 112 (i.e., a battery)). The slot contacts 132 of the controller 110 are then connected to one or more metal conductor elements provided on the earplug 120, thereby establishing an electrical metal connection. Each earplug 120 is defined by an earplug portion and a narrow tube portion extending downwardly from the bottom surface of the earplug portion for a certain length. In some implementations, the metal conductor element is set on the bottom surface of the narrow tube portion of each earplug 120. The electrical connection allows power from the power source (i.e., the external power source 111 or the internal power source 112) to flow to the earplug 120, thereby charging the earplug 120. Each slot 130 is also coupled to the processor 113 of the controller 110 to allow the controller 110 to control the communication between the earplug 120 and the controller 110 and between the earplug 120 and the computing device 100. A pair of earbuds 120 acts as an external audio source, and a microphone / speaker of the controller 110 acts as an internal audio source.

[0031] The controller 110 is first paired with the computing device 100 during the initial setup of the controller 110, and such pairing may be accomplished using a high-speed connection. The controller 110 may be communicatively connected to the computing device 100 using a wired or wireless connection. After the controller 110 is successfully paired with the computing device 100, the earbuds 120 are paired with the controller 110. The pairing of the earbuds 120 with the controller 110 allows the earbuds 120 to share the pairing of the controller 110 with the computing device 100 to allow communication between the earbuds 120 and the computing device 100. The communication of the earbuds 120 with the computing device 100 will use the high-speed communication connection established by the controller 110. In some implementations, the earbuds 120 are connected to the controller 110 via a wireless connection. In some embodiments, the wireless connection may be a proprietary wireless connection or any other widely available wireless connection.

[0032] The controller 110 with integrated earbud storage (i.e., slot 130) eliminates the need to have a separate earbud box to store and charge the earbuds 120. By reducing the number of devices to be tracked, gaming-centric headset storage is simplified because only the controller 110 (with the earbuds integrated therein) needs to be tracked. The integration of the earbuds 120 with the controller 110 also simplifies and reduces the number of pairing processes required to use the earbuds 120 to communicate with the controller 110 and the computing device 100. In addition to reducing the pairing process, the integration also improves and simplifies communication between different devices. For example, pairing is initially performed between the controller 110 and the computing device 100 to establish a communication connection between them. After the controller 110 is successfully paired with the computing device 100, the earbuds 120 are then paired with the controller 110. The pairing of the earbuds 120 with the controller 110 allows the earbuds 120 to communicate with the computing device 100 using the communication connection already established between the controller 110 and the computing device 100. Thus, the earbuds 120 access the high-speed communication connection between the controller 110 and the computing device 100 to provide and receive audio content. The earbuds can be configured to act as external audio sources and work in conjunction with the controller's internal audio sources (microphone / speakers). The audio content generated during the execution of the interactive application can be processed, using triangulation techniques to identify and isolate specific audio, or to identify and filter out or enhance certain ambient sounds. The processed audio is forwarded to a different device for rendering, allowing the user to obtain an enhanced user experience.

[0033] In addition, the power from the controller 110 is shared with the earbuds 120, eliminating the need to manage two different battery life states. Only the battery life and / or power state of the controller 110 needs to be managed. In addition to solving the duplication of pairing, storage, and charging of the earbuds 120, the integration also provides greater flexibility by allowing the microphones / speakers of the earbuds 120 and the controller 110 to be configured to work together to stream different audio to different devices. This flexibility allows the user to control which audio content is streamed to the earbuds for private rendering, and which audio content is streamed to the controller or computing device / display for public rendering. The earbuds 120 can be set, and the processor 113 of the controller 110 can be configured to stream chat audio to different earbuds 120 for private rendering, and stream game audio to the controller 110 and / or computing device 100 or display 115 for public rendering. The settings of the earbuds 120 and the configuration of the processor 113 of the controller 110 may be completed for or by the user during initial setup and dynamically updated during or after initial use or each use. In some embodiments, the earbud settings and processor configuration may be user-specific and / or interactive application-specific. Figure 2A and Figure 2B Further details of setting up the earbuds 120 and managing the different audio streams are discussed in detail.

[0034] Figure 2A Various modules of an earbud settings module 140 for setting up and managing earbuds 120 integrated with a controller 110 in some implementations are shown. The earbud settings module 140 may be software (i.e., program instructions) stored in a memory 114 of the controller 110 and executed by a processor 113 of the controller 110, or may be firmware. Figure 2B Additional submodules within the content stream configuration module 148 of the earbud setup module 140 in some implementations are shown, which may be used to manage the streaming of different audio content to different devices. Figure 2A and Figure 2B, the processor 113 of the controller 110 receives sensor data from one or more sensors 147 provided on the controller 110. In some implementations, some sensors 147 may be embedded in each slot 130 to detect the presence of an earbud in the corresponding slot. Additional sensors may be provided along the opening surface of the slot 130, other surfaces of the controller 110, and the earbud 120 itself. Thus, the sensor data may be used to detect the position of the earbud 120, the proximity of the earbud 120 to one or more facial features of the user, pairing data associated with the controller 110 and the earbud 120, and the like. The sensor data is received and interpreted by the earbud detection module 142 of the earbud setup module 140 to determine at least: (a) identification information of the controller 110 and the earbud 120, (b) whether one or both earbuds 120 are received in the corresponding slot 130, and (c) whether one or both earbuds 120 are worn by the user. The sensor data is provided as input to the pairing module 144 of the earbud setup module 140. In addition to location information, in some implementations, the sensor data may also be used to determine the battery life status of the controller 110 and the earbuds 120 .

[0035] The pairing module 144 receives the sensor data provided by the earbud detection module 142 and uses the identification information to identify the controller 110 and the earbud 120. In addition to identifying the controller 110 and the earbud 120, the sensor data is also used to determine whether the controller 110 is paired with the computing device 100 and whether the earbud 120 is paired with the controller 110. The pairing module 144 engages the controller and computer device pairing module 144a to query a database (not shown) associated with the controller 110 to determine whether the controller 110 has been paired with the computing device 100. If the controller 110 has not been paired with the computing device 100, the controller and computer device pairing module 144a initiates the pairing process between the controller 110 and the computing device 100. In some implementations, the controller 110 is paired with the computing device 100 by establishing a high-speed communication connection, and the pairing information is uploaded to the database. The communication connection can be wired or wireless. On the other hand, if the controller 110 has been paired with the computing device 100 , the controller-to-computer device pairing module 144 a returns the pairing information to the pairing module 144 .

[0036] Once the pairing between the controller 110 and the computing device 100 is established / verified, the pairing module 144 engages the earbud to controller pairing module 144b to verify whether the earbud 120 received in the slot 130 is associated with the controller 110 and whether the earbud 120 has been paired with the controller 110. The earbud to controller pairing module 144b queries the database of the controller 110 to obtain any earbud information associated with the controller 110. If the query returns any earbud information, the earbud to controller pairing module 144b verifies to see if the earbud information returned for the query matches the earbud information of the earbud 120 received in the slot 130. If no record is returned for the query, the earbud to controller pairing module 144b initiates a pairing process between the controller 110 and the earbud 120 received in the slot 130. As part of the earbud pairing process, the pairing between the controller 110 and the computing device 100 is shared with the earbud 120. Therefore, only by pairing the earbud 120 with the controller 110, the earbud 120 is able to utilize and use the pairing between the computing device 100 and the controller 110 to communicate with the computing device 100. When the earbud 120 is successfully paired with the controller 110, the earbud pairing information including the earbud identification information is updated to the database of the controller 110 for storage and subsequent verification. However, if a pairing record detailing the pairing of the earbud 120 with the controller 110 is returned in response to the query, the earbud information of the earbud 120 received in the slot 130 of the controller 110 is verified against the earbud information received in the pairing record of the controller 110 in response to the query. If there is no match between the earbud information retrieved from the database and the earbud information associated with the earbud 120 in the slot 130, the earbud and controller pairing module 144b returns an error message to indicate that the earbud 120 received in the slot 130 is not associated with the controller 110. An error message is returned to be rendered on the display screen 115, and the pairing process between the earbud 120 and the controller 110 ends. However, if there is a match between the earbud information retrieved from the database and the earbud information of the earbud 120 in the slot 130, it indicates that the earbud 120 has undergone a pairing process to pair with the controller 110. In this case, the earbud and controller pairing module 144b returns a successful matching signal to the pairing module 144. Upon successfully verifying the pairing between the earbud 120 received in the slot 130 and the controller 110, the pairing module 144 forwards the pairing information of the earbud 120 to the charging module 146 of the earbud setting module 140.

[0037] For example, the charging module 146 is used to verify that the controller 110 and the earbuds 120 are fully charged and ready for use in interacting with a video game. The charging module 146 interfaces with a controller charging module 146a that is configured to check the battery status of the controller 110 (i.e., check the charging status to determine whether the controller 110 is fully charged, partially charged, or uncharged). Based on the charging status, the controller charging module 146a can initiate a charging process to allow the controller 110 to charge or return an appropriate signal / message indicating the charging status of the controller 110.

[0038] In response to the charging module 146 receiving an appropriate signal (fully charged or sufficiently (e.g., about 75-90%) charged) from the controller charging module 146a, the charging module 146 engages the earbud charging module 146b to check the battery status of the earbud 120 received in the slot 130. Based on the charging status (i.e., battery status) of the earbud 120 received in the slot 130, the earbud charging module 146b is engaged to initiate the earbud charging process to charge the earbud 120 using the power source (111 or 112) of the controller 110, or return an appropriate signal / message indicating that the earbud 120 is sufficiently charged. It should be noted that in response to detecting the earbud 120 in the slot 130 of the controller 110, the pairing and charging process for the earbud is initiated. The detection of the earbud 120 in the slot 130 of the controller 110 may be verified using sensor data provided by one or more sensors 147 disposed in the slot 130, on the surface of the opening of the slot 130, on the surface of the controller 110, and on the earbud 120. For example, when the earbud 120 is placed in the slot 130 equipped with a spring-fit mechanism and pushed into place, a mechanical click may occur to indicate that the earbud 120 has been properly received in the slot 130. The occurrence of the mechanical click and the sensor data provided by the sensor 147 may be used to confirm the presence of the earbud 120 in the slot 130. The mechanical click may be a trigger event that may be used to initiate verification of pairing and charging of the controller 110 and the earbud 120.

[0039] After the controller 110 and the earbuds 120 are successfully paired and charged, the earbud setup module 140 engages the content stream configuration module 148 to determine how to stream the audio content generated during the video game play. In some implementations, the earbud setup module 140 engages the content stream configuration module 148 as part of the earbud setup or in response to a trigger event. In response to the trigger event, the trigger event detector 148a is engaged by the content stream configuration module 148 to evaluate the trigger event to determine whether the trigger event is related to the earbud 120. The trigger event may occur, for example, during the video game play, and may be triggered in response to detecting that the user is wearing one or two earbuds 120. Sensor data collected from sensors 147 disposed in the slot 130, on the surface of the earbud, on the surface of the opening of the slot 130, and on the surface of the controller 110 is used to verify the position of the earbud 120. The sensor data also provides detailed information on whether the user is wearing only one earbud 120 or two earbuds 120 and whether one earbud 120 or two earbuds 120 are located in the slot 130. The content stream configuration module 148 is configured to stream appropriate content generated during playing a video game (i.e., an interactive application) to different devices for rendering using sensor data and trigger event data. The content stream configuration module 148 first determines the different audio content generated during playing the video game. As described above, the audio content can include game audio, chat audio (i.e., social media audio), and voice input. Game audio is audio generated by game logic and provided for rendering during playing the game. Chat audio is provided by other users on a chat interface rendered with game content. Voice input is generated by a user and can include input commands to affect the game state of a video game, or can be chat audio for rendering on a chat interface.

[0040] The content stream configuration module 148 can identify, isolate and extract selected audio in the audio generated during the game, and stream the selected audio content of the audio content to the appropriate device. The audio content generated during the game includes game audio, chat (i.e., social media) audio, voice input, etc. Each of the computing device 100, the controller 110, the display screen 115 and the earplug 120 can include a microphone and a speaker embedded therein or associated therewith to render and receive audio content during the video game. In some implementations, the configuration of various devices can be provided by the device manufacturer and can be dynamically updated by the user during the use of the device. In an alternative implementation, the configuration of various devices can be controlled by the user and dynamically updated during use. The user input provided under the configuration of different devices or in order to update the configuration of different devices is received, interpreted and used to update the configuration of the corresponding device. In some implementations, the configuration of various devices can be device-specific, user-specific and / or interactive application-specific.

[0041] In some implementations, during initial setup of the various devices (i.e., during pairing), the devices may be configured to receive a game audio stream at the controller 110 or computing device or display screen 115 for rendering through associated speakers (i.e., game audio rendered for public consumption). The earbuds 120 may be configured to receive a chat audio stream for rendering via the speakers of the earbuds 120 when it is detected that the user is wearing the earbuds 120 during video game play (i.e., chat audio rendered for private consumption). When the user is not wearing the earbuds 120, then the chat audio may be configured to be streamed to the controller 110, computing device 100, or display screen 115 for rendering through the corresponding speakers. In this case, the device to which the chat audio is streamed may be different from the device to which the game audio is streamed.

[0042] In some implementations, the earbuds may be configured differently for single earbud use and dual earbud use. In some implementations, the earbuds 120 are configured such that when the user wears only one earbud 120, the chat audio is streamed to the earbud 120, and when the user wears two earbuds, the chat audio is streamed to both earbuds 120. In the case where the user wears two earbuds and the chat audio is streamed to both earbuds 120, when the user shares one of the earbuds (left earbud 120-L or right earbud 120-R) with a second user, the audio content streamed to the earbud 120 is updated so that the earbud (120-L or 120-R) shared with the second user is configured to receive game audio, and the earbud (120-R or 120-L) worn by the user is configured to receive chat audio. This configuration allows the user to continue to enjoy the chat audio privately while dynamically switching the chat audio to the game audio forwarded to the earbud shared with the second user. Alternatively, the audio content streamed to the earbud shared with the second user may continue to stream the chat audio, so that both the user and the second user can listen to the chat audio. In an alternative implementation, when the user wears two earbuds 120, both earbuds 120 can be configured to render game audio. In this case, when the user shares one of the earbuds (120-L or 120-R) with the second user, both earbuds (120-L, 120-R) can be configured to receive and render game audio, so that both the user and the second user can enjoy the game audio.

[0043] The content stream configuration module 148 can obtain the configuration of the computing device 100, controller 110, display screen 115, and earbuds 120 from the initial setup of each of these devices and direct the audio content accordingly. Once the configuration of the various devices has been completed, the content stream configuration module 148 identifies, extracts, and streams the appropriate audio content generated during the play of the video game to the different devices. For example, the earbud-related input module 151 is used to identify, extract, and forward earbud-related inputs to the appropriate earbuds or controller 110 / computing device 100. Similarly, the game-related input module 156 is used to identify, extract, and forward game-related inputs to the controller 110 or computing device, and the chat / social media-related input module 157 is used to extract and forward chat / social media-related inputs to the display screen 115 for rendering.

[0044] Figure 2B Various submodules of content stream configuration module 148 are shown, which are used to identify, extract appropriate audio content and forward it to different devices, video games to affect game states, and forward it to social media interfaces for rendering. A trigger event detector 148a is engaged to detect and analyze any trigger events generated during the play of a video game. The trigger event can be generated in response to input received from a user or in response to input from a video game that provides game content or in response to chat content received from a chat interface. The user can provide input to select and adjust the configuration of one or both earplugs, or provide voice input, where the voice input can be a voice command that affects the outcome of the video game, or a chat or social media content posted on a chat or social media interface presented with the video game. Based on the analysis, the content from the video game or the input from the user is processed by different submodules of content stream configuration module 148.

[0045] The input is processed by the input detection module 150 to identify the source and context of the input and classify the input so that appropriate actions can be taken. The input can be one or more finger taps at the opening of the slot (130-L or 130-R), and the input can be processed by the input detection module 150 to interpret the input as a selection action of the earplug (120-L or 120-R) received in the slot. In addition, the finger taps at the slot can be interpreted to determine whether there is or is not an earplug in the slot, and trigger a pairing action or a charging action. Alternatively, the finger taps at the slot can be interpreted as selecting the earplug associated with the slot to forward specific audio content. Similarly, the input can be a button press on the controller 110, and the input can be interpreted to perform a specific operation on an interactive application based on the action configured for each button selected. The input can be roughly classified into headphone-related input, game-related input, and chat / social media-related input. Earplug-related input can include input received for rendering, for forwarding, or for configuring earplugs. For example, user input may be provided using one or more control buttons and / or a touch screen interface of the controller 110, or using a microphone of the earbud, or at the surface of the opening of the slot 130. The earbud configuration input detector 152 is engaged to process configuration-related inputs. For example, the selection of one or both earbuds 120 (e.g., the left earbud or the right earbud (120-L, 120-R)) is detected by the earbud selection detector 152a, and the input for adjusting the streaming, volume, sensitivity, or other configuration settings of the selected earbud is processed by the earbud configuration adjustment module 152b. A visual indication of the selection of a particular earbud may be provided to the user by lighting up the appropriate slot associated with the selected earbud at the controller 110 (e.g., the right slot or the left slot (130-L, 130-R)) or rendering an image of the selected earbud 120 on the display 115, or providing tactile feedback on the relevant side of the controller 110, etc. For example, earbud-related input may be provided at any time before, during, or after playing a video game and configuration settings may be adjusted for the earbuds 120 for subsequent use of the earbuds 120 .

[0046] The earbud content selection module 153 is coupled to identify appropriate audio content to be streamed to each earbud 120 for rendering based on the configuration of the earbuds and based on whether the user is wearing one earbud or both earbuds 120. Figure 2BAs shown, the earbuds 120 can receive game audio provided by the game content module 156c or chat audio provided by the chat content module 157c for rendering through the speakers set in each earbud 120. In addition, the earbuds can include microphones to receive voice input / commands from the user wearing the earbuds 120 and provide them as input to the video game or chat / social media interface for publication. Therefore, based on the configuration settings of the earbuds 120 and when it is detected that the user is wearing one or two earbuds on their ears, the game audio or chat audio will be rendered to one earbud (120-L or 120-R) or both earbuds 120. When it is detected that the user is wearing only one earbud 120, the single earbud content stream module 154 is activated. The detection of whether the user is wearing only one earbud can be determined based on sensor data, which provides data related to the presence or absence of the earbud 120 in the corresponding slot 130 and the proximity of the earbud to one or more facial features of the user. Based on the configuration settings of each earbud 120, game content or chat content is selectively extracted and streamed to the appropriate earbud by the single earbud content stream module 154. If the user is wearing only the left earbud 120-L, and the configuration settings indicate that the left earbud 120-L only receives and renders chat audio, the left earbud content module 154a is engaged to activate the speaker of the left earbud 120-L, and the single earbud content stream module 154 extracts and forwards the chat audio for rendering via the left speaker. Similarly, if it is detected that the user is wearing only the right earbud 120-R, and the configuration settings indicate that the right earbud 120-R only receives game audio, the right earbud content module 154b is engaged to activate the speaker of the right earbud 120-R, and the single earbud content stream module 154 extracts and forwards the game audio for rendering via the right speaker.

[0047] When it is detected that the user is wearing two earbuds 120, the dual earbud content stream module 155 is activated. As with the single earbud content stream, the game audio or chat audio is selectively extracted and streamed to the appropriate earbud 120 (e.g., 120-L for game audio and 120-R for chat audio) based on the configuration settings of each earbud 120. Thus, based on the configuration settings, the dual earbud content stream module 155 receives the chat audio and renders the chat audio only to the left earbud 120-L and the game audio only to the right earbud 120-R, or renders the chat audio to both earbuds (120-L, 120-R), or renders the game audio to both earbuds (120-L, 120-R).

[0048] Similarly, the game-related input module 156 is used to identify the source and type of input for the game (i.e., interactive application) or generated by the game. The game-related input can be provided via the control buttons or touch screen interface of the controller 110, or can be a voice command provided by the microphone of the controller 110, or a voice input provided by the user through the microphone of the controller 110 or the earplug 120. Alternatively, the game input can be game content generated by the video game in response to the input provided by the user (for single-user games) or multiple users including the user (for multi-user games). The game-related input can include text input, voice input, control button and / or touch screen input provided by the user, and content generated by the video game, such as video content, audio content, tactile content, etc. The voice input module 156a is used to detect and process the voice input and forward the processed voice input as a game input to the game logic to affect the game state of the video game. Similarly, input provided through the controller 110 (i.e., through control buttons or touch screen (input) interface) is processed by the controller input to the game module 156b and forwarded to the game logic as game input to affect the game state of the video game. The game logic uses the input to update the game state of the video game and generate game content. The generated game content includes at least an audio portion and a video portion. The audio portion is extracted and forwarded to the earbud content selection module for rendering at the corresponding earbud 120 or forwarded to the controller content streaming module 158. The audio portion can be rendered via the speakers of the controller 110, or can be forwarded to the computing device for rendering through speakers associated with the computing device 100 or the display screen 115. The video portion of the game content is forwarded to the display screen 115 for rendering.

[0049] The chat / social media related input module 157 is used to detect the generated content for rendering or providing to the user at the chat or social media interface. The chat related input may include voice input provided by the user through the microphone of the earbud 120 or the controller 110, or text or other controller input provided by the user using the control buttons and / or touch screen interface of the controller 110. The voice input module 157a of the chat / social media related input module 157 is used to detect and process the voice input and forward the processed voice input to the social media publishing module 159 to be published as chat audio on the chat / social media interface. The chat content provided by other users can be similarly processed by the chat / social media related input module 157 to extract the chat audio included in the chat content and forward the chat audio to one or two earbuds 120 for rendering based on the configuration of the earbud 120. The controller input to the social media / chat interface module 157b is similarly used to process the chat related controller input and forward it to the social media publishing module 159 to be published on the chat / social media interface.

[0050] As mentioned earlier, Figure 2A and Figure 2B The earplug setting module 140 shown in Figure 2B The various modules of the content stream configuration module 148, the controller 110 with the integrated earbud 120 is a standalone multi-function device, which can simplify the pairing, charging, communication process and reduce the number of devices used to track location and battery status. In addition, the earbud 120 can communicate with the computing device using a high-speed communication connection established between the controller and the computing device, thereby providing low-latency communication between the earbud and the computing device.

[0051] FIG. 3A to FIG. 3E Various orientations and positions of slots for earbuds as set on a controller in some implementations are shown. Figure 3A The back of a controller is shown with slots for earbuds defined thereon. As shown, a left slot for receiving a left earbud and a right slot for receiving a right earbud are defined on the back of the controller near the top surface. Metal contacts (i.e., slot contacts) are defined in the portion of each slot that receives the bottom of the narrow tube portion of each earbud. The bottom of the narrow tube portion of each earbud includes a metal conductor element that contacts the slot contacts defined in each slot when the earbud is received in the corresponding slot. The slot contacts of each slot are coupled to a power source (such as a battery or an external power source) of the controller. The slots are also coupled to a processor of the controller to execute program instructions stored in a memory. The program instructions are configured to allow the processor to detect an earbud received in the slot, pair and charge it, and direct specific audio content generated during the execution of an interactive application at a computing device communicatively coupled to the controller. The slots of the earbuds are side-oriented. Thus, in Figure 3A In the first configuration shown, the slot for storing the earplug is set at slot position 1 (ie, close to the top surface) and has slot orientation 1 to receive the earplug in earplug orientation 1.

[0052] Figure 3B The front of the controller in some implementations is shown, with multiple control buttons and one or more touch screen interfaces disposed thereon. Some of the control buttons that may be disposed include trigger buttons, bumpers, directional pads with appropriate buttons for providing directional input, analog joysticks, guide buttons, a headphone jack 160 for wired connection, buttons for providing options, start, share, back buttons, and / or additional control buttons (162, 164-L, 164-R, 166a-d, 161-R, 161-L) for providing additional input. The controller also includes a touchpad (i.e., touch screen interface) 168 for providing input using finger presses and finger slides. This document states that Figure 3BThe control buttons and touch screen interface shown in may be typical button / touch input interfaces included in any type of controller for providing input to an interactive application.

[0053] Figure 3C A second configuration for setting the slots in some alternative implementations is shown, which is similar to Figure 3A In these implementations, the slot and the earplug received therein are set at slot position 1, with the slot orientation perpendicular to the top surface and earplug orientation 2.

[0054] Figure 3D A third configuration for setting the slots in some alternative implementations is shown, which is similar to Figure 3A and Figure 3C In these implementations, the slot and the earplug received therein are set at slot position 1 and have a slot orientation and earplug orientation 3 perpendicular to the top surface. Figure 3D The orientation of the slot 130 and the earplug 120 in Figure 3C The orientation shown in is opposite.

[0055] Figure 3E A fourth configuration for setting the slots in some implementations is shown, which is similar to Figure 3A , Figure 3C and Figure 3D The slot configurations shown are different. In these implementations, the slot is set at position 4 along the outer edge of the side of the controller so that the bottom of the slot is set to align with the outer edge of the lateral side wall. The slot 130 and the earplug 120 have a slot orientation and an earplug orientation 1. It should be noted that the position and orientation of the slot for receiving the earplug (i.e., the slot configuration) are provided as examples only and should not be considered limiting, and other orientations, surfaces, and positions are also contemplated.

[0056] It should be noted that the computing device can communicate with the controller and other computing devices located remotely or as part of the cloud gaming site through a network, which may include a local area network (LAN), a wide area network, a cellular network (e.g., 4G, 5G, etc.), or any other type of data network, including the Internet 300, and such communication may be performed through a wired or wireless connection. The embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, mobile devices, etc. The embodiments of the present disclosure may also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices linked based on wired or wireless networks. One or more embodiments may also be manufactured as computer-readable code on a computer-readable medium. The computer-readable medium may include a computer-readable tangible medium distributed on a network-coupled computer system so that the computer-readable code is stored and executed in a distributed manner.

[0057] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments should be considered as illustrative rather than restrictive, and the embodiments are not limited to the details given herein, but may be modified within the scope and equivalent range of the appended claims.

[0058] It should be understood that the various embodiments set forth herein may be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that may be set up by combining various elements to provide more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

Claims

1. A controller for providing input to an interactive application, comprising: a first surface having a pair of slots defined thereon, each slot of the pair of slots being configured to store an earbud, wherein each slot is coupled to a power source of the controller to charge the earbud when the earbud is received in each slot of the pair of slots; a plurality of control buttons and one or more touch input surfaces defined on one or more surfaces of the controller, wherein the one or more surfaces include the first surface, each of the plurality of control buttons and each of the one or more touch input surfaces being configured to provide input to perform an action in the interactive application; as well as A processor coupled to each of the slots and configured to execute program instructions to pair the earbud received therein with the controller and charge the earbud using the power source of the controller, the pairing of the earbud allowing the earbud to share a pairing between the controller and a computing device executing the interactive application to communicate with the computing device.

2. The controller of claim 1, wherein each slot includes one or more sensors to detect the presence of the earplug in the slot, wherein the one or more touch input surfaces include a touch input surface set on a surface of an opening of each slot, the touch input surface at the opening being equipped with one or more sensors to detect user input, and The processor is configured to detect user input at the touch input surface of the opening and interpret the user input to perform an action associated with the earbud, wherein the action is a pairing action, a charging action, or a forwarding action to direct audio content to the earbud for rendering.

3. The controller of claim 1 further comprising a memory to store program instructions executed by the processor of the controller.

4. The controller of claim 1, wherein the pair of slots are configured such that each slot in the pair is sized and shaped to allow the earbud to fit within the groove and flush with the first surface when received therein.

5. A controller as described in claim 1, wherein each slot includes one or more metal contacts, and the one or more metal contacts are coupled to the power supply of the controller, and each earbud of the pair includes a metal conductor element to connect with the one or more metal contacts in the corresponding slot, when the earbud is received in the slot, the connection allows the earbud to receive power from the power supply of the controller and be charged.

6. The controller of claim 1, wherein each slot is equipped with a spring-fit mechanism for aligning and securely retaining the earbud in the slot, and The spring engagement mechanism is configured such that when the earplug is aligned over the slot, pressing the earplug a first time causes the earplug to snap into place within the slot, and pressing the earplug a second time causes the earplug to pop out of the slot.

7. The controller of claim 1, wherein the pairing between the controller and the computing device is a high-speed communication connection, and each earbud of the pair employs the high-speed communication connection of the controller to communicate with the computing device, and Each earbud of the pair is coupled to the controller using a wireless connection.

8. The controller of claim 1, wherein the audio generated during execution of the interactive application comprises any one or a combination of interactive audio content and chat content received from the interactive application, and voice input received from a user wearing one or more earbuds of the pair, and The program instructions are configured to selectively extract specific audio from the audio and direct it to one or both earbuds of the pair, or to the controller or computing device for rendering.

9. A controller as described in claim 8, wherein the program instructions are configured to direct the interactive audio content generated during the execution of the interactive application to a first earbud and direct the chat content generated during the execution of the interactive application to a second earbud.

10. The controller of claim 8, wherein the program instructions are configured to direct the interactive audio content generated during execution of the interactive application to the controller and direct the chat content to one or both earbuds of the pair for rendering.

11. A controller as described in claim 8, wherein the program instructions are configured to process the audio content generated during the execution of the interactive application to recognize, extract voice input and transmit it to the interactive application to affect the state of the interactive application or to a chat interface to publish the voice input as a chat.

12. The controller of claim 1, wherein each earbud of the pair includes a speaker to render audio generated during execution of the interactive application.

13. The controller of claim 12, wherein each earbud of the pair includes a microphone to receive voice input from a user wearing the earbud and transmit the voice input to the controller, wherein the processor of the controller is configured to process the voice input and transmit it to the computing device for rendering as input audio.

14. The controller of claim 1, wherein the first surface of the controller is a back surface of the controller, and The one or more surfaces include a front side, one or more side sides, and the back side of the controller.

15. The controller of claim 1, wherein the first surface of the controller is a side surface of the controller, and The one or more surfaces include the front side, one or more side sides and a back side of the controller.

16. The controller of claim 1, wherein the controller includes one or more microphones and one or more speakers embedded therein, the one or more microphones and the one or more speakers coupled to the power source and the processor of the controller.

17. The controller of claim 1, wherein the controller is coupled to the computing device via a wired or wireless connection.

18. The controller of claim 1, wherein the controller includes one or more handles for holding the controller while providing input to the interactive application.

19. A system for executing an interactive application, comprising: a computing device for executing an instance of the interactive application, the interactive application generating streaming content for rendering, the streaming content comprising at least an audio portion and a video portion; a display screen associated with the computing device for rendering the video portion of the streaming content; A controller for providing input during execution of the interactive application, the controller comprising: a first surface having a pair of slots defined therein, each slot of the pair of slots being configured to store an earbud, wherein each slot is coupled to a power source of the controller to charge the earbud received therein, wherein the power source is used to charge the controller; a plurality of control buttons and one or more touch input surfaces defined on one or more surfaces of the controller, wherein the one or more surfaces include the first surface, each of the plurality of control buttons and each of the one or more touch input surfaces being configured to provide input to perform an action in the interactive application; a memory for storing program instructions for configuring the earplugs to be stored in the pair of slots; and A processor is configured to execute program instructions retrieved from a memory for configuring the earbud.

20. The system of claim 19, wherein the program instructions are configured to: detecting the presence of the earplug in a slot of the pair of slots; Initiating charging of the earbud using the power supply of the controller; verifying that the controller is paired with the computing device; initiating pairing of the earbud with the controller, the pairing of the earbud allowing the earbud to share the pairing of the controller with the computing device to communicate with the computing device after the earbud is successfully paired with the controller; detecting the presence of the earplug on the user; as well as Based on a configuration of the earbud, a selected portion of the audio portion of the streamed content is identified, extracted, and streamed to the earbud.

Images