System and method for coordinating gaming haptic responses across an ecosystem of peripheral devices

Abstract

A haptic peripheral device ecosystem coordination system may include: a display device that displays a three-dimensional (3D) gaming environment; a processor that identifies a gaming action event occurring in the 3D gaming environment based on a detected action event indicator and determines a location of the action event relative to a player avatar image within the 3D gaming environment; the processor transmits a haptic command to a controller of a triggered haptic peripheral device, the haptic command being selected from a plurality of haptic commands based on the identification of the gaming action event, the triggered haptic peripheral device being selected from a plurality of peripheral devices based on the location of the gaming action event, the controller of the triggered haptic peripheral causing haptic feedback at a location relative to a user at the triggered haptic peripheral device that is related to the location of the action event relative to the player avatar image.

Description

Technical Field

[0001] The present disclosure generally relates to an ecosystem of peripheral devices for information handling systems, such as keyboards, mice, display devices, and audio headsets. The present disclosure more specifically relates to coordinated generation of thermotactile feedback at one or more piezoelectric actuators within such an ecosystem of peripheral devices based on a game image displayed at a display device. Background Art

[0002] As the value and use of information continue to increase, individuals and businesses seek additional ways to process and store information. One option available to clients is an information handling system. An information handling system generally processes, compiles, stores and / or communicates information or data for commercial, personal or other purposes, thereby allowing clients to exploit the value of information. Since technology and information handling may vary between different clients or applications, information handling systems may also vary in terms of what information is handled, how the information is handled, how much information is processed, stored or communicated, and how quickly and efficiently the information can be processed, stored or communicated. Variations in information handling systems allow information handling systems to be general or configured for specific clients or specific uses (such as e-commerce, financial business processing, flight reservations, enterprise data storage or global communications). In addition, an information handling system may include a variety of hardware and software components that can be configured to process, store and communicate information, and may include one or more computer systems, data storage systems and networking systems. The information handling system may include telecommunications, network communications and video communications capabilities. In addition, the information handling system may include a keyboard or other input or output device, such as a cursor control device for a user to manually input information. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] It should be understood that for the sake of brevity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements. Embodiments incorporating the teachings of the present disclosure are shown and described with respect to the accompanying drawings herein, in which:

[0004] Figure 1 is a block diagram illustrating an information handling system according to one embodiment of the present disclosure;

[0005] Figure 2 is a side cross-sectional view of a tactile palm rest portion and a thermal tactile keyboard assembly according to one embodiment of the present disclosure;

[0006] Figure 3 is a perspective cutaway view of a tactile feedback mouse according to one embodiment of the present disclosure;

[0007] Figure 4is a perspective view of a haptic feedback audio headset according to one embodiment of the present disclosure;

[0008] Figure 5 is an image of a three-dimensional gaming environment in which game action events may occur according to one embodiment of the present disclosure;

[0009] Figure 6 is a block diagram of a haptic peripheral device ecosystem coordination system according to one embodiment of the present disclosure;

[0010] Figure 7 is an image of a haptic peripheral device ecosystem according to one embodiment of the present disclosure;

[0011] Figure 8 is a flow chart illustrating a method of identifying visual game action events according to one embodiment of the present disclosure;

[0012] Fig. 9 is a flow chart illustrating a method of initiating thermal haptic feedback at multiple peripheral devices according to one embodiment of the present disclosure; and

[0013] Fig.10 is a flow chart illustrating another method of presenting haptic feedback across multiple haptic zones of an input / output device according to one embodiment of the present disclosure.

[0014] The use of the same reference numbers in different drawings may indicate similar or identical items. DETAILED DESCRIPTION

[0015] The following description is provided in combination with the accompanying drawings to assist in understanding the teachings disclosed herein. The description focuses on specific embodiments and implementations of the teachings and is provided to help describe the teachings. This focus should not be interpreted as a limitation on the scope or applicability of the teachings.

[0016] The computer gaming industry is driven by interest in immersive gaming experiences. Such immersive gaming experiences operate to provide users with a multi-sensory perception of the action occurring during live gaming in which the users participate. Early developments in computer gaming involved low-level graphics and monophonic sound, such as experienced when playing all-in-one video games in an arcade. The video and computer gaming industry has evolved over the decades, incorporating ever-improving visual graphics, surround sound, high-definition sound, and in some cases, tactile motion, such as vibration. Early efforts to deliver such tactile motion included game console hand controllers that vibrated under certain conditions during live gaming. For example, when an avatar representing a player inputting commands via a controller suffers damage or injury during live gaming, or The controller provides the vibration pulses.

[0017] As computer-based games involving user input via keyboard, mouse or headset devices become more and more popular, there has been a need to incorporate such tactile motion or thermal tactile feedback into one or more parts of such keyboard, mouse or headset devices. Thermal tactile feedback can combine tactile motion (e.g., click, bump or vibration) with thermal changes in the temperature of the surface in contact with the user. The distributed surface area of ​​the peripheral device ecosystem including keyboard, mouse and headset devices makes it possible for thermal tactile feedback at one or more of these peripheral devices to represent events occurring in the game (e.g., the player avatar suffers damage). For example, a three-dimensional game environment displayed via a video display in accordance with the instructions of a game application in the embodiments described herein may include images of player avatars, images of non-player avatars (e.g., enemy avatars or ally avatars), and various objects (e.g., houses, cars, landscapes, weapons, etc.). Each of these avatars and objects can walk around in a three-dimensional game environment in which they are displayed based on the movement of the player avatar according to the user instructions input via any one of a plurality of peripheral input / output devices (such as keyboard, mouse or headset). Game action events may also occur within the three-dimensional game environment during game play, such as weapon firing, explosions, or other forms of attack between a game avatar and an object. A system is needed to provide and coordinate thermal haptic feedback (e.g., vibration, temperature change) associated with the type of game action event (e.g., the player avatar firing a weapon, the player avatar suffering damage) across multiple haptic zones of multiple peripheral input / output devices to represent such game action events.

[0018] The haptic peripheral device ecosystem coordination system in the embodiments of the present disclosure provides and coordinates such thermo-haptic feedback at the tactile zone related to the type of game action event within a three-dimensional game environment across multiple thermo-haptic input / output peripheral devices (e.g., keyboard, palm rest, mouse, headset). Piezoelectric actuators located below (or otherwise co-located with) an array of thermoelectric generators (TEGs) and below the housing of one or more of the peripheral input / output devices can provide thermo-haptic feedback in the embodiments described herein. The peripheral device controller in the embodiments described herein can transfer charge to a specific piezoelectric actuator through a circuit matrix (e.g., a printed circuit board assembly) based on the tactile zone (e.g., portion of a palm rest, portion of a mouse, portion of a headset) identified in the tactile command received from the haptic peripheral device ecosystem coordination system. After the controller of the identified peripheral device (e.g., a keyboard controller, mouse controller, or headphone controller) applies an electric charge to the piezoelectric actuator, the piezoelectric disk within the piezoelectric actuator can be mechanically stretched or compressed to produce a tactile motion event that can be sensed, such as the piezoelectric actuator warping up and down and returning to its pre-deformed state by way of a click, bump, or vibration.

[0019] Similarly, the peripheral device controller in the embodiments described herein can transfer charge to a specific thermoelectric generator (TEG) array based on the tactile area (e.g., portion of a palm rest, portion of a mouse, portion of a headset) identified in the tactile command received from the tactile peripheral device ecosystem coordination system. Such TEG arrays in the embodiments described herein can operate using the Peltier effect (also known as the thermoelectric effect), where applying a voltage to a p-doped semiconductor and an n-doped semiconductor pair of the TEG array causes a temperature change. In these embodiments, the top and bottom electrical insulators of the TEG array can be heated or frozen based on the amount of voltage or the polarity of the voltage applied at the TEG array.

[0020] The haptic peripheral device ecosystem coordination system in the embodiments of the present disclosure provides such thermal haptic feedback to specific portions of one or more of the plurality of peripheral devices within the peripheral device ecosystem based on the identified game action events occurring within the live game. The haptic peripheral device ecosystem coordination system in one embodiment may identify such live game game action events based on the identification of one or more of the visual action event indicators, the audio action event indicators, and the detected reception of the user input command. The visual action event indicators in one embodiment may be identified using a variety of different methods including, for example, trained image recognition models. In some embodiments, the haptic peripheral device ecosystem coordination system may capture a plurality of game images during a training cycle. The images captured by these training cycles may include images of various avatars or characters within the game environment. The avatars may be displayed in these captured training images at different depths and perspectives relative to the player's perspective, and the avatars may perform a variety of actions (e.g., firing or manipulating a weapon, redirecting weapon fire, moving toward or away from the player). The various avatars (which may also include avatars of the user / player himself) and the actions taken by the avatars within each of these captured training images may be labeled to form a training data set for the image recognition model.

[0021] These labeled captured training images can then be input into an image recognition model in order to train the model to accurately identify the avatars and the actions taken by those avatars within the captured training images. The image recognition model can be repeatedly trained on a variety of such training data sets until the model can accurately make such identifications within a preset tolerance (e.g., 70% of the time or more). In these embodiments, once the model has been trained, captured live game images (e.g., images that have not yet been labeled to identify an avatar or game action event) can be input into the trained image recognition model, and then visual action event indicators associated with game action events occurring within the live game can be identified when the game action events are displayed via a video display.

[0022] In another embodiment described herein, the haptic peripheral device ecosystem coordination system can identify such live game visual action event indicators by analyzing the changes in pixel brightness detected by the firmware of the video display device. In many game scenarios, damage to the player avatar or non-player game character avatar (e.g., enemy or ally) can be accompanied by bright appearances, such as weapon firing or explosions. In one embodiment, the firmware of the video display device can measure these brightness changes, their size, position and speed in real time during live game play. Such firmware real-time analysis methods exist in the art, for example, in order to record the lag between the display of the player inputting a user instruction (e.g., firing a weapon) and the player avatar taking an action according to such received user instructions. The haptic peripheral device ecosystem coordination system in one embodiment can access these real-time pixel brightness measurements when visual action event indicators associated with game action events (e.g., avatar firing a weapon, performing another type of attack, suffering damage) appear during live game play in some embodiments described herein to determine when the visual action event indicator has appeared.

[0023] In some embodiments described herein, the haptic peripheral device ecosystem coordination system may identify one or more audio action event indicators that are also associated with game action events. During live gameplay, the game software application may transmit an audio signal to a peripheral earphone speaker device, and this audio signal may be stored in a medium that can be accessed by the haptic peripheral device ecosystem coordination system. These audio signals in one embodiment may be analyzed to identify whether known audio action event indicators, such gunshots of different durations, explosions, or sounds associated with the player avatar engaging with a stationary object (e.g., wading) have occurred within the 3D game environment during live gameplay. Audio action event indicators identified in this manner in the embodiments described herein may also be associated with game action events.

[0024] The haptic peripheral device ecosystem coordination system may identify a game action event based on identification of a visual action event indicator, identification of an audio action event indicator, or a combination of the two in the embodiments described herein. In some embodiments, the haptic peripheral device ecosystem coordination system may also base such identification of a game action event on the receipt of a command or the lack of such a command input by a user via one or more of the thermal haptic input / output peripheral devices (e.g., a mouse or keyboard). For example, the haptic peripheral device ecosystem coordination system may distinguish between game action events initiated by a player avatar and game action events not initiated by a player avatar based on determining whether a user provided an input command (e.g., firing a weapon, moving the avatar in a particular direction) just before detecting a visual action event indicator or an audio action event indicator. By analyzing one or more of these visual action event indicators, audio action event indicators, and received user-input commands, in one embodiment, the haptic peripheral device ecosystem coordination system may identify that a game action event has occurred.

[0025] In an embodiment of the present disclosure, after determining that a particular game action event has occurred, the haptic peripheral device ecosystem coordination system may identify a haptic motion command, a haptic thermal command, or a combination of the two that, when executed by one of the thermal haptic input / output peripheral devices, can provide thermal haptic feedback specifically associated with the identified game action event. For example, an identified game action event in which a player avatar suffers damage may be associated with a haptic motion command that causes a single long motion pulse of high intensity (e.g., vibration) at the keyboard palm rest area, and with a haptic thermal command that causes a longer, high-intensity increase in heat also at the keyboard palm rest. The haptic peripheral device ecosystem coordination system in the embodiment may transmit such haptic motion command or haptic thermal command to a controller of the thermal haptic input / output peripheral device associated with such command, so as to initiate thermal haptic feedback of a portion of the peripheral device (e.g., palm rest, headset, mouse) associated with the type of game action event identified as occurring within the virtual three-dimensional game environment. In one embodiment, various tactile motion types (e.g., vibrations, clicks, bumps, or other mechanical motions) or thermal changes (heating or cooling) at different intensity levels, durations, or patterns may be associated with various types of game action events (e.g., the player avatar firing a weapon, the player avatar taking damage). The tactile motion types, thermal changes, or a combination of the two may be referred to herein as thermal tactile feedback.

[0026] In the embodiments described herein, the tactile commands transmitted to the peripheral device controller via the tactile peripheral device ecosystem coordination system can identify a portion of the surface of the peripheral device (e.g., palm rest, mouse, headset) in contact with the user where thermal tactile feedback will appear. The portion of the peripheral device in the embodiment where thermal tactile feedback appears may depend on the identified game action event and the avatar that takes such action or applies such action. For example, a game action event that affects the player's avatar (e.g., a nearby explosion or gunshot, suffering damage, moving into water) can be associated with thermal tactile feedback at the keyboard palm rest (e.g., vibrating motion pulses at different intensities, or heat increases or decreases of different lengths and intensities). As another example, a game action event initiated by a player (e.g., firing a weapon) can be associated with thermal tactile feedback at the mouse. As another example, a game action event that affects an enemy avatar (e.g., an enemy avatar suffering damage) can be associated with thermal tactile feedback at the headphone device.

[0027] In the embodiments described herein, the type of thermal tactile feedback initiated at the surface of the peripheral device in contact with the user can be varied so that each of the multiple available thermal tactile feedbacks is associated with a specific game action event, an avatar, or a combination of the above two. For example, thermal tactile feedback may include heat increase or heat reduction of different durations and intensities. In embodiments, certain game action events, such as the player avatar firing with a weapon, explosions or gunshots occurring near the player avatar, and the player avatar suffering damage, may be associated with heat increase, while other events, such as the player avatar moving into the water or the enemy avatar suffering damage, may be associated with heat reduction. In order to distinguish between each of these game action events, these heat increases or reductions may be applied at different durations or intensities. For example, the player avatar performing a single shot may be associated with a short heat increase of low intensity, and an explosion occurring near the player avatar may be associated with a long heat increase of high intensity. As another example, the player avatar moving into the water may be associated with a long heat reduction of high intensity, and the enemy avatar suffering damage may be associated with a short heat reduction of high intensity.

[0028] Thermal haptic feedback may also include haptic motions of different types, durations, and intensities, wherein each of the multiple available haptic motions is associated with at least one game action event. For example, certain game action events, such as the player avatar firing a weapon, an explosion or gunshot occurring near the player avatar, the player avatar suffering damage, may be associated with a single haptic motion or burst (e.g., pulse or cluster), while other events, such as the player avatar firing continuously or moving into water or the enemy avatar suffering damage, may be associated with multiple haptic motion bursts (e.g., pulse or cluster). In order to distinguish between each of these game action events, these haptic motions may be applied at different durations or intensities. For example, the player avatar firing continuously may be associated with multiple short motion pulses at low intensities, while an explosion occurring near the player avatar may be associated with a single long motion pulse at high intensities. In this way, embodiments of the present disclosure can identify the occurrence of a game action event, identify thermal tactile feedback associated with that game action event, and initiate the identified thermal tactile feedback at one of a plurality of peripheral input / output devices with which the user is in physical contact by executing one or both of a tactile motion command or a tactile thermal command. This can increase the user's immersion into the virtual 3D gaming environment of their avatar.

[0029] Turning now to the accompanying drawings, Figure 1 An information handling system 100 similar to an information handling system according to aspects of the present disclosure is shown. In the embodiments described herein, the information handling system may include any tool or collection of tools operable to compute, classify, process, transmit, receive, retrieve, initiate, switch, store, display, indicate, detect, record, reproduce, dispose of, or use any form of information, intelligence, or data for commercial, scientific, control, entertainment, or other purposes. For example, the information handling system 100 can be a personal computer, a mobile device (e.g., a personal digital assistant (PDA) or a smart phone), a server (e.g., a blade server or a rack server), a consumer electronic device, a network server or storage device, a network router, a switch or a bridge, a wireless router or other network communication device, a network connection device (a cellular phone, a tablet computer device, etc.), an IoT computing device, a wearable computing device, a set-top box (STB), a mobile information handling system, a handheld computer, a laptop computer, a desktop computer, a communication device, an access point (AP), a base station transceiver, a wireless telephone, a control system, a camera, a scanner, a printer, a pager, a personal trusted device, a network appliance, or any other suitable machine capable of executing a set of instructions (sequentially or otherwise) specifying the actions to be taken by the machine and of variable size, shape, performance, price, and functionality.

[0030] In one exemplary embodiment, the information handling system 100 may include a laptop or desktop gaming system executing a gaming application (e.g., as instructions 124). The gaming application may include any computer code executed by the processor 102 of the information handling system 100 to allow a user to engage with a gaming environment via an input / output (I / O) device (e.g., 112), a video / graphics display device 110, or any other input or output device. In various embodiments described herein, the thermotactile input / output peripheral device 112 may include various types of input / output devices. For example, the thermotactile input / output peripheral device in one embodiment may include a thermotactile headphone device, a thermotactile mouse, or a thermotactile keyboard and palm rest assembly.

[0031] As described herein, the thermotactile input / output peripheral 112 may also provide thermotactile feedback to the user in order to further immerse the user in the action presented to the user via the execution of the gaming application. This tactile feedback may originate from the selective activation of one or more thermoelectric generator (TEG) arrays 170-1, 170-2, 170-n, which heat or cool a portion of the housing of the thermotactile input / output peripheral 112. The TEG arrays 170-1, 170-2, 170-n may include two or more TEGs, which in the embodiments herein include a p-doped semiconductor and an n-doped semiconductor sandwiched between a top electrical insulator and a bottom electrical insulator, both of which are made of, for example, ceramic. In this specification and in the appended claims, the TEG arrays 170-1, 170-2, 170-n are defined as a group of TEGs that may or may not be coupled in series.

[0032] Additional tactile feedback may result from selective activation of one or more piezoelectric actuators (PEAs) 130-1, 130-2, 130-n that produce tactile motion (e.g., clicks, bumps, vibrations, or other mechanical motions) relative to a portion of the housing of the thermotactile input / output peripheral 112. These types of thermotactile feedback may be coordinated with actions performed in a gaming environment displayed on the video / graphics display device 110 during execution of a gaming application. Various exemplary gaming environments may result in how the TEG arrays 170-1, 170-2, 170-n and PEAs 130-1, 130-2, 130-n provide thermotactile feedback to a user via the thermotactile input / output peripheral 112. Details of activating the TEG arrays 170-1, 170-2, 170-n and PEAs 130-1, 130-2, 130-n based on the gaming environment are described in greater detail herein.

[0033] In a networked deployment, information handling system 100 may operate as a server or as a client computer in a server-client network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. In a particular embodiment, information handling system 100 may be implemented using an electronic device that provides voice, video, or data communications. For example, information handling system 100 may be any mobile or other computing device capable of executing a set of instructions (sequentially or otherwise) that specifies the actions to be taken by that machine. In addition, although a single information handling system 100 is shown, the term "system" should also be construed to include any collection of systems or subsystems that execute one or more sets of instructions, either individually or in combination, to perform one or more computer functions.

[0034] The information handling system 100 may include memory (volatile (e.g., random access memory, etc.), non-volatile (read-only memory, flash memory, etc.), or any combination thereof), one or more processing resources, such as a central processing unit (CPU), a graphics processing unit (GPU), hardware or software control logic, or any combination thereof. Additional components of the information handling system 100 may include one or more storage devices, one or more communication ports for communicating with external devices, and various input and output (I / O) devices 112, such as a keyboard and palm rest, a mouse, a headphone device incorporating one or more microphones and one or more speakers, a touch pad, an ambient light sensor, or any combination thereof. The information handling system 100 may also include: a power management unit 118, which supplies power to the information handling system 100 via a battery 117 or an alternating current (A / C) power adapter 119; a video / graphics display 110; and one or more buses operable to transmit communications between the various hardware components. Portions of the information handling system 100 may themselves be considered to be the information handling system 100.

[0035] The information handling system 100 may include a device or module that embodies one or more of the devices or executes instructions for one or more of the systems and modules described herein, and operates to perform one or more of the methods described herein. According to various embodiments herein, the information handling system 100 may execute code instructions 124 that may be operated on-board a server or system, a remote data center, or in a separate client information handling system. In some embodiments, it should be understood that any or all portions of the code instructions 124 may be operated on multiple information handling systems 100.

[0036] Information handling system 100 may include a processor 102, such as a central processing unit (CPU), control logic, or some combination thereof. Any of the processing resources may be operable to execute code as firmware or software code. Specifically, in one embodiment, processor 102 may be operable to execute code instructions for firmware of video / graphic display 110. Additionally, the information handling system 100 may include memory, such as main memory 104, static memory 106, or other memory (volatile (e.g., random access memory, etc.), non-volatile memory (read-only memory, flash memory, etc.), or any combination thereof, of a computer-readable medium 122 that stores instructions 124 for a gaming application or other application and a haptic peripheral device ecosystem coordination system 132 that drives the unit 116. The processor 102 may also provide a system clock to the information handling system for which a time-of-day clock may be tracked in conjunction with any location detector, such as a global positioning system, or in coordination with a network interface device 120 connected to one or more networks 128. The information handling system 100 may also include one or more buses 108 operable to transmit communications between various hardware components, such as any combination of various input and output (I / O) devices.

[0037] Information handling system 100 may also include video display 110. In one embodiment, video display 110 may function as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, or a solid state display. Additionally, information handling system 100 may include a thermotactile input / output peripheral 112, such as a cursor control device (e.g., a mouse, a touchpad, or gesture or touch screen input), a keyboard, a palm rest for a keyboard, or a headphone device incorporating a speaker or microphone. Various drivers (e.g., 114) and control electronics (e.g., controller 141) may be operatively coupled to operate an input device (e.g., thermotactile input / output peripheral 112) according to the embodiments described herein.

[0038] The network interface device 120 can provide connectivity to a network 128, such as a wide area network (WAN), a local area network (LAN), a wireless local area network (WLAN), a wireless personal area network (WPAN), a wireless wide area network (WWAN), or other network. Connectivity can be via a wired or wireless connection. The network interface device 120 can operate according to any wireless data communication standard. In order to communicate with a wireless local area network, a variety of standards can be used, including the IEEE 802.11 WLAN standard, the IEEE 802.15 WPAN standard, a WWAN such as 3GPP or 3GPP2, or similar wireless standards. In some aspects of the present disclosure, one network interface device 120 can operate two or more wireless links.

[0039] The network interface device 120 may connect to any combination of macrocellular wireless connections including 2G, 2.5G, 3G, 4G, 5G, etc. Utilization of radio frequency communication bands according to several exemplary embodiments of the present disclosure may include bands operable in licensed and unlicensed spectrum for use with WLAN standards and WWAN carriers.

[0040] In some embodiments, software, firmware, dedicated hardware implementations such as application specific integrated circuits, programmable logic arrays, and other hardware devices may be constructed to implement one or more of some of the systems and methods described herein. For example, the controller 141 may be operable to transfer an electrical charge to a piezoelectric actuator (PEA) (e.g., 130-1, 130-2, 130-n) located within the thermotactile input / output peripheral 112 in accordance with a tactile command received from the tactile peripheral ecosystem coordination system 132 to cause a tactile motion (e.g., vibration, click, bump, or other mechanical motion) beneath a specifically identified tactile zone of the thermotactile input / output peripheral (e.g., a mouse, keyboard palm rest, or headset). As another example, the controller 141 may be operable to transmit an electrical charge to the TEG arrays 170-1, 170-2, 170-n located within the thermal tactile input / output peripheral 112 in accordance with a tactile command received from the tactile peripheral ecosystem coordination system 132 to cause an increase or decrease in thermal temperature at a portion of a surface of the input / output peripheral 112.

[0041] As another example, the firmware for the video display 110 may be operable to measure and record the magnitude, size, and speed of pixel brightness changes during execution of a gaming application via the processor 102. In some embodiments, software code executable by the haptic peripheral device ecosystem coordination system 132 may use such brightness changes to determine gaming action events. Applications that may include the devices and systems of various embodiments may broadly include a variety of electronic systems and computer systems. One or more embodiments described herein may implement functionality using two or more specific interconnected hardware modules or devices and associated control and data signals that may be communicated between modules and communicated by the modules or as part of a dedicated integrated circuit. Therefore, the present system encompasses software, firmware, and hardware implementations.

[0042] According to various embodiments of the present disclosure, the methods described herein may be implemented by firmware or software programs that can be executed by a controller or processor system. In addition, in exemplary non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. Alternatively, a virtual computer system process may be constructed to implement one or more of the methods or functionalities as described herein.

[0043] The present disclosure contemplates a computer-readable medium that includes instructions, parameters, and profiles 124, or receives and executes instructions, parameters, and profiles 124 in response to propagated signals, so that devices connected to a network 128 can communicate voice, video, or data over the network 128. In addition, the instructions 124 can be transmitted or received over the network 128 via the network interface device 120.

[0044] The information handling system 100 may include a set of instructions 124 that can be executed to cause the computer system to perform any one or more of the methods or computer-based functions disclosed herein. As an example, the instructions 124 may execute a tactile peripheral device ecosystem coordination system 132, a peripheral device driver 114, a software agent, or other aspects or components. As another example, the peripheral device driver 114 may be operable to receive commands from or provide output to an input / output device driver 112, such as keystroke input, mouse input, light or sound output, etc. The operation of the peripheral device driver 114 and the tactile peripheral device ecosystem coordination system 132 may also be combined to provide constant temperature feedback to the input / output device 112 via a controller or controller 141. In various embodiments herein, the instructions 124 may execute any type of gaming application. Various software modules including the application instructions 124 may be coordinated by an operating system (OS) and / or via an application programming interface (API). An exemplary operating system may include and other OS types. Exemplary APIs may include Win32, Core Java APIs, or Android APIs. In some embodiments, instructions 124 may include another application, such as executable code for a variety of software game system applications. In some embodiments, software game system application 124 may transmit instructions to peripheral device driver 114 or controller 141 to indicate that an identified game action event associated with a portion of the thermal tactile input / output peripheral device 112 or the thermal tactile feedback of the tactile zone has occurred.

[0045] The disk drive unit 116 and the haptic peripheral device ecosystem coordination system 132 may include a computer-readable medium 122 in which one or more sets of instructions 124, such as software, may be embedded. Similarly, the main memory 104 and the static memory 106 may also include a computer-readable medium for storing one or more sets of instructions, parameters, or profiles 124. The disk drive unit 116 and the static memory 106 may also include space for data storage. In addition, the instructions 124 may embody one or more of the methods or logics described herein. For example, code instructions for instructions, various software game applications, software algorithms, processes, and / or methods related to the haptic peripheral device ecosystem coordination system 132 and the peripheral device driver 114 may be stored here. In a particular embodiment, the instructions, parameters, and profiles 124 may reside completely or at least partially in the main memory 104, the static memory 106, and / or the disk drive 116 during execution by the processor 102 of the information handling system 100. As illustrated, some or all of the haptic peripheral device ecosystem coordination system 132 and the peripheral device drivers 114 may be executed locally or remotely. The main memory 104 and the processor 102 may also include computer-readable media.

[0046] In an exemplary embodiment, the main memory 104 may include a computer-readable medium, such as RAM. Examples of the main memory 104 include: random access memory (RAM), such as static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NV-RAM), etc.; read-only memory (ROM); another type of memory; or a combination thereof. In some exemplary embodiments, the static memory 106 may include a computer-readable medium (not shown), such as NOR or NAND flash memory. In an exemplary embodiment, the haptic peripheral device ecosystem coordination system 132 and the peripheral device driver 114 may be stored in the static memory 106, or in a drive unit 116 on a computer-readable medium 122 such as flash memory or a disk. Although the computer-readable medium is shown as a single medium, the term "computer-readable medium" includes a single medium or multiple media (such as a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The term "computer-readable medium" should also include any medium capable of storing, encoding, or carrying a set of instructions for execution by a processor or causing a computer system to perform any one or more of the methods or operations disclosed herein.

[0047] In a specific non-limiting exemplary embodiment, the computer-readable medium may include a solid-state memory such as a memory card, or other packages that accommodate one or more non-volatile read-only memories. In addition, the computer-readable medium may be a random access memory or other volatile rewritable memory. In addition, the computer-readable medium may include a magneto-optical medium or an optical medium, such as a disk or tape or other storage device for storing information received via a carrier signal (such as a signal conveyed on a transmission medium). In addition, the computer-readable medium may store information received from a distributed network resource (such as from a cloud-based environment). The digital file attachment of an email or other independent information archive or archive set may be considered as a distribution medium equivalent to a tangible storage medium. Therefore, the present disclosure is considered to include any one or more of a computer-readable medium or a distribution medium and other equivalents and subsequent media in which data or instructions may be stored.

[0048] The information handling system 100 may also include a power management unit (PMU) 118, also known as a power supply unit (PSU). The PMU 118 may manage the power provided to components of the information handling system 100, such as the processor 102, a cooling system such as an array of fans, one or more drive units 116, a graphics processing unit (GPU), a video / graphics display device 110, and other components that may require power when a user has actuated a power button. In one embodiment, the PMU 118 may be electrically coupled to the bus 108 to provide this power. The PMU 118 may regulate power from a power source such as a battery 117 or an A / C power adapter 119. In one embodiment, the battery 117 may be charged via the A / C power adapter 119 and provide power to components of the information handling system 100 when the A / C power from the A / C power adapter 119 is removed.

[0049] The information handling system 100 may also include a haptic peripheral device ecosystem coordination system 132 that may be operably connected to the bus 108. The haptic peripheral device ecosystem coordination system 132 computer-readable medium 122 may also include space for data storage. According to the present specification, the haptic peripheral device ecosystem coordination system 132 may perform tasks related to the following operations: identifying a game action event within a live game of a gaming application; associating that game action event with a tactile zone of the thermal haptic input / output peripheral device 112; and causing thermal haptic feedback to occur at the identified tactile zone. In one embodiment, the haptic peripheral device ecosystem coordination system 132 may execute code instructions to perform the following operations based on the identified game action event and its location: identifying the game action event; and identifying the tactile zone of the thermal haptic input / output peripheral device where the tactile motion is appropriate.

[0050] In one embodiment, the haptic peripheral device ecosystem coordination system 132 may identify game action events displayed within a live game according to the various methods described herein. For example, the haptic peripheral device ecosystem coordination system 132 may detect game action events based on measured changes in pixel brightness in various regions of the video display 110 recorded by the firmware of the video display 110. As another example, the haptic peripheral device ecosystem coordination system 132 may train an image recognition model to recognize avatars, objects, and associated game action events based on a plurality of captured training game images (e.g., captured prior to a current or live game). In this embodiment, the training of the image recognition model may occur within a computing device incorporating the video display 110 and the thermal haptic input / output peripheral device 112, or may occur within a portion of the haptic peripheral device ecosystem coordination system 132 operating in a cloud environment and communicatively coupled to that computing device. Thus, in some embodiments, the haptic peripheral ecosystem coordination system 132 may partially operate after a computing device operated by a user executes code instructions of a gaming application displayed via the video display 110, and in some embodiments partially operate at a cloud-based server or server remote from the computing device operated by the user. After training the machine learning model, the haptic peripheral ecosystem coordination system 132 may detect gaming action events during gaming that warrant haptic feedback via machine learning inference model recognition of gaming action events or through encoded gaming instructions for haptic feedback during gaming. As described herein, in other embodiments, a software gaming system application (e.g., code 124) may have code instructions programmed into the gaming application to transmit instructions to the controller 141 indicating that an identified gaming action event associated with haptic movement of a portion or haptic zone of the thermal haptic input / output peripheral 112 has occurred to generate thermal haptic feedback.

[0051] According to the present specification, the haptic peripheral device ecosystem coordination system 132 may perform tasks related to providing as output an image recognition data set describing recognized game action event environment data experienced during execution of a game application. The image recognition data set may be used by the processor 102 executing the thermal haptic feedback model evaluation system to provide thermal haptic commands to the TEG arrays 170-1, 170-2, 170-n and one or more piezoelectric actuators (PEAs) 130-1, 130-2, 130-n to provide thermal haptic feedback commensurate with the game action environment experienced during the game. This may be done in real time as the user engages with the game environment through the inference model machine learning system for the game application. In this specification, the term "thermal tactile feedback" means any change in temperature, mechanical movement (e.g., click, bump, or vibration), or both, felt by a user due to activation of one or more of the TEG arrays 170-1, 170-2, 170-n and PEAs 130-1, 130-2, 130-n.

[0052] In one embodiment, the haptic peripheral device ecosystem coordination system 132 may have a convolutional neural network that is trained by receiving captured images obtained during the execution of the game application as training input. These captured images may be from any angle within the environment, and in some embodiments may include an avatar. In this specification and in the appended claims, the term "avatar" is intended to include any graphical representation of a user, a user's character, or a character placed in or interacting with a game environment. The captured images used to train the haptic peripheral device ecosystem coordination system 132 convolutional neural network or other machine learning model may also include representations of other environmental characteristics such as objects, actions, or other environmental characteristics. A non-exhaustive list of environmental characteristics may include weapon firing, explosions, avatars being attacked, interactions of avatars with objects within the game environment (cold snow, warm sand, shaking ground, etc.), noise, light, and other environmental characteristics. In one embodiment, each of these types of environmental characteristics may be captured in an image during a training cycle. The training cycle may be performed during the execution of the game application by the processor 102. This training cycle may be a dedicated duration, whether during or not during the user's actual game interaction. For example, during execution of a gaming application, the gaming application may direct the user to participate in a "demo" portion of the gaming environment, which allows the processor 102 to capture images and provide those images to the haptic peripheral device ecosystem coordination system 132. Alternatively, or in addition, the processor 102 may capture the images during execution of the gaming application and during the user's actual gaming. In this embodiment, the haptic peripheral device ecosystem coordination system 132 may be trained as the user participates in interactions with the gaming environment, thereby allowing the output from the haptic peripheral device ecosystem coordination system 132 to become more refined the longer the player engages with that gaming environment.

[0053] In one embodiment, the haptic peripheral device ecosystem coordination system 132 can be code instructions and operate with the main memory 104, the processor 102, the video display 110, the thermal haptic input / output peripheral device 112, and the NID 120 via the bus 108, and can use several forms of communication, including ACPI, SMBus, 24MHZ BFSK coded transmission channels or shared memory. Driver software, firmware, controllers, etc. can communicate with applications on the information handling system 100. During this process and after the haptic peripheral device ecosystem coordination system 132 has been trained, the processor 102 can receive output from the haptic peripheral device ecosystem coordination system 132, which defines an image recognition data set describing the recognized game action event environment data during the execution of the game application. The inference model training neural network or other machine learning model of the haptic peripheral device ecosystem coordination system 132 is operable to recognize game images in real time to identify game action events. After receiving this data set, the processor 102 can execute the model evaluation system. The model evaluation system may be computer code, or may be an ASIC that evaluates the accuracy of an image recognition data set to determine whether the accuracy reaches a threshold. For example, the model evaluation system may determine whether the environmental characteristics represent those of a gaming action event associated with thermotactile feedback at the thermotactile input / output peripheral 112. In the event that the model evaluation system has determined that the output from the haptic peripheral ecosystem coordination system 132 has reached a threshold accuracy, the processor 102 causes a signal to be sent to the TEG arrays 170-1, 170-2, 170-n or the PEAs 130-1, 130-2, 130-n to provide such thermotactile feedback described herein.

[0054] To send thermotactile signals from the processor 102, the information handling system 100 may implement a peripheral device driver 114. The peripheral device driver 114 may include any computer code that operates to control the thermotactile input / output peripheral device 112 by relaying signals from the processor 102 to the peripheral device controller 141. The thermotactile input / output peripheral device controller 141 may perform more functions such as cursor position, click selection, scrolling functions, and other functions understood for mouse pointing device operation according to the embodiments of this document. The thermotactile input / output peripheral device controller 141 may also control the relay of thermotactile signals from the processor 102 to each of the piezoelectric actuator (PEA) drivers 143-1, 143-2, 143-n and the thermoelectric generator (TEG) drivers 142-1, 142-2, 142-n. Similar to the peripheral device driver 114, the TEG drivers 142-1, 142-2, 142-n and the PEA drivers 143-1, 143-2, 143-n may include computer code that selectively activates the corresponding operatively coupled TEG arrays 170-1, 170-2, 170-n and PEAs 130-1, 130-2, 130-n when executed by the thermotactile input / output peripheral device controller 141. Each of the TEG arrays 170-1, 170-2, 170-n may be operatively coupled to the thermotactile input / output peripheral device controller 141 via, for example, a serial connector formed on a printed circuit board (PCB) 140. In one embodiment, the PCB 140 may be a dedicated PCB 140 in addition to the PCB for mounting the thermotactile input / output peripheral device controller 141, and it serves as a PCB for other functions on the thermotactile input / output peripheral device 112. For example, the PCB 140 in an embodiment where the thermotactile input / output peripheral 112 includes a mouse may be used for other functions, such as a mouse button actuator, a position tracking system (e.g., a trackball, an optical system, a gyroscope, etc.), a scrolling function, etc. As another example, the PCB 140 in an embodiment where the thermotactile input / output peripheral 112 includes a keyboard and palm rest assembly may be used for other functions, such as the registration of keystrokes at one or more keycaps operatively coupled to the PCB 140, or the initiation of tactile motion at one or more of such keycaps, etc. As another example, the PCB 140 in an embodiment where the thermotactile input / output peripheral 112 includes headphones (including a speaker or a microphone) may be used for other functions, such as capturing audio commands via a microphone, or playing back audio signals received from a gaming application via a speaker, etc.The haptic peripheral ecosystem coordination system 132 may present thermal haptic feedback based on the 3D gaming environment location or type of gaming action event and activate the TEG or PEA portion of the or each input / output device 112. Thus, the haptic peripheral ecosystem coordination system 132 may coordinate where in the haptic peripheral ecosystem available to the user when gaming to send thermal haptic feedback.

[0055] The TEG arrays 170-1, 170-2, 170-n can be arranged so as to apply a heating effect or a cooling effect to the surfaces of the housing of the thermotactile input / output peripheral 112. This can include those surfaces of the thermotactile input / output peripheral 112 that a user might touch. In one embodiment, the TEG arrays 170-1, 170-2, 170-n can form zones across the surface of the thermotactile input / output peripheral 112 that can be individually heated or cooled.

[0056] In one embodiment, each of the TEG arrays 170-1, 170-2, 170-n can be a TEG array formed on a flexible substrate. For example, the first TEG array 170-1 can include a plurality of TEGs, each TEG including a p-doped semiconductor and an n-doped semiconductor sandwiched between a top electrical insulator and a bottom electrical insulator both made of, for example, ceramic. Each of the p-doped semiconductor and the n-doped semiconductor can be welded between two ceramic plates and placed in electrical series and thermal parallel connection with each other to form each of the TEGs. In one embodiment, the TEG arrays 170-1, 170-2, 170-n can be operated using the Peltier effect (also known as the thermoelectric effect), in which applying a voltage to a p-doped semiconductor and an n-doped semiconductor pair causes a temperature change. In these embodiments, the top electrical insulator and the bottom electrical insulator can be heated or frozen based on the amount of voltage or the polarity of the voltage applied at the TEG leads of the arrays operatively coupled to the TEG arrays 170-1, 170-2, 170-n.

[0057] In another embodiment, the second TEG array 170-2 may also include multiple groups of p-doped semiconductors and n-doped semiconductors sandwiched between the top and bottom electrical insulators. These groups of p-doped semiconductors and n-doped semiconductors can be operably coupled together in series to receive the voltage at the first TEG lead. In this embodiment, the multiple groups of semiconductors each forming a TEG can be arranged in series and activated together to heat or cool the housing of the thermal tactile input / output peripheral device 112 depending on the magnitude or voltage applied via the first and second TEG leads.

[0058] In one embodiment, the second TEG array 170-2 (for example) may include a first group of TEGs coupled to a first TEG lead (forming a first TEG array) and a second group of TEGs coupled to a second first TEG lead (forming a second TEG array). Each of the first group of TEGs and the second group of TEGs may be operably coupled to a flexible substrate so that the first group of TEGs and the second group of TEGs are placed adjacent to the inner surface of the housing of the thermotactile input / output peripheral device 112. During operation, in this embodiment, the first group of TEGs of the second TEG array 170-2 may be heated or cooled independently of whether the second group of TEGs of the second TEG array 170-2 is heated or cooled. This may allow the first group of TEGs to be heated and then cooled independently (e.g., sequentially) while the second group of TEGs are heated and cooled. Thus, from the user's perspective, the presented thermotactile response feels like a heat wave has been across the outer surface of the thermotactile input / output peripheral device 112 in coordination with the game action event. As will be described herein, this selective and sequential heating and cooling of individual TEG arrays can indicate in real time the environmental characteristics occurring within the gaming environment. For example, the gaming application can be a first-person gaming application with a war genre. It should be understood that these war genre games can include explosions represented to the user on the video / graphic display device 110. Since such explosions will reach and pass over the user in real life, the processor 102 and the thermal tactile feedback model evaluation system can identify this environmental characteristic as the heat (and vibration) tactile feedback to be represented at the first TEG array 170-1, the second TEG array 170-2, and any additional TEG arrays 170-n. Since the heat wave can pass through the avatar in a wave motion, each of the TEG arrays 170-1, 170-2, 170-n can be sequentially activated to apply heat across the surface of the housing of the thermal tactile input / output peripheral device 112 in a wave motion. This can indicate to the user the direction and intensity of the explosion that has occurred near the avatar that is added to the output received by the user. Similar actions may occur when TEG arrays 170-1, 170-2, 170-n are selectively activated by the thermotactile input / output peripheral controller 141 to represent a cold wave passing through the avatar, such as an ice blast. It should be appreciated herein that a specific use case is described that defines activation of each of the TEG arrays 170-1, 170-2, 170-n in order to provide a user with a specific type of environmental characteristics of an action occurring within a gaming environment. Other use cases are contemplated herein and are further described herein.

[0059] As described herein, the thermotactile input / output peripheral 112 may include one or more PEAs 130-1, 130-2, 130-n. The PEAs 130-1, 130-2, 130-n may be used to apply vibrations or clicks to the inner surface of the housing of the thermotactile input / output peripheral 112 to be felt by the user. As with the TEG arrays 170-1, 170-2, 170-n, each of the PEAs 130-1, 130-2, 130-n may be operatively coupled to the PCB 140 via a serial connector that operatively couples the PEAs 130-1, 130-2, 130-n to a thermotactile input / output peripheral controller 141 and the processor 102 of the information handling system. After receiving a haptic command at the controller 141, the controller 141 may apply an electric charge (e.g., via one or more printed circuit boards 140 or other matrix circuits as described herein) to a piezoelectric actuator (e.g., 130-1, 130-2, 130-n) disposed within a haptic region of the thermotactile input / output peripheral 112 identified within the haptic command. The electric charge (i.e., having a particular current and voltage) applied to the PEA 130-1, 130-2, or 130-n may cause the PEA 130-1, 130-2, or 130-n to convert that charge into mechanical stress by, for example, causing the PEA 130-1, 130-2, or 130-n to flex upward or downward. The mechanical stress of the PEA 130-1, 130-2, or 130-n caused by the application of the electric charge may be felt by the user as a haptic movement of a portion of the surface of the thermotactile input / output peripheral 112.

[0060] In one embodiment, the haptic peripheral ecosystem coordination system 132 may also include one or more sets of instructions that, when executed by the controller 140, the processor, or both, adjust the polarity, voltage, or current of the charge applied to any PEA 130-1, 130-2, or 130-n. This adjustment may be done based on the desired haptic response from the PEA 130-1, 130-2, or 130-n, the age of the PEA 130-1, 130-2, or 130-n, the electrical characteristics of the PEA 130-1, 130-2, or 130-n, the mechanical characteristics of the PEA 130-1, 130-2, or 130-n, or a combination thereof. Because these characteristics may differ between piezoelectric actuators, the charge applied by controller 141 to any given PEA 130-1, 130-2, or 130-n may be customized to produce a specific level of tactile motion (e.g., tactile motion feedback, such as vibrations, clicks, bumps, or other mechanical motions of varying durations, intensities, or patterns) presented between input / output devices 112 at any given surface portion of the thermohaptic input / output peripheral 112 based on the location or type of game action event determined by the tactile peripheral ecosystem coordination system 132. In one embodiment, controller 141 of information handling system 100 may access one or more lookup tables (e.g., motion lookup tables). In this embodiment, controller 141 of information handling system 100 may access the lookup tables in order to determine the characteristics (e.g., voltage magnitude, frequency, polarity) of the charge applied to any given piezoelectric actuator to achieve a known level of tactile motion intensity. The one or more sets of instructions of the haptic peripheral device ecosystem coordination system 132 may also include one or more sets of instructions that, when executed by the controller 141, cause any number of subsequent voltage pulses to be applied to any PEA 130-1, 130-2, or 130-n to generate haptic motion feedback, such as vibrations, clicks, bumps, or other mechanical motions of varying durations, intensities, or patterns. Each of the PEAs 130-1, 130-2, 130-n may again be activated simultaneously or sequentially based on the data signal for the determined game action event received by the haptic peripheral device ecosystem coordination system 132 from the processor 102. For example, a rumble or tremor of an avalanche or structure that overturns a detected game action event may apply motion vibration haptic feedback presented at one or more input / output devices 112.

[0061] Training of the haptic peripheral ecosystem coordination system 132 may also provide the processor with those image recognition data sets describing the recognized game action event environmental data during the execution of the game application as output. The thermal tactile feedback model evaluation system may interpret this output as specific environmental characteristics associated with the game action event that have occurred or are occurring during the execution of the game application. The thermal tactile input / output peripheral controller 141 generates signals for activating individual PEAs 130-1, 130-2, 130-n to indicate those environmental characteristics during the game to the user. Similarly, it should be understood herein that the following specific use cases are described: it is defined to activate each of the PEAs 130-1, 130-2, 130-n to provide a specific type of environmental characteristics to the user based on the actions occurring within the game environment. Other use cases are contemplated herein and are further described herein (e.g., with respect to Table 1).

[0062] In a particular embodiment, the information handling system 100 may execute a basic input / output system (BIOS). In one embodiment, the haptic peripheral device ecosystem coordination system 132 may communicate with the main memory 104, the processor 102, the video display 110, the thermal haptic input / output peripheral device 112, and the network interface device 120 via the bus 108, and may use several forms of communication, including ACPI, SMBus, 24MHZ BFSK encoded transmission channels, or shared memory. Keyboard or touchpad driver software, firmware, controllers, etc. may communicate with applications on the information handling system 100. Similarly, in some embodiments herein, video display driver software, firmware, controllers, etc. may communicate with applications on the information handling system 100 and communicate with a piezoelectric keyboard driver. For example, the video display 110 may include display scalar hardware that operates to emit light of different brightness levels via multiple pixels of the video display 110. Furthermore, in one embodiment, firmware for the video display 110 is operable to monitor brightness changes at each of these pixels in order to detect regions of pixels within the video display 110 that experience brightness changes indicative of a gaming action event (e.g., due to displaying an explosion during live gameplay).

[0063] In other embodiments, dedicated hardware implementations such as application specific integrated circuits, programmable logic arrays, and other hardware devices may be constructed to implement one or more of the methods described herein. The applications that may include the equipment and systems of various embodiments may include a variety of electronic systems and computer systems in a broad sense. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices and related control and data signals that may be communicated between modules and communicated by the modules or as part of an application specific integrated circuit. Therefore, the present system encompasses software, firmware, and hardware implementations.

[0064] When referred to as a "system," "device," "module," "controller," etc., the embodiments described herein may be configured as hardware. For example, a portion of an information handling system device may be hardware, such as an integrated circuit (such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a structured ASIC, or a device embedded on a larger chip), a card (such as a peripheral component interface (PCI) card, a PCI-express card, a personal computer memory card international association (PCMCIA) card, or other such expansion card), or a system (such as a motherboard, a system-on-chip (SoC), or a stand-alone device). The system, device, controller, or module may include software, including firmware embedded at the device, such as Core processor, Brand processor, Snapdragon processor or other processor and chipset, or other such device, or software capable of operating the relevant environment of the information handling system. The system, device, controller or module may also include a combination of the foregoing examples of hardware or software. In one embodiment, the information handling system 100 may include an integrated circuit or a board-level product having its parts which may also be any combination of hardware and software. Devices, modules, resources, controllers or programs that communicate with each other do not need to communicate continuously with each other unless otherwise explicitly specified. In addition, devices, modules, resources, controllers or programs that communicate with each other may communicate directly or indirectly through one or more intermediaries.

[0065] Figure 2is a side cross-sectional view of a palm rest portion of a thermo-tactile keyboard and palm rest assembly 200 implementing a piezoelectric actuator and thermoelectric generator (TEG) array according to one embodiment of the present disclosure. As described herein, in the embodiments described herein, a piezoelectric actuator (PEA) and a thermoelectric generator (TEG) array located within the thermo-tactile keyboard and palm rest assembly 200 can provide thermo-tactile feedback in various zones of the thermo-tactile keyboard and palm rest assembly 200. In one embodiment, a tactile command transmitted by a tactile peripheral device ecosystem coordination system to a keyboard controller of an information handling system can initiate thermo-tactile feedback of a tactile zone of the thermo-tactile keyboard and palm rest assembly 200 associated with a type or relative location of a game action event detected as having occurred in a live game. The thermo-tactile feedback initiated by the controller in this manner can cause a tactile motion (e.g., a click, bump, vibration, or other mechanical motion) or thermal change to occur within the tactile zone so as to be felt by a user who can be in close contact with the identified tactile zone regarding the type or location of the game action event.

[0066] In some embodiments, thermo-tactile feedback including tactile motion may be provided via one or more piezoelectric actuators or thermoelectric generators located below one or more keys 205 of the keyboard portion of the thermo-tactile keyboard and palm rest assembly 200. For example, in one embodiment, the thermo-tactile keyboard and palm rest assembly 200 may include a plurality of tactile keys 205 having piezoelectric actuators or thermoelectric generators mounted below keycaps on key actuated switches or below key actuated switches, wherein the tactile motion is provided to the keys 205 by the key actuated switch mechanism.

[0067] The keyboard main PCB assembly 240 can be operably connected to a piezoelectric actuator or a thermoelectric generator under one or more keys 205 so that an electric charge can be transferred to the tactile key 205 via the keyboard main PCB assembly 240 in accordance with a tactile command received from a keyboard controller operably connected to the keyboard main PCB assembly 240. For example, the keyboard main PCB assembly 240 can be operably connected to each key 205 via a key switch circuit board mounted with the key support base 220. In one embodiment, the keyboard main PCB assembly 240 can receive a tactile command from the tactile peripheral device ecosystem coordination system via the keyboard controller. For example, in one embodiment, the tactile peripheral device ecosystem coordination system can detect that a tactile command is received via an information handling system (e.g., reference Figure 1 The haptic command is generated after a specific game action event occurs within a virtual 3D game environment displayed by a video display of the information handling system 100 described herein. The haptic command can be presented for feedback based on the type or location of the game action event, and the haptic command includes identifying a specific haptic zone of the haptic key 205 representing the thermal haptic keyboard where the thermal haptic feedback will appear.

[0068] In some embodiments, thermotactile feedback including tactile motion may also be provided via one or more piezoelectric actuators located across the width of the palm rest of the thermotactile keyboard and palm rest assembly 200. For example, in one embodiment, the thermotactile keyboard and palm rest assembly 200 may include a palm rest enclosed by a palm rest top cover 250 and a palm rest bottom cover 252. In one embodiment, a palm rest printed circuit board assembly 242 and a plurality of piezoelectric actuators (e.g., a first or left palm rest piezoelectric electric actuator 234 and a second or right palm rest piezoelectric actuator 236) may be incorporated into the palm rest of the thermotactile keyboard and palm rest assembly. In some embodiments, additional piezoelectric actuators (e.g., third and fourth palm rest piezoelectric actuators) may be provided on the right side of the palm rest (not shown due to cutaway view) for the user's right hand.

[0069] The palm rest PCB assembly 242 may be operably connected to the keyboard main PCB assembly 240 such that an electrical charge may be transferred to the palm rest PCB assembly 242 via the keyboard main PCB assembly 240 in accordance with a tactile command received from a keyboard controller operably connected to the keyboard main PCB assembly 240. For example, the keyboard main PCB assembly 240 may be operably connected to the engagement PCB assembly 242 via one or more spring-loaded connector pins (e.g., 241) (such as spring pins) that may be located between the palm rest PCB assembly 242 and the keyboard main PCB assembly 240. Thus, each of the spring pins 241 (and other spring pins), the palm rest PCB assembly 242, and the piezoelectric actuators (e.g., 234 and 236) located within the palm rest assembly may be electrically coupled such that an electrical pulse generated at the keyboard main PCB assembly 240 may be transferred to one of the piezoelectric actuators (e.g., 234) located within the palm rest through a portion of the palm rest PCB assembly 242 via one of the spring pins (e.g., 241).

[0070] In one embodiment, the keyboard main PCB assembly 240 can receive haptic commands from the haptic peripheral device ecosystem coordination system via the keyboard controller. For example, in one embodiment, the haptic peripheral device ecosystem coordination system can detect that the haptic commands are received via the information handling system (e.g., reference Figure 1 The haptic command is generated after a specific game action event occurs within a virtual 3D game environment displayed by a video display of the information handling system 100 described herein. The haptic command can be presented for feedback based on the type or location of the game action event, and the haptic command includes identifying a specific haptic zone 251 of the palm rest where the thermal haptic feedback will occur. Thus, the palm rest haptic zone 251 can be associated with the type of game action event identified based on the presence of a visual action event indicator, an audio action event indicator, or a combination of the two. A specific example is presented in Table 1 below:

[0071]

[0072] Table 1

[0073] In some embodiments, the tactile zone 251 of a tactile palm rest or other input / output device may have a single piezoelectric electric tactile actuator to form a single tactile zone, or may be divided into two or more tactile zones (e.g., 251a and 251b) within the tactile zone 251 with multiple piezoelectric electric actuators. The embodiments described below discuss one embodiment with multiple tactile zones within the tactile zone on a keyboard, keyboard palm rest, mouse, headset, or other input / output device. The embodiments may include, for example, tactile keys or groups of tactile keys of a keyboard as additional tactile zones. In one embodiment, the tactile peripheral device ecosystem coordination system may transmit this tactile command to a keyboard controller operably connected to the keyboard main printed circuit board assembly 240. In other embodiments, one or more controllers may be used to receive tactile commands for the tactile palm rest in the keyboard main printed circuit board assembly 240. Any combination of controllers is contemplated in various embodiments, as desired.

[0074] In one embodiment, each of the piezoelectric actuators (e.g., 234 and 236) located below the palm rest top cover 250 can perform tactile motions independently of each other according to the charge delivered via the keyboard main printed circuit board assembly 240. For example, in one embodiment, after receiving a tactile command that specifically identifies the left palm rest tactile zone 251, the keyboard controller can transmit the charge to the palm rest printed circuit board assembly 242 through the matrix circuit incorporated into the keyboard main printed circuit board assembly 240 to the piezoelectric actuator 234 or 236 to achieve tactile feedback to the user's left palm. If a tactile response to the left tactile area 251a of the tactile zone 251 is desired, the charge is sent to the piezoelectric actuator 234. In such an embodiment, the piezoelectric actuator 234 can then receive the charge from the palm rest printed circuit board assembly 242. After the keyboard controller applies the charge to the piezoelectric actuator 234, the ceramic disk within the piezoelectric actuator 234 may be mechanically stretched or compressed to produce a tactile motion event that can be sensed, such as the piezoelectric actuator warping up and down and returning to its pre-deformed state by means of clicks, bumps, vibrations, or other mechanical motions of varying intensities, durations, or patterns. This may cause the tactile motion 203 of the palm rest cover 250 at the tactile region 251a to be felt by the user's left hand placed on top of or in close contact with the palm rest cover 250. In one embodiment, the tactile commands received by the keyboard controller may identify various charge magnitudes or tactile motion types to be applied to the piezoelectric actuator 234 and in different pulse patterns.

[0075] If a tactile response is desired for the right tactile zone 251b of the tactile zone 251, a charge is sent to the piezoelectric actuator 236. In such an embodiment, the piezoelectric actuator 236 can then receive a charge from the palm rest printed circuit board assembly 242. After the keyboard controller applies the charge to the piezoelectric actuator 236, the ceramic disk within the piezoelectric actuator 236 can be mechanically stretched or compressed to produce a tactile motion event that can be sensed, such as the piezoelectric actuator warping up and down and returning to its pre-deformed state by means of clicks, bumps, vibrations, or other mechanical motions of different intensities, durations, or patterns. This can cause the tactile motion 204 of the palm rest top cover 250 at the tactile zone 251b to be felt by the left hand of the user placed on top of the palm rest top cover 250 or in close contact with the palm rest top cover. In one embodiment, the tactile command received by the keyboard controller can identify various charge magnitudes or tactile motion types to be applied to the piezoelectric actuator 236 and in different pulse patterns.

[0076] In some embodiments, thermal tactile feedback including changes in surface temperature at the palm rest top cover 250 may be provided via one or more TEG arrays positioned across the width of the palm rest of the thermal tactile keyboard and palm rest assembly 200. For example, in one embodiment, a left or first TEG array 270-1 and a second or right TEG array 270-2 may be incorporated into the palm rest of the thermal tactile keyboard and palm rest assembly. In some embodiments, additional TEG arrays (e.g., third and fourth TEG arrays) may be provided on the right side of the palm rest (not shown due to cutaway view) for the user's right hand.

[0077] As described herein, upon detecting the occurrence of a particular game action event within the virtual 3D game environment, the keyboard main PCB assembly 240 can receive a haptic command from the haptic peripheral device ecosystem coordination system identifying a particular tactile zone 251 of the palm rest where thermal tactile feedback is to occur. In some implementations, the tactile zone 251 can have a single TEG array to form a single tactile region, or can be segmented into two or more tactile regions (e.g., 251a and 251b) within the tactile zone 251 having multiple TEG arrays.

[0078] In one embodiment, each of the TEG arrays (e.g., 270-1 and 270-2) located below the palm rest top cover 250 can perform temperature changes independently of each other according to the charge delivered via the keyboard main printed circuit board assembly 240. For example, in one embodiment, after receiving a tactile command that specifically identifies the left palm rest tactile zone 251, the keyboard controller can transmit the charge to the palm rest printed circuit board assembly 242 through the matrix circuit incorporated into the keyboard main printed circuit board assembly 240 to reach the TEG array 270-1 or 270-2 to achieve thermal tactile feedback to the user's left palm. If a thermal tactile response to the left tactile area 251a of the tactile zone 251 is required, the charge is sent to the TEG array 270-1. In such an embodiment, the TEG array 270-1 can then receive the charge from the palm rest printed circuit board assembly 242. After the keyboard controller applies an electric charge to the p-doped semiconductor or n-doped semiconductor pair of the TEG array 270-1, the thermoelectric effect can cool or heat the electrical insulator within the TEG array 270-1. In one embodiment, the degree to which the temperature at the electrical insulator of the TEG array 270-1 increases or decreases can depend on the magnitude of the voltage applied by the keyboard controller in one embodiment. This can cause the increase or decrease in the temperature of the palm rest top cover 250 at the tactile region 251a to be felt by the user's left hand placed on top of the palm rest top cover 250 or in close contact with the palm rest top cover.

[0079] If a thermotactile response is needed to the left tactile zone 251b of the tactile zone 251, a charge is sent to the TEG array 270-2. In such an embodiment, the TEG array 270-2 can then receive the charge from the palm rest printed circuit board assembly 242. After the keyboard controller applies the charge to the p-doped semiconductor or n-doped semiconductor pair of the TEG array 270-2, the thermoelectric effect can freeze or heat the electrical insulator within the TEG array 270-2. In one embodiment, the degree of increase or decrease in temperature at the electrical insulator of the TEG array 270-2 can depend on the magnitude of the voltage applied by the keyboard controller in one embodiment. This can cause the increase or decrease in the temperature of the palm rest top cover 250 at the tactile zone 251b to be felt by the left hand of the user placed on top of the palm rest top cover 250 or in close contact with the palm rest top cover. In an exemplary embodiment, thermal tactile feedback may be presented between left and right tactile zones 251a and 251b, between tactile zones such as 251 and other tactile zones, or between various input / output devices based on the location or type of game action event.

[0080] Figure 3 is a perspective view of a haptic feedback mouse including designated areas for haptic motion or haptic thermal change according to one embodiment of the present disclosure. Figure 3Only a portion of the housing 371, 372 of the thermal tactile mouse 350 is shown removed to show the internal portion of the thermal tactile mouse 350. Figure 3 The removed portion of the housing 371, 372 shown in the figure may include a portion of the mouse bottom and side housing 372 or a portion of the mouse palm rest housing 371. The thermal tactile mouse 350 may be operatively coupled to an information processing system (not shown) via a wired or wireless connection. In the case where the thermal tactile mouse 350 is operatively coupled to the information processing system via a wired connection, the thermal tactile mouse 350 may include a signal and power line operatively coupled to the roller / LED PCB 364. The wired connection to the roller / LED PCB 364 may provide signal data from a processor of the information processing system and a voltage from a PMU at (for example) the information processing system. In the case where the thermal tactile mouse 350 is operatively coupled to the information processing system via a wireless connection, the roller / LED PCB 364 may include a transceiver for transmitting data to the information processing system and receiving data from the information processing system. In this embodiment, the thermal tactile mouse 350 may also include an independent power supply, such as a battery (for example, a lithium-ion rechargeable battery).

[0081] exist Figure 3 In the embodiment shown in FIG. 3 , the thermotactile mouse 350 may include a first PCB such as a roller / LED PCB 364 and a second PCB such as PCB 340. The roller / LED PCB 364 may include circuits for receiving input from one or more buttons 358 and a roller 368 used by a user to provide input to an information handling system. Additional buttons, such as one or more side buttons (not shown), may also be arranged along the surface of the thermotactile mouse 350 for user convenience. The roller / LED PCB 364 of the thermotactile mouse 350 may also include circuits associated with a position and motion detection system. Circuits for motion sensors and optical drives (or other types of acceleration and position sensors, such as rollers) may be placed on the roller / LED PCB 364.

[0082] In one embodiment, the roller / LED PCB 364 may also house a mouse controller for receiving input from actuation of the button 358, the roller wheel 368, the motion sensor, and the optical drive. This data received by the mouse controller may be processed into input data that the information handling system uses to interact with a graphical user interface at the information handling system. In the context of the present specification, the input data provided by the mouse controller may be used during execution of a gaming application to control actions within a gaming environment. For example, the processor may interpret input data from the mouse controller to a processor of the information handling system to control a cursor on a video / graphics display device, control the movement of an avatar within a gaming environment, or otherwise initiate gaming actions during gaming.

[0083] In one embodiment, the thermal tactile mouse 350 may also include a dedicated PCB 340 that is operatively coupled to the roller / LED PCB 364 and the mouse controller. The dedicated PCB 340 may include circuitry associated with any TEG drivers and PEA drivers as well as connection ports for operatively coupling to the roller / LED PCB 364 and the mouse controller. Figure 3 The first TEG array 370-1, the second TEG array 370-2, the first PEA 330-1 and the second PEA 330-2 in the embodiment of FIG. Figure 3 The scroll / LED PCB 364 and the dedicated PCB 340 are shown as two separate PCBs, but the present description contemplates that the two PCBs may be combined to reduce the footprint within the thermal tactile mouse 350. In one embodiment, either the scroll / LED PCB 364 or the dedicated PCB 340 may be divided into another number of PCBs to better arrange the circuitry within the thermal tactile mouse 350.

[0084] Figure 3 Also shown is a specific arrangement of the first TEG array 370-1, the second TEG array 370-2, the first PEA 330-1, and the second PEA 330-2 relative to each other. Figure 3 Only two TEG arrays and two PEAs are shown, but the present specification contemplates that any number of TEG arrays and PEAs may be placed within the housing of the thermotactile mouse 350 and against the inner surfaces of the housings 371, 372 of the thermotactile mouse 350. The locations of the TEG arrays and PEAs may also vary. In one embodiment, one or more TEG arrays and / or PEAs may be placed below the button 358 along the side of the thermotactile mouse 350 or at any other location of the thermotactile mouse 350 that the user may touch (even temporarily).

[0085] As described herein, either of the first TEG array 370-1 and the second TEG array 370-2 may include an array of p-doped semiconductors and n-doped semiconductor pairs. Figure 3 The first TEG array 370-1 and the second TEG array 370-2 shown in the figure can each be addressed individually by the mouse controller so that each of the first TEG array 370-1 and the second TEG array 370-2 can be activated individually. The p-doped semiconductor and the n-doped semiconductor pair can be sandwiched between the top and bottom electrical insulators, both made of, for example, ceramic. For ease of illustration, Figure 3 The second TEG array 370-2 in FIG. 3 is shown as not including these ceramic insulators. However, it is desirable that the second TEG array 370-2 or any other TEG array placed within the thermal tactile mouse 350 include the same ceramic insulators as in FIG. Figure 3 The first TEG array 370 - 1 is shown with a ceramic electrical insulator like the ceramic electrical insulator.

[0086] In this specification, the p-doped semiconductor and n-doped semiconductor pairs may each be considered a TEG by themselves, and each of the first TEG array 370-1 and the second TEG array 370-2 may be referred to herein as a TEG array. In one embodiment, each of the p-doped semiconductor and n-doped semiconductor pairs as a single TEG may be individually addressable so that each of the TEGs may provide specific tactile feedback to the user in some embodiments. In a specific example, adjacent TEG arrays may be activated simultaneously, sequentially, or otherwise to heat or cool the housing 371, 372 of the thermal tactile mouse 350. From the user's perspective, this selective activation of each of the TEG arrays feels like a wave of heat or cold has crossed the outer surface of the thermal tactile mouse 350 based on the presentation of the received tactile instructions. In one embodiment, multiple TEG arrays may be used to define specific zones on the thermal tactile mouse 350, and the multiple TEG arrays may be activated by the mouse controller to provide that thermal tactile feedback presented based on the location or type of game action event. Likewise, signals detected by a processor of an information handling system, such as those shown in Table 1, may be used to provide specific TEG array behavior, and this embodiment particularly incorporates the use of those types of detected signals.

[0087] In one embodiment, the first TEG array 370-1 or the second TEG array 370-2 may include multiple p-doped semiconductor and n-doped semiconductor pairs that are activated together to heat or cool the shell 371, 372 of the thermal tactile mouse 350. In one embodiment, the first TEG array 370-1 and the second TEG array 370-2 may be independently activated at different times to provide specific tactile feedback to the user. In one embodiment, in order to increase the number of zones that can be independently heated or cooled, the number of TEG arrays 370-1, 370-2 placed across the inner surface of the shell 371, 372 may be increased. In this embodiment, the number of connectors to and from the TEG arrays 370-1, 370-2 may increase as the number of TEG arrays 370-1, 370-2 increases, and a dedicated PCB 340 may be incorporated with these other connectors. In one embodiment, each of the TEG arrays 370-1, 370-2 may include a first lead that operatively couples the TEG arrays 370-1, 370-2 to a dedicated PCB 340 via a first connector. Additionally, in this embodiment, each of the TEG arrays 370-1, 370-2 may include a second lead that operatively couples the TEG arrays 370-1, 370-2 via another connector. In these embodiments, the connectors may be operatively coupled to a TEG driver and a mouse controller as described herein. In these embodiments, the first lead and the second lead may be used to provide a specific amount of voltage at a specific polarity across the TEGs within the TEG arrays to cause one of a heating effect or a cooling effect at the first TEG array 370-1 and the second TEG array 370-2.

[0088] like Figure 3 , the thermotactile mouse 350 also includes a first PEA 330-1 and a second PEA 330-2. Likewise, the number of PEAs placed within the housing of the thermotactile mouse 350 may be more or less than two, and the present specification contemplates these additional embodiments. Much like the TEG arrays 370-1, 370-2, the PEAs 330-1, 330-2 may be arranged so that they may apply vibrations, clicks, or other mechanical effects to the housings 371, 372 of the thermotactile mouse 350 in order to provide another different kind of tactile feedback to the user. Likewise, signals detected by a processor of an information handling system, such as the signals described with reference to Table 1, may be used to present specific PEA behaviors, and this embodiment particularly incorporates the use of those types of detected signals in order to provide specific tactile feedback to the user based on the type or location of a game action event, a game environment, and the like.

[0089] exist Figure 3In the embodiment shown in , the first PEA 330-1 has been placed behind the first TEG array 370-1 so that the first TEG array 370-1 is sandwiched between the first PEA 330-1 and the inner surface of the shell 371, 372 of the thermal tactile mouse 350. In this arrangement, the first PEA 330-1 can apply vibrations to the first TEG array 370-1 and the shell 371, 372 when activated. Activation of the first PEA 330-1 (and the second PEA 330-2) can be achieved when the mouse controller receives a signal to activate the first PEA 330-1 and sends a voltage to the piezoelectric material layer, causing the piezoelectric material to expand. When the voltage is removed, the piezoelectric material contracts back to its resting state. This expansion and contraction of the piezoelectric material after the voltage is selectively applied produces a vibration, click or other mechanical effect felt by the user at a location on the surface of the shell 371, 372.

[0090] In one embodiment, the first PEA 330-1 and the first TEG array 370-1 pair may be described herein as a thermal tactile module. Multiple thermal tactile modules (e.g., the second TEG array 370-2 and the second PEA 330-2) may be arranged anywhere within the housing 371, 372 and along the inner wall of the housing 371, 372. Likewise, in order to increase the granularity of the tactile feedback felt by the user, the number of thermal tactile modules may be increased, thereby creating more zones along the surface of the housing 371, 372.

[0091] In one embodiment, the thermotactile modules including TEG arrays 370-1, 370-2 and PEAs 330-1, 330-2 may be arranged along the inner surface of the mouse palm rest housing 371 where the user's palm is intended to rest. This may be one of many locations where these thermotactile modules may be placed and include sufficient surface area to produce multiple zones of thermotactile feedback that can be detected by the user during operation of the thermotactile mouse 350. Additional locations may also include button locations and side locations where thermotactile modules may be placed to apply thermotactile feedback to the user.

[0092] Figure 4is a perspective view of a haptic feedback audio headset including designated areas for tactile motion or tactile thermal changes according to one embodiment of the present disclosure. As described herein, in the embodiments described herein, a piezoelectric actuator (PEA) and a thermoelectric generation (TEG) array located within a thermohaptic headset 460 can provide thermohaptic feedback in various zones of the thermohaptic headset 460. In one embodiment, a haptic command transmitted to a headset controller by a haptic peripheral device ecosystem coordination system can initiate thermohaptic feedback of a tactile zone of the thermohaptic headset 460 associated with a type or location of a game action event detected as having occurred in a live game. The thermohaptic feedback initiated by the headset controller and presented from the haptic peripheral device ecosystem coordination system in this manner can cause a tactile motion (e.g., a click, bump, vibration, or other mechanical motion) or thermal change to occur within the tactile zone so as to be felt by a user of the thermohaptic headset 460 that can be in close contact with the identified tactile zone.

[0093] In some embodiments, thermotactile feedback including tactile motion can be provided via one or more piezoelectric actuators located below the surface of various portions of the thermotactile earphone 460. For example, in one embodiment, the thermotactile earphone 460 can include an earpiece that is designed to contact a portion of a user's ear or a portion of a user's head that surrounds the ear. In one embodiment, an earphone printed circuit board assembly and a plurality of piezoelectric actuators can be incorporated into one or more portions of these earpieces of the thermotactile earphone 460.

[0094] In one embodiment, the headset PCB assembly can receive haptic commands from the haptic peripheral device ecosystem coordination system via the headset controller, such as regarding Figure 1 For example, in one embodiment, the haptic peripheral device ecosystem coordination system may detect that a haptic peripheral device ecosystem coordination system is connected to the haptic peripheral device ecosystem coordination system via the information handling system (e.g., reference Figure 1A haptic command is generated for presentation at a headset haptic zone 461, 462, 463, or 464 after a specific game action event occurs within a virtual 3D game environment displayed by a video display of the described information handling system 100). The presented haptic command may be directed to the identification of a specific haptic zone at which thermal haptic feedback will be presented, such as headset left haptic motion zone 463, headset left haptic hot zone 461, headset right haptic hot zone 462, or headset right haptic motion zone 464. In another embodiment, each of the headset haptic zones (e.g., 461, 462, 463, or 464) may provide both haptic motion and haptic thermal feedback. In one embodiment, the headset haptic zone (e.g., 461, 462, 463, or 464) at which the haptic command may be presented may be associated with the type of game action event identified based on a visual action event indicator, an audio action event indicator, a game code instruction, or a combination, as described in more detail with respect to Table 1. In one embodiment, the haptic peripheral device ecosystem coordination system may transmit this haptic command to a headset controller operably connected to the headset printed circuit board assembly.

[0095] In one embodiment, each of the piezoelectric actuators located below the tactile motion area (e.g., 461, 462, 463, or 464) of the earphone 460 can perform tactile motion independently of each other according to the charge delivered via the earphone printed circuit board assembly. For example, in one embodiment, after receiving a tactile command that specifically identifies the left tactile motion area 463 of the earphone, the earphone controller can transmit the charge to the piezoelectric actuator below the surface of the earphone 460 disposed in the left tactile motion area 463 of the earphone through the matrix circuit incorporated into the earphone printed circuit board assembly to achieve tactile feedback to the user's left ear or left side of the head. After the earphone controller applies the charge to the piezoelectric actuator below the surface of the earphone 460 disposed in the left tactile motion area 463 of the earphone, the ceramic disk within the piezoelectric actuator can be mechanically stretched or compressed to generate a tactile motion event that can be sensed, such as the piezoelectric actuator warping up and down and returning to its pre-deformed state by means of clicks, bumps, vibrations, or other mechanical movements of different intensities, durations, or patterns. This can cause the left ear or left side of the head of the user that is in close contact with the surface of the earphone 460 within the earphone left tactile motion zone 463 to feel tactile motion of the surface of the earphone 460 within the earphone left tactile motion zone 463. In one embodiment, the tactile commands received by the earphone controller can identify various charge magnitudes or tactile motion types to be applied to the piezoelectric actuator and in different pulse patterns.

[0096] As another example, after receiving a tactile command that specifically identifies the right tactile motion zone 464 of the headset, the headset controller may transmit an electric charge to a piezoelectric actuator disposed below the surface of the headset 460 in the left tactile motion zone 464 of the headset through a matrix circuit incorporated into the headset printed circuit board assembly to achieve tactile feedback to the user's right ear or right side of the head. After the headset controller applies the electric charge to the piezoelectric actuator disposed below the surface of the headset 460 in the right tactile motion zone 464 of the headset, the ceramic disk in the piezoelectric actuator may be mechanically stretched or compressed to generate a tactile motion event that can be sensed, such as the piezoelectric actuator warping up and down and returning to its pre-deformed state by means of clicks, collisions, vibrations, or other mechanical movements of different intensities, durations, or patterns. This may cause the right ear or right side of the head of the user, which is in close contact with the surface of the headset 460 in the right tactile motion zone 464 of the headset, to feel the tactile movement of the surface of the headset 460 in the right tactile motion zone 464 of the headset.

[0097] In some embodiments, thermal haptic feedback can be provided via one or more TEG arrays disposed below the surface of the headset 460, the thermal haptic feedback comprising surface temperature changes at one or more headset haptic thermal zones (e.g., 461, 462, 463, or 464). As described herein, upon detecting the occurrence of a particular game action event within the virtual 3D gaming environment, the headset printed circuit board assembly can receive a haptic command from the haptic peripheral device ecosystem coordination system identifying a particular haptic zone (e.g., 461, 462, 463, or 464) of the headset where thermal haptic feedback will occur, and the haptic command is presented by the haptic peripheral device ecosystem coordination system to that particular haptic zone (e.g., 461, 462, 463, or 464).

[0098] In one embodiment, each of the TEG arrays located below the earpiece of the headset 460 can perform temperature changes independently of each other according to the charge delivered via the headset printed circuit board assembly. For example, in one embodiment, after receiving a tactile command that specifically identifies the headset left tactile hot zone 461, the headset controller can transmit the charge to the TEG array below the surface of the headset 460 disposed in the headset left tactile hot zone 461 through the matrix circuit incorporated in the headset printed circuit board assembly to achieve tactile feedback to the user's left ear or left side of the head. After the charge is applied by the headset controller to the TEG array below the surface of the headset 460 disposed in the headset left tactile hot zone 461, after the charge is applied by the headset controller to the p-doped semiconductor or n-doped semiconductor pair of the TEG array below the surface of the headset 460 disposed in the headset left tactile hot zone 461, the thermoelectric effect can freeze or heat the electrical insulator within the TEG array below the surface of the headset 460 disposed in the headset left tactile hot zone 461. In one embodiment, the degree to which the temperature at the electrical insulator of the TEG array increases or decreases may depend on the magnitude of the voltage applied by the keyboard controller in one embodiment. This may cause the left ear or left side of the head of the user in close contact with the headset 460 to feel an increase or decrease in the temperature of the surface of the headset 460 within the left tactile hot zone 461 of the headset.

[0099] As another example, after receiving a haptic command specifically identifying the headset right tactile hot zone 462, the headset controller may transmit a charge to a TEG array disposed below the surface of the headset 460 within the headset right tactile hot zone 462 through a matrix circuit incorporated within the headset printed circuit board assembly to achieve tactile feedback to the user's right ear or right side of the head. After the charge is applied by the headset controller to the TEG array disposed below the surface of the headset 460 within the headset right tactile hot zone 462, after the charge is applied by the headset controller to a p-doped semiconductor or n-doped semiconductor pair of the TEG array disposed below the surface of the headset 460 within the headset right tactile hot zone 462, the thermoelectric effect may freeze or heat the electrical insulator within the TEG array disposed below the surface of the headset 460 within the headset right tactile hot zone 462. In one embodiment, the degree to which the temperature at the electrical insulator of the TEG array increases or decreases may, in one embodiment, depend on the magnitude of the voltage applied by the keyboard controller. This may cause the right ear or right side of the head of the user in close contact with the earphone 460 to feel an increase or decrease in the temperature of the surface of the earphone 460 within the right tactile hot zone 462 of the earphone.

[0100] Figure 55 is an image of a three-dimensional game environment 500 in which game action events associated with multiple thermal tactile feedback may occur according to one embodiment of the present disclosure. In one embodiment, such a three-dimensional game environment 500 may display a player avatar 510 (often represented by an avatar holding a weapon) performing actions (e.g., firing a weapon or performing some other player attack action event 511) in accordance with user instructions input via a keyboard or mouse. One or more non-player avatars, such as an enemy avatar (e.g., 520 or 530) or an ally avatar 540, may also be displayed within the three-dimensional game environment 500. In one embodiment, the actions of the enemy avatar (e.g., 520 or 530) may be controlled by a game application in some embodiments. In other embodiments, the actions of the enemy avatar (e.g., 520 or 530) and the actions of the ally avatar 540 may be controlled by input received from other users (e.g., not users of the thermal tactile keyboard and palm rest assembly described herein) communicating with the game application via a network. The three-dimensional game environment 500 may also include objects such as buildings, weapons, non-interactive characters, or vehicles (e.g., 550 and 560).

[0101] Each of the avatars and objects may be displayed within the three-dimensional game environment 500 at a specific location (e.g., orientation and depth or Euclidean coordinates (X, Y, Z)) relative to the player avatar 510. In one embodiment, one or more of the avatars may participate in a certain type of game action event, such as firing a weapon or performing some other form of attack and receiving damage from such attack. For example, the player avatar 510 may perform a player attack game action event 511, which is displayed as a spray 511 of a specific color leaving the player avatar's weapon 510 (e.g., simulating the firing of a gun). As another example, an enemy avatar 520 may initiate an enemy attack game action event 521 on the player's avatar, which is displayed as a bright pulse simulating lightning. In other examples, another enemy avatar 530 may initiate an enemy avatar attack game action event 531 on a non-player or ally avatar 540, or a non-player or ally avatar 540 may initiate an ally attack game action event 541 on an enemy avatar 520. In some embodiments, a game action event may occur when an avatar (e.g., a player, an ally, or an enemy) is damaged from an incoming attack. For example, an ally avatar attack 541 or a player avatar attack 511 may cause an enemy avatar 520 to trigger a damage game action event 522. These are merely examples of game action events that may occur within a game, and are intended to be illustrative rather than restrictive. Other game action events may include any action allowed by any avatar or character within the game application, including wielding any weapon, initiating dialogue, throwing spells, using telekinesis, entering water or other liquids, etc.

[0102] As described herein, in one embodiment, the haptic peripheral device ecosystem coordination system may use a variety of different methods, including, for example, trained image recognition models, to recognize game action events occurring within the three-dimensional game environment 500. In this embodiment, the haptic peripheral device ecosystem coordination system may capture multiple images of the game during a training period, such as during Figure 5 The images captured during these training cycles may include images of various avatars (e.g., 510, 520, 530, or 540) displayed at different depths and viewing angles relative to the player's perspective. The avatars may perform or be affected by a variety of game action events (e.g., 511, 521, 522, 531, or 541). The various avatars (e.g., 510, 520, 530, or 540) (which may also include the user / player's own avatar 510) and the game action events (e.g., 511, 521, 522, 531, or 541) within each of these captured training images may be labeled to form a training data set for the image recognition model.

[0103] These labeled captured training images may then be input into an image recognition model to train the model to accurately identify the avatars (e.g., 510, 520, 530, or 540) and game action events (e.g., 511, 521, 522, 531, or 541) within the captured training images. The image recognition model may be repeatedly trained on a variety of such training data sets until the model can accurately make such identifications within a preset tolerance (e.g., 70% of the time or more). In these embodiments, once the model has been trained, captured live game images (e.g., images that have not yet been labeled to identify an avatar or game action event) may be input into the trained image recognition model, and the avatars (e.g., 510, 520, 530, or 540), avatar positions, and game action events (e.g., 511, 521, 522, 531, or 541) that appear within the live game may then be identified when displayed via a video display. If the identified game action event (e.g., 511, 521, 522, 531, or 541) is associated with thermal tactile feedback (e.g., tactile movement or temperature change at one or more tactile areas of one or more thermal tactile input / output peripheral devices), the tactile peripheral device ecosystem coordination system may transmit a tactile command to a controller of the thermal tactile input / output peripheral device to initiate thermal tactile feedback within the tactile area of ​​the thermal tactile input / output peripheral device (e.g., a thermal tactile keyboard palm rest, a thermal tactile mouse, or a thermal tactile headphone device).

[0104] In another embodiment described herein, the haptic peripheral device ecosystem coordination system can identify such live game action events (e.g., 511, 521, 522, 531, or 541) by analyzing changes in pixel brightness detected by the firmware of the video display device. In many game scenarios, damage to a player avatar or a non-player game character avatar (e.g., an enemy or an ally) can be accompanied by a bright appearance (e.g., 511, 521, 522, 531, or 541), such as a weapon firing or an explosion. In one embodiment, the firmware of the video display device can measure these brightness changes, their size, position, and speed in real time during live game play. Such firmware real-time analysis methods exist in the art, for example, to record the lag between a player inputting a user instruction (e.g., firing a weapon) and the display of the player avatar 510 taking an action 511 in accordance with such received user instructions. In one embodiment, when a game action event occurs during live gameplay, the haptic peripheral ecosystem coordination system may, in some embodiments described herein, access these real-time pixel brightness measurements to determine when the game action event has occurred (e.g., avatar 510 fires weapon 511, performs another type of attack, suffers damage) and the location (e.g., depth and angle relative to the player) of the game action event (e.g., 511).

[0105] In another embodiment, the game application itself may generate instructions or indicators that identify the occurrence of a game action event within a live game and the location or type of the game action event. For example, the game application that generates the three-dimensional game environment 500 may be pre-programmed to transmit an indication that a specific game action event has occurred to the haptic peripheral device ecosystem coordination system. In this embodiment, the haptic peripheral device ecosystem coordination system may then identify a tactile zone within one or more of the peripheral devices of the peripheral device ecosystem associated with the game action event identified in the instruction or indicator received from the game application (e.g., in Table 1). In such an embodiment, the haptic peripheral device ecosystem coordination system may then transmit a tactile command to present the tactile feedback so identified to the associated tactile zone of the peripheral device (e.g., with reference to Table 1). In another embodiment, the game application may be pre-programmed to identify a tactile zone within one or more of the peripheral devices of the peripheral device ecosystem associated with the game action event (e.g., in Table 1), and instruct the peripheral device driver to present that tactile feedback to the associated tactile zone of the peripheral device.

[0106] The location or positioning of the game action event within the displayed image of the 3D gaming environment 500 may inform the type of game action event detected. For example, the displayed image may be divided into four quadrants, including an upper left quadrant 581, an upper right quadrant 582, a lower right quadrant 583, and a lower left quadrant 584. In a first-person shooter game, the player avatar's weapon 510 may conventionally be displayed in the lower right quadrant 583, so that a bright flash or image-recognized game action event within this lower right quadrant 583 may most likely be associated with the player avatar firing its weapon 511. In another example, game action events displayed in the upper left quadrant 581 and upper right quadrant 582 may, in one embodiment, reflect game action events that occur at a distance from the player avatar 510, and therefore may not be associated with game action events caused by user input commands (e.g., the player avatar firing a weapon 511) or damage to the player avatar, which may be displayed in the lower right quadrant 583 or lower left quadrant 584.

[0107] In one embodiment, the haptic feedback associated with the identified game action event may initiate haptic movement or thermal haptic feedback of a portion of an input / output device (e.g., keyboard, keyboard palm rest, haptic mouse, haptic headset) having a position relative to the user that is related to the position of the game action event (e.g., 511, 521, 522, 531, or 541) relative to the player avatar within the virtual three-dimensional game environment 500. For example, a game action event 511 located in the lower right corner of the image of the virtual three-dimensional game environment 500 and indicating an event occurring at or very near the player avatar 510 may be associated with haptic movement or thermal haptic feedback within a haptic zone toward the right of one or more input / output devices (such as a haptic mouse or keyboard palm rest) positioned closest to the user. As another example, a game action event 531 located in the top center of the image of the virtual three-dimensional game environment 500 and indicating an event occurring relatively far from the player avatar 510 may be associated with haptic movement or thermal haptic feedback of one or more function or number keys of the keyboard or a haptic headset.

[0108] In some embodiments, the detected game action event may be associated with multiple tactile zones. For example, in one embodiment, an ally attacking game action event 541 may start at an ally avatar 540 and move toward an enemy avatar 520. In such exemplary embodiments, the game action event 541 may be associated with a tactile motion or thermal tactile feedback that starts in a tactile zone located on the left side of an input / output device (e.g., a keyboard, a keyboard palm rest, or a tactile headset) and advances in a straight line across multiple tactile zones toward the right side of the input / output device. In such a manner, the tactile peripheral device ecosystem coordination system may cause tactile feedback at one or more locations across multiple input / output devices that reflect the location of the game action event 540 within the virtual three-dimensional game environment 500. For example, a heat wave or shock wave may be felt at or across one or more keys of a keyboard, and the heat wave or shock wave may advance to the tactile zone of the keyboard palm rest.

[0109] In some embodiments, the haptic peripheral device ecosystem coordination system may also determine whether the avatar is a player avatar (e.g., 510) or a non-player avatar (e.g., 520, 530, or 540). If the identified game action event (e.g., player avatar 510 firing weapon 511) is associated with thermohaptic feedback (e.g., haptic movement or temperature change at one or more haptic regions of one or more thermohaptic input / output peripheral devices), the haptic peripheral device ecosystem coordination system may transmit a haptic command to a controller of the thermohaptic input / output peripheral device to initiate thermohaptic feedback within a haptic region of the thermohaptic input / output peripheral device (e.g., a thermohaptic keyboard palm rest, a thermohaptic mouse, or a thermohaptic headphone device).

[0110] Figure 6is a block diagram of a haptic peripheral device ecosystem coordination system that coordinates haptic commands across multiple peripheral devices according to one embodiment of the present disclosure. As described herein, in one embodiment, the haptic peripheral device ecosystem coordination system 632 can coordinate thermal haptic feedback across multiple thermal haptic input / output peripheral devices such as a thermal haptic mouse 650, a thermal haptic headset 660, or a thermal haptic keyboard and palm rest assembly 642 based on a recognized game action event occurring within a live game of a gaming software application 620. In one embodiment, the haptic peripheral device ecosystem coordination system 632 can perform such coordination by: determining that a visual action event indicator or an audio action event indicator associated with a game action event has occurred, or receiving an indication from a gaming application that a game action event has occurred; and identifying a haptic motion command or a haptic thermal command associated with that game action event. In one embodiment, the haptic peripheral device ecosystem coordination system 632 can identify haptic feedback associated with a game action event that has just occurred, and present that haptic feedback at one or more input / output devices (e.g., 640, 650, or 660) within an ecosystem of input / output devices accessible to the user. In this embodiment, the haptic peripheral device ecosystem coordination system 632 can retrieve executable code instructions for such haptic motion commands or haptic thermal commands from the language repository 690, the executable code instructions being written in a format that can be executed by one of the multiple thermal haptic input / output peripheral devices that are also associated with the identified game action event. In one embodiment, the haptic peripheral device ecosystem coordination system 632 can transmit such executable code instructions to the identified thermal haptic input / output peripheral device for execution, thereby causing thermal haptic feedback indicating the identified game action event to be presented at that peripheral device.

[0111] In one embodiment, the gaming software application 620 may generate instructions or indicators that identify the occurrence of a gaming action event within a live game and the location or type of the gaming action event. For example, the gaming application 620 that generates a three-dimensional gaming environment may be preprogrammed to transmit an indication to the haptic peripheral device ecosystem coordination system 632 that a particular gaming action event has occurred. In this embodiment, the haptic peripheral device ecosystem coordination system 632 may then identify a haptic zone within one or more of the peripheral devices (e.g., 640, 650, or 660) of the peripheral device ecosystem that is associated with the gaming action event identified in the instructions or indicators received from the gaming application 620 (e.g., within Table 1). In such embodiments, the haptic peripheral device ecosystem coordination system may then transmit a haptic command to present the haptic feedback so identified (e.g., with reference to Table 1) to the associated haptic zone of the peripheral device (e.g., 640, 650, or 660). In another embodiment, the gaming application may be preprogrammed to identify a tactile zone within one or more of the peripheral devices (e.g., 640, 650, or 660) of the peripheral device ecosystem that is associated with a gaming action event (e.g., within Table 1), and direct the peripheral device driver 614 to present that tactile feedback to the associated tactile zone of the peripheral device (e.g., 640, 650, or 660).

[0112] In one embodiment, the haptic peripheral ecosystem coordination system 632 can determine that a visual action event indicator associated with a game action event has occurred within a live game controlled by the game software application 620. In one embodiment, the game software application 620 can transmit display instructions to the video display 610, which can generate and display an image of the live game, such as a reference image. Figure 5 In one embodiment, the gaming software application 620 may also transmit audio commands or audio signals to the controller 661 of the thermotactile earphone device 660, which may play the audio signals via one or more speakers incorporated therein.

[0113] In one embodiment, the haptic peripheral ecosystem coordination system 632 may identify the occurrence of a visual action event indicator associated with a game action event via one or more of a variety of contemplated methods. For example, in one embodiment, the visual action event indicator may be identified by analyzing captured live game images through a trained image recognition model, or analyzing pixel brightness during live game play to detect the occurrence of a game action event, such as with respect to Figure 8Described in more detail. As another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 can identify visual action event indicators by analyzing changes in pixel brightness detected by the firmware 611 of the video display device 610. In many game scenarios, damage to a player avatar or a non-player game character avatar (e.g., an enemy or an ally) can be accompanied by bright appearances, such as weapon firing or explosions. In one embodiment, the firmware 611 of the video display device can measure these brightness changes, their size, position, and speed in real time during live game play. In such embodiments, the firmware 611 can transmit these measured changes, their size, position, and speed for storage at the graphics data module 693 of the language repository 690 that can be accessed by the haptic peripheral device ecosystem coordination system 632. In one embodiment, when a game action event occurs during live gameplay, the haptic peripheral ecosystem coordination system 632 may, in some embodiments, access the real-time pixel brightness measurements stored at 693 to determine when a visual action event indicator associated with the game action event has occurred (e.g., the avatar fired a weapon, performed another type of attack, suffered damage) and the location of the game action event (e.g., depth and angle relative to the player), such as with respect to Figure 8 In another embodiment, the gaming application 620 may transmit an indication that a visual motion event has occurred during live gaming directly to the haptic peripheral ecosystem coordination system 632 .

[0114] By using these or other methods, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may identify the occurrence of a visual action event indicator associated with a game action event within the haptic data 694 stored at the language repository 690. For example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate a single large flash at the player avatar with the player avatar taking a single shot (as indicated in Table 1). As another example, the haptic peripheral device ecosystem coordination system 632 may associate repeated large flashes at the player avatar with the player avatar continuously shooting. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate a single large flash near the player avatar and the enemy avatar with an explosion near the player avatar. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate a single medium flash on either side of the player avatar with an enemy bullet landing near the player avatar. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate a water splash image with the player avatar moving into water. In another example, in one embodiment, the haptic peripheral ecosystem coordination system 632 may associate a single flash blast at the player avatar and a large flash blast or blood splatter image near the player avatar with the player avatar suffering damage. In another example, in one embodiment, the haptic peripheral ecosystem coordination system 632 may associate a single large flash blast at the player avatar and a flash blast at an enemy avatar with the enemy avatar suffering damage.

[0115] In some embodiments, the identification of the occurrence of a game action event can also be informed by the following operation: analyzing the audio signal played via the thermohaptic earphone device 660 in accordance with the command received from the game software application 620 to identify the audio action event indicator. After the earphone controller 661 receives the audio signal from the game software application 620, in one embodiment, the controller 661 can forward the audio signal or a processed version of the audio signal (e.g., converted to a frequency spectrum) to the language repository 690 for storage in the audio data 692. The haptic peripheral device ecosystem coordination system 632 can analyze such stored audio data 692 to identify when an audio action event indicator (e.g., a gunshot, an explosion, an interaction of a player avatar with water) has occurred. In one embodiment, the occurrence of such an audio action event indicator can be associated with one or more game action events. For example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 can analyze such stored audio data 692 to identify an audio action event indicator, such as a spike in the volume of the audio signal, the duration of such a spike, or a known pattern of such a spike (e.g., a gun audio wave pattern, an explosion audio wave pattern, a water audio wave pattern). As described in more detail with respect to Table 1, in one embodiment, such audio action event indicators may be associated with one or more thermotactile feedbacks at one or more thermotactile input / output peripherals. In one embodiment, the wave pattern may be identified by comparing the amplitude of the audio signal with the amplitude of a known pre-tagged audio signal such as a gun, explosion, or water audio wave pattern. This comparison may be performed according to known audio analysis methods in various embodiments, and may include, for example, using a neural network or audio analysis software application operating at an information handling system. In another embodiment, the gaming application 620 may transmit an indication that an audio action event has occurred during live gaming directly to the haptic peripheral ecosystem coordination system 632.

[0116] By using these or other methods, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may identify within the haptic data 694 stored at the language repository 690 the occurrence of an audio action event indicator associated with a game action event. For example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate repeated short loud audio spikes in a gun audio wave pattern with the player avatar continuously firing (as indicated in Table 1). As another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate repeated short loud audio spikes in an explosion audio wave pattern with explosions occurring near the player avatar. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate short loud audio spikes in a water splash audio wave pattern with the player avatar moving into water. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate short loud audio spikes in a blood splatter audio wave pattern with the player avatar sustaining damage. In another example, in one embodiment, the haptic peripheral ecosystem coordination system 632 may associate a short, gentle audio spike in a gun audio wave pattern with an enemy avatar taking damage.

[0117] In other embodiments, the identification of the occurrence of a game action event can also be informed by analyzing a user input command received via a thermal tactile keyboard 640 or via a thermal tactile mouse 650 via a peripheral device driver 114. In some embodiments, this can be performed in conjunction with the following operations: a game action event is identified based on the appearance of a video action event indicator or an audio action event indicator. A single video action event indicator or an audio action event indicator can be associated with multiple game action events, each of which can be associated with a separate thermal tactile feedback (as described by either or both of the tactile motion commands or tactile thermal commands and tactile zones identified in Table 1). For example, as shown in Table 1 above, a short loud flash or audio spike in a gun audio wave pattern can be associated with both a game action event in which a player avatar performs a single shot and a game action event in which an enemy bullet falls near the player avatar. In addition, the game action event of the player avatar firing a single shot can be associated with thermal tactile feedback appearing at the mouse (as described by the tactile motion commands, tactile thermal commands, and tactile areas identified in Table 1), while the game action event of an enemy bullet landing near the player avatar can be associated with thermal tactile feedback appearing at the keyboard palm rest.

[0118] In one embodiment, in order for the haptic peripheral ecosystem coordination system 632 to distinguish between these two game action events and thus identify the thermo-haptic input / output peripheral (e.g., mouse 650 or keyboard 640) at which the corresponding haptic motion command or haptic thermal command is executed, in one embodiment, the haptic peripheral ecosystem coordination system 632 may analyze the user input command received from the keyboard 640 or mouse 650. Input from the thermo-haptic mouse 650 may be detected to determine that the player avatar is shooting. For example, in one embodiment, the haptic peripheral ecosystem coordination system 632 may identify whether the user input command is received from the mouse 650 or the keyboard 640 just before the flash or analyzed audio signal including the identified visual or audio action event indicator. If such a user input command is received immediately before the analyzed audio signal is played back, this may indicate that the identified visual or audio action event indicator is associated with a game action event initiated by the player (such as the player avatar firing a weapon). If such user input command is not received immediately prior to the detected game action event, this may indicate that the identified visual or audio action event indicator is associated with a non-player initiated game action event (such as an enemy avatar firing a weapon at a player avatar).

[0119] After determining that a game action event has occurred, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command, a haptic thermal command, or both associated with the identified game action event. In one embodiment, the haptic peripheral device ecosystem coordination system 632 may make this determination via reference to a data structure stored within the haptic data 694 of the language repository 690, such as the data structure reflected at Table 1 above. For example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for executing a single short motion pulse of low intensity at the thermal haptic mouse 650, and a haptic thermal command for executing a short thermal increase of low intensity at the thermal haptic mouse 650 based on determining that the player avatar has fired a single shot. As another example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for executing multiple short motion pulses of low intensity at the thermal haptic mouse 650, and a haptic thermal command for executing a long thermal increase of low intensity at the thermal haptic mouse 650 based on determining that the player avatar is firing continuously.

[0120] In one embodiment, such haptic motion commands and haptic thermal commands may be executed at one or more of the thermal haptic input / output peripherals (e.g., 640, 650, 660). For example, the haptic peripheral ecosystem coordination system 632 may identify a haptic motion command to execute three sets of short motion pulses of high intensity at the thermal haptic headset 660 based on determining that an enemy avatar has suffered damage, and a haptic thermal command to execute a short thermal decrease of high intensity at the thermal haptic headset 660. As another example, the haptic peripheral ecosystem coordination system 632 may identify a haptic motion command to execute a single long motion pulse of high intensity at the keyboard palm rest 642 based on determining that an explosion has occurred near the player avatar, and a haptic thermal command to execute a long thermal increase of high intensity at the keyboard palm rest 642. In another example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for performing a high-intensity single short motion pulse at the keyboard palm rest 642, and a haptic thermal command for performing a high-intensity short thermal increase at the keyboard palm rest 642 based on determining that an enemy bullet has landed near the player avatar. In another example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for performing a low-intensity multiple short motion pulse at the keyboard palm rest 642, and a haptic thermal command for performing a high-intensity long thermal decrease at the keyboard palm rest 642 based on determining that the player avatar has moved into water. In another example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for performing a high-intensity single long motion pulse at the keyboard palm rest 642, and a haptic thermal command for performing a high-intensity long thermal increase at the keyboard palm rest 642 based on determining that the player avatar has suffered damage. In various embodiments, similar haptic feedback may be provided to the haptic keys of the keyboard 640 or the haptic mouse 650.

[0121] In some embodiments, a tactile motion command or a tactile thermal command can be associated with a tactile zone within a thermal tactile input / output peripheral device incorporating multiple tactile zones. For example, a tactile motion command for a thermal tactile keyboard 640 can identify a specific piezoelectric actuator or TEG array within one or more key switch structures (e.g., disposed between a keycap and a main keyboard printed circuit board) of a single key or a group of keys of the keyboard 640. As another example, a tactile motion command for a thermal tactile keyboard palm rest 642 can identify a specific piezoelectric actuator within a plurality of piezoelectric actuators disposed within the palm rest, such as with respect to a key switch structure. Figure 2 As another example, a tactile thermal command for a thermal tactile keyboard palm rest 642 may identify a specific TEG array within a plurality of TEG arrays disposed within the palm rest, as also described with respect to Figure 2 In another example, a tactile motion command for the thermal tactile mouse 650 may identify a specific piezoelectric actuator within a plurality of piezoelectric actuators disposed within the mouse 650, such as Figure 3 As another example, a tactile thermal command for a thermal tactile mouse 650 may identify a specific TEG array within a plurality of TEG arrays disposed within the mouse 650, as also described with respect to Figure 3 In another example, a haptic motion command for the thermohaptic earphone 660 may identify a specific piezoelectric actuator within a plurality of piezoelectric actuators disposed within the earphone 660, such as Figure 4 As another example, a tactile thermal command for a thermohaptic headset 660 may identify a specific TEG array within a plurality of TEG arrays disposed within the headset 660, as also described with respect to Figure 4 Describe in more detail.

[0122] The tactile motion commands or tactile thermal commands stored in the tactile data 694 may be written in an extensible markup language (XML) that can be executed by each of the headset controller 661, the keyboard controller 641, and the mouse controller 651 (e.g., as shown in Table 1). The XML description module 691 is operable to translate the XML tactile motion commands or XML tactile thermal commands into code instructions that can be executed by one of the headset controller 661, the keyboard controller 641, or the mouse controller 651. In other embodiments, the tactile motion commands and tactile thermal commands identified in this manner (e.g., with reference to Table 1) may include pointers or hyperlinks to the code instructions stored in the XML description module 691. In such a manner, the tactile peripheral device ecosystem coordination system 632 may retrieve executable code instructions for initiating thermal tactile feedback at one or more of the thermal tactile input / output peripheral devices (e.g., 640, 650, or 660) from the language repository 690 based on the identification of a game action event occurring within a live game. The haptic peripheral ecosystem coordination system 632 may then transmit executable code instructions for initiating thermal haptic feedback associated with the identified gaming action event to one or more of the thermal haptic input / output peripherals (eg, 640 , 642 , 650 , or 660 ).

[0123] Figure 7is an image of a haptic peripheral device ecosystem that initiates coordinated haptic feedback at multiple peripheral devices based on live game action events according to one embodiment of the present disclosure. As described herein, in one embodiment, the haptic peripheral device ecosystem coordination system can coordinate thermal haptic feedback across multiple thermal haptic input / output peripheral devices such as a thermal haptic mouse 750, a thermal haptic headset 760, or a thermal haptic keyboard and palm rest assembly 740 based on the types of recognized game action events and their locations that occur within a live game of a gaming software application. In one embodiment, the haptic peripheral device ecosystem coordination system can perform such coordination by: determining that a visual action event indicator or an audio action event indicator associated with a game action event has occurred; and identifying a haptic motion command, a haptic thermal command, and a haptic zone associated with that game action event. In this embodiment, the haptic peripheral ecosystem coordination system may transmit executable code instructions to a thermal tactile input / output peripheral device (e.g., 740, 750, or 760) associated with the identified tactile area for execution, resulting in thermal tactile feedback at that peripheral device (e.g., 740, 750, or 760) indicating the identified gaming action event.

[0124] In one embodiment, a gaming software application that generates a three-dimensional gaming environment displayed via video display 710 may generate instructions or indicators that identify the occurrence of a gaming action event within a live game and the location or type of the gaming action event. For example, the gaming application that generates the three-dimensional gaming environment may be preprogrammed to transmit an indication to the haptic peripheral device ecosystem coordination system that a particular gaming action event has occurred. In this embodiment, the haptic peripheral device ecosystem coordination system may then identify a haptic zone within one or more of the peripheral devices (e.g., 740, 750, or 760) of the peripheral device ecosystem that is associated with the gaming action event identified in the instructions or indicators received from the gaming application (e.g., within Table 1). In such embodiments, the haptic peripheral device ecosystem coordination system may then transmit a haptic command to present the haptic feedback so identified (e.g., with reference to Table 1) to the associated haptic zone of the peripheral device (e.g., 740, 750, or 760). In another embodiment, the gaming application may be preprogrammed to identify a tactile zone within one or more of the peripheral devices (e.g., 740, 750, or 760) of the peripheral device ecosystem that is associated with a gaming action event (e.g., within Table 1), and to direct the peripheral device driver to present that tactile feedback to the associated tactile zone of the peripheral device (e.g., 740, 750, or 760).

[0125] In one embodiment, the haptic peripheral device ecosystem coordination system may determine that a visual action event indicator associated with a game action event has occurred within a live game controlled by a gaming software application. For example, in one embodiment, the haptic peripheral device ecosystem coordination system may associate a single large flash at a player avatar with a game action event 711 in which the player avatar fires a single shot (as indicated in Table 1). In another example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may associate a single large flash at a player avatar and flashes at an enemy avatar with a game action event 722 in which the enemy avatar suffers damage. In another embodiment, the gaming application may transmit an indication that a visual action event has occurred during live gaming directly to the haptic peripheral device ecosystem coordination system.

[0126] In some embodiments, the identification of the occurrence of a game action event can also be informed by the following operation: analyzing the audio signal played via the thermohaptic earphone device 760 in accordance with the command received from the game software application to identify the audio action event indicator. In one embodiment, the occurrence of such an audio action event indicator can be associated with one or more game action events. For example, in one embodiment, the haptic peripheral device ecosystem coordination system can associate repeated short loud audio spikes in the gun audio wave pattern with the player avatar continuously shooting (as indicated in Table 1). As another example, in one embodiment, the haptic peripheral device ecosystem coordination system can associate repeated short loud audio spikes in the explosion audio wave pattern with explosions occurring near the player avatar. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system can associate short loud audio spikes in the water splash audio wave pattern with the player avatar moving into the water. In another example, in one embodiment, the haptic peripheral device ecosystem coordination system can associate short loud audio spikes in the blood splatter audio wave pattern with the player avatar suffering damage. In another embodiment, a gaming application may transmit an indication that an audio motion event has occurred during live gameplay directly to the haptic peripheral ecosystem coordination system.

[0127] After determining that a game action event has occurred, in one embodiment, the haptic peripheral ecosystem coordination system may identify a haptic motion command, a haptic thermal command, or both associated with the identified game action event and transmit the haptic motion command, the haptic thermal command, or both to the identified thermal haptic input / output peripheral (e.g., 740, 750, or 760). For example, the haptic peripheral ecosystem coordination system presents the detected game action events by type or location, and may transmit to the thermal haptic mouse 750 a haptic motion command for executing a single short motion pulse of low intensity at the haptic region 751, and a haptic thermal command for executing a short thermal increase of low intensity at the haptic region 751 based on a determination that the game action event is that the player avatar has fired a single shot. As another example, the tactile peripheral ecosystem coordination system may transmit to the thermal tactile mouse 750 tactile motion commands for performing multiple short motion pulses of low intensity at the tactile area 751 and tactile thermal commands for performing long thermal increases of low intensity at the tactile area 751 based on determining that the game action event is that the player avatar is continuously shooting.

[0128] In one embodiment, such haptic motion commands and haptic thermal commands may be executed at one or more of the thermal haptic input / output peripherals (e.g., 740, 750, 760). The haptic peripheral ecosystem coordination system coordinates the haptic feedback by determining which input / output devices are active; and presenting haptic feedback to the haptic zones to produce a three-dimensional haptic experience in the input / output devices. For example, the haptic peripheral ecosystem coordination system may transmit to the thermal haptic headset 760 a haptic motion command to execute three sets of short motion pulses of high intensity in one or more of the haptic zones 761 or 762, and a haptic thermal command to execute a short thermal reduction of high intensity at the thermal haptic headset 760 based on determining that an enemy avatar has suffered damage. As another example, the haptic peripheral ecosystem coordination system may transmit to the thermal tactile keyboard and palm rest assembly 740 a tactile motion command for performing a high-intensity single long motion pulse at one or more of the tactile zones 745 or 746, and a tactile thermal command for performing a high-intensity long thermal increase at one or more of the tactile zones 745 or 746 or at one or more of the keys 747 or 748 based on determining that an explosion has occurred near the player avatar.

[0129] In one embodiment, the tactile feedback associated with the identified game action event may initiate tactile movement or thermal tactile feedback of a portion of an input / output device (e.g., 740, 750, or 760) having a position relative to the user that is related to the position of the game action event (e.g., 711, 722, 742) relative to the player avatar within the virtual three-dimensional game environment. For example, a game action event 711 located in the lower right corner of the image of the virtual three-dimensional game environment and indicating an event occurring at or very near the player avatar 710 may be associated with tactile movement or thermal tactile feedback within a tactile zone toward the right of one or more input / output devices positioned closest to the user. In one such exemplary embodiment, the game action event 711 may be associated with tactile movement or thermal tactile feedback at the right tactile zone 746 of the tactile keyboard palm rest or at the tactile zone 751 of the tactile mouse 750. As another example, a game action event 742 located at the top center of the image of the virtual three-dimensional gaming environment and indicating an event occurring relatively farther away from the player avatar 710 may be associated with tactile motion or thermal tactile feedback of one or more function or number keys (e.g., 747 or 748) of a keyboard or zones 761 or 762 of a tactile headset.

[0130] In some embodiments, the detected game action event may be associated with multiple tactile zones. For example, in one embodiment, an ally attacking game action event 742 may begin at an ally avatar 741 and move toward an enemy avatar 743. In such exemplary embodiments, the game action event 742 may be associated with a tactile motion or thermal tactile feedback that begins in a tactile zone located on the left side of the input / output device and advances in a straight line across multiple tactile zones toward the right side of the input / output device. For example, the tactile motion or thermal tactile feedback associated with the game action event 742 may be presented at key 747 and then advance to key 748 across the keyboard 740. As another example, the tactile motion or thermal tactile feedback associated with the game action event 742 may be presented at the left tactile zone 762 of the headset 760 and then advance to the right tactile zone 761 of the tactile headset 760. In such a manner, the haptic peripheral device ecosystem coordination system can result in haptic feedback at one or more locations across multiple input / output devices (e.g., 740, 750, 760) that reflect the location of a game action event (e.g., 711 or 742) within a virtual three-dimensional gaming environment.

[0131] Figure 8is a flow chart illustrating a method of identifying visual action events via a trained image recognition model or via display firmware image analysis according to one embodiment of the present disclosure. As described herein, in one embodiment, a haptic peripheral ecosystem coordination system may use a variety of different methods to identify live game action events that warrant thermal haptic feedback at one of a plurality of thermal haptic input / output peripherals, including, for example, a trained image recognition model and a display brightness model.

[0132] At block 802, a display device may display an image of a three-dimensional gaming environment. Figure 6 In the depicted embodiment, the gaming software application 620 may transmit display instructions to the video display 610, which may generate and display images of live gaming, such as those shown in FIG. Figure 5 The image described. As reference Figure 5 As described, in one embodiment, a three-dimensional game environment 500 displayed via a video display may display a player avatar 510 (often represented by an avatar holding a weapon) performing actions (e.g., firing a weapon or performing some other player attack action event 511) in accordance with user instructions input via a keyboard or mouse. One or more non-player avatars, such as an enemy avatar (e.g., 520 or 530) or an ally avatar 540, may also be displayed within the three-dimensional game environment 500. In one embodiment, the actions of the enemy avatar (e.g., 520 or 530) may be controlled by a game application in some embodiments. In one embodiment, one or more of the avatars may participate in some type of game action event, such as firing a weapon or performing some other form of attack and receiving damage from such an attack.

[0133] In one embodiment, the haptic peripheral device ecosystem coordination system may determine whether to recognize a visual action event indicator using a display brightness model or an image recognition model at block 804. This decision may be made based on user input or based on the availability of various models. For example, a display brightness model may reduce responsiveness or increase lag between user input and display of an avatar action performed in response. In this embodiment, the user may choose to employ an image recognition model (if one is available). In another embodiment, such as when an image recognition model has not been trained, the model may not be available. In this embodiment, the haptic peripheral device ecosystem coordination system may default to using a display brightness model. If the haptic peripheral device ecosystem coordination system uses a display brightness model to recognize a visual action event indicator, the method may proceed to block 806 to execute such a display brightness model. If the haptic peripheral device ecosystem coordination system uses an image recognition model to recognize a visual action event indicator, the method may proceed to block 814 to execute such an image recognition model.

[0134] At frame 806, in the embodiment that adopts display brightness model to identify visual action event indicator, display firmware can identify the variation of brightness in the pixel area identified and the degree and speed of this type of brightness variation.In one embodiment, display firmware can at least partially operate to measure the variation of pixel brightness over time during the live game that displays via video display.In one embodiment, the variation of pixel brightness can indicate the generation of game action event in some cases.In one embodiment, the size, position and magnitude of this type of variation of pixel brightness can indicate the type of game action event and the position of described game action event relative to player avatar, as described in more detail below about frame 812.

[0135] In one embodiment, the firmware for the display device may identify the size and location of the pixel area where a brightness change has been detected at box 808. For example, in one embodiment, the display monitoring scalar firmware may be able to access or generate a two-dimensional array of pixel values ​​that associate each of the pixels in the display with a specific brightness value (e.g., between 0 and 255 for 8-bit RGB colors) at which a single image of the live game is emitted. In one embodiment, each frame of the live game may be associated with a different two-dimensional pixel brightness array. In one embodiment, the display monitoring scalar firmware may compare a plurality of these two-dimensional pixel brightness arrays to each other in order to identify one or more pixels that have experienced or are experiencing a change in pixel brightness value that satisfies a threshold (e.g., a change of 100 or more). In this embodiment, for example, the display monitoring scalar firmware may determine the magnitude of the brightness change by determining the number of pixels so identified that are located next to each other.

[0136] Once one or more pixels are identified in this manner, the location of the pixel within the display can be determined by reference to the identification number of the pixel given within the two-dimensional array of pixel brightness values. In one embodiment, each pixel within a video display can be assigned a number based on its location within the display. For example, the top left pixel within a standard full high definition 1080p computer video display comprising 2,073,600 pixels can be given the identification number 0000001, while the bottom right pixel can be given the identification number 2073600.

[0137] In some implementations, the display monitoring scalar firmware may associate this identification number with a particular portion or zone of the displayed image. For example, in one implementation, the display monitoring scalar firmware may divide the display view into four separate quadrants, with all pixels associated with only one of the four quadrants. In this implementation, the quadrant associated with a pixel may include the determined location of the pixel.

[0138] At block 810, in one embodiment, the display device firmware may transmit a notification to the processor of the magnitude, speed, and pixel region location of the brightness change. Figure 1 , a video display 110 including display monitoring scalar firmware may be operably connected to a haptic peripheral device ecosystem coordination system 132 via one or more buses 108. In this embodiment, the display monitoring scalar firmware may transmit the determined size and location of the pixel experiencing the identified brightness change to the haptic peripheral device ecosystem coordination system 132 using the bus 108. The display monitoring scalar firmware may also transmit the magnitude of such change and the speed of such change, the magnitude of the change being determined by comparing the value of a single pixel across multiple two-dimensional arrays (each two-dimensional array representing a different frame of live game play), and the speed of the change being determined by dividing the magnitude by the number of frames so compared.

[0139] In one embodiment, at block 812, the haptic peripheral ecosystem coordination system may associate a brightness change with a particular avatar or visual action event indicator based on the brightness change magnitude, speed, or pixel region location. As described herein, in some cases, a change in pixel brightness may indicate the occurrence of a game action event. In one embodiment, the size, location, and magnitude of such a change in pixel brightness may indicate the type of game action event and the location of the game action event relative to the player's avatar. For example, in reference to Figure 5 In the depicted embodiment, a player attack game action event 511 may be displayed as a different color than an ally attack game action event 541 or an enemy avatar attack game action event 521 or 531. As another example, in one embodiment, different weapons fired by the player avatar 510 may be associated with different colors and therefore different brightness values ​​(e.g., numbers between 0 and 255 for 8-bit RGB colors). Additionally, certain weapons may be associated with longer duration bursts, resulting in slower changes in pixel brightness than the firing of a single bullet, which may appear as a very rapid flash, for example.

[0140] The size of the change in pixel brightness, or the number of pixels that experience such a change in brightness at any given moment, can also be used to identify game action events or the location of the game action events within the virtual three-dimensional game environment (e.g., 500). For example, a game action event (e.g., enemy attack game action event 531) that occurs farther away from the player avatar 510 may appear smaller in comparison to a similar game action event (e.g., enemy attack game action event 521). As another example, damage caused by a grenade launcher may appear larger than damage caused by a rifle shot.

[0141] In one embodiment, the location of the pixel that experiences the brightness change can be further used to identify the location of the game action event in one embodiment. For example, in one embodiment, if the pixel that experiences the brightness change is located in the upper part of the video display, the brightness change can be associated with the game action event that occurs relatively farther from the player avatar 510 compared to the game action event depicted by the brightness change of the pixel in the lower part of the video display. In one embodiment, it may be so because the object (e.g., enemy avatar 520) located closer to the player avatar 510 in the virtual three-dimensional game environment 500 is more depicted toward the bottom of the screen than the object (e.g., enemy avatar 530) located farther from the player avatar 510. In some embodiments, the determination of the game action event and its location can be determined based on a combination of these factors. The method for identifying the visual action event indicator using the display brightness model can then end.

[0142] At block 814, in an embodiment where the haptic peripheral device ecosystem coordination system uses an image recognition model to identify visual action event indicators, the haptic peripheral device ecosystem coordination system may capture images of the training game. In one embodiment, the haptic peripheral device ecosystem coordination system may capture images of the training game that include game action events and various avatars placed at different depths and at various perspectives relative to the player avatar. For example, in reference to Figure 5 In the depicted embodiment, the haptic peripheral device ecosystem coordination system can capture images displayed within the virtual three-dimensional game environment 500. In one embodiment, the captured images can include flat or two-dimensional images or views of the three-dimensional game. These captured images of the training game can include one or more avatars, such as player avatar 510, ally avatar 540, enemy avatar 520, or enemy avatar 530. One or more game action events can also be included within the captured images of the training game, such as player attack game action event 511, enemy attack game action event 521, enemy damage game action event 522, enemy attack game action event 531, or ally attack game action event 541. In addition, the avatars and game action events can occur at different locations relative to the player avatar 510 within the virtual three-dimensional game environment 500, and can therefore be viewed from various angles. For example, enemy avatar 520 and enemy avatar 530 may have different positions relative to player avatar 510 (e.g., enemy avatar 520 is positioned closer to player avatar 510 than enemy avatar 530), thereby causing player avatar 510 to view each of the enemy avatars (e.g., 520 and 530 at different angles).

[0143] In some cases, the captured images of the training game may also include images of the avatars or action events at various angles. For example, one captured image may include the player avatar 510 looking at the right side of the ally avatar 540 as the ally avatar 540 fires from the left peripheral vision of the player avatar 510 toward the enemy avatar 520 within the right peripheral vision of the player avatar 510. As another example, a second captured image may include the player avatar 510 looking at the back of the ally avatar 540 as the ally avatar 540 fires from a position relatively closer to the player avatar 510 toward the enemy avatar 530 positioned relatively farther away from the player avatar 510.

[0144] In one embodiment, the captured image may be described pixel by pixel, with each of the pixels in the image being associated with a value between, for example, 0 and 255. In other embodiments, particularly with respect to high-definition images, the pixel values ​​may adhere to a larger scale representing a greater granularity between pixel color and brightness. In one embodiment, a single captured image may thus be represented by a two-dimensional array, with one column identifying the pixel location within the captured image and another column identifying the color value of that pixel (e.g., between 0 and 255 for a typical 8-bit red / green / blue (RGB) image).

[0145] In one embodiment, the haptic peripheral device ecosystem coordination system may mark the visual action event indicators and avatars within the image captured by the training game at block 816. For example, in one embodiment, the haptic peripheral device ecosystem coordination system may mark the player avatar 510, ally avatar 540, enemy avatar 520, or enemy avatar 530 displayed within the captured image of the virtual three-dimensional game environment 500. One or more game action events may also be marked, such as a player attack game action event 511, an enemy attack game action event 521, an enemy damage game action event 522, an enemy attack game action event 531, or an ally attack game action event 541.

[0146] As described above, the marked captured images can show each of the avatars and game action events at multiple angles. These marked and captured images can form a set of images for training image recognition models for presenting tactile feedback across tactile keyboards and tactile palm rests, as described herein. In one embodiment, markings can be used to compare attempts with untrained image recognition models to identify avatars and game action events during a training session of the image recognition model, and are associated with tactile feedback types, locations, durations, intensity, or motion patterns, as described in more detail below. Training an image recognition model on the marked captured images showing each of the avatars and game action events at various angles can allow the image recognition model to later recognize these avatars and game action events, regardless of the position or angle at which they appear relative to the player's avatar in a virtual three-dimensional game environment.

[0147] At box 818, in one embodiment, the tactile peripheral device ecosystem coordination system can train an image recognition model to recognize avatars and visual action events within the captured training game image. In one embodiment, the image recognition model may include a feedforward convolutional neural network (CNN) having multiple layers, the multiple layers including an input layer, one or more hidden layers, and an output layer. In some embodiments, a single image recognition model (e.g., a single CNN) can be trained to recognize multiple features (e.g., objects, avatars, game action events) within a single captured image. In other embodiments, separate image recognition models can be formed, each of which operates as a classifier to detect whether the image includes only one recognized feature (e.g., the category of the game action event of the associated position or type of tactile tactile feedback to be presented on the tactile keyboard or tactile palm rest area). For example, one image recognition model can be operated to identify an image in which a player attacks a game action event (e.g., 511), while another image recognition model operates to identify an image in which an ally attacks a game action event (e.g., 541).

[0148] Various pre-processing steps may be applied to the images captured at block 814 in order to convert the images into inputs that are understood by the feed-forward neural network. For example, in some embodiments, convolution methods may be applied to the captured images, as well as linear correction or merging methods. Such pre-processing may begin with a two-dimensional array of values ​​describing pixel values ​​representing one or more images captured at block 814. Each of these pixels may be grouped into a matrix of a preset size (e.g., 3x3 pixels). In one embodiment, the convolution step may involve a mathematical convolution of two of such matrices that overlap with each other (e.g., share rows or columns of pixels with each other).

[0149] Conventional convolution methods known in the art can be used to perform this step. For example, Python programming language provides the functionality of convolution operations for applying user-defined pixel values ​​in two-dimensional arrays. Such convolution methods can produce feature maps of two-dimensional images of virtual three-dimensional game environments (e.g., 500) captured at frame 814 and marked at frame 704. The convolution applied can be limited by the user by the following operation: before matrix multiplication, one or more types of filters that select operation to weight pixel values ​​(e.g., between 0 and 255) by various means. The type of filter selected and applied by the user in this way can change the pixel value of the captured image so as to make image detection easier. For example, the filter can sharpen or focus the image, blur the image or enhance the edge of the object, avatar or game action event displayed in the captured image.

[0150] In one embodiment, the convolution method may be followed by a pooling step, or in some embodiments, by a rectified linear unit activation function (ReLU) step between the convolution and the pooling. In one embodiment, the ReLU step may apply a rectified linear unit activation function to the product of the convolution step in order to overcome the vanishing gradient problem inherent in many image recognition models, thereby allowing the model to learn faster and perform better. Various models are contemplated.

[0151] In one embodiment, the merging step is operable to reduce the size of the feature map, increase the computational speed, and avoid overfitting. In one embodiment, merging involves separating the captured image that forms the basis of the input layer into non-overlapping and consistent pixel blocks, and discarding all pixels except the maximum value of all pixels across that block (e.g., the node value of the brightest pixel in that block). This greatly reduces the size of the feature map, and increases the computational speed of the image recognition model. There are various merging functions in the art, including maximum merging (e.g., as described above), average merging (taking the average value of all pixel values ​​in the pixel block) and sum merging (each of the pixel values ​​of the pixels in the block is added together). Regardless of the merging function applied, the merging layer can reduce the size of the information that the image recognition model uses to analyze the captured image. In some embodiments, the merging step can be skipped, especially when computing resources increase or when the convolution method evolves to produce smaller feature maps.

[0152] The feature map produced by such convolution (and optionally by additional linear correction or merging) can be flattened into a single vector of the first of multiple fully connected layers forming a feedforward neural network. Each of the values ​​within this single vector can constitute the value of a node within the input layer of the neural network. As described above, in one embodiment, the neural network is operable to identify whether an image includes one or more specific features (e.g., avatars, game action events, objects). In one embodiment, a neural network that operates to determine whether a single identified feature (e.g., an enemy avatar, a player initiates an attack game action event, an enemy avatar suffers damage game action event) is displayed within an image may include a first hidden layer after the input layer, wherein each of the nodes of the first hidden layer provides a best guess as to whether the image includes a particular identified feature (e.g., 0 for no and 1 for yes). Any number of layers can be used to model the neural network of the image recognition model, and the nodes in each additional layer can be determined based on the values ​​of the nodes in the previous layer and a weight matrix describing the correlation between each of the values ​​within the flattened feature map and the identified features. In other words, each new layer in the neural network may include multiple nodes, where each node represents a best guess as to whether a value within the flattened feature map is associated with a particular identified feature within the image of the virtual three-dimensional gaming environment captured at box 814.

[0153] A greater number of layers within the neural network topology may reduce the likelihood of divergence (producing unusable results), but may increase processing time. The neural network of the image recognition model may then generate an output layer comprising a plurality of nodes, each node representing a best guess (0 or 1) indicating the presence of a particular identified feature (e.g., an enemy avatar or a particular game action event) within an image captured at box 814, labeled at box 816, and input into the untrained model at box 818 during the training session. In one embodiment, each of the node values ​​within the output layer (e.g., each node value represented by a binary 0 or 1) may be averaged to produce a percentage likelihood (e.g., a value between 0 and 1) that the input training image depicts the particular feature that the model is being trained to recognize or classify. The process of generating an output layer based on a known set of input layer values ​​may be described herein as forward propagation.

[0154] In one embodiment, the haptic peripheral ecosystem coordination system may compare the percentage likelihood (e.g., a value between 0 and 1) considered as an average of the node values ​​across the output layer to a known value indicating whether the image input into the model at block 818 includes (e.g., represented by a value of 1) or does not include (e.g., represented by a value of 0) a particular feature that the model was trained to recognize based on the labeling of the image at block 816. In this embodiment, the difference between the percentage likelihood and the known value may be referred to herein as an error signal for this initial forward propagation. For example, in one embodiment, the initial forward propagation of the image recognition model during a training session may produce an output layer with an average of 0.76 likelihood that the image captured at block 814 includes a player attack game action event 511. In this embodiment, the image captured at block 814 may have been associated with a value of 1 at block 816, indicating that the captured image includes a player attack game action event 511. In this embodiment, the error function for the initial forward propagation may be 0.24. In one embodiment, each forward propagation that occurs during such a training session may produce a separate error signal as the training session proceeds, as described in more detail below.

[0155] The haptic peripheral device ecosystem coordination system may then use this known error function to adjust the weight matrices associated with each layer of the modeled neural network, which weight matrices describe the interrelationships between each of the nodes within the input layer, hidden layer, and output layer. For example, the haptic peripheral device ecosystem coordination system may perform a back-propagation method to adjust each of the weight matrices to more accurately reflect the correlation between each of the values ​​within the feature map represented within the input layer and the likelihood that such value indicates the presence of a feature that the model is trained to recognize (e.g., a player attack game action event 511). The haptic peripheral device ecosystem coordination system may then forward propagate the same input node values ​​used in the first forward propagation through each of the weight-adjusted hidden layers to produce a new output layer and a new error signal. The haptic peripheral device ecosystem coordination system may repeatedly perform this forward propagation and back-propagation until the output layer produces a likelihood that matches a known value (e.g., as applied during labeling of the image at block 816): the image contains the feature that the model is trained to recognize. For example, if an image is known to contain a feature that the model is trained to detect (e.g., a player attack game action event 511), the image may be associated with a value of 1, and the haptic peripheral device ecosystem coordination system may perform a forward propagation method and a backpropagation method until the model output node has an average value sufficiently close to a value of 1 so that the model can be considered to accurately identify the feature within the image or its absence. In one embodiment, the average output node value may be sufficiently close to the known value of 1 if the average output node value meets a preset threshold (e.g., 99%, 99.9%, etc.). In other words, if the model can, for example, identify a feature within a single image or its lack thereof with 99% accuracy, the model may be said to accurately identify a feature (e.g., a player attack game action event 511) or its absence within a single training image.

[0156] In one embodiment, the haptic peripheral device ecosystem coordination system may determine whether the image recognition model accurately recognizes the visual action event and the avatar within a preset threshold accuracy percentage at block 820. In one embodiment, the haptic peripheral device ecosystem coordination system may repeatedly perform the forward propagation and back propagation described with respect to block 818 using different input node values ​​to fine tune the weight matrix of the image recognition model. In other words, the image recognition model may be trained on multiple different captured images, each captured image depicting (or not depicting) a particular recognized feature (e.g., a player attack game action event 511) at various locations within the image or seen from various angles.

[0157] In one embodiment, the image recognition model may be considered fully trained at box 818 to recognize a given feature (e.g., a player attack game action event 511) when the image recognition model accurately recognizes the feature or the lack thereof within a preset threshold percentage of the captured images forming the training set (as described above with respect to box 818). For example, in one embodiment, in an embodiment where the training set includes 100 captured and labeled images, the preset threshold accuracy percentage may be set to 70%. In this embodiment, the haptic peripheral device ecosystem coordination system may determine at box 820 whether the image recognition model can accurately recognize the feature or the lack thereof for which the image recognition model was trained in 70 of the 100 captured and labeled images comprising the training set. This is merely an exemplary threshold accuracy percentage value and is intended to be illustrative rather than limiting. Other embodiments may require a higher (e.g., 90%, 95%, etc.) or lower (e.g., 65%, 60%, etc.) preset threshold accuracy percentage value. If the image recognition model can accurately recognize game action events and avatars within the tolerance at block 820, the method can proceed to block 822: applying the trained image recognition model to live game images to identify live game action events. If the image recognition model cannot accurately recognize game action events and avatars within the tolerance at block 820, the method can return to block 814 to further train the image recognition model on additional training game images under modified weighting, etc.

[0158] At block 822, in one embodiment, the haptic peripheral ecosystem coordination system may use a trained image recognition model to identify avatars or visual action event indicators within images captured by live gameplay. In one embodiment, live game images may depict multiple avatars or multiple game action events, each located at various positions relative to the player's avatar. In one embodiment, live game images may be described by a two-dimensional array of pixel values, similar to the captured images within the training set. In addition, the live game images may undergo the pre-processing convolution, ReLU, merging, or flattening steps described above with respect to the captured images within the training set to produce a flattened feature map of the live game images. This step may be performed repeatedly and in real time to detect the occurrence of specific features (e.g., game action events) as they occur during gameplay.

[0159] In one embodiment, the haptic peripheral ecosystem coordination system may input the captured live game image into a trained image recognition model or a trained inference model. In this embodiment, a flattened feature map of the live game image may be input into the image recognition model. The image recognition model may then output a percentage probability that the live game image includes a feature (e.g., a player attack game action event 511) that the image recognition model has been trained to detect or recognize. If the percentage probability that the live game image includes a feature meets a preset threshold detection value (e.g., 90%, 95%, etc.), the 3D haptic keyboard and palm rest system may determine that a specific game action event (e.g., a player attack game action event) has occurred.

[0160] In some embodiments, multiple portions of a single captured live game image may be continuously input into a trained model to determine whether the features that the model is trained to detect are displayed within a specific portion of the entire image. In one embodiment, the haptic peripheral device ecosystem coordination system may perform such segmentation to identify the portion of the virtual 3D game environment 500 where a specific game action event occurs during live game play. For example, in one embodiment, the captured image may depict a view of a player avatar of the entire virtual 3D game environment 500. In some embodiments, such captured images may be segmented into multiple smaller images, each image representing a zone or quadrant within the entire captured image. More specifically, the entire captured image may be segmented into four separate images, including an upper right portion, an upper left portion, a lower left portion, and a lower right portion. This is only an example of such segmentation, and it is expected that segmentation into a larger number of zones or portions may be employed. In one embodiment, increasing the number of segmented zones may increase the granularity of game action event location determination, but may also increase the computing resources required to perform such determinations.

[0161] In this embodiment, after such segmentation, the segmented images (e.g., the four separate segmented images in the example described above) can each be pre-processed, and their flattened feature maps can be input into an image recognition model to determine whether a particular game action event has occurred within a particular segmented zone or portion of the entire image. In this embodiment, the image recognition model can then provide an output indicating whether a particular game action event (e.g., player attack 511) has occurred within a particular segmented zone or portion of the player avatar view of the virtual 3D game environment (e.g., 500). This can provide a rough orientation of the game action event relative to the player avatar (e.g., to the left of the left two quadrants, or further from the top two quadrants). In this way, in one embodiment, the haptic peripheral device ecosystem coordination system can measure the location of the identified game action event within the captured live game image.

[0162] In some embodiments, an image of a live game may be input into multiple image recognition models, each of which is trained to detect different features. In this embodiment, each of the multiple image recognition models may be trained for one of the multiple game action events that the tactile peripheral device ecosystem coordination system is associated with a particular type of tactile feedback. For example, a first image recognition model may be trained to identify a player attack game action event (e.g., 511), a second image recognition model may be trained to identify an enemy attack game action event (e.g., 521 or 531), a third image recognition model may be trained to identify an enemy damage game action event (e.g., 522), and a fourth image recognition model may be trained to identify an ally attack game action event (e.g., 541). In other embodiments, a single image recognition model may be trained to identify each of these game action events or features. Regardless of the specific embodiment adopted herein, in one embodiment, the tactile peripheral device ecosystem coordination system may be operable to identify the presence or absence of multiple visual action event indicators associated with tactile feedback within a single captured live game image. In one embodiment, block 822 may be continuously repeated throughout such live game play to allow the haptic peripheral device ecosystem coordination system to detect game action events associated with haptic feedback in real time as they occur during live game play. The method for identifying visual action event indicators using a trained image recognition model may then end.

[0163] Fig. 9 is a flow chart illustrating a method of initiating haptic motion or thermal feedback at a peripheral device within a haptic peripheral ecosystem based on an identified visual or audio action event occurring during live gaming according to one embodiment of the present disclosure. As described herein, in one embodiment, a haptic peripheral device ecosystem coordination system may coordinate rendering of thermal haptic feedback across multiple thermal haptic input / output peripheral devices such as a thermal haptic mouse, a thermal haptic headset, a thermal haptic keyboard, or a thermal haptic palm rest assembly based on an identified gaming action event occurring within a live gaming of a gaming software application. In one embodiment, a gaming application may transmit an indication that a gaming action event has occurred to the haptic peripheral device ecosystem coordination system. In another embodiment, the haptic peripheral device ecosystem coordination system may perform such coordination by: determining that a visual action event indicator or an audio action event indicator associated with a gaming action event has occurred; and transmitting a haptic motion command or a haptic thermal command associated with that gaming action event to a thermal haptic input / output peripheral device also associated with that gaming action event. Execution of such a command at a thermotactile input / output peripheral may thus result in thermotactile feedback at that peripheral indicative of the identified gaming action event.

[0164] At block 902, the gaming software application may transmit instructions to a headset controller to play an audio signal via a headset speaker of an audio headset peripheral device. Figure 6 In the depicted embodiment, the gaming software application 620 may transmit audio commands or audio signals to a controller 661 of a thermotactile earphone device 660, which may play the audio signals via one or more speakers incorporated therein.

[0165] In one embodiment, the controller of the audio headset peripheral device may transmit an indication that an audio signal has been played via the left speaker or the right speaker and a description of the audio signal to the haptic peripheral device ecosystem coordination system at block 904. For example, the thermohaptic headset device 660 may also transmit these audio signals to the language repository 690 for storage in the audio data 691, which may be accessed by the haptic peripheral device ecosystem coordination system 632.

[0166] At box 906, the haptic peripheral device ecosystem coordination system may identify an audio action event indicator based on the received description of the audio signal. For example, the haptic peripheral device ecosystem coordination system 632 may analyze the stored audio data 692 to identify when an audio action event indicator has occurred (e.g., a gunshot, an explosion, an interaction of a player avatar with water). In one embodiment, the occurrence of such an audio action event indicator may be associated with one or more game action events. For example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may analyze such stored audio data 692 to identify an audio action event indicator, such as a spike in the volume of the audio signal, the duration of such a spike, or a known pattern of such a spike (e.g., a gun audio wave pattern, an explosion audio wave pattern, a water audio wave pattern). As described in more detail with respect to Table 1, in one embodiment, such an audio action event indicator may be associated with one or more thermal tactile feedback at one or more thermal tactile input / output peripheral devices.

[0167] Using these or other methods, in one embodiment, the haptic peripheral ecosystem coordination system 632 may identify the occurrence of audio action event indicators associated with game action events within the haptic data 694 stored at the language repository 690. For example, in one embodiment, the haptic peripheral ecosystem coordination system 632 may associate repeated short loud audio spikes in a gun audio wave pattern with the player avatar continuously firing (as indicated in Table 1).

[0168] At block 908, in one embodiment, the haptic peripheral ecosystem coordination system may identify visual motion events within the live game image. Figure 8In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.In some embodiments, the method of the present invention can be used to identify the game action event of the visual action event indicator.

[0169] In one embodiment, the haptic peripheral device ecosystem coordination system may determine at block 910 whether a user input command is received via a keyboard or mouse just before the audio action event indicator is detected. Figure 6 As described in the embodiments of , a single audio action event indicator or a single visual action event indicator can be associated with multiple game action events, each of which can be associated with a separate thermal haptic feedback (as described by either or both of the haptic motion commands or haptic thermal commands and haptic zones identified in Table 1). For example, in one embodiment, the haptic peripheral device ecosystem coordination system 632 can identify whether a user input command is received from the mouse 650 or from the keyboard 640 just before a detected visual action event indicator or an identified audio action event indicator (e.g., a short loud audio spike in a gun audio wave pattern).

[0170] In one embodiment, in order for the haptic peripheral device ecosystem coordination system 632 to distinguish between two game action events associated with the same audio action event indicator (e.g., a short loud audio spike in a gun audio wave pattern) or the same visual action event indicator, and thus identify the thermal haptic input / output peripheral device (e.g., mouse 650 or keyboard 640) at which the corresponding haptic motion command or haptic thermal command is executed, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may analyze the user input command received from the keyboard 640 or mouse 650. If such user input command is received immediately before the detected visual action event indicator or audio action event indicator, this may indicate that the identified audio action event indicator or visual action event indicator is associated with a player-initiated game action event (such as the player avatar firing a weapon). Therefore, if the user input command is received just before the visual or audio action event indicator is detected, the method may proceed to block 912 to identify the game action event as player-initiated. If such user input command is not received immediately before the visual or audio action event indicator is detected, this may indicate that the identified audio action event indicator is associated with a non-player initiated game action event (such as an enemy avatar firing a weapon at a player avatar). Therefore, if no user input command is received just before the audio or visual action event indicator is detected, the method may proceed to block 914 to identify the game action event as non-player initiated. In embodiments where the audio action event indicator identified at block 906 or the visual action event indicator identified at block 908 is associated with only one game action event, the method may skip blocks 910, 912, and 914 and may proceed directly to block 916.

[0171] At block 912, in an implementation where a user input command is received just before an audio or visual action event indicator occurs, the haptic peripheral device ecosystem coordination system may associate the audio or visual action event indicator with a player-initiated game action event. For example, the haptic peripheral device ecosystem coordination system may associate a short loud audio spike in a gun audio wave pattern that occurs immediately after receiving the user input command with a player-initiated game action event of a single shot fired by the player avatar, rather than with a non-player-initiated game action event of an enemy bullet landing near the player avatar. The method may then proceed to block 916 for identifying a haptic motion command or a haptic thermal command.

[0172] In one embodiment, in embodiments where a user input command is not received just prior to the occurrence of an audio or visual action event indicator, the haptic peripheral device ecosystem coordination system may associate the audio or visual action event indicator with a non-player-initiated game action event at block 914. For example, the haptic peripheral device ecosystem coordination system may associate a short loud audio spike in a gun audio wave pattern that occurs without an accompanying user input command with a non-player-initiated game action event of an enemy bullet landing near the player avatar, rather than with a player-initiated game action event of the player avatar firing a single shot. The method may then proceed to block 916 for identifying a haptic motion command or a haptic thermal command.

[0173] In one embodiment, the haptic peripheral device ecosystem coordination system may identify a haptic motion command or a haptic thermal command based on the identified audio event type or visual motion event at block 916. Figure 6 In the depicted embodiment, after determining that a game action event has occurred, in one embodiment, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command, a haptic thermal command, or both associated with the identified game action event. In one embodiment, the haptic peripheral device ecosystem coordination system 632 may make this determination via reference to a data structure stored within the haptic data 694 of the language repository 690, such as the data structure reflected at Table 1 above. For example, the haptic peripheral device ecosystem coordination system 632 may identify a haptic motion command for executing a single short motion pulse of low intensity at the thermal haptic mouse 650 and a haptic thermal command for executing a short thermal increase of low intensity at the thermal haptic mouse 650 based on determining that the player avatar has fired a single shot.

[0174] At block 918, in one embodiment, the haptic peripheral device ecosystem coordination system may retrieve code instructions associated with a haptic motion command or a haptic thermal command that can be executed by one of the plurality of peripheral devices from a language repository. Figure 6In the depicted embodiment, the haptic motion commands and haptic thermal commands identified at block 916 may include pointers or hyperlinks to code instructions stored within XML description module 691. In one embodiment, such code instructions may be written in a format (e.g., Extensible Markup Language (XML)) that can be executed by each of headset controller 661, keyboard controller 641, and mouse controller 651. In other embodiments, the haptic motion commands or haptic thermal commands stored within haptic data 694 may be written in XML (e.g., as shown in Table 1), and XML description module 691 may be operable to translate the XML haptic motion commands or XML haptic thermal commands into code instructions that can be executed by one of headset controller 661, keyboard controller 641, or mouse controller 651. In such a manner, the haptic peripheral ecosystem coordination system 632 may retrieve executable code instructions from the language repository 690 for initiating thermal haptic feedback at one or more of the thermal haptic input / output peripherals (e.g., 640, 650, or 660) based on recognition of game action events occurring within a live game.

[0175] In one embodiment, at block 920, the haptic peripheral ecosystem coordination system may transmit a haptic motion command or a haptic thermal command executable code instruction to a controller of the triggered peripheral device. The haptic peripheral ecosystem coordination system 632 may transmit executable code instructions for initiating thermal haptic feedback associated with the identified game action event to one or more of the thermal haptic input / output peripheral devices (e.g., 640, 650, or 660). For example, in one embodiment, the haptic peripheral ecosystem coordination system may transmit code instructions executable by the thermal haptic mouse 650 to generate a single short motion pulse of low intensity at a piezoelectric actuator within the thermal haptic mouse 650 and a short thermal increase of low intensity at a TEG array within the thermal haptic mouse 650. In this embodiment, the haptic peripheral ecosystem coordination system may transmit such executable code instructions to the thermal haptic mouse 650 in response to determining that a game action event of a single shot fired by a player avatar has occurred based on analysis of one or both of the visual action event indicator or the audio action event indicator.

[0176] As another example, in one embodiment, the haptic peripheral ecosystem coordination system may transmit code instructions executable by the thermal haptic keyboard and palm rest assembly 642 to generate a high intensity single long motion pulse at the piezoelectric actuators within the thermal haptic keyboard and palm rest assembly 642 and to generate a high intensity long thermal increase at the TEG array within the thermal haptic keyboard and palm rest assembly 642. In this embodiment, the haptic peripheral ecosystem coordination system may transmit such executable code instructions to the thermal haptic keyboard and palm rest assembly 642 in response to determining that a game action event of an explosion near a player avatar has occurred.

[0177] In another example, in one embodiment, the haptic peripheral device ecosystem coordination system may transmit code instructions executable by the thermohaptic headset 660 to generate three short motion pulses of high intensity at a piezoelectric actuator within the thermohaptic headset 660 and generate short thermal decreases of high intensity at a TEG array within the thermohaptic headset 660. In this embodiment, the haptic peripheral device ecosystem coordination system may transmit such executable code instructions to the thermohaptic headset 660 in response to determining that a game action event in which an enemy avatar suffers damage has occurred.

[0178] At block 922, the controller of the triggered peripheral device may initiate a temperature change in the TEG array based on the received tactile thermal command. Figure 1 In the depicted embodiment, a controller 141 of a thermotactile input / output peripheral device 112 (which in various embodiments may include a thermotactile mouse, a thermotactile keyboard and palm rest assembly, or a thermotactile headset) may receive haptic motion commands or haptic thermal commands from a haptic peripheral device ecosystem coordination system 132 via bus 108 and peripheral device driver 114. In various embodiments, controller 141 may include information about Figure 2 , Figure 3 and Figure 4 The mouse controller, keyboard controller or headphone controller are described in more detail.

[0179] In reference Figure 2In the described embodiment, for example, the keyboard controller may execute code instructions to generate a single long motion pulse of high intensity at a piezoelectric actuator (e.g., 234 or 236) within the thermal tactile keyboard and palm rest assembly 200, and generate a long thermal increase of high intensity at a TEG array (e.g., 270-1 or 270-2) within the thermal tactile keyboard and palm rest assembly 200. The keyboard controller may perform such execution by causing the keyboard main printed circuit board 240 to transmit a first charge of a preset duration, magnitude, or polarity to a piezoelectric actuator (e.g., 234 or 236) identified by the keyboard controller based on the received tactile motion command. The keyboard controller may also cause the keyboard main printed circuit board 240 to transmit a second charge of a preset duration, magnitude, or polarity to a TEG array (e.g., 270-1 or 270-2) identified by the keyboard controller based on the received tactile thermal command.

[0180] As another example, in reference Figure 3 In the described embodiment, the mouse controller can execute code instructions to generate a single short motion pulse of low intensity at the piezoelectric actuator (e.g., 330-1 or 330-2) and a short thermal increase of low intensity at the TEG array (e.g., 370-1 or 370-2) within the thermal tactile mouse 350. The mouse controller can perform such execution by causing the mouse printed circuit board 340 to transmit a first charge of a preset duration, magnitude, or polarity to the piezoelectric actuator (e.g., 330-1 or 330-2) identified by the mouse controller based on the received tactile motion command. The mouse controller can also cause the mouse printed circuit board 340 to transmit a second charge of a preset duration, magnitude, or polarity to the TEG array (e.g., 370-1 or 370-2) identified by the mouse controller based on the received tactile thermal command.

[0181] In another example, in reference Figure 4In the described embodiment, the headset controller can execute code instructions to generate three sets of short motion pulses of high intensity at a piezoelectric actuator located under one of the headset tactile motion zones (e.g., 463 or 464), and generate short thermal reductions of high intensity at a TEG array located under one of the headset tactile thermal zones (e.g., 461 or 462) within the thermal tactile headset 460. The headset controller can perform such execution by causing the headset printed circuit board to transmit a first charge of a preset duration, magnitude, or polarity to the piezoelectric actuator located under one of the headset tactile motion zones (e.g., 463 or 464) and identified by the headset controller based on the received tactile motion command. The headset controller can also cause the headset printed circuit board 340 to transmit a second charge of a preset duration, magnitude, or polarity to the TEG array located under one of the headset tactile thermal zones (e.g., 461 or 462) and identified by the headset controller based on the received tactile thermal command. The method for initiating haptic motion or thermal feedback across multiple peripheral devices based on an identified occurrence of a game action event within a live game may then end.

[0182] Fig.10 is a flow chart illustrating a method for presenting tactile feedback across multiple tactile zones at one or more input / output devices based on a detected type or location of a game action event according to one embodiment of the present disclosure. As described herein, in one embodiment, a tactile peripheral device ecosystem coordination system for an information handling system may include a processor that identifies a game action event and determines the location of the action event relative to a player avatar image for an action event indicator within a 3D game environment. The tactile peripheral device ecosystem coordination system may also transmit a tactile command selected from a plurality of tactile commands based on the type or location of the action event indicator to a controller of a triggered tactile peripheral device. In this embodiment, the controller of the triggered tactile peripheral device may execute the received tactile command to cause tactile feedback within a tactile zone of the triggered tactile peripheral device associated with the location or type of the action event.

[0183] At block 1002, a user may launch a gaming application. Figure 6 In the depicted embodiment, a user may launch a gaming software application 620. For reference Figure 5 In another example described above, the game software application may then generate a live game displayed within the three-dimensional virtual game environment 500. Figure 7 In the depicted embodiment, video display 710 may display a three-dimensional virtual gaming environment generated by the gaming application during live gameplay as a user provides input via one or more input / output devices (eg, 740, 750, or 760).

[0184] In one embodiment, at block 1004, the haptic peripheral device ecosystem coordination system may detect the currently operating input / output device. Figure 7 In the described embodiment, the haptic peripheral device ecosystem coordination system can detect that one or more of the input / output devices (e.g., haptic keyboard or haptic keyboard palm rest assembly 740, haptic mouse 750, or haptic headset 760) is currently being used by the user. Game application settings can also be used to determine which input / output devices are used, such as determining which haptic keys 747 or 748 are designated for game actions or whether to engage the settings of the haptic mouse 751 or haptic headset 760. The use of a haptic keyboard key such as 747 can further indicate that the user's left hand can utilize the haptic zone 745 of the haptic keyboard palm rest. Similarly, the use of a haptic keyboard key such as 748 can further indicate that the user's right hand can utilize the haptic zone 746 of the haptic keyboard palm rest. The use of the mouse 750 can indicate that one of the sides of the haptic keyboard and palm rest assembly 740 may not be used.

[0185] At block 1006, in one embodiment, the haptic peripheral device ecosystem coordination system or the game application itself may detect the occurrence of a game action event within the live game. For example, the haptic peripheral device ecosystem coordination system may use a reference Figure 8 As another example, the haptic peripheral device ecosystem coordination system may use reference Fig. 9 The method described herein can be used to detect audio action identifiers. In some embodiments, the game application code itself can transmit the identification of audio or visual action identifiers for game action events to the haptic peripheral device ecosystem coordination system during the ongoing game. In some embodiments, such instructions or some portion of instructions for game action events or environments can be encoded into the game application.

[0186] In these embodiments, the haptic peripheral device ecosystem coordination system can identify game action events associated with one or both of these visual or audio action event identifiers (e.g., by referring to Table 1). In some embodiments, such identification can also take into account input received from one or more input / output devices, such as in Fig. 9 In other embodiments, the game application itself can identify the occurrence of a game action event and transmit an indication that the identified game action event has occurred to the haptic peripheral device ecosystem coordination system.

[0187] At block 1008, in one embodiment, the haptic peripheral ecosystem coordination system or the gaming application may evaluate the type or location of the gaming action event. Figure 7 In the depicted embodiment, the haptic peripheral device ecosystem coordination system may evaluate the location of the action event 711 within the lower right quadrant of the three-dimensional gaming environment when the player avatar fires its weapon. In this embodiment, the haptic peripheral device ecosystem coordination system may detect that the type of the action event 711 is the player avatar firing its weapon, and may detect that the location of the action event 711 is occurring near or at the player avatar.

[0188] At block 1010, in one embodiment, the haptic peripheral device ecosystem coordination system may present haptic feedback to a controller of a haptic input / output device associated with the evaluated type or location of the game action event. For example, the haptic peripheral device ecosystem coordination system presents the detected game action event by type or location, and may transmit to the thermal haptic mouse 750 a haptic motion command for executing a single short motion pulse of low intensity at the haptic region 751, and a haptic thermal command for executing a short thermal increase of low intensity at the haptic region 751, based on determining that the game action event is that the player avatar has fired a single shot. As another example, the haptic peripheral device ecosystem coordination system may transmit to the thermal haptic mouse 750 a haptic motion command for executing multiple short motion pulses of low intensity at the haptic region 751, and a haptic thermal command for executing a long thermal increase of low intensity at the haptic region 751, based on determining that the game action event is that the player avatar is firing continuously.

[0189] In one embodiment, such haptic motion commands and haptic thermal commands may be executed at one or more of the thermal haptic input / output peripherals (e.g., 740, 750, 760). The haptic peripheral ecosystem coordination system coordinates haptic feedback by determining which input / output devices are active; and presenting haptic feedback to the haptic zones (e.g., 745, 746, 747, 748, 751, 761, or 762) to produce a three-dimensional haptic experience in the input / output devices. For example, the haptic peripheral ecosystem coordination system may transmit to the thermal haptic headset 760 a haptic motion command for executing three sets of short motion pulses of high intensity in one or more of the haptic zones 761 or 762, and a haptic thermal command for executing a short thermal reduction of high intensity at the thermal haptic headset 760 based on determining that the enemy avatar has suffered damage. As another example, the haptic peripheral ecosystem coordination system may transmit to the thermal tactile keyboard and palm rest assembly 740 a tactile motion command for performing a high-intensity single long motion pulse at one or more of the tactile zones 745 or 746, and a tactile thermal command for performing a high-intensity long thermal increase at one or more of the tactile zones 745 or 746 or at one or more of the tactile keys 747 or 748 based on determining that an explosion has occurred near the player avatar.

[0190] In one embodiment, the tactile feedback associated with the identified game action event may initiate tactile movement or thermal tactile feedback of a portion of an input / output device (e.g., 740, 750, or 760) having a position relative to the user that is related to the position of the game action event (e.g., 711, 722, 742) relative to the player avatar within the virtual three-dimensional game environment. For example, a game action event 711 located in the lower right corner of the image of the virtual three-dimensional game environment and indicating an event occurring at or very near the player avatar 710 may be associated with tactile movement or thermal tactile feedback within a tactile zone toward the right of one or more input / output devices positioned closest to the user. In one such exemplary embodiment, the game action event 711 may be associated with tactile movement or thermal tactile feedback at the right tactile zone 746 of the tactile keyboard palm rest or at the tactile zone 751 of the tactile mouse 750. As another example, game action event 742, located at the top center of the image of the virtual three-dimensional gaming environment and indicating an event occurring relatively farther away from the player avatar 710, may be associated with tactile motion or thermal tactile feedback of one or more keys of a keyboard (e.g., 747 or 748) or zones 761 or 762 of a tactile headset.

[0191] In some embodiments, the detected game action event may be associated with multiple tactile zones. For example, in one embodiment, an ally attacking game action event 742 may begin at an ally avatar 741 and move toward an enemy avatar 743. In such exemplary embodiments, the game action event 742 may be associated with a tactile motion or thermal tactile feedback that begins in a tactile zone located on the left side of the input / output device and advances in a straight line across multiple tactile zones toward the right side of the input / output device. For example, the tactile motion or thermal tactile feedback associated with the game action event 742 may be presented at key 747 and then advance to key 748 across the keyboard 740. As another example, the tactile motion or thermal tactile feedback associated with the game action event 742 may be presented at the left tactile zone 762 of the headset 760 and then advance to the right tactile zone 761 of the tactile headset 760. In such a manner, the haptic peripheral device ecosystem coordination system can result in haptic feedback at one or more locations across multiple input / output devices (e.g., 740, 750, 760) that reflect the location of a game action event (e.g., 711 or 742) within a virtual three-dimensional gaming environment.

[0192] At block 1012, in one embodiment, the controller of the tactile input / output device may provide power to a piezoelectric actuator or a thermoelectric generator of the presented tactile region of the tactile input / output device. Figure 1In the described embodiment, the haptic peripheral device ecosystem coordination system 132 may transmit instructions to the peripheral device driver 114 via the bus 108 to initiate specific thermal haptic feedback or haptic motion at one or more thermal haptic input / output peripheral devices 112 (e.g., haptic mouse 750, haptic keyboard key 747 or 748, haptic keyboard palm rest haptic zone 745 or 746, haptic headset 760). According to various embodiments herein, any array of haptic zones in the input / output peripheral device 112 is contemplated. In one embodiment, the peripheral device driver 114 of one or more of these thermal haptic input / output peripheral devices 112 may transmit instructions to the controller 141 of the triggered input / output device 112 to present thermal haptic feedback by supplying power to one or more TEG arrays (e.g., 170-1, 170-2, 170-n), or present haptic motion by supplying power to one or more PEAs (e.g., 130-1, 130-2, 130-n).

[0193] In reference Figure 2 In the depicted embodiment, the keyboard controller can provide power to the piezoelectric actuator 234 via the keyboard main printed circuit board 240 to cause tactile motion 203 within the tactile region 251a of the keyboard palm rest 250. As another example, the keyboard controller can provide power to the TEG array 271-1 via the keyboard main printed circuit board 240 to cause thermal tactile feedback within the tactile region 251a of the keyboard palm rest 250. Similarly, the piezoelectric actuator and the TEG array can be mounted on Figure 2 The 200-key keyboard is provided under the keycaps to provide a tactile zone at the keys used during gaming. Figure 3 In another exemplary embodiment described, a mouse controller may provide power to a piezoelectric actuator 330-1 via a mouse printed circuit board 340 to cause tactile motion within a tactile region 371 of a haptic mouse 350. As another example, a mouse controller may provide power to a TEG array 370-1 via a mouse printed circuit board 340 to cause thermal tactile feedback within a tactile region 371 of a haptic mouse 350. In another embodiment, a haptic headset controller may provide power to a piezoelectric actuator or TEG array to cause tactile motion or thermal tactile feedback within a tactile region 461 of a haptic headset 460. The method may then end.

[0194] Do not need to be executed in any given or specified order Figure 8 , Fig. 9 and Fig.10 The blocks of the flowcharts or steps and aspects of the operations of the embodiments discussed herein. It is contemplated that additional blocks, steps or functions may be added, some blocks, steps or functions may not be performed, blocks, steps or functions may occur simultaneously, and blocks, steps or functions from one flowchart may be performed within another flowchart.

[0195] Unless explicitly specified otherwise, devices, modules, resources, or programs that communicate with each other need not be in continuous communication with each other. In addition, devices, modules, resources, or programs that communicate with each other may communicate directly or indirectly through one or more intermediaries.

[0196] Although only a few exemplary embodiments have been described herein, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of the embodiments of the present disclosure. It is therefore intended that all such modifications be included within the scope of the embodiments of the present disclosure as defined in the appended claims. In the claims, means-plus-function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents, but also equivalent structures.

[0197] The subject matter described herein is to be considered illustrative rather than restrictive, and the appended claims are intended to cover any and all such modifications, enhancements, and other embodiments that fall within the scope of the invention. Therefore, to the maximum extent permitted by law, the scope of the present invention is to be determined by the broadest permissible interpretation of the appended claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

1. A tactile peripheral input / output device ecosystem coordination system for an information handling system, comprising: A display device, wherein the display device displays a three-dimensional 3D game environment; a processor that identifies a gaming action event occurring in the 3D gaming environment and determines, for an action event indicator, a position of the action event relative to a player avatar image within the 3D gaming environment; the processor transmitting a haptic command to a controller of the triggered haptic peripheral input / output device, the haptic command being selected from a plurality of haptic commands based on the motion event indicator; the triggered haptic peripheral input / output device being selected from a plurality of peripheral input / output devices within a haptic peripheral input / output device ecosystem based on a known location of the motion event relative to the location of the player avatar image within the 3D gaming environment relative to a haptic area on the triggered haptic peripheral input / output device within the haptic peripheral input / output device ecosystem; as well as The controller of the triggered tactile peripheral input / output device executes the received tactile command to cause tactile feedback within the tactile area of ​​the triggered tactile peripheral input / output device at a position relative to the user related to the position of the action event relative to the player avatar image.

2. The haptic peripheral input / output device ecosystem coordination system for an information handling system as recited in claim 1, wherein the triggered haptic peripheral input / output device is a keyboard palm rest.

3. The haptic peripheral input / output device ecosystem coordination system for an information handling system as claimed in claim 1, wherein the triggered haptic peripheral input / output device is a mouse.

4. The haptic peripheral input / output device ecosystem coordination system for an information handling system of claim 1, wherein the triggered haptic peripheral input / output device is an audio headphone device.

5. A tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 1, wherein the tactile feedback is tactile movement caused by the controller applying a voltage across a piezoelectric actuator within the triggered tactile peripheral input / output device in accordance with the execution of the tactile command.

6. The tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 1, wherein the tactile feedback is a tactile thermal change caused by the controller applying a voltage to a thermoelectric generator TEG within the triggered tactile peripheral input / output device in accordance with the execution of the tactile command.

7. The haptic peripheral input / output device ecosystem coordination system for an information handling system as claimed in claim 1, further comprising: The processor identifies the gaming action event based on a combination of a detected visual action event indicator and a detected audio action event indicator.

8. A method for coordinating haptic feedback across an ecosystem of peripheral input / output devices, comprising: Displaying a three-dimensional 3D game environment via a display device; identifying, via a processor, a game action event occurring in the 3D game environment to generate a detected action event indicator; Determining the location and type of the game action event within the 3D game environment; transmitting a haptic command selected from a plurality of haptic commands based on the recognition of the game action event to a controller of a triggered haptic peripheral input / output device selected from a plurality of peripheral input / output devices within a haptic peripheral input / output device ecosystem based on the location and type of the game action event within the 3D game environment relative to a known location of a haptic area on the triggered haptic peripheral input / output device within the haptic peripheral input / output device ecosystem; as well as The received haptic command is executed via the controller of the triggered haptic peripheral input / output device to cause haptic feedback within the haptic area of ​​the triggered haptic peripheral input / output device associated with the type of the game action event.

9. The method of claim 8, further comprising: The received haptic command is executed to cause a haptic thermal decrease in temperature within the haptic region by applying a voltage to a thermoelectric generator within the triggered haptic peripheral input / output device.

10. The method of claim 8, further comprising: The received haptic command is executed to cause a haptic thermal increase in temperature within the haptic region by applying a voltage to a thermoelectric generator within the triggered haptic peripheral input / output device.

11. The method of claim 8, further comprising: The gaming action event is identified, via the processor, based on an action event indicator received from gaming application code to generate the haptic command.

12. The method of claim 8, further comprising: The game action event is identified, via the processor, based on a combination of a detected visual action event indicator and the position of the game action event relative to a player avatar image within the 3D gaming environment.

13. The method of claim 8, further comprising: executing, via the processor, code instructions of the trained image recognition model to identify the game action events and the movement of the game action events within the 3D game environment within the captured live game images displayed via the display device; as well as A moving haptic response is generated across a plurality of haptic areas of the triggered haptic peripheral input / output device corresponding to the motion of the game action event.

14. The method of claim 8, further comprising: executing, via the processor, code instructions of the trained image recognition model to identify the game action events and the movement of the game action events within the 3D game environment within the captured live game images displayed via the display device; as well as A movement haptic response corresponding to the motion of the game action event is generated across the plurality of peripheral input / output devices within the haptic peripheral input / output device ecosystem.

15. The method of claim 8, further comprising: receiving from firmware of the display device a measured magnitude and rate of change of pixel brightness of an identified pixel region of the display device; identifying, via the processor, the game action event and movement of the game action event within the 3D game environment based on the measured magnitude and speed of pixel brightness changes and the identified pixel regions; as well as A moving haptic response is generated across a plurality of haptic zones of the triggered haptic peripheral input / output device corresponding to the motion of the game action event.

16. A haptic peripheral input / output device ecosystem coordination system for an information handling system, comprising: A display device, wherein the display device displays a three-dimensional 3D game environment; a processor that identifies a game action event occurring in the 3D game environment and determines a location and type of the game action event within the 3D game environment to generate a detected action event indicator; The processor transmits the haptic thermal commands and the haptic motion commands to one or more controllers of the triggered haptic peripheral input / output device; the triggered haptic peripheral input / output device being selected from a plurality of peripheral input / output devices within a haptic peripheral input / output device ecosystem based on the location and type of the game action event within the 3D game environment relative to a known location of a haptic area on the triggered haptic peripheral input / output device within the haptic peripheral input / output device ecosystem; the one or more controllers of the triggered haptic peripheral input / output device executing the received haptic motion commands to cause haptic motion within the haptic region of the triggered haptic peripheral input / output device and executing the haptic thermal commands to cause haptic thermal changes within the haptic region; as well as The tactile region is located at a position relative to a user that is related to the location and type of the game action event.

17. A tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 16, wherein the one or more controllers apply voltage pulses across the piezoelectric actuators within the triggered tactile peripheral input / output devices to cause the tactile motion in accordance with the execution of the tactile motion command.

18. A tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 16, wherein the tactile motion command identifies the magnitude of a voltage applied by the one or more controllers across a piezoelectric actuator within the triggered tactile peripheral input / output device to cause the tactile motion at one of a plurality of preset intensity levels.

19. A tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 16, wherein the tactile motion command identifies the duration of voltage applied by the one or more controllers across a piezoelectric actuator within the triggered tactile peripheral input / output device to cause the tactile motion within one of a plurality of preset durations.

20. The tactile peripheral input / output device ecosystem coordination system for an information handling system as described in claim 16, wherein the tactile thermal command identifies the duration of voltage applied by the one or more controllers to the thermoelectric generator TEG within the triggered tactile peripheral input / output device to cause the tactile thermal change within one of a plurality of preset durations.

Images