Input processing using encoding and decoding

Abstract

This invention provides a system and method for generating an input feed containing multiple input data. [Solution] The system generates an input feed containing multiple input data, which includes at least inertial sensing data generated by an inertial sensing circuit. The input feed is integrated into an encoder state, the encoder state is pushed into a state buffer, and feedback information is obtained based on the communication state between the first and second devices. The decoder side may decode the payload data to generate output data and use the output data (for example, for game processing).

Description

BACKGROUND ART

[0001] Controller technology for video game systems has evolved significantly since the advent of the first game systems. For many years, the most reliable controllers had to be physically connected to the game system (or computer) using wires and connectors to enable data transfer between the game controller and the game system. Different wireless communication protocols have allowed game controllers to operate more reliably and wirelessly from the game system. For example, many wireless controller configurations utilize the Bluetooth (registered trademark) communication protocol to communicate data between the system and the controller. SUMMARY OF THE INVENTION

[0002] While these communication protocols improve the ability of controllers to operate wirelessly from game systems, certain drawbacks still exist in these communication protocols. In particular, controller configurations are providing more input data at higher frequencies. For example, 6-axis inertial measurement unit (IMU) data, which consists of three angular velocity axes and three acceleration axes, can be difficult to transmit over conventional communication channels (e.g., Bluetooth (registered trademark)) that have limited feedback. Missing samples due to packet loss cause integration errors, leading to integration drift (e.g., incorrect orientation). Accordingly, certain drawbacks exist in the design of communication protocols and / or software associated with these conventional communication channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] [Figure 1] One non-limiting example of a system 1 in which a user can operate one or more controllers 10 is shown. [Figure 2] One non-limiting example of the encoder 20 operating in cooperation with the decoder 30 in the system 1 is shown. [Figure 3A]A non-limiting flowchart example of the processes related to encoder 20 and / or decoder 30 is also shown. [Figure 3B] A non-limiting flowchart example of the processes related to encoder 20 and / or decoder 30 is also shown. [Figure 4A] An example timing diagram between encoder 20 and decoder 30 is shown, illustrating at least two layers of packet loss management. [Figure 4B] An example timing diagram between encoder 20 and decoder 30 is shown, illustrating at least two layers of packet loss management. [Figure 5A] An example of a non-restrictive data structure 500 is shown, which illustrates the header 510 of the generated payload. [Figure 5B] An example of a non-restrictive data structure 500 for an exemplary data 520 block of payload data is shown. [Figure 5C] Here is another example of an unrestricted data structure 550. [Figure 6A] An example of a non-restrictive block diagram illustrating the changes in the internal state of encoder 20 is shown. [Figure 6B] An example of a non-restrictive block diagram illustrating the changes in the internal state of encoder 20 is shown. [Figure 6C] An example of a non-restrictive block diagram illustrating the changes in the internal state of encoder 20 is shown. [Figure 6D] An example of a non-restrictive block diagram illustrating the changes in the internal state of encoder 20 is shown. [Figure 6E] An example of a non-restrictive block diagram illustrating the changes in the internal state of encoder 20 is shown. [Figure 7] This shows an example of a non-restrictive block diagram of hardware components related to the system. [Figure 8] A non-restrictive example diagram is shown. [Figure 9] A non-restrictive illustrative diagram showing different buffer configurations is provided. [Figure 10] A non-restrictive simplified diagram is shown illustrating the difference between quantization compensation with and without it. [Modes for carrying out the invention]

[0004] Game systems can communicate with wireless controllers using various wireless communication protocols. Some game systems are designed so that controllers are physically connected to the game system, then detached from the system, and the user can operate them wirelessly from the game system. These systems may use the Bluetooth® communication protocol to communicate data between the controllers and the main system unit.

[0005] Certain shortcomings in current wireless communication protocols can lead to the loss of data packets transmitted between the controller and the main system unit, potentially causing various processing errors. In the Bluetooth® communication protocol, various bandwidth limitations, necessary to match the legacy 200Hz Application Programming Interface (API), prevent the transmission of complete signals between devices, and simple downscaling significantly degrades the quality of integration during input processing. Therefore, a system is needed that provides high-quality downsampling and quantization and is tolerant of packet loss. Furthermore, a system that favorably utilizes packet loss feedback (but can also operate without it) is required.

[0006] This summary is provided to introduce, in a simplified form, a selection of concepts that will be further described in the detailed description below. This summary is not intended to identify any important or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Rather, this summary is intended to provide an overview of the subject matter described herein. Therefore, the features described above are merely examples, and other features, aspects, and advantages of the subject matter described herein will become apparent from the detailed description, drawings, and claims below.

[0007] In the following description, specific details such as certain nodes, functional entities, technologies, and protocols are provided for illustrative and non-limiting purposes to provide an understanding of the described technology. Those skilled in the art will see that other embodiments are possible, separate from the specific details described below. In other examples, detailed descriptions of well-known methods, apparatus, technologies, etc., have been omitted to avoid obscuring the description with unnecessary details.

[0008] The sections used in this detailed description are intended to guide the reader through the general subject matter of each section. As shown below, descriptions of many features span multiple sections, and headings should not be read as influencing the meaning of descriptions contained within any particular section. Exemplary systems using encoding and / or decoding for input processing

[0009] Figure 1 shows a non-limiting example of a system 1 in which a user can operate one or more controllers 10. System 1 includes at least one game system 100, which is wirelessly connected to the controllers 10. In one embodiment, the game system 100 can be connected to a separate display device 1000. Of course, the game system 100 may include an integrated display device, and the user can play the game using the integrated display device of system 100.

[0010] In one embodiment, the controller 10 may be configured to be attachable to and detachable from the game system 100. For example, the controller 10 may be attached to the game system 100, allowing a user to hold multiple controllers 10 and game systems 100 to play one or more video games. The user may also detach the controller 10 from the game system 100 and operate the controller 10 in free space. The controller 10 may include various input devices (e.g., directional sticks, buttons, touch sensor areas) that the user can use to provide input by operating the devices on the controller. The controller 10 may also include various inertial sensor components (e.g., accelerometers, angular velocity sensors, compasses) and other components that can transmit various information (e.g., temperature) to the system 100. These examples are, of course, not limiting, and the technology described herein assumes all kinds of inputs and arrangements for the controller 10.

[0011] Figure 2 shows a non-limiting example of how the encoder 20 works in conjunction with the decoder 30 in System 1. Figures 3A and 3B also show non-limiting flowchart examples of processes related to the encoder 20 and / or decoder 30. Although Figure 2 shows only one encoder 20 and one decoder 30, this example is non-limiting, and the techniques described herein assume one or more encoders 20 and / or one or more decoders 30. In one embodiment, the encoder 20 may reside on or within the controller 10, and the decoder 30 may reside on or within the system 100. This example is also non-limiting, and the techniques described herein assume any form in which the encoder 20 and / or decoder 30 may exist. For example, the controller 10 and / or system 100 may include any combination of encoders 20 and / or decoder 30. Furthermore, the encoders 20 and / or decoder 30 may reside separately from the controller 10 and / or system 100 (e.g., in a separate network device and / or cloud computing environment).

[0012] In one embodiment, the encoder 20 is capable of acquiring input data from the controller 10 (e.g., step 301), and the input data may be generated from the IMU sensor of the controller 10. The input data may include the configuration at startup (e.g., of the system 1). The input data may also include a timestamp / output data rate (ODR) (which may be equivalent to a sample index). The input data may also include angular velocity samples (3-axis, floating-point, radians / second), acceleration samples (3-axis, floating-point, meters / second 2 )), temperature samples (floating-point, degrees), and / or other metadata.

[0013] The encoder 20 is capable of generating one or more encoded data payloads (e.g., on demand) (e.g., step S303), and the payloads can be transmitted to the decoder 30 (e.g., step 304). These payloads may have a variable size (e.g., 30 to 40 bytes) depending on the amount of data to be encoded, and the number of payloads may be generated based on a specific feedback mode. The system 1 is operable in one or more of the following feedback modes: (1) No feedback (one payload is generated and transmitted to the decoder 30 without feedback regarding whether the packet containing the payload is received), (2) Prompt feedback (one payload is generated and transmitted to the decoder 30, and the reception status of the packet is known before the next packet generation is performed - the encoder 20 is notified as to whether the packet is lost), and / or (3) Delayed feedback (the status of the previously transmitted packet is known, but only after the next packet is generated - in this situation, the encoder 20 generates two payloads, one to be used if the previous packet was successfully transmitted, and the other to be used if the previous packet was lost). These examples are of course non-limiting, and the techniques described herein contemplate any type or number of feedback modes.

[0014] The decoder 30 is capable of obtaining an input stream of the payload from the encoder 20 (e.g., step 305), and generating a stream of output samples as an output (e.g., step 306). In one embodiment, the decoder 30 (or a device associated with the decoder 30) is capable of polling the encoder 20 (or a device associated with the encoder 20) (e.g., step 305) to obtain payload data generated at various time intervals. The decoder 30 can then decode the payload data (e.g., step 306), and the decoded payload data can be used for various processes (e.g., video game processing) (e.g., step 307).

[0015] The stream of output samples may include timestamps, durations, and sample indices. The stream of output samples may also include angular velocity data (3-axis, floating-point, radians / second), acceleration data (3-axis, floating-point, meters / second 2 ), temperature data (floating-point, degrees), and / or other metadata. In certain embodiments, the feedback provided to the encoder 20 may be provided by the transport layer between the encoder 20 and the decoder 30 (i.e., not by the decoder 30 alone).

[0016] In certain embodiments, most of the encoder 20 and the decoder 30 internally rely on some type of timestamp and duration calculation. In the encoder 20 and (in most parts of the decoder 30), the timestamps and durations are referred to as timestamp (ODR) and duration (ODR), and one unit of the timestamp or duration corresponds to one period of the sensor ODR (Output Data Rate). For example, when the sensor ODR is 800 Hz, one unit of timestamp (ODR) represents 1.25 ms (1 / 800 second), and when the sensor ODR is 960 Hz, one unit of timestamp (ODR) represents 1.04 ms (1 / 960 second).

[0017] The encoder 20 includes at least two functions: Feed() and GeneratePayload(). The Feed() function is called for each input high-frequency sample (e.g., every 1.25 ms). GeneratePayload() can be called at any time, and in one embodiment, the GeneratePayload() function may be called every 5 ms (or every 15 ms). The Feed() function preprocesses the input samples and stores the results in an internal state for future GeneratePayload() calls. The GeneratePayload() call generates a payload from all new samples since the last generated state.

[0018] During Feed(), each input sample (e.g., acceleration, angular velocity, temperature) is supplied to the encoder 20, the samples are integrated into an internal encoder state, and the state is pushed into a state buffer (e.g., step 302). A new internal encoder state is generated from the previous state and input samples. The state stores data internally in a specific way. For example, from two states A and B separated by an arbitrary duration (e.g., 5ms, 30ms, or 1 second), it is possible to reconstruct a sample with angular velocity, acceleration, temperature, and / or other data that summarizes as accurately as possible what happened between states A and B. The newly generated state is pushed into the state buffer.

[0019] The specific information stored in each state may include any of the following: timestamp (ODR) (timestamp of the input sample), orientation (quaternion representing the orientation without linking to the orientation of the real-world controller), acceleration accumulator (sum of rotated accelerations), and / or temperature accumulator (sum of temperature).

[0020] The timestamp (ODR) of the new state may include the timestamp (ODR) of the input sample used to merge the new state. The difference from the previous state is always 1, and the timestamp (ODR) is primarily used to calculate the duration between two states by subtracting the timestamp. The timestamp may be stored as a 32-bit unsigned integer, resulting in ambiguous differences for durations of approximately 50 days (e.g., longer than the Bluetooth® disconnection timeout).

[0021] State orientation represents a virtual orientation that does not have a link to the real-world controller orientation. State orientation is initialized with a unit quaternion, and each new state orientation is calculated by integrating the angular velocity of the input sample into the previous state orientation. Because the angular velocity of the input sample is biased and contains noise, the orientation relative to the real-world orientation drifts over time. However, given two states with orientations, it is possible to calculate the angle between the quaternions and, by dividing by the duration between the states, calculate the average angular velocity, which provides a good approximation of the input signal. State orientation may be stored as a 32-bit floating-point quaternion.

[0022] Storing the actual velocity in the state is generally impractical because gravity needs to be removed from the signal, which requires very precise integration with unbiased orientation (and the decoder 30 needs to output acceleration, so velocity may not be necessary). The state stores an acceleration accumulator, which is the sum of the quantized rotational accelerations. The accumulator is a 64-bit integer and is freely overflow-enabled. With the sum of accelerations, system 1 provides a good average value of the input signal by calculating the difference between any two states and dividing by the duration between the states.

[0023] However, it should be noted that simple addition tends to degrade the signal over time if rotational motion is present. For example, if there is no rotation in a 1g acceleration signal, adding this signal for 100 samples will result in a state from (0,0,0) to (0,0,981). In this case, dividing the state difference between the acceleration accumulators by the duration (ODR) gives the expected signal of (0,0,9.81). However, if the signal is a 1g acceleration and rotates 360 degrees around the Y-axis during 100 samples, the acceleration signal is local, so the gravity vector goes from (0,0,9.81) to (0,9.81,0), then to (0,0,-9.81), (0,-9.81,0), and back to (0,0,9.81). In this case, the signals cancel each other out, and the sum after 100 samples is (0,0,0), and the average acceleration is zero. If a 180-degree rotation occurs during the gap, the Z-axis component cancels out, leaving only the Y-axis acceleration at a low value. An example is shown in Figure 8.

[0024] In either case, the input signal is 9.81 m / s 2 Despite the acceleration, a degraded signal is obtained, and a lower frequency is expected as the output. To minimize this problem, the acceleration of the input samples is rotated by state orientation before summation. This allows the summation to be performed on a standard that is not rotated as much as the full world standard (ignoring bias and noise errors). To calculate the average acceleration between two states, the difference between the two rotating accumulators is calculated, and that value is rotated back to the local standard using the inverse of state orientation. If the duration between states is tens of milliseconds (or hundreds of milliseconds), the integral error of orientation is small, and the signal degradation due to rotation is low. When a 1g signal is rotated 360 degrees, the rotated acceleration remains approximately the same value for 100 samples, so the magnitude of the acceleration remains 1g.

[0025] The state buffer contains a history of past states that can be used in the subsequent payload generation process. The state buffer includes at least two different ring buffers: a high-density buffer containing 20 states and a low-density buffer containing 16 states. When a state is pushed into the state buffer, it is pushed to the top of the high-density buffer. The high-density buffer may always hold the most recent 20 states (approximately 20 ms at a 960 Hz ODR sample). When the high-density buffer is full and an old state is removed from the end of the buffer, that sample is pushed to the top of the low-density buffer if the remainder of the timestamp (ODR) divided by 16 is zero. The low-density buffer stores 16 states at 16 time intervals (ODR), thus covering a low-density history of 256 input samples (approximately 270 ms at a 960 Hz ODR). An example illustrating a different buffer configuration is reproduced in Figure 9.

[0026] A key characteristic of the state buffer is that its contents are deterministic. That is, given the last sample number N, it is possible to calculate the sample number of each state in the state buffer. This characteristic is used by the decoder 30 to know which states were available when the encoder 20 generated the payload, and to know the sample duration without transmitting individual sample durations.

[0027] After a certain number of Feed() calls, it is possible to call the GeneratePayload() function to generate the payload to be sent to the decoder 30. For example, GeneratePayload() may be called every 5ms (200Hz), and Feed() may be called at 1660Hz, or after 500ms in the case of packet loss in some feedback modes. The encoder 20 can store the last transmitted state, which may be stored separately from the state buffer. This allows for a configuration that is slightly different from stored states with the same timestamp (ODR), and the system 1 can retain states at timestamps (ODR) that were not held in the low-density buffer or were removed from the low-density buffer.

[0028] When GeneratePayload() is called, encoder 20 generates a payload that represents what happened between the last transmitted state and the latest state. First, it calculates the payload duration (ODR) by calculating the difference between the start state and the end state timestamps (ODR). Depending on the payload duration, encoder 20 determines the payload pattern, and depending on the payload pattern, the payload will contain 0 to 3 samples. The encoder attempts to generate samples with a duration close to 5ms. Examples of generated samples include at least: (1) no samples or data if the duration is 0, (2) pattern #0 (1 sample + temperature) if the duration is ≤7.5ms, (3) pattern #1 (2 samples + temperature) if the duration is ≤12.5ms, (4) pattern #2 (3 samples + temperature) if the duration is ≤17.5ms, and (5) pattern #3 (3 samples) if the duration is >17.5ms.

[0029] Encoded samples are generated based on the start and end states. If the payload contains one sample, the sample is generated from the last transmitted state (A) and the most recent state (B). However, in the case of two or three samples, intermediate states are used from the state buffer. Here we describe the process for three samples, but the process for two samples is similar, starting from step 3, except that S1 is A: (1) The encoder 20 calculates the target timestamp (ODR) at one-third of the position between A and B (rounded up) and searches the state buffer for the nearest available sample T1 that matches this target sample or is thereafter; (2) The encoder 20 generates encoded samples from state A to state T1 (this process ensures that the last transmitted state is A, T1 has the same timestamp (ODR) but is slightly different) (1) The state is updated to state S1; (2) The encoder 20 searches the state buffer again for a target timestamp (ODR) between state S1 and state B and extracts state T2; (3) The encoder generates encoded samples from state S1 to state T2 and calculates state S2; (4) The encoder generates encoded samples from state S2 to state B and calculates state S3 which has the same timestamp (ODR) as B but the data is slightly different (depending on the feedback mode, S3 will be the last sent state for the next GeneratePayload() call).

[0030] The state buffer has a high-density portion that lasts approximately 20 ms in one duration (ODR) interval, so the samples have an accurate target match from pattern #0 to pattern #2. For pattern #3, a low-density buffer may be used, so the timestamp (ODR) of the target sample may not be available, and the three samples will not be spaced regularly apart. In the case of a very long payload duration that exceeds the duration of the low-density buffer, the first sample may be stretched and become much longer than the other two samples (i.e., from A to the older sample in the state buffer). Details of the sample generation process are described below.

[0031] Specifically, the sample is serialized from two states, A and B, and some contextual metadata. If the sample is the first sample in the payload, the orientation of state B is encoded directly using context-aware bit assignments. Assuming the quaternion is normalized, the encoded quaternion is rescaled so that the largest component is 1, and only the three smallest components are serialized. The index of the largest component is encoded with 2 bits, and the remaining bits are divided into three to encode the smallest component. If the sample is not the first sample in the payload, the angular velocity is calculated as the difference between the orientations of state A and state B. This orientation is quantized using the maximum attainable value (angular velocity × sample duration, or 2π) as the maximum possible angle. Table 1 below shows an example of bit assignments for angular velocity / orientation encoding: [Table 1]

[0032] Each sample encodes an equivalent of a three-component signal (here, the equivalent as the average number of bits per component). Table 2 below shows such samples: [Table 2]

[0033] For acceleration, the difference between the acceleration accumulators of state A and state B is calculated and rotated in local space using the orientation of state A. This local acceleration accumulator difference is quantized and encoded using the maximum accumulator difference attainable during the sample duration. Table 3 below shows the bit allocation for each component of the acceleration encoding: [Table 3]

[0034] The GeneratePayload() function generates a variable-size payload ranging from 4 to 40 bytes. A typical structure includes a 4-byte header and variable-size payload data ranging from 0 to 36 bytes. Figure 5A shows a non-restrictive data structure 500, illustrating an exemplary header 510 of a generated payload. In one embodiment, the header 510 may be of a fixed size and contain all the metadata necessary to interpret the payload data.

[0035] The header 510 includes multiple fields, each containing different bit sizes of information, from byte 0 to byte 3. In one embodiment, the header 510 includes at least an end timestamp field 511, a duration field 512, a sample pattern field 513, an angular velocity saturation field 514, and an acceleration saturation field 515. The end timestamp field 511 may include the timestamp (ODR) of state B used to generate the payload. Since this timestamp uses 12 bits, it may loop every 4.3 seconds for a 960Hz ODR.

[0036] The duration field 512 may include the difference (ODR) between the timestamp (ODR) of state A and the timestamp (ODR) of state B. Since this duration uses 12 bits, it will overflow with a duration exceeding 4.3 seconds at an ODR of 960 Hz. The sample pattern field 513 includes 2 bits indicating the type of pattern in the data (e.g., pattern #0, #1, #2, or #3). The angular velocity saturation field 514 and the acceleration saturation field 515 are used for secondary functions described herein.

[0037] Figure 5B shows an exemplary data structure 500 of a data 520 block of payload data. As shown in Figure 5B, data 520 contains various patterns (e.g., patterns #0 to #3) where each pattern contains high-density usage values, multiple sample values, temperature values, and total values. In one embodiment, if the duration is zero, the sample pattern is set to #0, but the data section is empty. If the duration is greater than zero, the data section contains the data structure shown in the exemplary data 520 of Figure 5B. In one exemplary embodiment, the payload shown in the data structures of Figures 5A and 5B is output from the encoder 20 to the decoder 30.

[0038] It should be noted that the decoding process is nominally simpler than the encoding process. The decoder 30 can obtain the encoded payload whether or not it knows the data size (i.e., it can always supply it in a 42-byte buffer even if only 4 bytes are used). Decoding the payload generates 0 to 4 samples in the output queue, and the decoder 30 begins extracting all metadata from the header 510.

[0039] Depending on the payload duration (ODR) and sample pattern, the decoder 30 determines how the payload data was encoded and estimates the number of samples, the number of bits per component, and other values. If the payload contains one sample, the duration is equal to the payload duration. However, if there are two or three samples, the duration must be divided among the output samples. The decoder 30 does not have access to the high-density and low-density buffers of the encoder 20, but the logic is designed so that the timestamp ODR of each state in these buffers can be estimated from the timestamp LSB of state B (the timestamp LSB is stored in the payload's "end timestamp" 12-bit field). This timestamp of state B allows the decoder 30 to determine which samples were available when the payload was generated: state B (end timestamp), state A (end timestamp - payload duration), all the latest states in the high-density buffer (from end timestamp -1 to end timestamp -19), and all the states in the low-density buffer (the 16 last states with timestamp -20%16=0).

[0040] The decoder 30 can divide the payload duration (ODR) into two or three intervals that end in an available state, using the same algorithm as the encoder 20. For each sample, the decoder 30 reads an appropriate amount of data depending on the sample pattern and sample index. For the first sample of the payload (e.g., sample #0), the decoder 30 reads and decodes the encoded quaternion. If the first sample is received by the decoder 30, the decoder 30 sets a flag to skip output samples for this payload because it is not possible to calculate the angular velocity for this sample. If the first sample is not the first sample received by the decoder 30, the angular velocity of the sample is calculated by dividing the difference between the decoded quaternion and the last sample orientation by the sample duration. The quaternion is then stored in the decoder 30, and the state is updated with the last sample orientation using the new quaternion.

[0041] For other samples (e.g., sample #1 or sample #2), the angular velocity is obtained by decoder 30 by using the inverse of the encoding operation and using the maximum reachable range over the duration of the sample as the quantized maximum value. The final sample orientation is updated by integrating this decoded angular velocity. For acceleration, for each sample, the value is decoded (similar to acceleration coding) by using the reachable range as the quantized maximum value. Then, if the skip output flag is not set for this sample, the sample is pushed to the output sample queue. Since the number of bits per component for each sample index and pattern is shared knowledge, it is not necessary to include this information in the payload.

[0042] One of the objectives of the technology described herein is to ensure optimal integration quality in the event of packet loss. Without packet loss management, transmitting only downsampled raw data would result in integration errors due to missing samples, which may not be accurately corrected by simple interpolation on the decoder 30 side. The technology described herein includes at least two cumulative packet loss management layers. One layer includes an active packet loss management system that requires one bit of feedback information about the previously transmitted packet. The other layer includes passive packet loss management that implements decoder 30 logic for fully (or at least partially) restoring the data using payload information.

[0043] Figures 4A and 4B show example timing diagrams between encoder 20 and decoder 30, depicting at least two packet loss management layers. Figure 4A shows an example timing diagram 400 related to a “passive” packet loss management process, and Figure 4B shows an example timing diagram related to an “active” packet loss management process. In the example shown in Figure 4A, encoder 20 transmits packets containing the generated payload data to decoder 30.

[0044] The process begins with operation 401, in which encoder 20 generates payload data as packet P1. In one embodiment, decoder 30 periodically polls encoder 20, and encoder 20 sends payload data in different packets (e.g., operation 402). Upon receiving packet P1, decoder 30 processes packet P1 (operation 403) and uses the processed data (e.g., for video game processing). Decoder 30 may send a response packet acknowledging successful receipt of packet P1 (operation 404). Encoder 20 prepares another packet P2 containing new payload data (operation 405) and sends packet P2 to decoder 30 (operation 406).

[0045] In the example shown in Figure 4A, the decoder 30 may not be able to successfully receive packet P2 transmitted by encoder 20 (operation 407). In this example, the decoder 30 may not send any acknowledgment (or, when polling encoder 20, it may send a message to encoder 20 indicating that the packet has not been received). Even if the decoder 30 fails to receive packet P2, encoder 20 continues to generate payload data and prepares the next packet P3 (operation 408). Encoder 20 sends packet P3 to decoder 30 (operation 409), and if decoder 30 receives it successfully, it processes packet P3 (operation 410) and confirms secure reception (operation 411).

[0046] In the passive recovery process, the encoder 20 generates a packet containing additional information so that the decoder 30 can effectively reconstruct the lost packet. Figure 5C shows an example of an unrestricted packet 550 sent from the encoder 20 to the decoder 30. In particular, packet 550 may include a first packet 551 and a second packet 552, the first packet 551 may not have been received by the decoder 30. The first packet 551 contains a first quaternion value 551A and a first angular velocity value 551B. Similarly, the second packet 552 contains a second quaternion value 552A and a second angular velocity value 552B. This example is, of course, unrestricted, and the first packet 551 and the second packet 552 may contain any kind of value.

[0047] In the example in Figure 4A, the first packet 551 may not have been received by the decoder 30, and the decoder 30 may have successfully received the second packet 552. By using a combination of the second quaternion 552A (which may contain an angle value) and the second angular velocity value 552B, the decoder 30 can reconstruct the final orientation from packet 551. This process may be repeated across various packets so that the decoder 30 can reconstruct the final orientation using the most recent combination of payload data.

[0048] Figure 4B shows an example of the non-limited timing of active packet loss recovery shown in Figure 450. In the example shown in Figure 4B, the encoder 20 generates various packets to be received by the decoder 30. The encoder 20 generates packet P1 (operation 451), where a first payload PfA and a second payload PfN are generated. In one embodiment, the PfA payload is used when the last transmitted packet was received by the decoder 30, and the PfN payload is used when the last transmitted packet was not received by the decoder 30.

[0049] The encoder 20 sends packet P1 to the decoder 30 (operation 452), the decoder 30 processes packet P1 (operation 453), and confirms receipt of packet P1 (operation 454). The encoder 20 then generates packet P2 (containing a PfN or PfA payload depending on the previous packet reception) (operation 455) and sends packet P2 to the decoder 30 (operation 456). Since the packet was successfully received by the decoder 30 (operation 453), the encoder 20 sends packet P2 with a PfA payload to the decoder 30.

[0050] In the example in Figure 4B, the decoder 30 fails to receive packet P2 (operation 457), and the encoder 20 has certain feedback information available to generate and send a suitable next packet to the decoder 30. In particular, the encoder 20 does not necessarily know (at the time of operation 458) whether the decoder 30 successfully received the previous packet, but still generates the next packet containing either a PfA or PfN payload. The encoder 20 generates the PfA and PfN payloads (operation 458), and upon receiving feedback that the previous packet was not received by the decoder 30, it sends packet 3 with the PfN payload (operation 459). The decoder 30 receives and processes packet 3 (operation 460) and sends an acknowledgment of receipt to the encoder 20 (operation 461). The active recovery process is described in more detail below and is further illustrated in the examples in Figures 6A-B. Both the active and passive recovery processes of System 1 are also described in more detail below.

[0051] More specifically, System 1 can communicate feedback about the transmission status (e.g., lost or received) of the previous packet when a new connection event occurs. In a conventional controller configuration, when a connection event occurs, the controller is immediately required to send a packet, and even if the status of the previous packet is known at this point, it is too late to use this previous packet status to generate a complete packet. For simple types of data, such as bit fields, if the previous packet was not received, the controller quickly updates the prepared packet by performing a logical OR operation between the packet's bit field and the old field value. This operation superimposes the two states, and since the OR operation is applied until the packet is received, the "1" is never lost.

[0052] Regarding IMU data, a simple logical OR operation cannot be performed because the difference in payloads required for "the last payload was received" (PfA) and "the last payload was lost" (PfN) is too large to be generated during the connection event. This technology pre-generates two IMU payloads between two connection events to prepare for both cases: one for PfA and one for PfN. Two packets are prepared using these payloads, and when a connection event occurs, either a PfA or PfN packet is sent depending on the state of the previous packet. Only after the connection event is complete and the packets have been sent can the encoder 20 receive notification about the state of the previous packet and update its internal state (the state of the immediately preceding packet remains unknown until the next connection event). Subsequently, the encoder 20 can prepare the next two payloads to prepare for two possible outcomes for the immediately preceding payload. The encoder 20 should consider some past events, as a long history may affect the next packet. The two payloads will not be identical if the last packet is lost for the first time and if the previous 10 packets were lost.

[0053] In some cases, polling packets from system 100 may be lost, in which case "no connection event" occurs from the controller 10's perspective, and the PfA / PfN status for the last transmitted packet is not transmitted to encoder 20 at this point (multiple payloads may need to be received before the status of the last actually transmitted packet can be confirmed). System 1 is able to handle this complexity, and as a result, from the perspective of decoder 30 on the system 100 side, the packet stream appears as a continuous payload stream with no interruptions or discontinuities in the timestamps (the only indication of packet loss is longer-lasting payload content).

[0054] As described herein, the encoder 20 generates at least two payloads in preparation for each connection event. The PfA payload is used if the last transmitted packet was received by system 100, and the PfN payload is used if the last transmitted packet was not received by system 100. After each connection event, three possibilities may occur: (1) the controller 10 notifies the encoder 20 that a PfA packet was used, (2) the controller 10 notifies the encoder 20 that a PfN packet was used, or (3) no notification is made to the encoder 20 because the polling packet for the connection event was lost (such a case is called a "timeout" and means the absence of an event).

[0055] System 1 can store various integrated states, including any of the following: (1) a state buffer containing a history of the last integrated state (including high-density and low-density sections); (2) a last state containing the most recent integrated state (which may also contain the last state in the high-density buffer, but is clearly identified for readability); (3) a sent state containing the most recent state in which encoder 20 was notified to have been transmitted (which may also be lost); (4) a confirmed state containing the most recent state in which encoder 20 was confirmed to have been received by system 100; and / or (5) a lastPayload state containing the most recent state used for payload generation.

[0056] Various events related to System 1 can change these integrated states. One event may include the feeding of input samples, in which case new samples are integrated and a new integrated state is generated (the new state is pushed into the state buffer and the last state is set to this new state). Another event may include the generation of payloads, in which case both PfA and PfN payloads are generated and lastPayload is set to last. These two payloads are generated using last as state B and different state A according to the method described herein (for PfA packets, sent is used as state A, and for PfN packets, confirmed is used as state B). Another event may include the signaling of PfA, in which case confirmed is set to sent and sent is set to lastPayload. Yet another event may include the signaling of PfN, in which case sent is set to lastPayload. Note that no values ​​are updated during timeout events.

[0057] Note that the payload covers a range of states or samples, and system 1 must ensure that decoder 30 receives uninterrupted coverage (payloads can cover long durations if necessary, and each received payload must cover the gap since the last received sample / state). All generated payloads cover a range going back to the last state, and therefore, in any sequence of PfA, PfN, or timeout, encoder 20 can know that system 100 has acquired this state if the previously transmitted packet has been received (in the case of PfA). Thus, in new PfA generation, encoder 20 can use this state as state A. However, since the last state is likely to change by Feed() between payload generation and signaling, or between payload generation and new payload generation, the last state actually used for payload generation is stored as lastPayload to maintain reference. In the event of a timeout, no payload has been sent, and encoder 20 should behave as if the previous generation had never occurred, and no state is used as state A (this is why the sent state is set to lastPayload only when a PfA or PfN signal is received, confirming actual packet transmission).

[0058] When a PfA signal is sent to encoder 20 immediately after a connection event that transmits payload N, encoder 20 obtains "assurance" that system 100 has received a state leading to the last state of payload N-1. This state is called the transmit state and is guaranteed to be received, and all subsequent payloads must start from at least this state (this is why the confirmed state is set to the sent state value when the PfA signal is received). For PfN payloads, encoder 20 must then cover the gap by assuming that the last packet was lost, and cover the interval between the confirmed state and the last state.

[0059] Figures 6A and 6B are examples of non-restrictive block diagrams showing how the internal state of encoder 20 changes after at least three connection events 601–603. For each connection event 601–603, at least three cases (i.e., PfA, PfN, and timeout) are considered, leading to multiple final internal states. States 610–624 are identified by numbers that increase by one between connection events 601–603 (these can be considered state timestamp ODRs when the feed frequency of the input sample matches the frequency of the connection event). Each box representing states 610-624 includes at least the following items: (1) Buffer states representing all states present in the encoder 20's state buffer; (2) System 100 (e.g., console) states representing the encoder 20's knowledge of states received by System 100 (solid lines represent state intervals in which the encoder 20 is aware that it has been received by System 100, and dashed lines represent state intervals in which the encoder 20 has sent to System 100 but it is unclear whether it has been received); (3) PfA representing the contents of the generated PfA payload; and (4) PfN representing the contents of the generated PfN payload.

[0060] In one embodiment, when the encoder 20 enters each of the different connection states 601 to 603, different items may be generated for each case. For example, when system 1 enters connection state 602, the encoder 20 generates a payload PfA (the result of which is represented as state 613) and a payload PfN (the result of which is represented as state 614). Each of these payloads contains different data that provides recovery cases in case the last packet was transmitted successfully or failed. For example, in state 614, PfN may contain a value of length 4 (i.e., 0 to 3), and in state 613, PfN may contain a value of length 3 (i.e., 1 to 3). Different payloads PfA and PfN are generated by the encoder 20, and the appropriate payload (PfA or PfN) is sent to the decoder 30 based on knowledge of the state of the previously transmitted packet (e.g., known based on feedback information).

[0061] With the recovery system described herein, even if packet loss occurs, the decoder 30 should not perceive a data gap. This is because the first packet after the gap structurally covers the entire duration of the gap (this eliminates the need for the decoder 30 to perform passive recovery interpolation). In one embodiment, if the nominal connection event interval is 5ms, system 1 generates one sample per PfA payload to cover 5ms (pattern #1). If one or two packets are lost (or as predicted in the PfN payload), system 1 generates packets (patterns #1 and #2) containing two or three samples to cover longer durations. Even if many more packets are lost, the encoder 20 continues to generate three samples, but in pattern #3, each sample covers a longer duration to fill the gap. The longer samples result in more quantized and noisier signal quality, but have little effect on the integrated orientation.

[0062] Even in situations where feedback is impossible, or where packet loss occurs in the system 100 itself even when PfA / PfN is used correctly (e.g., overflow due to system scheduling problems caused by excessive CPU or I / O load), the decoder 30 is built robust against actual undetected packet loss. There can be at least two different cases of packet loss: (1) detected packet loss, where the packet loss is detected by the radio protocol or transport layer and a PfA or PfN signal is sent to the encoder 20 (in this case, there is no discontinuity in the timestamp); and (2) undetected packet loss, where the packet loss is not detected by the radio protocol or transport layer and a PfA signal is sent to the encoder 20, and the encoder 20 "believes" that the decoder 30 has received the payload, but the packet is actually lost (in this case, there is a discontinuity in the timestamp). A passive recovery process may be used to address the case of undetected packet loss.

[0063] It should be noted that the payload format may include at least two partially redundant pieces of information: (1) payload duration (ODR) and (2) end timestamp (ODR) (LSB only). In the nominal case, from the perspective of decoder 30, the new payload duration is equal to the new payload end timestamp minus the end timestamp of the previous payload: duration(N) = endTimestamps(N) - endTimestamps(N-1). If there is one or more undetected packet losses, this "duration(N)" will be less than "endTimestamps(N) - endTimestamps(N-1)", and part of the timestamp difference will be the duration of the packet gap.

[0064] If such a difference occurs, a recovery procedure may be initiated. During the recovery procedure, the decoder 30 stores the difference between the timestamp-based duration and the payload duration as recoveryDuration, and the decoder 30 continues to decode the payload as usual. When decoding the first sample of the payload, in order to obtain the angular velocity, the decoder 30 calculates the difference between the sample quaternion and the last orientation quaternion and divides it by the sample duration + recoveryDuration. This result provides an average velocity that covers both the gap and the actual first payload sample.

[0065] In order to maintain the orientation integral when the system user provides corrections, this angular velocity must be integrated over the entire duration (thus, before the nominal sample, an additional recovery sample with a duration matching recoveryDuration is generated using the calculated angular velocity). Since acceleration may not be as important, system 1 may not perform a complex pre-configured recovery process, and the recovery sample may simply use the same acceleration as the nominal sample. The output of the recovery sample is also flagged to allow system 100 to perform specific actions when using the recovery sample.

[0066] By using this recovery logic, System 1 can guarantee a very low orientation integration error even in the case of undetected packet loss. The recovery sample has a variable size to cover the gap with one sample, similar to conventional data recovery techniques. An exemplary scenario involves operation between a detected or undetected gap with a long packet loss gap of 20ms for a payload duration of 5ms at 1kHz ODR. In a conventional recovery system, encoder 20 generates a 25ms payload containing three samples. Encoder 20 attempts to distribute them evenly, resulting in three samples with durations of 9ms, 8ms, and 8ms. Decoder 30 outputs three samples maintaining these durations. In the recovery system described herein, after the gap, decoder 30 receives a 5ms payload with a 20ms mismatch. Decoder 30 outputs a 20ms recovery sample and a 5ms nominal sample. Since both samples have the same angular velocity and acceleration, the available trajectory is less detailed than in a conventional recovery system.

[0067] System 1 may include other additional advanced features. For example, System 1 may include adaptation of the connection event frequency. Conventional controller connection event frequencies are not fixed in time depending on specific parameters (e.g., the number of connected controllers, Wi-Fi usage). System 100 can switch between different frequencies at any time, with no other instructions other than the change in the frequency of the connection event pole packets. Typically, in the nominal case, the connection event interval is 5ms (200Hz), but it can be switched to 10ms or 15ms. The architecture of System 1 can easily handle these frequency changes (they are automatically handled as "timeout" cases).

[0068] PfA and PfN payload generation can be maintained every 5ms, even if the connection event interval is switched to 10ms or 15ms. In this case, the generated payload is not always used when the next payload generation occurs. Since there is no usage and no PfA or PfN signal, the internal changes of encoder 20 are the same as before (only the lastPayload state is used, but it is not used as a payload generation input), and thus it is equivalent to calling payload generation on encoder 20 after 10ms or 15ms (as if no intermediate unused payloads were generated at all). If the firmware of controller 10 has information on the change in connection event frequency, payload generation can be scheduled less frequently to save CPU usage, but the payloads received from decoder 30 will be the same. This may also mean that system 1 can seamlessly handle any combination of pole packet loss and connection events, or can support new connection event frequencies without modification.

[0069] System 1 may also include certain improvements related to quantization compensation. In particular, the integration of encoder states is performed using high-resolution values ​​(e.g., 32 bits per component) to enable low calculation errors. However, except for pattern #0, it is not possible to transmit values ​​at high resolution, and since they are encoded into values ​​with fewer bits per component within the payload, quantization errors occur. Without special processing, these quantization errors during the encoding process are translated into small errors in the decoded output angular velocity and acceleration. These errors generate integral drift, such as yaw drift, in addition to drift due to sensor noise and errors.

[0070] To minimize these errors, each time the encoder 20 generates a sample from state A to state B, it decodes the encoded quantized value and integrates state A with respect to the quantized angular velocity and acceleration. The encoder 20 gives a new state B' (very close to state B, but representing what the decoder actually uses). For the next sample, the encoder 20 uses state B' as the new state A, partially correcting the error and preventing drift by encoding the difference from the state containing the quantization error, rather than state B. Figure 10 provides a simple diagram illustrating the difference with and without quantization compensation, showing an example where the signal for one axis is constant at 3.22 units and the quantized LSB is 1 unit. Without quantization compensation, the decoder 30 accumulates an error of 2 units every 9 samples and generates only 3-unit samples. With quantization compensation, the difference is calculated from the decoder's integral estimate, so the error is corrected when it reaches 1 LSB.

[0071] Regarding angular velocity, a quaternion is sent to the first sample of the payload, allowing for correction of quantization errors, so the importance of correction is not very high, but system 1 benefits from obtaining better intermediate angular velocity samples and acceleration integrals for samples 2 and 3. Quantization compensation interacts to some extent with the conventional recovery system. To avoid accumulation errors, lastPayload is set not to last, but to last', which is calculated by the integral of the quantized encoded samples. Since two payloads are generated, even though both PfA and PfN payloads target the last state, the encoded samples are slightly different, generating two different last' states (one for the PfA payload and one for the PfN payload). Since encoder 20 cannot immediately know which payload is used, the two states (lastPayloadIfPfA and lastPayloadIfPfN) are actually stored by encoder 20 in order to set lastPayload to last'. The state change logic includes the following: (1) In payload generation, both PfA and PfN payloads are generated (for PfA packets, lastPayloadIfPfA is set to last', and for PfN packets, lastPayloadIfPfN is set to last'), (2) In the PfA signal, confirmed is set to sent, and sent is set to lastPayloadIfPfA, (3) In the PfN signal, sent is set to lastPayloadIfPfN.

[0072] It should be noted that System 1 may be configured in a high-density range. Due to the limited available bandwidth within the payload, encoder 20 must quantize the samples, resulting in a quality degradation for payload patterns #1, #2, or #3 (for payload pattern #0, there is only one sample to encode, so the number of bits per component is very high, and there is no quality degradation compared to the internal state). The quality of quantization depends on at least two parameters: (1) the number of bits per component limited by the allocated payload size within the packet, and (2) the encoding range or maximum encoded value selected by the desired maximum amplitude.

[0073] Quantization quality may be expressed as 2 raised to the power of bitsPerComponents divided by the encoding range. For example, switching angular acceleration from 2000 / s to 200 / s with the same number of bits improves signal quality tenfold. Sensors often support wide amplitudes for angular velocity and acceleration, which users may reach during certain gameplay. Expected amplitudes are often as follows: maximum angular velocity: ±2000 / s, maximum acceleration: ±16G. For the vast majority of times, the IMU signal may have much smaller values ​​(e.g., around 200 / s or less for small movements, plus about ±1G in addition to 1G acceleration in any direction, thus an envelope of 2G).

[0074] To improve quality most of the time, System 1 uses high-density range flags for patterns #1 to #3 within the payload data. In the configuration of System 1, the developer can configure two ranges: (a) maximum range and (b) high-density range. For example, recommended values ​​for angular velocity may include a maximum range of 2000 / s and a high-density range of 250 / s (i.e., 8 times smaller). Recommended values ​​for acceleration may include a maximum range of 16G and a high-density range of 2G (i.e., 8 times smaller). When payload generation is requested, encoder 20 performs a first pass to calculate approximations of angular velocity and acceleration for two or three samples of the future payload. If all angular velocities are less than 95% of the high-density range, the angular velocity high-density range use flag is set to 1 (otherwise set to 0). If all accelerations are less than 95% of the high-density range, the acceleration high-density range use flag is set to 1 (otherwise set to 0).

[0075] The two flags are written as one bit each in the payload, and the actual sample coding is performed using the maximum range or high-density range depending on the state of each flag. The decoder 30 may start by reading the flags in the received payload to determine the range to use for sample decoding. Using the recommended range as an example, this simple compression improves the quality of angular velocity by eight times and the quality of acceleration by four times in most cases. To achieve an equivalent effect in quality, an additional 15 bits would be required to increase the number of bits per component, but this method requires only 2 bits.

[0076] It should be noted that System 1 may include certain improvements regarding clock compensation. In particular, although the IMU chip generates samples at regular intervals, this regularity is not perfect. In some cases, the rate of timestamp increment may be slightly slower or faster than the timestamp increment rate of System 100 (this rate difference can reach ±5%). In other cases, the rate difference may change over time due to temperature changes or other reasons. One solution is to assume that the clock is accurate and maintain the output timestamp / duration at a theoretical value, but this presents at least two problems: (1) the output sample duration is slightly inaccurate, introducing bias errors into the output signal, and (2) the output sample timestamp drifts from the clock of System 100, making it unusable for developers to immediately synchronize the IMU sensor with other sensors.

[0077] System 1 can integrate a clock correction system for outputting samples with the following objectives: (1) a new sample timestamp is always equal to "previous sample timestamp + new sample duration" (no discontinuity in duration integral), (2) the sample timestamp is close to the real-time timestamp with relatively fixed latency and does not drift over time, and (3) the sample duration matches the real-time interval represented by the sample during acquisition (no rapid clock correction due to large changes in duration, no sensitivity to processing scheduling). To achieve these objectives, the clock correction of System 1 can maintain the following states: payloadTimestampUs (system 100 timestamp of the current payload feed in microseconds), previousPayloadTimestampUs (system 100 timestamp of the previous payload feed), previousOutputSampleTimestampUs (timestamp of the last output sample), previousTargetTimestampUs (correction timestamp target of the last output sample), timestampOdrSum (ODR time accumulator), timestampPayloadSum (system 100 time accumulator), pendingCorrection, pendingCorrectionAge (for verification), activeTimestampRatio, pendingTimestampRatio, and / or pendingTimestampRatioAge (for verification). These states are updated at different times.When the first payload is received, the state is initialized as follows: (1) previousPayloadTimestampUs, previousOutputSampleTimestampUs, and previousTargetTimestampUs are set to payloadTimestampUs - the current payload duration (ODR) converted using a theoretical ODR value (e.g., 1250 μs per unit (ODR)) that can simulate a virtual previous packet; (2) timestampOdrSum is set to a large value (e.g., 1000) to stabilize the initial timestamp ratio; (3) timestampPayloadSum.

[0078] It should be noted that System 1 can encode the IMU sensor temperature within the packet. Since temperature is low-frequency data and not necessarily critical, only one 16-bit temperature (-103 to 153°C, 0.004 / LSB) is used, and only for payload patterns #0, #1, or #2 (#3 is typically used only for packet gaps where 16 bits are used to improve signal quality to compensate for longer payload durations). The encoded temperature is the average temperature of all samples between state A and state B used to generate the payload. Since the history is low-density and some states between state A and state B may be unavailable, the state stores a temperature accumulator rather than the actual temperature (for each input sample, the sample temperature is added to the temperature accumulator of the previous state to obtain the accumulator value for the new state). The accumulator is a 32-bit integer accumulator and is safely overflow-safe as it stores the 16-bit encoded temperature. The average temperature is calculated by dividing the temperature accumulator difference by the payload duration.

[0079] It should also be noted that input samples may correspond to data acquired during IMU sensor saturation. This type of saturation data does not represent the actual motion and can cause problems for the integrator. The output samples of decoder 30 have two Boolean flags indicating whether saturation angular velocity values ​​or saturation acceleration values ​​were used to generate these output samples. To transmit this information, encoder 20 is supplied with input samples containing the Boolean saturation flag. Similar to temperature, the state of encoder 20 stored in the state buffer or other stored states includes two accumulators: (1) saturation angular velocity count and (2) saturation acceleration count. These counters are incremented by 1 during state integration if the corresponding flag is set in the input sample. During payload generation, encoder 20 compares the counters of state A and state B used for payload generation and sets the corresponding saturation flag in the payload header if the saturation counts are different. Decoder 30 reads the flag in the payload header and applies it to all output samples generated in this payload. While this might flag output samples as oversaturated, over-flagging isn't a major issue, and using 2 bits per sample is too expensive compared to the gain obtained.

[0080] System 1 may face processed limitations such as input sample saturation, long-duration payloads, corrupted payloads, and duplicate or inverted payloads, while limitations such as long undetected gaps may not be processed. Regarding input sample saturation, encoder 20 can process samples with values ​​higher than the configured range. If this continues for a long time, the acceleration accumulator may overflow, potentially causing a saturated output as a configured value (however, this issue can be addressed by quantization compensation). Note that long-duration packets will be processed by the ts loop estimator even after 2 seconds. Also, if a corrupted payload is meaningful, it may not crash and may produce a bad output (meaningless payloads are discarded). Duplicate or inverted payloads are discarded, and in the case of inverted payloads, a recovery sample is generated. Undetected gaps exceeding a duration of 4096 ODR will lead to unexpected results, but note that this should exceed the 2-second radio protocol timeout.

[0081] The System 1 API can have two separate and independent aspects: an API for encoder 20 and an API for decoder 30. In each aspect, the API consists of a software class (e.g., a C++ class) for instantiation. Encoder 20 and decoder 30 share similar lifecycle-related methods: void Initialize(Config config)—initializes the internals of encoder 20 or decoder 30 (receiving configuration as a parameter); bool IsInitialized()—checks whether the class has been initialized; and void Finalize()—destroys the internals. Encoder 20 or decoder 30 must be initialized before any other function calls.

[0082] Note that the Config structure may be identical for encoder 20 and decoder 30. Encoder 20 and decoder 30 must be initialized with the same values ​​(otherwise, output data quality cannot be guaranteed). These values ​​include at least: uint32_t timestampOdrToUs—the duration of the input sample for encoder 20 in microseconds (e.g., 1042 μs per sample for a 960 Hz ODR sensor); float accelerationRange—the maximum expected acceleration for the input sample (m / s²). 2 )(Samples may actually have higher acceleration, but the transmitted signal is "clamped"); float angularVelocityRange—Maximum expected angular velocity (radians / sec) for input sample (Samples may actually have higher angular velocity, but the transmitted signal is "clamped"); float highDensityAccelerationRange—Acceleration threshold for acceleration high density mode; and float highDensityAngularVelocityRange—Angular velocity threshold for angular velocity mode.

[0083] Encoder 20 may include at least the following values: void Feed(InputSample const& sample) - supplies one input sample to encoder 20; void GeneratePayloads(uint8_t* pfaPacket,uint8_t* pfnPacket,size_t payloadPfAMaxSize,size_t payloadPfNMaxSize,size_t* pfaPacketSize,size_t* pfnPacketSize) - generates output payloads for the delayed decision cases of PfA and PfN; void GenerateNotOredPayload(uint8_t* payload,size_t payloadMaxSize,size_t* payloadSize) - generates output payloads for the use case without feedback; void SignalPfA() - notifies that the PfA payload has been used; and void SignalPfN() - notifies that the PfN payload has been used. The input sample structure may include at least the following data: uint3 timestampOdr - sequence number of the sample; float acceleration[3] - m / s 2Acceleration in units; float angularVelocity[3]—angular velocity in units of rad / s-1; float temperature—temperature in degrees Celsius; bool saturatedAcceleration—true if the sample's acceleration has reached saturation; and bool saturatedAngularVelocity—true if the sample's angular velocity has reached saturation. The decoder 30 may include at least the following values: size_t Feed(uint8_t const* payload,size_t packetSize,uint64_t packetTimestampUs)—supplies the transmitted payload to the decoder 30; size_t GetPayloadUsedSize(uint8_t const* payload,size_t packetSize)—decodes only a portion of the header to calculate the effective payload size in the buffer; size_t GetAvailableOutputSampleCount()—returns the number of output samples in the output queue; and size_t GetSamples(OutputSixAxisSample* samples,size_t maxSampleCount)—reads and processes samples from the output queue. These examples are, of course, non-limiting, and the techniques described herein assume any kind of data and input values ​​related to the encoder 20 and / or decoder 30.

[0084] Figure 7 shows an example of a non-limiting block diagram of the hardware architecture of system 1260. In the example shown in Figure 7, the encoder device 1210 communicates with the decoder device 1200 via a communication link 1240. The communication link 1240 may include a peer-to-peer connection between the encoder device 1210 and the decoder device 1200. In one embodiment, the communication link 1240 may include a short-range wireless communication link using a communication protocol such as Bluetooth®. This example is non-limiting, and the communication link 1240 may include any other form of communication between the decoder device 1200 and the encoder device 1210. For example, the communication link 1240 may include a network of interconnected computing devices such as the Internet. The communication link 1240 may constitute a local area network (LAN). As described below, the hardware elements shown in Figure 7 may be used to implement various software components and operations described as being included in or running in the encoder device 1210 and the decoder device 1200.

[0085] In one embodiment, the encoder device 1210 (also referred to herein as the “encoder system”) may include any or more of the following: one or more processors 1212; one or more memory devices 1214; one or more network interface devices 1216; and one or more user input adapters 1220. As described below, these elements (e.g., processors 1212, memory devices 1214, network interface devices 1216, user input adapters 1220) are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions for the encoder device 1210.

[0086] In some embodiments, each or any of the processors 1212 may include, for example, a single-core or multi-core processor, a microprocessor (for example, sometimes called a central processing unit or CPU), a digital signal processor (DSP), a microprocessor combined with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system-on-a-chip (SOC) (for example, an integrated circuit including a CPU and memory, network interfaces, and other hardware components). Alternatively, in some embodiments, each or any of the processors 1212 may use an instruction set architecture such as x86 or Advanced RISC Machine (ARM).

[0087] In one embodiment, each or any of the memory devices 1214 may include random access memory (RAM) (e.g., dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (e.g., based on NAND or NOR technology), hard disk, magneto-optical medium, optical medium, cache memory, registers (e.g., holding instructions), or other types of devices that store data and / or instructions (e.g., on or executed by the processor 1212) whether volatile or nonvolatile. Memory devices 1214 are an example of a nonvolatile computer-readable storage medium.

[0088] In one embodiment, each or any of the network interface devices 1216 may include one or more circuits (e.g., a baseband processor and / or a wired or wireless transceiver) and implement Layer 1, Layer 2, and / or higher layers for one or more wired communication technologies (e.g., Ethernet® (IEEE 802.3)) and / or wireless communication technologies (e.g., Bluetooth®, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), and / or other short-range, medium-range, and / or long-range wireless communication technologies). The transceiver may include circuits for a transmitter and a receiver. The transmitter and receiver may share a common housing and may share some or all of the circuits within the housing to perform transmission and reception. In one embodiment, the transmitter and receiver of the transceiver may not share a common circuit and may reside in the same or separate housings.

[0089] In one embodiment, each or any of the user input adapters 1220 may include one or more circuits for receiving and processing user input data from one or more user input devices (not shown in Figure 7) that are included in, attached to, or otherwise communicate with the encoder device 1210, and output data to the processor 1212 based on the received input data. Alternatively, in one embodiment, each or any of the user input adapters 1220 may include, for example, a PS / 2 interface, a USB interface, a touchscreen controller, etc., or the user input adapters 1220 may enable input from user input devices (not shown in Figure 7) such as a keyboard, mouse, trackpad, touchscreen, etc.

[0090] The user input adapter 1220 may include various input elements, although not particularly limited, such as inertial sensors (e.g., accelerometers, gyroscopes, compasses). In one embodiment, the user input adapter 1220 may include an acceleration sensor that detects the acceleration of the encoder device 1210 (including gravitational acceleration), i.e., the force (including gravity) applied to the device body 1210. The acceleration sensor is capable of detecting the value of linear acceleration along the detection axis direction (linear acceleration) in the acceleration applied to the detection unit of the acceleration sensor. The acceleration sensor is capable of detecting the magnitude of linear acceleration along a predetermined three axis direction. As the acceleration sensor, a three-axis acceleration sensor may be used, a combination of a two-axis acceleration sensor and a one-axis acceleration sensor may be used, or three one-axis acceleration sensors may be used. For example, in the case of a multi-axis acceleration sensor having two or more axes, the acceleration components along each axis are detected as acceleration applied to the detection unit of the acceleration sensor. The acceleration sensor may also detect acceleration in one axis direction or two axis directions.

[0091] The user input adapter 1220 may further include an angular velocity sensor configured to detect angular velocity around three axes and output data (e.g., angular velocity data) indicating the detected angular velocity to the processor 1212. The angular velocity sensor may be configured using a three-axis gyro sensor or a combination of two or fewer gyro sensors. The angular velocity sensor detects angular velocity (per unit time) around the x-axis, angular velocity (per unit time) around the y-axis, and angular velocity (per unit time) around the z-axis. The rotation directions around the x, y, and z axes are referred to as the roll direction, pitch direction, and yaw direction, respectively, based on the positive z-axis direction of the encoder device 1210. Accelerometers or gyro sensors are generally referred to as inertial sensors. Note that the encoder device 1210 can also output various other information, such as temperature data, which provides temperature values ​​using a temperature sensor.

[0092] In one embodiment, the encoder device 1210 may include one, two, three, four, or more of each or any of the above elements (e.g., the processor 1212, the memory device 1214, the network interface device 1216, and the user input adapter 1220). Alternatively, in one embodiment, the encoder device 1210 may include one or more of the following: a processing system including the processor 1212; a memory or storage system including the memory device 1214; and a network interface system including the network interface device 1216.

[0093] The encoder device 1210 may be configured in a variety of ways in various embodiments. As one example, the encoder device 1210 may be configured such that the processor 1212 includes: a multi-core (or single-core) processor; a first network interface device (e.g., implementing WiFi, Bluetooth®, NFC, etc.); a second network interface device implementing one or more cellular communication technologies (e.g., 3G, 4G LTE, CDMA, etc.); and a memory or storage device (e.g., RAM, flash memory, or hard disk). The processor, the first network interface device, the second network interface device, and the memory device may be integrated as part of the same SOC (e.g., a single integrated circuit chip). As another example, the encoder device 1210 may be configured as follows: the processor 1212 includes two, three, four, five or more multi-core processors; the network interface device 1216 includes a first network interface device implementing Ethernet® and a second network interface device implementing WiFi and / or Bluetooth®; and the memory device 1214 includes RAM and flash memory or a hard disk.

[0094] The decoder device 1200 also includes various hardware components used to implement the software elements of the decoder device. In one embodiment, the decoder device 1200 (also referred to herein as the “decoder device”) may include any or more of the following: one or more processors 1202; one or more memory devices 1204; and one or more network interface devices 1206. As described below, these elements (e.g., processors 1202, memory devices 1204, network interface devices 1206) are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions for the decoder device 1200.

[0095] In some embodiments, each or any of the processors 1202 may include, for example, a single-core or multi-core processor, a microprocessor (which may be called, for example, a central processing unit or CPU), a digital signal processor (DSP), a microprocessor combined with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system-on-a-chip (SOC) (for example, an integrated circuit including a CPU and memory, network interfaces, and other hardware components). Alternatively, in some embodiments, each or any of the processors 1202 may use an instruction set architecture such as x86 or Advanced RISC Machine (ARM).

[0096] In one embodiment, each or any of the memory devices 1204 may include random access memory (RAM) (e.g., dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (e.g., based on NAND or NOR technology), hard disk, magneto-optical medium, optical medium, cache memory, registers (e.g., holding instructions), or other types of devices that store data and / or instructions (e.g., on or executed by the processor 1202) whether volatile or nonvolatile. Memory devices 1204 are an example of a nonvolatile computer-readable storage medium.

[0097] In one embodiment, each or any of the network interface devices 1206 may include one or more circuits (e.g., a baseband processor and / or a wired or wireless transceiver) and implement Layer 1, Layer 2, and / or higher layers for one or more wired communication technologies (e.g., Ethernet® (IEEE 802.3)) and / or wireless communication technologies (e.g., Bluetooth®, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), and / or other short-range, medium-range, and / or long-range wireless communication technologies). The transceiver may include circuits for a transmitter and a receiver. The transmitter and receiver may share a common housing and may share some or all of the circuits within the housing to perform transmission and reception. In one embodiment, the transmitter and receiver of the transceiver may not share a common circuit and may reside in the same or separate housings.

[0098] In various embodiments, the decoder device 1200 may include one, two, three, four, or more of each or any of the above elements (e.g., the processor 1202, the memory device 1204, the network interface device 1206). Alternatively, in some embodiments, the decoder device 1200 may include one or more of the following: a processing system including the processor 1202; a memory or storage system including the memory device 1204; and a network interface system including the network interface device 1206.

[0099] The decoder device 1200 may be configured in a variety of ways in various embodiments. As one example, the decoder device 1200 may be configured such that the processor 1202 includes: a multi-core (or single-core) processor; a first network interface device (e.g., implementing WiFi, Bluetooth®, NFC, etc.); a second network interface device implementing one or more cellular communication technologies (e.g., 3G, 4G LTE, CDMA, etc.); and a memory or storage device (e.g., RAM, flash memory, or hard disk). The processor, the first network interface device, the second network interface device, and the memory device may be integrated as part of the same SOC (e.g., a single integrated circuit chip). As another example, the decoder device 1200 may be configured as follows: the processor 1202 includes two, three, four, five or more multi-core processors; the network interface device 1206 includes a first network interface device implementing Ethernet® and a second network interface device implementing WiFi and / or Bluetooth®; and the memory device 1204 includes RAM and flash memory or a hard disk.

[0100] As stated above, wherever it is stated in this specification that a software module or software process performs some operation, that operation is actually performed by the underlying hardware elements in accordance with the instructions that constitute the software module. Accordingly, in various embodiments, each or any combination of the encoder device 1210 or decoder device 1200 (referred to individually as “components” in the following paragraphs for clarity) is implemented using the example of encoder device 1210 or decoder device 1200 shown in Figure 7.

[0101] In such embodiments, the following applies to each component: (a) the elements of the encoder device 1210 shown in Figure 7 (i.e., one or more processors 1212, one or more memory devices 1214, one or more network interface devices 1216, and one or more user input adapters 1220) and the elements of the decoder device 1200 (i.e., one or more processors 1202, one or more memory devices 1204, one or more network interface devices 1206), or any suitable combination or subset thereof, are configured, adapted, or programmed to implement each or any combination of the operations, activities, or functions described as being performed by any software module described as being contained within the component. (b) or, additionally, to the extent that it is stated herein that one or more software modules are present in a component, in a given embodiment, such software modules (and any data that is stated herein to be processed and / or used by such software modules) are stored in their respective memory devices (e.g., in a given embodiment, a volatile memory device such as RAM or an instruction register, or a non-volatile memory device such as flash memory or a hard disk), and all operations that are stated herein to be performed by such software modules are performed by their respective processors, in conjunction with other elements in or connected to the encoder device 1210 or decoder device 1200, as necessary;(c) Or or additionally, to the extent that the component processes and / or otherwise handles data as described herein, in some embodiments, such data is stored in the respective memory device (e.g., in some embodiments, a volatile memory device such as RAM or a non-volatile memory device such as flash memory or a hard disk), or processed / handled by the respective processor in conjunction with the encoder device 1210 or decoder device 1200 or other elements connected thereto, as necessary; (d) Or or additionally, in some embodiments, the respective memory device may, when executed by the processor, store instructions that cause the encoder device 1210 or decoder device 1200 or other elements connected thereto to perform each or any operation performed by the component and / or any software module described as being contained within the component;

[0102] The hardware configuration shown in Figure 7 and described above is provided as an example, and the subject matter described herein may be used in combination with various different hardware architectures and elements. For example, while individual function / operation blocks are shown in many of the figures in this book, in various embodiments the functions of these blocks may be implemented by (a) using individual hardware circuits, (b) using application-specific integrated circuits (ASICs) specifically configured to perform the described functions / operations, (c) using one or more digital signal processors (DSPs) specifically configured to perform the described functions / operations, (d) using the hardware configuration described above in relation to Figure 7, (e) via other hardware arrangements, architectures, and configurations, or via a combination of the techniques described in (a) to (e).

[0103] In many places in this book, software modules and the actions performed by them are described. This is done for the sake of ease of description, and it should be understood that where it is stated in this book that a software module performs some action, that action is actually performed by the underlying hardware components (e.g., the processor and memory) according to the instructions and data that make up the software module. Technical advantages

[0104] The technology described herein provides an improvement over existing technologies for performing input processing between devices. In particular, the technology utilizes a system for encoding and decoding data (e.g., in a wireless communication process) between a first device (e.g., a controller) and a second device (e.g., a game system). In one embodiment, the system employs both active and / or passive recovery processes for packet loss management. During the recovery process, the encoder generates at least two different payloads and transmits one of the two payloads based on whether the previously transmitted packet was received by the decoder (using feedback information). In this way, the technology advantageously improves communication processes using conventional wireless communication protocols, achieving little to no data loss between devices. Selected definition

[0105] In this specification, when an item is described as existing in expressions such as “some embodiments,” “various embodiments,” “specific embodiments,” “specific exemplary embodiments,” “some exemplary embodiments,” or “exemplary embodiments,” it should be understood that the item exists in at least one embodiment, but not necessarily in all embodiments. Similarly, in this specification, when an operation is described as “possible,” “can be performed,” or “may be performed,” when a feature, element, or component is described as “may be included,” or “may be applicable,” when an item is described as “may have an attribute,” or when expressions such as “possible,” “can be,” or “may be possible” are used, it should be understood that the operation, feature, element, component, attribute, etc., exists in at least one embodiment, but not necessarily in all embodiments. The terms and expressions used herein, and their variations thereof, should be interpreted openly and not restrictively unless expressly limited. As examples of this: “and / or” includes any combination of one or more related enumeration items (for example, “a and / or b” means a, b, or a and b); the singular forms “a,” “an,” and “the” should be interpreted as meaning “at least one,” “one or more,” etc.; the term “examples” provides examples of the subject being discussed and is not an exhaustive or restrictive list; “include” and “inclusive” (and their conjugations and variations) indicate the presence of related enumeration items but do not exclude the presence or addition of other items; if an item is described as “optional,” it does not mean that other items are also optional.

[0106] As used herein, the term “non-temporary computer-readable storage medium” includes registers, cache memory, ROM, semiconductor memory devices (e.g., D-RAM, S-RAM, and other RAMs), magnetic media such as flash memory, hard disks, magneto-optical media, optical media such as CD-ROMs, DVDs, and Blu-ray discs, or other devices for non-temporary electronic data storage. The term “non-temporary computer-readable storage medium” does not include temporary and propagating electromagnetic signals.

[0107] Even if process steps, algorithms, or similar elements are described or requested in a particular order, including when described or requested with reference to drawings, the process may be configured to operate in a different order. In other words, the order or arrangement of steps expressly described or requested herein does not indicate that the steps must be performed in that order; rather, the steps of the processes described herein may be performed in any possible order. Furthermore, even if a step is described or suggested to occur non-simultaneously because it is described after another step, the steps may be performed simultaneously (or in parallel). In addition, the illustration of a process with drawings does not mean that the illustrated process excludes other variations and modifications, nor does it mean that the illustrated process or any of its steps are necessary, nor does it mean that the illustrated process is preferred.

[0108] While various embodiments are shown and described in detail, the claims are not limited to any particular embodiment or example. Nothing described above should be read to suggest that any particular element, step, scope, or function is essential. All structural and functional equivalents to the elements of the embodiments described above, known to those skilled in the art, are expressly incorporated and intended to be included herein. Furthermore, a device or method is not required to be included in the invention if it does not address all the problems that the invention seeks to solve. Nothing described herein is intended to be exclusive to the public.

Claims

1. It is a system, A first device having a processing circuit including at least one processor and an inertial sensing circuit; and The device includes a second device having a processing circuit that includes at least one processor, The first device described above is configured to perform the following: To generate an input feed that includes multiple input data, wherein the input data includes at least inertial sensing data generated by the inertial sensing circuit; Integrating the aforementioned input feed into the encoder state and pushing the encoder state into the state buffer; Obtaining feedback information based on the communication status between the first device and the second device; and To generate payload data based on the state information in the state buffer and the feedback information, The second device described above is configured to perform the following: Polling the first device and obtaining the payload data; Decoding the aforementioned payload data to generate output data; and The aforementioned output data is to be used for game processing.

2. A system according to claim 1, wherein the second device is configured to reconstruct input data in the payload data when a gap occurs in transmission between the first device and the second device.

3. The system according to claim 1, wherein the input feed is generated by the first device for a first time duration.

4. The system according to claim 3, wherein the payload data is generated by the first device for a second time duration that is longer than the first time duration.

5. The system according to claim 1, wherein the payload data includes a first payload relating to the last transmitted payload received by the second device.

6. The system according to claim 5, wherein the payload data includes a second payload relating to the last transmitted payload that was not received by the second device.

7. The system according to claim 1, wherein the encoder state is pushed to a high-density state buffer covering a first time sample, and the encoder state is pushed to a low-density state buffer covering a second time sample.

8. The system according to claim 1, wherein the plurality of input data include any of the following: timestamp data, angular velocity data, acceleration data, temperature data, and metadata.

9. The system according to claim 1, wherein the generated output data includes any of the following: timestamp and duration data, sample index data, angular velocity data, acceleration data, temperature data, and metadata.

10. A non-temporary computer-readable storage medium configured to store computer-readable instructions configured to provide the following execution when executed by the processor of an information processing device: The method involves generating an input feed that includes multiple input data, wherein the input data includes inertial sensing data generated by at least a first input device's inertial sensing circuit; Integrating the aforementioned input feed into the encoder state and pushing the encoder state into the state buffer; Obtaining feedback information based on the communication status between the first device and the second device; and To generate payload data based on the state information in the state buffer and the feedback information.

11. A non-temporary computer-readable storage medium according to claim 10, wherein the second device is configured to reconstruct input data in the payload data when a gap occurs in transmission between the first device and the second device.

12. A non-temporary computer-readable storage medium according to claim 10, wherein the input feed is a non-temporary computer-readable storage medium generated in a first time duration.

13. A non-temporary computer-readable storage medium according to claim 12, wherein the payload data is generated in a non-temporary computer-readable storage medium with a second time duration longer than the first time duration.

14. A non-temporary computer-readable storage medium according to claim 10, wherein the payload data includes a first payload relating to the last transmitted payload received by the second device.

15. A non-temporary computer-readable storage medium according to claim 14, wherein the payload data includes a second payload relating to the last transmitted payload that was not received by the second device.

16. A method for encoding data, including the following: The method involves generating an input feed that includes multiple input data, wherein the input data includes inertial sensing data generated by at least a first input device's inertial sensing circuit; Integrating the aforementioned input feed into the encoder state and pushing the encoder state into the state buffer; Obtaining feedback information based on the communication status; and To generate payload data based on the state information in the state buffer and the feedback information.

17. A method according to claim 16, wherein the second device is configured to reconstruct input data in the payload data when a gap occurs in the transmission between the first device and the second device.

18. A method according to claim 16, wherein the input feed is generated over a first time duration.

19. A method according to claim 18, wherein the payload data is generated over a second time duration that is longer than the first time duration.

20. A method according to claim 16, wherein the payload data includes a first payload relating to the last transmitted payload received by a second device.

Images