Cloud game interaction method and system based on voice recognition

Abstract

This invention discloses a cloud gaming interaction method and system based on speech recognition, belonging to the field of speech recognition and interaction technology. The method includes: Step 1, predefining multiple discrete non-semantic interaction commands, and pre-setting an acoustic fingerprint template for each interaction command based on the formant position, energy envelope shape, and duration on the time-frequency spectrum; Step 2, on the client side, matching the real-time audio stream with the acoustic fingerprint template, and outputting the corresponding interaction command identifier when the matching degree exceeds a predetermined threshold; Step 3, sequentially combining multiple interaction commands within a preset time window. This method enables acoustic fingerprint matching and command stream recognition of voice commands to be completed on the client side, avoiding the two network round trips between the original audio and the command, as well as the serial processing in the cloud, in traditional architectures. This significantly reduces end-to-end response latency, achieving real-time interaction comparable to physical peripherals, effectively adapting to the high real-time control requirements of cloud gaming, and improving user responsiveness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition and interaction, and more particularly to a cloud game interaction method and system based on speech recognition. BACKGROUND

[0002] With the rapid development of cloud computing and network technology, the computing load of application programs is increasingly migrated to cloud servers, forming a distributed computing paradigm in which the client is responsible for input and output and the server is responsible for core computing. Under this paradigm, building an efficient and natural human-computer interaction channel to reduce end-to-end system response delay is the key to improving the user experience of distributed applications. Traditional human-computer interaction mainly relies on physical peripherals such as keyboards, mice, and touch screens, which have explicit input signals and extremely low local processing delay. In order to provide a more free and multi-dimensional interaction experience, speech recognition technology is introduced as a natural interaction means, with the ideal vision of allowing users to directly control application programs or operating systems through voice commands, thereby freeing up both hands and enriching the interaction dimension.

[0003] However, in the classic client and cloud architecture, implementing voice interaction usually follows a serial data processing key path: the client collects raw audio data, which is uploaded to the cloud server through the network; the server needs to sequentially perform a series of computationally intensive tasks such as endpoint detection, automatic speech recognition, and natural language understanding, and finally maps the parsed semantic intent to a predefined control instruction. This processing link has significant defects: first, the ASR and NLU module algorithms are complex and time-consuming, constituting an inherent processing delay bottleneck; second, the two-way network transmission of raw audio and control instructions further introduces a non-negligible network delay; the superposition of the two results in end-to-end response delay, making it difficult to meet the needs of high real-time interaction scenarios such as real-time control, dynamic interface operation, and instant feedback applications. Although marginal improvements can be achieved by optimizing a single algorithm or improving hardware computing power, the inherent delay introduced by the fixed data processing sequence and remote computing architecture cannot be eliminated.

[0004] Based on this, the present application proposes a cloud game interaction method and system based on speech recognition, which can achieve low-delay voice command interaction comparable to local physical peripherals in a distributed computing environment. SUMMARY

[0005] In view of the problems existing in the prior art, the purpose of the present application is to provide a cloud game interaction method and system based on speech recognition, which can achieve acoustic fingerprint matching and voice command stream recognition on the client, avoiding the two-way network transmission of raw audio and instructions and the cloud serial processing in the traditional architecture, greatly reducing the end-to-end response delay, achieving real-time interaction comparable to physical peripherals, effectively adapting to the high real-time control requirements of cloud games, and improving the user operation hand tracking degree.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] Firstly, the cloud gaming interaction method based on speech recognition, in a client-server computing environment, includes the following steps:

[0008] Step 1: Predefine multiple discrete non-semantic interactive tone commands, and pre-set an acoustic fingerprint template for each interactive tone command based on the formant position, energy envelope shape and duration on the time spectrum;

[0009] Step 2: On the client side, the real-time audio stream is matched with the acoustic fingerprint template. When the matching degree exceeds a predetermined threshold, the corresponding interactive voice command identifier is output.

[0010] Step 3: Combine multiple interactive voice commands sequentially within a preset time window to form a voice command stream, and pre-compile the voice command stream pattern library; temporarily store the interactive voice command identifiers and compare them with the pre-compiled voice command stream pattern library in real time to identify the voice command stream code;

[0011] Step 4: Synchronize the interactive audio commands, audio command stream mode library and the addresses of the in-application functions mapped to them, as well as the application state from the server.

[0012] Step 5: Validate the audio stream code based on the application state, and package the valid audio stream code, timestamp, and user identity into a data packet;

[0013] Step 6: The server receives the data packet and, based on the application function address mapped synchronously from Step 4, directly calls the corresponding application function using the audio stream code.

[0014] Further, step 1 includes:

[0015] Step 11: Generate sound waves based on a combination of multiple preset physical sound generation parameters;

[0016] Step 12: Extract the center frequency sequence of the formant peaks, the slope of the energy envelope, and the total duration of the event from the sound wave, and encode them into an acoustic skeleton vector;

[0017] Step 13: Verify the acoustic skeleton vector by playback and re-extraction, and adjust the combination of physical sound generation parameters accordingly until the predetermined standard is met.

[0018] Further, step 2 includes:

[0019] Step 21: Detect energy mutation events based on the background noise baseline and cache the corresponding audio data to obtain suspicious segments;

[0020] Step 22: Calculate the coarse contour vector of the suspicious segment and compare it with the pre-stored template coarse contour vector using a matching path tree to filter candidate templates;

[0021] Step 23: After performing local elastic deformation on the time spectrum of the candidate template, calculate the local energy cross-correlation of the key resonance peak trajectory band;

[0022] Step 24: Calculate the matching confidence based on the cross-correlation results and path matching degree. When the matching confidence exceeds a predetermined threshold, output the corresponding game sound command identifier.

[0023] Further, step 3 includes:

[0024] Step 31: Within the sliding window, dynamically assign and update the temporal confidence weights of the game sound command identifiers;

[0025] Step 32: Perform multi-path parallel matching and backtracking pruning based on weighted identifier sequences and pre-compiled pattern libraries;

[0026] Step 33: When the cumulative activation level of a path is the highest and exceeds the dynamically adjusted confirmation threshold, output the audio stream code.

[0027] Further, step 4 includes:

[0028] Step 41: Send the instance fingerprint to the server and receive the returned context anchor.

[0029] Step 42: Obtain the minimum necessary dataset from the server based on the context anchor, which includes interactive audio commands, a subset of the audio command stream pattern library, and their mapping address offsets.

[0030] Furthermore, step 4 also includes:

[0031] Step 43: Subscribe to game logic state variables based on context anchors and receive incremental encoding updates;

[0032] Step 44: Send a heartbeat carrying the version tag hash to receive the data block patch package, and trigger a resynchronization request based on the context anchor if the validity check fails.

[0033] Further, step 5 includes:

[0034] Step 51: Parse the semantic condition expression based on the audio stream code, and perform logical operations in conjunction with the game logic state snapshot to mark the initial validity;

[0035] Step 52: Assign execution urgency priority tags to the initially effective audio stream codes based on the state transition probability matrix;

[0036] Step 53: Package the audio stream code, timestamp, and role identifier according to the priority marker and select the corresponding network channel to send it.

[0037] Further, step 6 includes:

[0038] Step 61: Receive the data packet, query the calling credentials based on the user's identity identifier, and generate the expected calling address;

[0039] Step 62: Search for the machine code features of the matching function signature verification code within the preset offset range to confirm the valid call address.

[0040] Furthermore, step 6 also includes:

[0041] Step 63: Write the call parameters into the shared memory mapping area and trigger the proxy thread to execute the call using the valid call address;

[0042] Step 64: Capture the game state after the call is executed as a side effect snapshot, compare and reconcile it with the synchronized game logic state, and generate a compensation instruction to push if there is a discrepancy.

[0043] Secondly, cloud gaming interaction systems based on speech recognition include:

[0044] The audio command template module is used to predefine multiple discrete non-semantic interactive audio commands and preset an acoustic fingerprint template for each interactive audio command based on the formant position, energy envelope shape and duration on the time spectrum;

[0045] The real-time matching module is used on the client side to perform matching calculations between the real-time audio stream and the acoustic fingerprint template. When the matching degree exceeds a predetermined threshold, the corresponding interactive audio command identifier is output.

[0046] The stream recognition module is used to sequentially combine multiple interactive audio commands within a preset time window to form an audio command stream, and pre-compile an audio command stream pattern library; it temporarily stores the interactive audio command identifiers and compares them with the pre-compiled audio command stream pattern library in real time to identify the audio command stream code;

[0047] The data synchronization module is used to synchronize interactive audio commands, audio command stream mode libraries and their mapped application function addresses, as well as application status from the server.

[0048] The verification and packaging module is used to verify the validity of the audio stream code based on the application state, and to package the valid audio stream code with the timestamp and user identity into a data packet.

[0049] The function call module is used by the server to receive data packets and, based on the application function addresses mapped from the data synchronization module, directly call the corresponding application internal functions from the audio stream code.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] (1) This solution completes the acoustic fingerprint matching and voice command stream recognition of voice commands on the client side, avoiding the two network round trips and cloud serial processing of the original audio and commands in the traditional architecture, greatly reducing the end-to-end response latency, realizing real-time interaction comparable to physical peripherals, effectively adapting to the high real-time control requirements of cloud games, and improving the responsiveness of user operation.

[0052] (2) This solution adopts the minimum necessary dataset synchronization and incremental update strategy, which transmits only the audio commands, mode library subsets and state change data required by the current game scene. Combined with the version hash verification and patch update mechanism, it greatly reduces network transmission overhead, reduces bandwidth usage, and improves the interaction stability in weak network environments.

[0053] (3) This solution constructs a multi-level validity verification system, from sound command matching confidence screening and semantic verification based on game state to server identity credential verification and accurate function address confirmation, to avoid invalid calls and unauthorized operations in all aspects. Combined with state snapshot comparison and compensation instruction push, it ensures that the client and server states are consistent and improves the reliability of interaction.

[0054] (4) This scheme optimizes the recognition efficiency by using sliding window time-series weight allocation, multi-path parallel matching and backtracking pruning, and uses proxy thread to execute function calls to avoid blocking the main process. It achieves reasonable allocation of resources throughout the recognition, transmission and calling process, while ensuring interactive performance, reducing the hardware load of the server and client, and improving the overall operating efficiency of the system. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0056] Figure 1 This is a flowchart of the cloud gaming interaction method based on speech recognition according to the present invention;

[0057] Figure 2 This is a flowchart illustrating the relationships between the various modules in the cloud gaming interaction system based on speech recognition of this invention. Detailed Implementation

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] Please see Figure 1 The cloud gaming interaction method based on speech recognition, in a client and server computing environment, includes the following steps:

[0061] Step 1: Predefine multiple discrete non-semantic interactive tone commands, and pre-set an acoustic fingerprint template for each interactive tone command based on the formant position, energy envelope shape and duration on the time spectrum;

[0062] Step 1 further includes the following steps:

[0063] Step 11: Generate sound waves based on a combination of preset physical sound parameters. The specific operation is as follows:

[0064] First, the key physical acoustic parameters affecting the acoustic properties of sound waves are identified. These parameters typically include the frequency range, initial amplitude, amplitude decay rate, phase change pattern, and duration of sound emission. Based on the requirements of cloud gaming interaction scenarios for voice command recognition efficiency and anti-interference, multiple sets of different physical acoustic parameter combinations are preset. Significant differences in key parameters are ensured between each set to avoid confusion in the acoustic characteristics of the subsequently generated sound waves. Then, sound waves are synthesized according to each set of preset physical acoustic parameter combinations. During the synthesis process, each parameter in the parameter combination directly determines the core properties of the sound wave. For example, the frequency range determines the pitch range of the sound wave, the amplitude parameter determines the loudness characteristics of the sound wave, and the phase change pattern affects the waveform shape of the sound wave. Finally, multiple basic sound waves with unique acoustic properties are generated.

[0065] Step 12: Extract the formant center frequency sequence, energy envelope slope, and total event duration from the sound wave, and encode them into an acoustic skeleton vector. The specific operations are as follows:

[0066] First, time-frequency spectrum analysis is performed on the generated fundamental sound wave. Signal processing techniques such as short-time Fourier transform are used to convert the time-domain sound wave signal into a time-frequency domain signal to clearly present the frequency distribution characteristics of the sound wave at different time points. Based on this, three types of acoustic features are accurately extracted: the first is the formant center frequency sequence. Formants, as energy concentration regions in the sound wave spectrum, are core indicators characterizing the acoustic properties of sound waves. The center frequencies of each order of formants are located using a spectral peak detection algorithm and arranged in chronological order to form a formant center frequency sequence, reflecting the variation of formants over time; the second is the energy envelope slope, which is obtained by analyzing the energy distribution of the time-frequency domain signal... The fitting process yields the energy envelope curve. The slope of the curve in different time intervals is then calculated to characterize the change in sound wave energy over time. Sound waves generated by different emission parameters exhibit different rates of energy decay or enhancement, resulting in significantly different energy envelope slopes. Thirdly, the total event duration is the time span from the initial sound wave to when the energy decays to the background noise level. Finally, the extracted formant center frequency sequence, energy envelope slope, and total event duration are standardized and encoded to eliminate dimensional differences between the various feature parameters. This process ultimately integrates these features to form an acoustic skeleton vector that comprehensively and uniquely characterizes the acoustic properties of the corresponding sound wave.

[0067] Step 13: Verify the acoustic skeleton vector by playback and re-extraction, and adjust the combination of physical sound generation parameters accordingly until the predetermined standard is met. The specific operations are as follows:

[0068] First, the basic sound wave corresponding to the acoustic skeleton vector obtained in step 12 is replayed to simulate the sound wave propagation process in an actual interactive scenario. Then, during the replay, the sound wave signal is reacquired, and the acoustic skeleton vector is extracted again following the same feature extraction and encoding process as in step 12, resulting in a re-extracted vector. The original acoustic skeleton vector and the re-extracted vector are compared for similarity, and the deviation values ​​in various feature dimensions such as the formant center frequency sequence, energy envelope slope, and total event duration are calculated. If the deviation value is within a preset allowable range, the sound wave generated by the current combination of physical sound parameters and the extracted acoustic skeleton vector are determined to meet a predetermined standard; if the deviation value exceeds... If the deviation falls outside the allowable range, it indicates that the stability of the sound wave generated by the current combination of physical acoustic parameters is insufficient, or that there are unreliable factors in the feature extraction process. In this case, based on the specific distribution of the deviation value, the relevant parameters in the combination of physical acoustic parameters are adjusted accordingly. For example, if the deviation of the center frequency of the resonant peak is too large, the frequency range parameter is adjusted; if the deviation of the energy envelope slope exceeds the threshold, the amplitude decay rate parameter is adjusted. The above process of playback, re-extraction, comparison and adjustment is repeated until the deviation value of the acoustic skeleton vector corresponding to the generated sound wave is stable within the predetermined standard range after multiple playbacks and re-extractions, ensuring that the finally determined combination of physical acoustic parameters can generate sound waves with stable acoustic characteristics.

[0069] In a preferred embodiment of the present invention, step 2 is further included: on the client side, the real-time audio stream is matched with the acoustic fingerprint template, and when the matching degree exceeds a predetermined threshold, the corresponding interactive audio command identifier is output.

[0070] Step 2 further includes the following steps:

[0071] Step 21: Detect energy mutation events based on the background noise baseline and cache the corresponding audio data to obtain suspicious segments. The specific operations are as follows:

[0072] First, a background noise baseline needs to be established. This is achieved by continuously collecting audio data for a preset duration during the initial stage of cloud gaming interaction or during gaps when no user is speaking. Energy statistical analysis is then performed on this sample to calculate the mean energy μ and standard deviation σ of the background noise. The background noise baseline is set as the sum of μ and k times σ, where k is an empirical coefficient, typically ranging from 1.5 to 2.5, to balance thorough noise filtering with the capture rate of effective signals. Subsequently, the energy of the audio stream collected in real-time by the client is monitored frame by frame. When the energy value of a frame exceeds the preset background noise baseline, an energy mutation event is identified. This mutation event indicates the possible presence of interactive audio commands from the user. At this point, an audio data caching mechanism is activated, starting from the beginning of the energy mutation and caching continues until the audio energy falls below the background noise baseline and remains below it for a preset duration. Caching then stops, and the cached audio data segment is identified as a suspicious fragment, thus achieving the initial capture and separation of effective signals.

[0073] Step 22: Calculate the coarse contour vector of the suspicious segment and compare it with the pre-stored template coarse contour vector using a matching path tree to filter candidate templates. The specific operations are as follows:

[0074] First, simplified feature extraction is performed on the suspicious segments obtained in step 21 to generate coarse contour vectors. Specifically, it is not necessary to perform full-dimensional fine feature analysis on the suspicious segments. Only key parameters that can reflect their core acoustic contours are extracted, such as the simplified formant range, the trend features of the overall energy envelope, and the approximate duration of the segment. These parameters are standardized and encoded to form coarse contour vectors. At the same time, each pre-stored acoustic fingerprint template has a corresponding coarse contour vector generated in advance. A matching path tree is constructed based on these template coarse contour vectors. The path tree is hierarchically divided according to the similarity of each feature dimension, so that template coarse contour vectors with similar features are located in similar tree node positions. Then, the coarse contour vector of the suspicious segment is input into the matching path tree. Feature comparison is performed layer by layer starting from the root node. The similarity between the vector to be matched and the corresponding template coarse contour vector of each node is calculated. Through path traversal and pruning strategies, template branches with similarity far below the preset coarse matching threshold are removed. Only templates with high similarity are retained as candidate templates, completing the process of quickly screening the candidate set from a large number of templates.

[0075] Step 23: After performing local elastic deformation on the time spectrum of the candidate template, calculate the local energy cross-correlation of the key resonance peak trajectory band. The specific operation is as follows:

[0076] Considering that users' voices may have slight fluctuations in the time or frequency domain in real-world scenarios, direct rigid comparison is prone to matching deviations. Therefore, it is necessary to first perform local elastic deformation processing on the time spectrum of the candidate template. The adjustment range of elastic deformation is set based on the statistical law of actual voice fluctuations. For example, ±10% duration stretching is allowed in the time domain, and ±50Hz formant frequency shift is allowed in the frequency domain. By stretching or compressing the local area, the time spectrum of the candidate template can better adapt to the feature changes caused by the difference in actual voice of the suspicious segment. After the deformation adjustment is completed, the focus is on the key formant trajectory band that characterizes the core features of the interactive sound command. This trajectory band is the core identification area of ​​the preset acoustic fingerprint template in step 1, which contains key information on the change of formant position over time. By calculating the local energy cross-correlation between the suspicious segment and the candidate template after elastic deformation on the key formant trajectory band, the cross-correlation calculation can accurately reflect the degree of fit of the energy distribution of the two in the core area. The higher the cross-correlation value, the closer the core acoustic features of the two are.

[0077] Step 24: Calculate the matching confidence based on the cross-correlation results and path matching degree. When the matching confidence exceeds a predetermined threshold, output the corresponding game sound command identifier. The specific operation is as follows:

[0078] First, the calculation logic of the matching confidence is clarified. The local energy cross-correlation result obtained in step 23 is normalized to obtain a normalized cross-correlation value. At the same time, the path matching degree between the candidate template and the coarse contour vector of the suspicious segment in step 22 is extracted. These two indicators are weighted and summed according to a preset weight ratio to obtain the final matching confidence. The weight of the normalized cross-correlation value is usually set to 0.6, and the weight of the path matching degree is set to 0.4. This weight allocation is determined based on the contribution of the core fine features and coarse contour features to the matching accuracy. Then, the calculated matching confidence is compared with the preset final matching threshold. This threshold is determined through training with a large number of samples, for example, set to 0.8, to balance the recognition accuracy and the missed recognition rate. When the matching confidence exceeds the preset threshold, the suspicious segment is determined to be successfully matched with the corresponding candidate template. At this time, the game sound command identifier corresponding to the candidate template is output. If the matching confidence does not reach the preset threshold, the current suspicious segment is determined to be an invalid signal, and no sound command identifier is output, ensuring that only real and valid interactive sound commands can be recognized and trigger subsequent processes.

[0079] In a preferred embodiment of the present invention, step 3 is further included: sequentially combining multiple interactive audio commands within a preset time window to form an audio command stream, and precompiling an audio command stream pattern library; temporarily storing the interactive audio command identifier and comparing it with the precompiled audio command stream pattern library in real time to identify the audio command stream code;

[0080] Step 3 further includes the following steps:

[0081] Step 31: Within the sliding window, dynamically assign and update the temporal confidence weights of the game sound command identifiers. The specific operations are as follows:

[0082] First, the duration parameter of the sliding window is set. This parameter is determined based on the combined duration statistics of common interactive sound commands in cloud gaming, for example, set to 2 seconds, to ensure complete coverage of multiple consecutive interactive sound commands issued by the user. When the game sound command identifier output in step 2 is temporarily stored, it is immediately included in the identifier sequence within the current sliding window, and a time-series confidence weight is dynamically assigned according to the interval between its output time and the current time. Typically, a weight decay mechanism is used, that is, the sound command identifier that has just been output receives the highest initial weight, for example, an initial weight value of 2 seconds. =0.9. As time moves towards the back of the sliding window, the weight decreases frame by frame according to the preset attenuation coefficient w. The value of w is usually 0.95, which makes the weight of the earlier output audio tokens gradually decrease, thereby reflecting the impact of temporal freshness on matching reliability. When a new audio token enters the sliding window and causes the number of tokens in the window to exceed the threshold, or when the earliest audio token exceeds the window duration range, the expired audio token is automatically removed, and the temporal confidence weight of the remaining tokens is updated synchronously to ensure that a valid audio token sequence with temporal weight is always maintained in the sliding window.

[0083] Step 32: Perform multi-path parallel matching and backtracking pruning based on the weighted identifier sequence and the pre-compiled pattern library. The specific operations are as follows:

[0084] First, the pre-compiled audio stream pattern library has pre-stored all preset valid audio combination sequences in the cloud game. Each sequence corresponds to specific game control logic. During the matching process, the weighted identifier sequence obtained in step 31 is used as input, and a multi-path parallel matching mechanism is started. That is, starting from the first audio identifier in the identifier sequence, it is compared with the pattern path corresponding to the initial audio identifier that can be matched in the pattern library. During the comparison process, as each audio identifier is matched, the cumulative weight of the current path is calculated and its adaptability is judged with the standard weight distribution of the corresponding path in the pattern library. To avoid invalid paths occupying too many computing resources, a backtracking pruning strategy is introduced. When the cumulative weight of a matching path is lower than the preset pruning threshold T, the subsequent matching process of the path is immediately terminated. The value of T is determined according to the average of the lowest cumulative weight of each standard path in the pattern library, for example, set to 0.3. By advancing the multi-path parallel process, it is ensured that no valid matching possibility is missed. At the same time, invalid paths are quickly eliminated through backtracking pruning, thereby improving matching efficiency while ensuring the comprehensiveness of the matching.

[0085] Step 33: When the cumulative activation level of a path is the highest and exceeds the dynamically adjusted confirmation threshold, output the audio stream code. The specific operation is as follows:

[0086] First, the calculation logic for the cumulative activation of a path is clarified. The temporal confidence weights of each command identifier on each unpruned matching path are accumulated, and then weighted by the fit coefficient f between the path and the standard path in the pattern library to obtain the cumulative activation of each path. The fit coefficient f is determined based on the consistency between the matching order of the command identifiers on the path and the standard order. When they are completely consistent, f = 1.0. When there is a deviation in order, f decreases according to the degree of deviation. Then, the confirmation threshold is dynamically adjusted. This threshold is not a fixed value, but is adaptively adjusted according to the number of command identifiers in the current sliding window and the dispersion of the cumulative activation of each path. For example, when there are many command identifiers in the window, the confirmation threshold is appropriately increased and set to the average of the cumulative activation of each path. 1.2 times the standard deviation The sum; when the quantity is small, the threshold is adjusted to With 1.0 times The sum of these values ​​is used to balance the recognition accuracy and the missed recognition rate. The cumulative activation of all paths is sorted, and the path with the highest cumulative activation is selected. If the cumulative activation of a path exceeds the dynamically adjusted confirmation threshold, the corresponding audio stream pattern is considered to have matched successfully, and the corresponding audio stream code is output. If it does not exceed the threshold, the audio identifier sequence in the current sliding window is considered not to constitute a valid audio stream, and no audio stream code is output, ensuring the validity and reliability of the output results.

[0087] In a preferred embodiment of the present invention, step 4 is further included: synchronizing interactive audio commands, audio command stream mode library and the addresses of in-application functions mapped thereto, and application status from the server.

[0088] Step 4 further includes the following steps:

[0089] Step 41: Send the instance fingerprint to the server and receive the returned context anchor. The specific operations are as follows:

[0090] During implementation, the client first generates a unique instance fingerprint. This fingerprint is obtained through a hash operation using key information such as the currently running cloud gaming application instance identifier, client device hardware information, and application version number. For example, the device identifier D, application version number V, and instance startup time are hashed together. Combinatorial operations are performed to generate a fixed-length instance fingerprint F, ensuring that the fingerprint of each client application instance is unique. Subsequently, the client sends the instance fingerprint to the server. After receiving it, the server retrieves the corresponding game service instance and associated context information through the instance fingerprint, including the running process of the game instance, the currently loaded game scene, and the configured interaction rules. After the retrieval is completed, the server generates and returns a context anchor point. This anchor point is a string of index information with positioning function, which can accurately point to the storage area and related process resources in the server that store the data to be synchronized by the client. Through this process, the client and the server establish a one-to-one precise association, avoiding confusion or mismatch during data synchronization.

[0091] Step 42: Based on the context anchor, obtain the minimum necessary dataset from the server, which includes interactive audio commands, a subset of the audio command stream pattern library, and their mapping address offsets. The specific operations are as follows:

[0092] After obtaining the context anchor returned in step 41, the client sends the anchor to the server as the retrieval basis to request the corresponding data resources. After locating the associated data of the corresponding game instance based on the context anchor, the server does not return the full amount of interactive commands and command stream pattern library. Instead, it filters out a core subset of data related to the current game scene, user role permissions, and recent interactive behaviors, namely the minimum necessary dataset. This dataset specifically includes the interactive commands available in the current scene, a subset of command stream pattern libraries adapted to the scene, and the address offsets of the in-application functions mapped by these commands and patterns. The address offsets are used to calculate the accurate function call address in combination with the base address, avoiding the security risks and data redundancy caused by directly transmitting the complete address. The server compresses and encodes the filtered minimum necessary dataset and sends it to the client. After receiving it, the client decompresses and stores the data locally.

[0093] Step 43: Subscribe to game logic state variables based on context anchors and receive incremental encoding updates. The specific operations are as follows:

[0094] Based on the context anchor point obtained in step 41, the client sends a subscription request for game logic state variables to the server. The type of variable to be subscribed is dynamically determined according to the current game scene, including key logic variables such as character health, game level progress, current executable operation permissions, and the state of interactive objects within the scene. After receiving the subscription request, the server binds the client to the corresponding game state variable update mechanism. When the subscribed state variable changes, the server no longer transmits the full state data, but instead uses incremental encoding to process the changed part, extracting only the changed value, change timestamp, and variable identifier of the state variable for encoding. Subsequently, the server actively pushes the incrementally encoded update data to the client. After receiving it, the client locates the corresponding state variable in its local storage based on the variable identifier and uses the incremental data to complete the local state update, ensuring that the game logic state stored on the client is consistent with that on the server in real time.

[0095] Step 44: Send a heartbeat carrying the version tag hash to receive the data block patch package, and trigger a resynchronization request based on the context anchor if the validity check fails. The specific operations are as follows:

[0096] The client sends heartbeat packets to the server at preset time intervals. The heartbeat packets carry the hash of the version tag of the currently stored synchronization data. This hash value is generated by hashing the locally synchronized interactive commands, a subset of the pattern library, and game state data. For example, the version hash H is calculated by combining the data version number Vd and the data content digest C. This is used by the server to quickly verify the integrity and timeliness of the client's data. After receiving the heartbeat packet, the server compares the version tag hash sent by the client with the corresponding data version hash stored in its own memory. If there is a difference, a corresponding data block patch package is generated. The patch package only contains the missing or inconsistent data fragments of the client and is sent to the client over the network for the client to update. At the same time, the client performs validity verification after receiving the synchronization data or patch package, including data integrity verification, specifically through hash value comparison, and data format verification. If the verification finds an anomaly, such as the received data hash being inconsistent with the server's declaration, or the data format being unparseable, the client will immediately trigger a resynchronization request to the server based on the context anchor point obtained in step 41. The server relocates and sends the complete minimum necessary dataset based on the context anchor point to ensure that the client always holds accurate and valid synchronization data.

[0097] In a preferred embodiment of the present invention, step 5 is further included: validating the audio stream code based on the application state, and packaging the valid audio stream code with the timestamp and user identity into a data packet;

[0098] Step 5 further includes the following steps:

[0099] Step 51: Parse the semantic conditional expression based on the audio stream code, and perform logical operations in conjunction with the game logic state snapshot to mark the initial validity. The specific operations are as follows:

[0100] First, the audio command stream code output in step 3 is parsed to extract its semantic conditional expression. This expression predefines the prerequisite game state requirements for the audio command stream code to execute, such as the character's health being higher than a preset threshold, the current scene being interactive, and the user having the corresponding operation permissions. Then, a snapshot of the game logic state at the current moment is obtained. This snapshot is a collection of instantaneous states generated based on the latest game logic state variables synchronized in step 43, containing key information such as character attributes, scene parameters, and interaction permissions. The parsed semantic conditional expression is logically operated on with the game logic state snapshot, verifying one by one whether each requirement in the conditional expression matches the state variables in the snapshot. If all conditions are met, the audio command stream code is marked as initially valid; if any condition is not met, such as the character's health being lower than the threshold required by the conditional expression, it is marked as invalid, and the subsequent processing of the audio command stream code is terminated.

[0101] Step 52: Based on the state transition probability matrix, assign execution urgency priority tags to the initially effective audio stream codes. The specific operations are as follows:

[0102] First, a state transition probability matrix is ​​introduced. This matrix is ​​pre-trained using a large number of game interaction scenarios to represent the probability distribution of the game state transitioning from the current state to various target states after the execution of different audio command stream codes. It includes information related to the urgency of state transitions. For audio command stream codes marked as initially valid in step 51, the system queries the state transition probability matrix to obtain the state transition probability distribution corresponding to the audio command stream code. The system focuses on analyzing the probability value of transitioning to an emergency state, specifically the probability values ​​of states such as character avoidance and key task triggering. Based on the magnitude of this probability value, execution urgency priority tags are assigned. For example, when the probability value of transitioning to an emergency state is greater than a preset threshold P, such as P=0.7, the highest priority tag is assigned. When the probability value is between 0.3 and 0.7, assign a medium priority label. When the probability value is less than 0.3, assign the lowest priority flag. The allocation of priority markers is essentially a quantitative sorting based on the urgency of the impact of the audio stream code on the game process.

[0103] Step 53: Package the audio stream code, timestamp, and role identifier according to the priority marker, and select the corresponding network channel for transmission. The specific operation is as follows:

[0104] First, based on the priority markers assigned in step 52, the corresponding audio command stream code and key auxiliary information are packaged together. The auxiliary information includes the timestamp of the audio command stream code generation and the user role identifier. The timestamp ensures that the server processes commands in the order they are generated, preventing command execution order errors due to network transmission delays. The role identifier is used by the server to verify the legitimacy of the command sender, ensuring that operation permissions match the role. After packaging, the corresponding network channel is selected for transmission based on the priority markers. Preset network channels are divided according to transmission priority, such as the highest priority marker. The corresponding audio stream data packets will be transmitted through a low-latency dedicated channel, which uses a smaller transmission buffer and a higher transmission frequency to minimize transmission latency; medium priority. The corresponding data packets are transmitted through the regular transmission channel; lowest priority. The corresponding data packets are transmitted through a batch transmission channel, and are sent in batches without affecting the transmission of high-priority instructions. By selecting differentiated channels, the real-time performance of critical instructions is guaranteed, and efficient use of network resources is achieved.

[0105] In a preferred embodiment of the present invention, step 6 is further included: the server receives the data packet and, based on the application function address synchronously mapped from step 4, directly calls the corresponding application function using the audio stream code;

[0106] Step 6 further includes the following steps:

[0107] Step 61: Receive the data packet, retrieve the invocation credentials based on the user's identity identifier, and generate the expected invocation address. The specific operations are as follows:

[0108] The server first receives the data packet transmitted by the client through the corresponding network channel. It extracts the core identity information, the user role identifier, from the data packet. Based on this identity identifier, the server queries the corresponding calling credentials in the preset user permission database. The calling credentials contain key information such as the user's current game character's operation permission scope and the list of authorized function calls. By verifying the validity of the credentials, it can be confirmed that the data packet was not generated by illegal intrusion or unauthorized operation. At the same time, combined with the function address mapping relationship within the application synchronized to the server in step 4, the server extracts the address offset corresponding to the audio stream code in the data packet. This offset is superimposed with the application's basic loading address in the server's memory to generate the expected calling address. This expected address initially points to the memory area where the application's internal function to be triggered by the audio stream code is located.

[0109] Step 62: Search for the machine code features of the function signature checksum within the preset offset range to confirm the valid call address. The specific operation is as follows:

[0110] After obtaining the expected call address generated in step 61, the server sets a preset offset range. This range is determined based on the address offset error statistics synchronized in step 4. For example, it is set to a memory region of ±16 bytes before and after the expected call address to cover possible minor offset deviations. Subsequently, the server traverses the machine code in memory within this preset offset range and extracts the function signature checksum at each location. The function signature checksum is generated by hashing the function's parameter type, return value type, and function name, and is unique. The extracted machine code features are compared with the pre-stored target function signature checksum. When the two match completely, the current location can be confirmed as a valid call address. If no matching checksum is found after traversing the preset range, the address location is determined to be abnormal, the current call process is terminated, and the exception handling mechanism is triggered.

[0111] Step 63: Write the call parameters into the shared memory mapping area, triggering the proxy thread to execute the call using the valid call address. The specific operations are as follows:

[0112] The server first parses the audio stream code in the data packet, extracting the function call parameters contained therein, including key information such as parameter type and parameter value. To avoid interference with the application's main memory area during parameter transmission and to improve parameter transmission efficiency, these call parameters are written to a preset shared memory mapping area. This area is an independent memory space shared by the server application process and the proxy thread, possessing isolation and efficient read / write characteristics. After the parameters are written, the pre-created proxy thread is triggered. The proxy thread reads the call parameters in the shared memory mapping area and obtains the valid call address confirmed in step 62. Through this valid call address, the execution of the internal function of the application is directly triggered. The design of using a proxy thread to execute the call avoids process blocking caused by directly executing the call in the main process, ensuring the normal operation of other logic in the application.

[0113] Step 64: Capture the game state after the call execution as a side effect snapshot, compare and reconcile it with the synchronized game logic state, and generate a compensation command to push if there is a discrepancy. The specific operation is as follows:

[0114] After the proxy thread triggers and executes the function call, it immediately captures the game state changes that occur after the call and encapsulates them as a side effect snapshot. This snapshot contains the instantaneous values ​​of all relevant game states, such as character attributes, scene parameters, and task progress, after the function call. Subsequently, this side effect snapshot is compared with the game logic state synchronized to the server in step 4. The values ​​of each state variable are checked one by one to see if they are consistent. If the comparison results are completely consistent, it is determined that the function call did not produce a state deviation and no additional processing is required. If there are inconsistent state variables, such as a difference between the character's health snapshot value and the synchronized state value, a compensation instruction is generated based on the consistent baseline state. The compensation instruction contains the correct value of the deviation state variable and the update instruction, and is pushed to the client. After receiving it, the client updates its local game state according to the compensation instruction, ensuring that the game state of the client and the server always remains synchronized and avoiding the impact of state deviation on the effectiveness of subsequent interaction instructions.

[0115] Example 2

[0116] Please see Figure 2 Based on Example 1, this example provides a cloud gaming interaction system based on speech recognition, including:

[0117] The audio command template module is used to predefine multiple discrete non-semantic interactive audio commands and preset an acoustic fingerprint template for each interactive audio command based on the formant position, energy envelope shape and duration on the time spectrum;

[0118] The real-time matching module is used on the client side to perform matching calculations between the real-time audio stream and the acoustic fingerprint template. When the matching degree exceeds a predetermined threshold, the corresponding interactive audio command identifier is output.

[0119] The stream recognition module is used to sequentially combine multiple interactive audio commands within a preset time window to form an audio command stream, and pre-compile an audio command stream pattern library; it temporarily stores the interactive audio command identifiers and compares them with the pre-compiled audio command stream pattern library in real time to identify the audio command stream code;

[0120] The data synchronization module is used to synchronize interactive audio commands, audio command stream mode libraries and their mapped application function addresses, as well as application status from the server.

[0121] The verification and packaging module is used to verify the validity of the audio stream code based on the application state, and to package the valid audio stream code with the timestamp and user identity into a data packet.

[0122] The function call module is used by the server to receive data packets and, based on the application function addresses mapped from the data synchronization module, directly call the corresponding application internal functions from the audio stream code.

[0123] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A cloud gaming interaction method based on speech recognition, characterized in that, In a client-server computing environment, the following steps are included: Step 1 involves predefining multiple discrete non-semantic interactive tone commands, and for each interactive tone command, pre-setting an acoustic fingerprint template based on the formant position, energy envelope shape, and duration in the time-spectrum. This also includes: Step 11: Generate sound waves based on a combination of multiple preset physical sound generation parameters; Step 12: Extract the center frequency sequence of the formant peaks, the slope of the energy envelope, and the total duration of the event from the sound wave, and encode them into an acoustic skeleton vector; Step 13: Verify the acoustic skeleton vector by playback and re-extraction, and adjust the combination of physical sound generation parameters accordingly until the predetermined standard is met. Step 2: On the client side, the real-time audio stream is matched against the acoustic fingerprint template. When the matching degree exceeds a predetermined threshold, the corresponding interactive voice command identifier is output, which also includes: Step 21: Detect energy mutation events based on the background noise baseline and cache the corresponding audio data to obtain suspicious segments; Step 22: Calculate the coarse contour vector of the suspicious segment and compare it with the pre-stored template coarse contour vector using a matching path tree to filter candidate templates; Step 23: After performing local elastic deformation on the time spectrum of the candidate template, calculate the local energy cross-correlation of the key resonance peak trajectory band; Step 24: Calculate the matching confidence based on the cross-correlation results and path matching degree. When the matching confidence exceeds a predetermined threshold, output the corresponding game sound command identifier. Step 3: Combine multiple interactive voice commands sequentially within a preset time window to form a voice command stream, and pre-compile the voice command stream pattern library; temporarily store the interactive voice command identifiers and compare them with the pre-compiled voice command stream pattern library in real time to identify the voice command stream code; Step 4 involves synchronizing interactive audio commands, audio command stream mode libraries, and their mapped application function addresses from the server, as well as the application state. This also includes: Step 41: Send the instance fingerprint to the server and receive the returned context anchor. Step 42: Obtain the minimum necessary dataset from the server based on the context anchor, which includes interactive audio commands, a subset of the audio command stream pattern library, and their mapping address offsets. Step 5: Validate the audio stream code based on the application state, and package the valid audio stream code, timestamp, and user identifier into a data packet. This also includes: Step 51: Parse the semantic condition expression based on the audio stream code, and perform logical operations in conjunction with the game logic state snapshot to mark the initial validity; Step 52: Assign execution urgency priority tags to the initially effective audio stream codes based on the state transition probability matrix; Step 53: Package the audio stream code, timestamp, and role identifier according to the priority marker and select the corresponding network channel for transmission; Step 6: The server receives the data packet and, based on the application function address mapped synchronously from Step 4, directly calls the corresponding application function using the audio stream code.

2. The cloud gaming interaction method based on speech recognition according to claim 1, characterized in that, Step 3 includes: Step 31: Within the sliding window, dynamically assign and update the temporal confidence weights of the game sound command identifiers; Step 32: Perform multi-path parallel matching and backtracking pruning based on weighted identifier sequences and pre-compiled pattern libraries; Step 33: When the cumulative activation level of a path is the highest and exceeds the dynamically adjusted confirmation threshold, output the audio stream code.

3. The cloud gaming interaction method based on speech recognition according to claim 2, characterized in that, Step 4 also includes: Step 43: Subscribe to game logic state variables based on context anchors and receive incremental encoding updates; Step 44: Send a heartbeat carrying the version tag hash to receive the data block patch package, and trigger a resynchronization request based on the context anchor if the validity check fails.

4. The cloud gaming interaction method based on speech recognition according to claim 3, characterized in that, Step 6 includes: Step 61: Receive the data packet, query the calling credentials based on the user's identity identifier, and generate the expected calling address; Step 62: Search for the machine code features of the matching function signature verification code within the preset offset range to confirm the valid call address.

5. The cloud gaming interaction method based on speech recognition according to claim 4, characterized in that, Step 6 also includes: Step 63: Write the call parameters into the shared memory mapping area and trigger the proxy thread to execute the call using the valid call address; Step 64: Capture the game state after the call is executed as a side effect snapshot, compare and reconcile it with the synchronized game logic state, and generate a compensation instruction to push if there is a discrepancy.

6. A cloud gaming interaction system based on speech recognition, applied to any one of the cloud gaming interaction methods based on speech recognition as described in claims 1-5, characterized in that, include: The audio command template module is used to predefine multiple discrete non-semantic interactive audio commands and preset an acoustic fingerprint template for each interactive audio command based on the formant position, energy envelope shape and duration on the time spectrum; The real-time matching module is used on the client side to perform matching calculations between the real-time audio stream and the acoustic fingerprint template. When the matching degree exceeds a predetermined threshold, the corresponding interactive audio command identifier is output. The stream recognition module is used to sequentially combine multiple interactive audio commands within a preset time window to form an audio command stream, and pre-compile an audio command stream pattern library; it temporarily stores the interactive audio command identifiers and compares them with the pre-compiled audio command stream pattern library in real time to identify the audio command stream code; The data synchronization module is used to synchronize interactive audio commands, audio command stream mode libraries and their mapped application function addresses, as well as application state from the server. The verification and packaging module is used to verify the validity of the audio stream code based on the application state, and to package the valid audio stream code with the timestamp and user identity into a data packet. The function call module is used by the server to receive data packets and, based on the application function addresses mapped from the data synchronization module, directly call the corresponding application internal functions from the audio stream code.

Images