Handle key tactile feedback control system and method based on deep learning

Abstract

The invention provides a handle key tactile feedback control system and method based on deep learning, and relates to the technical field of deep learning model application, and the method comprises the steps: a tactile feedback mode matching module is used for receiving button operation data from a handle, and combining a host scene type to obtain a tactile feedback mode matching module; inputting a preset tactile feedback matching database for tactile feedback configuration to obtain a selected tactile feedback mode; the expected mode generation module is used for retrieving a historical handle motion sample set to obtain an expected tactile feedback mode when the number of the selected tactile feedback modes is equal to 0 and the host scene type does not belong to a tactile feedback taboo type; and the optimization execution module is used for calling the tactile feedback mode analysis network bound with the handle, performing control optimization on the handle motor array, and executing handle key tactile feedback control. The technical problems that in the prior art, key tactile feedback is poor in flexibility, a tactile feedback mode needs to be set for each scene, and unset scenes cannot be responded are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning model application, and in particular to a handle button tactile feedback control system and method based on deep learning. BACKGROUND

[0002] Tactile feedback is a hardware-based technology designed to enhance game immersion by simulating real physical sensations. It replaces the traditional handle's single motor vibration mode with a more refined linear motor and dynamic algorithm that generates different vibration frequencies and amplitudes based on the game scenario. For example, in a shooting game, the handle vibration when the weapon is fired will vary depending on the type of weapon; in a racing game, the subtle vibration difference can also be transmitted when the tires press over grass or mud.

[0003] However, traditional button tactile feedback usually binds fixed scenes and tactile feedback modes in advance. When the preset scene is triggered, the corresponding tactile feedback is started. The disadvantage is poor flexibility, as the tactile feedback mode needs to be set for each scene, and the unconfigured scene cannot be responded to. SUMMARY

[0004] The present application provides a handle button tactile feedback control system and method based on deep learning to solve the technical problems of poor flexibility of button tactile feedback in the prior art, the need to set tactile feedback mode for each scene, and the inability to respond to unconfigured scenes.

[0005] The technical solution of the present application to solve the above technical problems is as follows:

[0006] In a first aspect, the present application provides a handle button tactile feedback control system based on deep learning, comprising:

[0007] A tactile feedback mode matching module is configured to receive button operation data from the handle, combine the main scene type, input the preset tactile feedback matching database for tactile feedback configuration, and obtain the selected tactile feedback mode. The tactile feedback matching database is constructed by pre-stored triple arrays, and any one triple array includes button operation data, main scene type, and tactile feedback mode.

[0008] An expected mode generation module is configured to, when the number of selected tactile feedback modes is equal to 0, and when the main scene type does not belong to the tactile feedback taboo type, retrieve the historical handle motion sample set with button operation data and main scene type as constraints, perform high-frequency running state statistics, and obtain the expected tactile feedback mode.

[0009] An optimization execution module is configured to call the tactile feedback mode analysis network bound to the handle, control optimization of the handle motor array based on the expected tactile feedback mode, obtain the motor array target control parameter, and execute the handle button tactile feedback control.

[0010] The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes motor array control parameters for a preset handle model and labels that identify the haptic feedback pattern of the handle.

[0011] Secondly, the present invention provides a deep learning-based haptic feedback control method for gamepad buttons, comprising:

[0012] The system receives button operation data from the controller, combines it with the host scene type, inputs it into a preset haptic feedback matching database to configure haptic feedback, and obtains the selected haptic feedback mode. The haptic feedback matching database is constructed by pre-stored triple arrays, and any triple array includes button operation data, host scene type, and haptic feedback mode.

[0013] When the number of selected haptic feedback modes is 0, and the host scene type does not belong to the prohibited haptic feedback type, the historical controller motion sample set is retrieved based on the button operation data and the host scene type, and high-frequency running state statistics are performed to obtain the desired haptic feedback mode.

[0014] The haptic feedback pattern analysis network bound to the controller is invoked. Based on the desired haptic feedback pattern, the controller motor array is optimized to obtain the target control parameters of the motor array and execute the haptic feedback control of the controller buttons.

[0015] The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes motor array control parameters for a preset handle model and labels that identify the haptic feedback pattern of the handle.

[0016] The beneficial effects of this invention are:

[0017] Compared to existing technologies, this application first uses a haptic feedback pattern matching module to receive button operation data from the controller, combine it with the host scene type, and input it into a preset haptic feedback matching database to configure haptic feedback and obtain a selected haptic feedback pattern. This can provide instant and accurate haptic feedback patterns for most common scenarios. Secondly, through a desired pattern generation module, when the number of selected haptic feedback patterns is zero and the host scene type is not a prohibited type for haptic feedback, the module uses button operation data and host scene type as constraints to retrieve historical controller motion sample sets, perform high-frequency operating state statistics, and obtain the desired haptic feedback pattern, providing a reliable data foundation for subsequent control optimization. Finally, through the optimization execution module, the haptic feedback pattern analysis network bound to the controller is invoked. Based on the desired haptic feedback pattern, the controller motor array is optimized for control to obtain the target control parameters of the motor array. The haptic feedback control of the controller buttons is then executed. Through the haptic feedback pattern analysis network built based on deep learning, the motor array control parameters are input and the predicted haptic feedback pattern is output. Then, by constructing an fitness evaluation function, control optimization is performed to find the motor array control parameters with the highest fit to the desired haptic feedback pattern, thus solving the problem that traditional fixed parameters cannot adapt to dynamic scenes.

[0018] Through the above technical solution, this application achieves instant response by quickly matching haptic feedback patterns in conventional scenarios through a preset haptic feedback matching database. When there is no preset matching, it retrieves historical samples and performs outlier filtering based on button operation data and host scene type to generate the desired haptic feedback pattern that fits the user's habits. Then, through a haptic feedback pattern analysis network built based on deep learning, it optimizes the motor array control parameters in reverse according to the fitness evaluation function. In this way, this application not only ensures the response speed in conventional scenarios, but also solves the problem of difficult adaptation when there are no preset scenarios, thus improving the flexibility of the haptic feedback of the gamepad buttons. Attached Figure Description

[0019] Figure 1 A schematic diagram of the deep learning-based haptic feedback control system for gamepad buttons provided by the present invention;

[0020] Figure 2 This is a flowchart illustrating the deep learning-based haptic feedback control method for gamepad buttons provided by the present invention.

[0021] In the attached diagram, the components represented by each number are as follows:

[0022] Haptic feedback pattern matching module 11, expected pattern generation module 12, and optimization execution module 13. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0026] Example 1

[0027] like Figure 1 As shown, this embodiment of the invention provides a deep learning-based haptic feedback control system for gamepad buttons. The system includes: a haptic feedback pattern matching module 11, a desired pattern generation module 12, and an optimization execution module 13.

[0028] The haptic feedback pattern matching module 11 is used to receive button operation data from the controller, combine it with the host scene type, input the preset haptic feedback matching database to configure haptic feedback, and obtain the selected haptic feedback mode. The haptic feedback matching database is constructed by pre-stored ternary arrays, and any ternary array includes button operation data, host scene type and haptic feedback mode.

[0029] Button operation data and console scene type are important parameters that determine the generation of the controller's button haptic feedback mode. Button operation data includes button type (such as trigger button, shoulder button, joystick, etc.), operation intensity (such as trigger button press depth, etc.), operation frequency (such as number of consecutive clicks, etc.), operation trajectory (such as joystick tilt angle and direction, etc.). Button operation data provides accurate operation feature reference for the feedback mode. For example, a rapid click action button may need to be matched with a high-frequency vibration haptic feedback mode. The console scene type affects the style of the haptic feedback mode. For example, a racing game with a bumpy road scene may need to be matched with a continuous low-frequency vibration haptic feedback mode.

[0030] To address the aforementioned issues, this application receives button operation data from the controller, combines it with the host scene type, inputs it into a preset haptic feedback matching database for haptic feedback configuration, and obtains the selected haptic feedback mode. The haptic feedback matching database is constructed using pre-stored ternary arrays, where each ternary array includes button operation data, host scene type, and haptic feedback mode. Specifically:

[0031] First, button operation data is received from the controller. This data includes the target user's specific actions on various controller buttons, such as tilting the left joystick 30° to the left or clicking the A button 5 times consecutively. Second, the current console scene type is considered, such as driving on asphalt in a racing game or shooting in a shooting game. Finally, the button operation data and console scene type are used as search criteria and input into a preset haptic feedback matching database to match the selected haptic feedback mode. The selected haptic feedback mode is the specific haptic feedback that should be triggered under the target user's current button operation data and console scene type, such as low-frequency small-amplitude vibration or high-frequency pulse vibration.

[0032] The haptic feedback matching database is constructed from pre-stored ternary arrays. Each ternary array includes the binding relationship between button operation data, host scene type, and haptic feedback mode. For example, a typical ternary array might be: button operation data (such as pressing the trigger button all the way down), host scene type (such as shooting game - shotgun firing), and haptic feedback mode (such as high-frequency strong vibration + grip radial vibration). That is, when the target user presses the trigger button all the way down in the shotgun firing scene of a shooting game, the haptic feedback matching database will directly match the preset haptic feedback mode of high-frequency strong vibration + grip radial vibration. The preset haptic feedback matching database can quickly respond to the corresponding haptic feedback mode in common scenarios to ensure the real-time nature of the feedback.

[0033] For example, button operation data (such as clicking action key A 5 times in a row) and host scene type (such as shooting game) are input into a preset haptic feedback matching database to configure haptic feedback and obtain the selected haptic feedback mode (such as high-frequency pulse vibration).

[0034] In this way, by matching the preset haptic feedback with the corresponding relationships between button operation data, host scene type and haptic feedback mode in the database, it is possible to provide instant and accurate haptic feedback modes for most common scenarios.

[0035] The desired haptic feedback mode generation module 12 is used to retrieve the historical handle motion sample set, perform high-frequency running state statistics, and obtain the desired haptic feedback mode when the number of selected haptic feedback modes is equal to 0 and the host scene type does not belong to the haptic feedback taboo type, with the button operation data and host scene type as constraints.

[0036] Traditional button haptic feedback usually involves pre-binding fixed scenes and haptic feedback modes. When a scene is triggered, the corresponding haptic feedback is activated. This mode cannot respond to undefined scenes.

[0037] To address the aforementioned issues, this application, when the number of selected haptic feedback modes is equal to 0 and the host scene type does not belong to the haptic feedback prohibition type, uses button operation data and host scene type as constraints to retrieve historical controller motion sample sets, performs high-frequency operation state statistics, and obtains the desired haptic feedback mode. Here, the haptic feedback prohibition type is the scene type that the user presets and does not require haptic feedback.

[0038] Specifically, the expected pattern generation module 12 includes:

[0039] The data extraction unit is used to extract button operation data and joystick operation data from button operation data. The button operation data includes a list of clicked button types and a list of clicked button frequencies. The joystick operation data includes a list of joystick operation types and a list of operation directions. The clicked button type and joystick operation type represent the function type, which is predefined by the user terminal according to the host scenario type.

[0040] The data retrieval unit is used to retrieve historical controller motion sample sets based on constraints such as a list of clicked button types, a list of clicked button frequencies, a list of operation joystick types, a list of operation directions, and host scene types.

[0041] The desired pattern acquisition unit is used to decompose the historical handle motion sample set into functional areas, obtain several functional area motion sample sets, perform high-frequency operation state statistics on each, and obtain the desired tactile feedback pattern.

[0042] In this embodiment, button operation data and joystick operation data are first extracted from the button operation data. The button operation data includes a list of click button types (such as trigger button, shoulder button L2, etc.) and a list of click button frequencies (such as 2 times / second, 3 times / second, etc.). The joystick operation data includes a list of joystick operation types (such as left joystick, right joystick, etc.) and a list of operation directions (such as 30° tilt, vertical downward, etc.). The click button type and joystick operation type represent the function type. The function type is predefined by the user terminal according to the host scene type. For example, in a shooting scene, the user terminal predefines the trigger button as firing and the right joystick as viewing angle rotation to ensure the correlation between operation features and scene requirements and avoid confusion of the function of the same button in different scenes.

[0043] Secondly, using the lists of click button types, click button frequencies, joystick types, operation directions, and host scene types as constraints, a historical controller motion sample set is retrieved. This historical controller motion sample set stores a large amount of user button operation data and corresponding haptic feedback pattern records for various host scenes, and can be constructed by collecting historical data from a large number of users. For example, using the lists of click button types (e.g., trigger buttons, shoulder buttons L2), click button frequencies (e.g., 2 times / second, 3 times / second), joystick types (e.g., left joystick, right joystick), operation directions (e.g., 30° tilt, vertical downward), and host scene types (e.g., shooting scenes) as constraints, the historical controller motion sample set is retrieved to obtain all samples of the same host scene type, ensuring the relevance of the search results.

[0044] Finally, the historical controller motion sample set was functionally decomposed to obtain motion sample sets for the left joystick area, right joystick area, action button area, directional pad area, touchpad area, shoulder buttons, trigger buttons, left grip, right grip, palm contact surface, top edge of the controller, and around the indicator lights. High-frequency operating state statistics were then performed on each of these sets to obtain the desired haptic feedback mode. This is because the feedback requirements of different functional areas of the controller are independent; for example, the vibration modes of the trigger buttons and joysticks are not directly related. Therefore, the retrieved historical controller motion sample set needs to be decomposed into 12 sub-sample sets according to functional areas. High-frequency operating state statistics were performed on each sub-sample set, and finally, the high-frequency operating states of the 12 sub-sample sets were combined to form the desired haptic feedback mode, ensuring that the feedback conforms to the operating habits of most users and adapts to the functional requirements of the current scenario.

[0045] Specifically, the expected pattern acquisition unit includes:

[0046] The region extraction unit is used to extract the first functional region motion sample set from several functional region motion sample sets;

[0047] The pattern extraction unit is used to traverse the motion sample set of the first functional area and extract the sample tactile feedback pattern set. Each sample tactile feedback pattern in the sample tactile feedback pattern set includes vibration frequency, vibration direction and vibration amplitude.

[0048] The outlier analysis unit is used to perform outlier analysis on the vibration frequency, vibration direction, and vibration amplitude of the first sample based on the sample tactile feedback pattern set, and to obtain the outlier factor of the vibration frequency, the outlier factor of the vibration direction, and the outlier factor of the vibration amplitude of the first sample.

[0049] The high-frequency factor acquisition unit is used to calculate the mean values ​​of the vibration frequency outlier factor, the vibration direction outlier factor, and the vibration amplitude outlier factor of the first sample. The ratio of 1 to the mean value is set as the high-frequency factor of the first sample.

[0050] The computational execution unit is used to obtain the high-frequency factors of the Nth sample.

[0051] The anomaly removal unit is used to remove tactile feedback patterns whose high-frequency factors are less than or equal to the high-frequency factor threshold from the motion sample set of the first functional area based on the high-frequency factors of the first sample up to the high-frequency factors of the Nth sample, to obtain the desired tactile feedback pattern of the first functional area, and add it to the desired tactile feedback pattern.

[0052] In this embodiment, a first functional area motion sample set is randomly extracted from several functional area motion sample sets. For example, a functional area motion sample set is randomly extracted from the motion sample sets of the left joystick area, right joystick area, action button area, directional key area, touchpad area, shoulder buttons, trigger buttons, left grip, right grip, palm contact surface, top edge of the handle, and the area around the indicator lights. For instance, the left joystick area motion sample set is selected as the first functional area motion sample set.

[0053] Secondly, the motion sample set of the first functional area is traversed to extract a set of sample tactile feedback patterns. Each sample tactile feedback pattern in this set includes vibration frequency, vibration direction, and vibration amplitude. For example, the motion sample set of the left joystick area is traversed, and the tactile feedback pattern of each sample, including the left joystick area vibration frequency (e.g., 50–200 Hz), left joystick area vibration direction (e.g., horizontal / vertical / radial), and left joystick area vibration amplitude (e.g., an intensity gradient of 0–1), is extracted to form a set of sample tactile feedback patterns. For instance, the sample tactile feedback patterns of the left joystick area in a shooting game sample within the left joystick area motion sample set include: a left joystick area vibration frequency of 100 Hz, a left joystick area vibration direction of horizontal, and a left joystick area vibration amplitude of 0.3.

[0054] Furthermore, based on the sample tactile feedback pattern set, outlier analysis is performed on the vibration frequency, vibration direction, and vibration amplitude of the first sample to obtain the outlier factors of the vibration frequency, vibration direction, and vibration amplitude of the first sample. For example, from the set of sample tactile feedback patterns, the set of vibration frequencies in the left joystick area is extracted. Based on the vibration frequency of the left joystick area of ​​the first sample, the absolute value of the frequency deviation is calculated by traversing the set of vibration frequencies in the left joystick area to obtain a set of vibration frequency deviations. Based on the set of vibration frequency deviations from smallest to largest, k vibration frequency deviations are selected, the mean is calculated, and then the ratio of 1 to the mean is calculated and set as the local density of vibration frequency in the left joystick area of ​​the first sample. This is added to the set of local density of vibration frequency in the left joystick area of ​​all samples. The mean of the set of local density of vibration frequency in the left joystick area of ​​all samples is calculated and compared with the local density of vibration frequency in the left joystick area of ​​the first sample to obtain the outlier factor of vibration frequency in the first sample. For example, the outlier factor of vibration frequency in the first sample is calculated to be 0.89, and the outlier factor of vibration direction in the first sample is calculated to be 0.71 and the outlier factor of vibration amplitude in the first sample is calculated to be 0.80 using the same method.

[0055] Further, the mean values ​​of the outlier factors for the vibration frequency, vibration direction, and vibration amplitude of the first sample are calculated. The ratio of 1 to the mean is set as the high-frequency factor of the first sample. This step is repeated until the high-frequency factor of the Nth sample is obtained. The smaller the mean, the more the characteristics of the first sample conform to the commonalities of most samples, meaning the probability that the first sample is an outlier tactile feedback pattern is lower, and the larger the calculated high-frequency factor of the first sample. For example, if the outlier factor for the vibration frequency of the first sample is 0.89, the outlier factor for the vibration direction of the first sample is 0.71, and the outlier factor for the vibration amplitude of the first sample is 0.80, then the mean of these three factors = (0.89 + 0.71 + 0.80) / 3 = 0.8. The high-frequency factor of the first sample is then calculated as 1 / 0.8 = 1.25. The calculation is continued using the same method until the high-frequency factor of the Nth sample is obtained.

[0056] Furthermore, based on the high-frequency factors of the first sample up to the Nth sample, tactile feedback patterns with high-frequency factors less than or equal to the high-frequency factor threshold are removed from the motion sample set of the first functional area to obtain the desired tactile feedback patterns for the first functional area. These patterns are then added to the desired tactile feedback patterns. This is because the high-frequency factor can reflect the degree of deviation between the current sample and the majority of samples; the more the characteristics of the current sample conform to the commonalities of the majority of samples, the larger the high-frequency factor. For example, the high-frequency factor threshold can be dynamically adjusted according to the size, distribution characteristics, and actual scene requirements of the historical controller motion sample set. For instance, by setting the high-frequency factor threshold to 1.2, tactile feedback patterns with high-frequency factors ≤ 1.2 are removed from the motion sample set of the left joystick area to obtain the desired tactile feedback patterns for the first functional area, which are then added to the desired tactile feedback patterns.

[0057] Finally, following the same method as the motion sample set of the left joystick area, the same analysis was performed on the motion sample sets of the action button area, directional pad area, touchpad area, shoulder buttons, trigger buttons, left grip, right grip, palm contact surface, top edge of the controller, and indicator light area to obtain the corresponding desired haptic feedback patterns, which were then added to the desired haptic feedback patterns.

[0058] Furthermore, the outlier analysis unit includes:

[0059] The frequency analysis unit is used to extract the set of vibration frequencies from the set of sample tactile feedback patterns;

[0060] The deviation calculation unit is used to calculate the absolute value of the frequency deviation by traversing the set of vibration frequencies based on the vibration frequency of the first sample, and to obtain the set of vibration frequency deviations.

[0061] The local density calculation unit is used to sort k vibration frequency deviations from smallest to largest based on the vibration frequency deviation set, calculate the mean, and then calculate the ratio of 1 to the mean, which is set as the local density of the vibration frequency of the first sample and added to the local density set of vibration frequencies of all samples.

[0062] The outlier factor acquisition unit is used to calculate the ratio of the mean of the local density set of vibration frequencies of all samples to the local density of vibration frequencies of the first sample, and to obtain the outlier factor of vibration frequency of the first sample.

[0063] The calculation process for the outlier factors of the first sample vibration direction and the first sample vibration amplitude is the same as that for the outlier factor of the first sample vibration frequency.

[0064] In this embodiment, a set of vibration frequencies is first extracted from the set of sample tactile feedback patterns. For example, the vibration frequencies of the left joystick area of ​​all samples are extracted from the set of sample tactile feedback patterns to form a set of left joystick area vibration frequencies, such as {80Hz, 90Hz, 85Hz, 300Hz, ...}.

[0065] Secondly, based on the vibration frequency of the first sample, the absolute value of the frequency deviation is calculated by traversing the set of vibration frequencies to obtain a set of vibration frequency deviations. The first sample is a tactile feedback pattern randomly selected from the set of sample tactile feedback patterns. For example, a sample is randomly selected from the set of sample tactile feedback patterns as the first sample, and its left joystick area vibration frequency, such as 90Hz, is obtained as the left joystick area vibration frequency of the first sample. Then, the set of left joystick area vibration frequencies is traversed, and the absolute value of the frequency deviation between the first sample's left joystick area vibration frequency and all other vibration frequencies in the set is calculated. For example, |90-80|=10, |90-120|=30, etc., to obtain a set of vibration frequency deviations, such as {10, 30, 20, 15, 15, 35, ...}.

[0066] Secondly, based on the vibration frequency deviation set from smallest to largest, k vibration frequency deviations are selected, the mean is calculated, and then the ratio of 1 to the mean is calculated, which is set as the local density of the vibration frequency of the first sample. This value is then added to the set of local density of vibration frequencies of all samples. Here, k is the number of nearest neighbor deviations selected when calculating the local density of the sample. k can be taken as 20% to 30% of the total number of vibration frequency deviations. For example, if the total number of vibration frequency deviations is 25, then k can be taken as 5 to 7. This is because if k is too small, the mean calculation is easily affected by individual extreme deviations, resulting in large fluctuations and poor stability in the local density calculation. If k is too large, too many absolute values ​​of frequency deviations that are far apart will be included, and the calculated mean will be closer to the overall deviation level than the true local distribution characteristics of the sample. Selecting 20% ​​to 30% of the total number of vibration frequency deviations can simultaneously take into account local distribution characteristics and stability. For example, the vibration frequency deviation set is sorted in ascending order. For instance, {10, 30, 20, 15, 15, 35, ...} is sorted in ascending order to {10, 15, 15, 20, 30, 35, ...}, which contains 25 data points. Then, five vibration frequency deviations are selected from these in ascending order: 10, 15, 15, 20, and 30. The mean is then calculated as (10 + 15 + 15 + 20 + 30) / 5 = 18. This mean reflects the vibration frequency of the left rocker area of ​​the first sample. The mean value was calculated as the difference between the frequency and the average of most similar samples. Finally, the local density of the vibration frequency in the left joystick area of ​​the first sample was calculated as 1 / 18 = 0.056. The smaller the mean value, the greater the calculated local density of the vibration frequency in the left joystick area of ​​the first sample. This is because the smaller the mean value, the smaller the absolute value of the frequency deviation between the vibration frequency in the left joystick area of ​​the first sample and other vibration frequencies in the set of vibration frequencies in the left joystick area. This further indicates that the vibration frequency in the left joystick area of ​​the first sample is more common in the set of vibration frequencies in the left joystick area, and the lower the probability that the vibration frequency in the left joystick area of ​​the first sample is an outlier.

[0067] Furthermore, the mean of the set of local densities of vibration frequencies of all samples is calculated, and the local density of vibration frequency of the first sample is compared with that of the first sample to obtain the outlier factor of vibration frequency of the first sample, where the outlier factor of vibration frequency of the first sample = (mean of the set of local densities of vibration frequencies of all samples) / (local density of vibration frequency of the first sample). For example, following the same method described above, the next sample is randomly selected from the set of sample tactile feedback patterns as the second sample. The local density of the vibration frequency in the left joystick area of ​​the second sample is calculated until the set of sample tactile feedback patterns is traversed to obtain the set of local density of the vibration frequency in the left joystick area of ​​all samples. For example, the set of local density of the vibration frequency in the left joystick area of ​​all samples is {0.056, 0.042, 0.046, 0.071, 0.044, 0.052, ...}. Its mean is calculated to be 0.05. Then the outlier factor of the vibration frequency of the first sample is 0.05 / 0.056 = 0.89. The outlier factor of the vibration frequency of the first sample is less than 1, which means that the local density of the first sample is higher than the overall level, and the probability that the first sample is an outlier is smaller. The outlier factor of the vibration frequency of the first sample is greater than 1, which means that the local density of the first sample is lower than the overall level, and the first sample may be abnormal.

[0068] Finally, following the same calculation process as the outlier factor for the vibration frequency of the first sample, the outlier factors for the vibration direction and vibration amplitude of the first sample are calculated. For example, the outlier factor for the vibration direction of the first sample is calculated to be 0.71, and the outlier factor for the vibration amplitude of the first sample is calculated to be 0.80. In this way, abnormal feedback patterns can be accurately identified and filtered to ensure that the desired haptic feedback pattern can truly reflect the operating feedback habits of most users in similar scenarios.

[0069] In summary, compared with the prior art, this application, when the number of selected haptic feedback modes is equal to 0 and the host scene type does not belong to the prohibited haptic feedback type, uses button operation data and host scene type as constraints to retrieve historical handle motion sample sets, performs high-frequency operating state statistics, and obtains the desired haptic feedback mode. In this way, a reliable data foundation is provided for subsequent control optimization.

[0070] The optimization execution module 13 is used to call the haptic feedback pattern analysis network bound to the gamepad, optimize the control of the gamepad motor array based on the desired haptic feedback pattern, obtain the target control parameters of the motor array, and execute the haptic feedback control of the gamepad buttons. The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes the motor array control parameters of the preset gamepad model and a label identifying the haptic feedback pattern of the gamepad.

[0071] When haptic feedback cannot be configured from the preset haptic feedback matching database, the aforementioned steps obtain the desired haptic feedback mode, which can be used to optimize the control of the handle motor array.

[0072] To address the aforementioned issues, this application retrieves a haptic feedback pattern analysis network bound to the controller, optimizes the control of the controller motor array based on the desired haptic feedback pattern, obtains the target control parameters of the motor array, and executes haptic feedback control of the controller buttons. The haptic feedback pattern analysis network is generated using deep learning training through multiple sets of data. Each set of data includes motor array control parameters for a preset controller model and a label identifying the controller's haptic feedback pattern.

[0073] Specifically, the optimization execution module 13 includes:

[0074] Rule building units are used to construct fitness evaluation rules:

[0075] The fitness acquisition unit is used to calculate the Euclidean distance based on the normalized values ​​of frequency deviation, amplitude deviation, and direction angle deviation as the three-axis coordinate deviations, and set it as the fitness evaluation value.

[0076] The comprehensive fitness acquisition unit is used to construct several regional fitness evaluation functions based on the expected tactile feedback pattern and in combination with fitness evaluation rules, traversing several functional areas, and summing the several regional fitness evaluation functions to obtain the comprehensive fitness evaluation function.

[0077] The pattern prediction unit is used to randomly configure several control parameters of the handle motor array, analyze them through the haptic feedback pattern analysis network, and obtain several predicted haptic feedback patterns.

[0078] The optimization unit is used to process several predicted haptic feedback modes based on the comprehensive fitness evaluation function, obtain several comprehensive fitness evaluation values, perform minimum value extraction, and obtain the target control parameters of the motor array.

[0079] In this embodiment, based on the desired haptic feedback pattern in the user's current host scenario, a pre-trained haptic feedback pattern analysis network is invoked to optimize the control of the controller motor array, obtain the target control parameters of the motor array, and execute the haptic feedback control of the controller buttons accordingly. Specifically:

[0080] First, an fitness evaluation function is constructed to evaluate different predictive tactile feedback patterns. The fitness evaluation rule is to calculate the Euclidean distance based on the normalized values ​​of frequency deviation, amplitude deviation, and direction angle deviation as the three-axis coordinate deviations, and set it as the fitness evaluation value. For example, the expected frequency, expected amplitude, and expected direction angle are obtained from the expected haptic feedback pattern, and the predicted frequency, predicted amplitude, and predicted direction angle are obtained from the haptic feedback pattern analysis network. The frequency deviation between the expected frequency and the predicted frequency, the amplitude deviation between the expected amplitude and the predicted amplitude, and the direction angle deviation between the expected direction angle and the predicted direction angle are calculated. Then, the deviations are normalized, for example, by Z-Score normalization, to obtain normalized values ​​for frequency deviation, amplitude deviation, and direction angle deviation. The normalized values ​​for frequency deviation, amplitude deviation, and direction angle deviation are used as the three-axis coordinate deviations, and the Euclidean distance is calculated and set as the fitness evaluation value. The smaller the Euclidean distance, the closer the predicted haptic feedback pattern is to the expected haptic feedback pattern, the higher the fitness, and the smaller the fitness evaluation value.

[0081] Secondly, based on the expected haptic feedback pattern and combined with the fitness evaluation rules, several functional areas are traversed to construct several area fitness evaluation functions. These functions are then summed to obtain a comprehensive fitness evaluation function. For example, following the aforementioned method, the left joystick area, right joystick area, and action button area are traversed, and a fitness evaluation function is constructed for each area. All fitness evaluation functions are then summed to obtain a comprehensive fitness evaluation function. The closer the predicted haptic feedback pattern of each area is to the expected haptic feedback pattern, the higher the fit, the smaller the fitness evaluation value, and the smaller the summed comprehensive fitness evaluation function. This indicates that the overall predicted haptic feedback pattern is closer to the expected haptic feedback pattern.

[0082] Next, several control parameters for the handle motor array are randomly configured and analyzed by a haptic feedback pattern analysis network to obtain several predicted haptic feedback patterns. For example, several control parameters for the handle motor array are randomly configured, input into the haptic feedback pattern analysis network, and several predicted haptic feedback patterns are output.

[0083] Finally, based on the comprehensive fitness evaluation function, several predicted haptic feedback patterns are traversed and processed to obtain several comprehensive fitness evaluation values. Minimum value extraction is then performed to obtain the target control parameters of the motor array. For example, following the same method described above, several predicted haptic feedback patterns are traversed, several comprehensive fitness evaluation values ​​are calculated, and minimum value extraction is performed to obtain the target control parameters of the motor array. The smaller the comprehensive fitness evaluation value, the closer the overall predicted haptic feedback pattern is to the desired haptic feedback pattern. Minimum value extraction can find the current optimal target control parameters of the motor array.

[0084] Furthermore, the optimization execution module 13 also includes:

[0085] The data acquisition unit is used to extract several labels that identify the haptic feedback patterns of functional areas from the labels that identify the haptic feedback patterns of the handle.

[0086] The model architecture unit is used to configure a fully connected network as the backbone network, and to configure a fully connected network as multiple branch networks for several functional areas. The output nodes of the backbone network and the input nodes of the multiple branch networks are merged to obtain the haptic feedback pattern analysis network architecture.

[0087] The model training unit is used to train the haptic feedback pattern analysis network by using labels representing haptic feedback patterns of several functional areas as the corresponding branch supervision ground truths of multiple branch networks, and using motor array control parameters as the input of the backbone network.

[0088] In this embodiment, the motor array control parameters of a preset controller model are obtained, and the actual tactile feedback patterns under different motor array control parameters are collected as labels to identify the controller's tactile feedback patterns. For example, for a preset controller model, sensors (such as vibration sensors, pressure sensors, etc.) can be used to collect in real time the actual tactile feedback characteristics (such as actual vibration frequency, direction, amplitude, etc.) of each functional area of ​​the controller (such as the left joystick area, right joystick area, etc., a total of 12 areas) when the motor array operates according to specific control parameters (such as current, frequency, phase, etc.). These characteristics are then organized into a structured label to identify the controller's tactile feedback patterns, such as a left joystick area vibration frequency of 50Hz, lateral vibration, amplitude of 0.5, etc., as labels to identify the controller's tactile feedback patterns. Further, this application trains a tactile feedback pattern analysis network using the motor array control parameters of the preset controller model and the labels to identify the controller's tactile feedback patterns. Specifically:

[0089] First, from the labels identifying the haptic feedback patterns of the controller, several labels identifying the haptic feedback patterns of functional areas are extracted. For example, the labels identifying the haptic feedback patterns of the controller describe the overall feedback pattern; however, the feedback patterns of different functional areas of the controller are independent of the requirements. Twelve independent labels for functional areas are extracted from the labels identifying the haptic feedback patterns of the controller. For instance, the label identifying the haptic feedback pattern of the left joystick area might include a vibration frequency of 50Hz, lateral vibration, and an amplitude of 0.5. In this way, the network can learn the mapping relationship between the motor array control parameters and the haptic feedback pattern of each area independently, avoiding interference between the features of different areas.

[0090] Secondly, the model architecture is constructed. Specifically, a fully connected network is configured as the backbone network, and several fully connected networks are configured as branch networks for each functional area. The output nodes of the backbone network and the input nodes of the branch networks are merged to obtain the haptic feedback pattern analysis network architecture. For example, a fully connected network is configured as the backbone network, and the motor array control parameters of the preset controller model, such as the voltage, frequency, phase, and other electrical signal parameters of the motor, are input to extract the global features of the control parameters. Then, a fully connected network is configured for the left joystick area, right joystick area, action button area, directional button area, touchpad area, shoulder buttons, trigger buttons, left grip, right grip, palm contact surface, top edge of the controller, and around the indicator lights. The input of each branch is the output feature of the backbone network, which is responsible for refining the output features of the backbone network that can reflect global features into the feedback pattern features of the corresponding area, such as the vibration frequency and direction of the left joystick area. Furthermore, the output nodes of the backbone network are directly connected to the input nodes of the branch networks to form an information flow of backbone output → branch input, ensuring that the targeted influence of global parameters on each area is effectively learned.

[0091] Finally, model training is performed. Specifically, labels representing haptic feedback patterns in several functional areas are used as ground truth values ​​for the corresponding branches of multiple branch networks. Motor array control parameters are used as input to the backbone network to train the haptic feedback pattern analysis network. For example, the core of model training is to teach the network the mapping relationship between motor array control parameters and haptic feedback patterns in each area. For instance, inputting motor array control parameters for a preset handle model (e.g., motor A with 3V voltage and 100Hz frequency, motor B with 2V voltage and 50Hz frequency, etc.), using the labels of the haptic feedback patterns of each branch network as the ground truth values ​​for the corresponding branches, optimizing the network weights through backpropagation, and using the mean squared error (MSE) loss function to minimize the difference between the output of each branch (predicted haptic feedback patterns in the predicted areas) and the labels (real haptic feedback patterns). Ultimately, the trained haptic feedback pattern analysis network can output predicted feedback patterns for 12 areas based on any set of motor control parameters.

[0092] In summary, compared to existing technologies, this application invokes a haptic feedback pattern analysis network bound to the controller, optimizes the control of the controller's motor array based on the desired haptic feedback pattern, obtains the target control parameters of the motor array, and executes haptic feedback control of the controller buttons. Thus, by using a deep learning-based haptic feedback pattern analysis network, inputting motor array control parameters, and outputting a predicted haptic feedback pattern, and then constructing an fitness evaluation function for control optimization, the motor array control parameters with the highest fit to the desired haptic feedback pattern are found, solving the problem that traditional fixed parameters cannot adapt to dynamic scenarios.

[0093] Furthermore, this application also includes:

[0094] A standard matching unit is used to perform haptic feedback control of the gamepad buttons according to the selected haptic feedback mode when the number of selected haptic feedback modes is not equal to 0.

[0095] The forbidden stop unit is used to stop the process when the host scene type belongs to the forbidden type of haptic feedback.

[0096] The haptic feedback disallowed types are user-preset scenario types where haptic feedback is not needed. In normal scenarios, if button operation data is received from the controller, combined with the host scenario type, a preset haptic feedback matching database is entered to configure haptic feedback and obtain the selected haptic feedback mode. If the number of selected haptic feedback modes is not zero, the controller button haptic feedback control is executed directly according to the selected haptic feedback mode. If the host scenario type is a user-preset scenario type where haptic feedback is not needed, the process stops. This improves the response speed in normal scenarios.

[0097] In summary, the embodiments of this application have at least the following technical effects:

[0098] Compared to existing technologies, this application first receives button operation data from the controller, combines it with the host scene type, inputs it into a preset haptic feedback matching database to configure haptic feedback, and obtains the selected haptic feedback mode. In this way, it can provide instant and accurate haptic feedback modes for most common scenarios.

[0099] Secondly, when the number of selected haptic feedback modes is equal to 0, and when the host scene type does not belong to the prohibited haptic feedback type, this application uses button operation data and host scene type as constraints to retrieve historical handle motion sample sets, perform high-frequency operating state statistics, and obtain the desired haptic feedback mode. In this way, a reliable data foundation is provided for subsequent control optimization.

[0100] Finally, this application invokes the haptic feedback pattern analysis network bound to the controller. Based on the desired haptic feedback pattern, it optimizes the control of the controller's motor array to obtain the target control parameters for the motor array and executes haptic feedback control for the controller buttons. Thus, by using a haptic feedback pattern analysis network built based on deep learning, inputting motor array control parameters, and outputting a predicted haptic feedback pattern, and then constructing an fitness evaluation function for control optimization, the application finds the motor array control parameters with the highest fit to the desired haptic feedback pattern, solving the problem that traditional fixed parameters cannot adapt to dynamic scenes.

[0101] Through the above technical solution, this application achieves instant response by quickly matching haptic feedback patterns in conventional scenarios through a preset haptic feedback matching database. When there is no preset matching, it retrieves historical samples and performs outlier filtering based on button operation data and host scene type to generate the desired haptic feedback pattern that fits the user's habits. Then, through a haptic feedback pattern analysis network built based on deep learning, it optimizes the motor array control parameters in reverse according to the fitness evaluation function. In this way, this application not only ensures the response speed in conventional scenarios, but also solves the problem of difficult adaptation when there are no preset scenarios, thus improving the flexibility of the haptic feedback of the gamepad buttons.

[0102] Example 2

[0103] like Figure 2 As shown, embodiments of the present invention also provide a deep learning-based haptic feedback control method for gamepad buttons, including:

[0104] The system receives button operation data from the controller, combines it with the host scene type, inputs it into a preset haptic feedback matching database to configure haptic feedback, and obtains the selected haptic feedback mode. The haptic feedback matching database is constructed by pre-stored triple arrays, and any triple array includes button operation data, host scene type, and haptic feedback mode.

[0105] When the number of selected haptic feedback modes is 0, and the host scene type does not belong to the prohibited haptic feedback type, the historical controller motion sample set is retrieved based on the button operation data and the host scene type, and high-frequency running state statistics are performed to obtain the desired haptic feedback mode.

[0106] The haptic feedback pattern analysis network bound to the controller is invoked. Based on the desired haptic feedback pattern, the controller motor array is optimized to obtain the target control parameters of the motor array and execute the haptic feedback control of the controller buttons.

[0107] The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes motor array control parameters for a preset handle model and labels that identify the haptic feedback pattern of the handle.

[0108] Specifically, "using button operation data and host scene type as constraints, retrieving historical controller motion sample sets, performing high-frequency operation state statistics, and obtaining the desired haptic feedback mode" includes:

[0109] From the button operation data, extract the key operation data and joystick operation data. The key operation data includes a list of clicked key types and a list of clicked key frequencies. The joystick operation data includes a list of joystick operation types and a list of operation directions. The clicked key type and joystick operation type represent the function type, which is predefined by the user terminal according to the host scenario type.

[0110] Using the list of clicked button types, the list of clicked button frequencies, the list of joystick types, the list of operation directions, and the host scene type as constraints, retrieve the historical gamepad motion sample set;

[0111] The historical handle motion sample set is decomposed into functional areas to obtain several functional area motion sample sets. High-frequency operation state statistics are performed on each of them to obtain the desired tactile feedback mode.

[0112] Specifically, "the historical handle motion sample set is decomposed into functional areas to obtain several functional area motion sample sets, and high-frequency operation state statistics are performed on each to obtain the desired haptic feedback mode," including:

[0113] Extract the motion sample set of the first functional region from several functional region motion sample sets;

[0114] Traverse the motion sample set of the first functional area and extract the sample tactile feedback pattern set. Each sample tactile feedback pattern in the sample tactile feedback pattern set includes vibration frequency, vibration direction and vibration amplitude.

[0115] Based on the sample tactile feedback pattern set, outlier analysis was performed on the vibration frequency, vibration direction, and vibration amplitude of the first sample to obtain the outlier factor of the vibration frequency, the outlier factor of the vibration direction, and the outlier factor of the vibration amplitude of the first sample.

[0116] Calculate the mean values ​​of the vibration frequency outlier factor, vibration direction outlier factor, and vibration amplitude outlier factor of the first sample, and use 1 to compare with the mean value, which is set as the high frequency factor of the first sample.

[0117] Until the high-frequency factor of the Nth sample is obtained;

[0118] Based on the high-frequency factors of the first sample up to the high-frequency factors of the Nth sample, tactile feedback patterns with high-frequency factors less than or equal to the high-frequency factor threshold are deleted from the motion sample set of the first functional area to obtain the desired tactile feedback pattern of the first functional area, and then added to the desired tactile feedback pattern.

[0119] Furthermore, "based on the sample tactile feedback pattern set, outlier analysis is performed on the vibration frequency, vibration direction, and vibration amplitude of the first sample to obtain the outlier factors for the vibration frequency, vibration direction, and vibration amplitude of the first sample," including:

[0120] Extract the set of vibration frequencies from the set of tactile feedback patterns in the samples;

[0121] Based on the vibration frequency of the first sample, the absolute value of the frequency deviation is calculated by traversing the set of vibration frequencies to obtain the set of vibration frequency deviations.

[0122] Based on the vibration frequency deviation set from small to large, k vibration frequency deviations are selected, the mean is calculated, and then the ratio of 1 to the mean is calculated and set as the local density of the vibration frequency of the first sample, which is then added to the set of local density of vibration frequency of all samples.

[0123] Calculate the ratio of the mean of the local density set of vibration frequencies of all samples to the local density of vibration frequencies of the first sample to obtain the outlier factor of vibration frequency of the first sample.

[0124] The calculation process for the outlier factors of the first sample vibration direction and the first sample vibration amplitude is the same as that for the outlier factor of the first sample vibration frequency.

[0125] Specifically, "retrieving the haptic feedback pattern analysis network bound to the controller, optimizing the control of the controller motor array based on the desired haptic feedback pattern, obtaining the target control parameters of the motor array, and executing the haptic feedback control of the controller buttons" includes:

[0126] Extract several labels that identify the haptic feedback modes of functional areas from the labels that identify the haptic feedback modes of the handle;

[0127] Configure a fully connected network as the backbone network, and configure a fully connected network as multiple branch networks for several functional areas. Merge the output nodes of the backbone network and the input nodes of the multiple branch networks to obtain the haptic feedback pattern analysis network architecture.

[0128] The labels of several functional areas of tactile feedback patterns are used as the corresponding branch supervision ground truth values ​​of multiple branch networks. The motor array control parameters are used as the input of the backbone network to train the tactile feedback pattern analysis network.

[0129] Specifically, "retrieving the haptic feedback pattern analysis network bound to the controller, optimizing the control of the controller motor array based on the desired haptic feedback pattern, obtaining the target control parameters of the motor array, and executing the haptic feedback control of the controller buttons" also includes:

[0130] Construct fitness evaluation rules:

[0131] Using the normalized values ​​of frequency deviation, amplitude deviation, and direction angle deviation as the three-axis coordinate deviations, the Euclidean distance is calculated and set as the fitness evaluation value.

[0132] Based on the expected tactile feedback pattern, combined with the fitness evaluation rules, several functional areas are traversed to construct several area fitness evaluation functions. The fitness evaluation functions of several areas are summed to obtain the comprehensive fitness evaluation function.

[0133] Several control parameters for the motor array of the handle are randomly configured and analyzed through a haptic feedback pattern analysis network to obtain several predicted haptic feedback patterns.

[0134] Based on the comprehensive fitness evaluation function, several predicted tactile feedback modes are processed to obtain several comprehensive fitness evaluation values. The minimum value is extracted to obtain the target control parameters of the motor array.

[0135] Furthermore, the deep learning-based haptic feedback control method for gamepad buttons also includes:

[0136] When the number of selected haptic feedback modes is not equal to 0, the haptic feedback control of the gamepad buttons is executed according to the selected haptic feedback mode;

[0137] The process is stopped when the host scene type belongs to the haptic feedback prohibited type.

[0138] In summary, the embodiments of this application have at least the following technical effects:

[0139] Compared to existing technologies, this application first uses a haptic feedback pattern matching module to receive button operation data from the controller, combine it with the host scene type, and input it into a preset haptic feedback matching database to configure haptic feedback and obtain a selected haptic feedback pattern. This can provide instant and accurate haptic feedback patterns for most common scenarios. Secondly, through a desired pattern generation module, when the number of selected haptic feedback patterns is zero and the host scene type is not a prohibited type for haptic feedback, the module uses button operation data and host scene type as constraints to retrieve historical controller motion sample sets, perform high-frequency operating state statistics, and obtain the desired haptic feedback pattern, providing a reliable data foundation for subsequent control optimization. Finally, through the optimization execution module, the haptic feedback pattern analysis network bound to the controller is invoked. Based on the desired haptic feedback pattern, the controller's motor array is optimized for control, obtaining the target control parameters for the motor array. This enables the execution of haptic feedback control for the controller buttons. Using a haptic feedback pattern analysis network built based on deep learning, the motor array control parameters are input, and the predicted haptic feedback pattern is output. Then, by constructing an fitness evaluation function, control optimization is performed to find the motor array control parameters with the highest fit to the desired haptic feedback pattern, solving the problem that traditional fixed parameters cannot adapt to dynamic scenes. In this way, response speed in conventional scenes is guaranteed, while the adaptation difficulty when there are no preset scenes is solved, improving the flexibility of the controller's button haptic feedback.

[0140] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxesFigure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Although preferred embodiments of the invention have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0146] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A deep learning-based haptic feedback control system for gamepad buttons, characterized in that, Applied to the host, including: The haptic feedback pattern matching module is used to receive button operation data from the controller, combine it with the host scene type, input it into a preset haptic feedback matching database to configure haptic feedback, and obtain the selected haptic feedback pattern. The haptic feedback matching database is constructed by pre-stored ternary arrays, and each of the ternary arrays includes button operation data, host scene type and haptic feedback pattern. The desired haptic feedback mode generation module is used to retrieve the historical controller motion sample set, perform high-frequency running state statistics, and obtain the desired haptic feedback mode when the number of selected haptic feedback modes is equal to 0 and the host scene type does not belong to the haptic feedback taboo type, with the button operation data and the host scene type as constraints. The optimization execution module is used to call the haptic feedback pattern analysis network bound to the controller, optimize the control of the controller motor array based on the desired haptic feedback pattern, obtain the target control parameters of the motor array, and execute the haptic feedback control of the controller buttons. The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes motor array control parameters of a preset handle model and a label identifying the haptic feedback pattern of the handle.

2. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 1, characterized in that, The desired pattern generation module includes: The data extraction unit is used to extract key operation data and joystick operation data from the button operation data. The key operation data includes a list of clicked key types and a list of clicked key frequencies. The joystick operation data includes a list of joystick operation types and a list of operation directions. The clicked key types and joystick operation types represent function types, which are predefined by the user terminal according to the host scenario type. The data retrieval unit is used to retrieve the historical controller motion sample set based on the list of click button types, the list of click button frequencies, the list of operation joystick types, the list of operation directions, and the host scene type. The desired pattern acquisition unit is used to decompose the historical handle motion sample set into functional areas to obtain several functional area motion sample sets, and perform high-frequency operation state statistics on each to obtain the desired tactile feedback mode.

3. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 2, characterized in that, The desired pattern acquisition unit includes: The region extraction unit is used to extract a first functional region motion sample set from the plurality of functional region motion sample sets; The pattern extraction unit is used to traverse the motion sample set of the first functional area and extract the sample tactile feedback pattern set, wherein any sample tactile feedback pattern in the sample tactile feedback pattern set includes vibration frequency, vibration direction and vibration amplitude. The outlier analysis unit is used to perform outlier analysis on the vibration frequency, vibration direction, and vibration amplitude of the first sample based on the set of tactile feedback patterns of the samples, and to obtain the outlier factor of the vibration frequency, the outlier factor of the vibration direction, and the outlier factor of the vibration amplitude of the first sample. The high-frequency factor acquisition unit is used to calculate the mean of the vibration frequency outlier factor, the vibration direction outlier factor, and the vibration amplitude outlier factor of the first sample, and to use 1 to calculate the ratio of the mean and set it as the high-frequency factor of the first sample. The computational execution unit is used to obtain the high-frequency factors of the Nth sample. An anomaly removal unit is used to delete tactile feedback patterns whose high-frequency factors are less than or equal to a high-frequency factor threshold from the motion sample set of the first functional area based on the high-frequency factors of the first sample up to the high-frequency factors of the Nth sample, to obtain the desired tactile feedback pattern of the first functional area, and add it to the desired tactile feedback pattern.

4. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 3, characterized in that, The outlier analysis unit includes: The frequency analysis unit is used to extract a set of vibration frequencies from the set of tactile feedback patterns in the samples; The deviation calculation unit is used to calculate the absolute value of the frequency deviation by traversing the set of vibration frequencies based on the vibration frequency of the first sample, so as to obtain the set of vibration frequency deviations. The local density calculation unit is used to sort k vibration frequency deviations from small to large based on the vibration frequency deviation set, calculate the mean, and then calculate the ratio of 1 to the mean, which is set as the local density of the vibration frequency of the first sample and added to the local density set of vibration frequencies of all samples. The outlier factor acquisition unit is used to calculate the ratio of the mean of the local density set of vibration frequencies of all samples to the local density of vibration frequencies of the first sample, and to obtain the outlier factor of vibration frequency of the first sample. The calculation process for the outlier factor of the first sample vibration direction and the outlier factor of the first sample vibration amplitude is the same as that for the outlier factor of the first sample vibration frequency.

5. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 1, characterized in that, The optimization execution module includes: The data acquisition unit is used to extract several labels indicating the haptic feedback modes of functional areas from the labels indicating the haptic feedback modes of the handle. The model architecture unit is used to configure a fully connected network as the backbone network, and to configure a fully connected network as multiple branch networks for several functional areas. The output nodes of the backbone network and the input nodes of the multiple branch networks are merged to obtain the haptic feedback pattern analysis network architecture. The model training unit is used to train the haptic feedback pattern analysis network by using labels representing haptic feedback patterns of several functional areas as the corresponding branch supervision ground truth values ​​of the multiple branch networks, and using the motor array control parameters as the input of the backbone network.

6. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 1, characterized in that, The optimization execution module also includes: Rule building units are used to construct fitness evaluation rules: The fitness acquisition unit is used to calculate the Euclidean distance based on the normalized values ​​of frequency deviation, amplitude deviation, and direction angle deviation as the three-axis coordinate deviations, and set it as the fitness evaluation value. The comprehensive fitness acquisition unit is used to construct several regional fitness evaluation functions by traversing several functional areas based on the desired tactile feedback mode and in combination with the fitness evaluation rules, and to sum the several regional fitness evaluation functions to obtain the comprehensive fitness evaluation function. The pattern prediction unit is used to randomly configure several control parameters of the handle motor array, analyze them through the haptic feedback pattern analysis network, and obtain several predicted haptic feedback patterns. An optimization unit is used to process the several predicted tactile feedback modes based on the comprehensive fitness evaluation function, obtain several comprehensive fitness evaluation values, perform minimum value extraction, and obtain the target control parameters of the motor array.

7. The deep learning-based haptic feedback control system for gamepad buttons as described in claim 1, characterized in that, Also includes: A standard matching unit is used to perform haptic feedback control of the handle buttons according to the selected haptic feedback mode when the number of selected haptic feedback modes is not equal to 0. The taboo stop unit is used to stop the process when the host scene type belongs to the haptic feedback taboo type.

8. A deep learning-based haptic feedback control method for gamepad buttons, applied to the deep learning-based haptic feedback control system for gamepad buttons as described in any one of claims 1-7, characterized in that, include: The system receives button operation data from the controller, combines it with the host scene type, inputs it into a preset haptic feedback matching database to configure haptic feedback, and obtains the selected haptic feedback mode. The haptic feedback matching database is constructed by pre-stored ternary arrays, and each of the ternary arrays includes button operation data, host scene type, and haptic feedback mode. When the number of selected haptic feedback modes is equal to 0, and when the host scene type does not belong to the haptic feedback prohibition type, the historical controller motion sample set is retrieved with the button operation data and the host scene type as constraints, high-frequency running state statistics are performed, and the desired haptic feedback mode is obtained. The haptic feedback pattern analysis network bound to the controller is invoked, and the controller motor array is optimized based on the desired haptic feedback pattern to obtain the target control parameters of the motor array and execute the haptic feedback control of the controller buttons. The haptic feedback pattern analysis network is generated by deep learning training through multiple sets of data. Each set of data includes motor array control parameters of a preset handle model and a label identifying the haptic feedback pattern of the handle.

Images