Control method, chip and electronic equipment
By detecting the amplitude of audio data in real time and generating vibration commands, the problem that tactile feedback products in the prior art cannot generate corresponding vibration sensations in vibrating scenes is solved, and the adaptation and user experience of different game manufacturers are achieved.
Patent Information
- Application Number
- CN202411713238.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to use a general solution to enable the haptic feedback product to generate corresponding vibration sensation in vibrating scenarios, especially when adapting and processing in games of different game manufacturers, the workload is large and the versatility is poor.
The control terminal detects the amplitude of the target audio data in real time. If it is greater than the vibration threshold, a corresponding vibration command is generated and the state of the vibration terminal is judged. If it is in the target state, a vibration command is sent to drive the vibration terminal to vibrate.
It is realized that when determining that the scene corresponding to the target audio data is a vibration scene, the vibration terminal vibration is accurately controlled to avoid mistriggering, and to ensure that the vibration command with a longer vibration time can be fully executed, improving the user experience.
Smart Images
Figure CN119633362A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of audio processing, and in particular to a control method, a chip and an electronic device. Background Art
[0002] With the development of virtual reality (VR) and augmented reality (AR) technologies, users have higher and higher requirements for immersive experience. By combining auditory information and tactile information, the user's immersive experience can be enhanced, making the user feel as if they are in the scene. Therefore, more and more tactile feedback products are emerging, such as handles, VR headsets, smart gloves, smart glasses, etc.
[0003] Currently, for vibration scenes in games that need to integrate auditory information and tactile information, vibration instructions for vibration scenes can be pre-made. However, the workload of pre-making vibration instructions for vibration scenes is large, and it is impossible to adapt games from different game manufacturers through a universal solution.
[0004] How to enable tactile feedback products to generate corresponding vibration sensations in vibration scenarios is a problem that needs to be solved at present. Summary of the invention
[0005] In order to enable a tactile product to generate a corresponding vibration sensation in a vibration scene, an embodiment of the present application provides a control method, a chip, and an electronic device.
[0006] In a first aspect, an embodiment of the present application provides a control method, which includes being applied to a control terminal, wherein the control method includes: the control terminal establishes a connection with a vibration terminal; the control terminal obtains and plays target audio data; when the control terminal detects that the amplitude of the target audio data is greater than a vibration threshold during the process of playing the target audio data, the control terminal generates a corresponding vibration instruction based on the target audio data; and the control terminal determines whether the vibration terminal is in a target state, wherein the target state is: at least one button of the vibration terminal is in use; or all buttons of the vibration terminal are in an idle state for an idle time that is less than a preset target time; if the judgment result is yes, the control terminal sends a vibration instruction to the vibration terminal to drive the vibration terminal to vibrate.
[0007] In this way, when it is determined that the scene corresponding to the target audio data is a vibration scene and the user is using a vibration terminal, the vibration terminal can be controlled to vibrate to avoid false triggering. In addition, when the idle time of all keys of the vibration terminal being in an idle state is less than a preset target time, the present application can still drive the vibration terminal to vibrate based on the vibration instruction, thereby ensuring that the vibration instruction with a longer vibration time can be fully executed, thereby ensuring the user experience.
[0008] In a possible implementation, it includes: detecting that the vibration terminal collects pressing information of a pressing operation, and determining that the vibration terminal is in use; detecting that the vibration terminal does not collect pressing information of a pressing operation, and determining that the vibration terminal is in an idle state.
[0009] In one possible implementation, the vibration threshold is determined by the following method: determining a target time period; determining the vibration threshold based on the amplitude mean and amplitude variance of the target audio data within the target time period, wherein the target time period includes any one of N frames before the current frame, N frames after the current frame, P frames before the current frame, and Q frames after the current frame, N>0, P>0, Q>0, and P+Q=N.
[0010] In a possible implementation, if the judgment result is no, the control terminal does not send a vibration instruction to the vibration terminal.
[0011] In one possible implementation, the control terminal is any one of a computer, a mobile phone, a drone, a robot, a tablet computer, and a personal computer, and the vibration terminal is any one of a handle, a virtual reality headset, an augmented reality glasses, a tactile glove, a smart bracelet, and a smart watch.
[0012] In a possible implementation, generating a corresponding vibration instruction based on the target audio data includes: extracting audio features of the target audio data to obtain audio feature information; and performing vibration mapping based on the audio feature information to obtain a vibration instruction corresponding to the target audio data.
[0013] In one possible implementation, the audio feature information includes the audio amplitude and the spectral centroid, the vibration instruction includes the motor vibration intensity and the motor vibration frequency, and vibration mapping is performed based on the audio feature information to obtain the vibration instruction corresponding to the target audio data, including: mapping the audio amplitude to obtain the corresponding motor vibration intensity; mapping the spectral centroid to obtain the corresponding motor vibration frequency.
[0014] In one possible implementation, the target audio data includes multiple sub-audio data of different types, and when it is detected that the amplitude of the target audio data is greater than a vibration threshold, a corresponding vibration instruction is generated based on the target audio data, including: when it is detected that the amplitudes of at least two sub-audio data are greater than the vibration threshold; determining the sub-audio data with the largest amplitude among the at least two sub-audio data; and generating the vibration instruction based on the sub-audio data with the largest amplitude.
[0015] In a possible implementation, the different types of sub-audio data include at least two of ambient sound audio data, human voice audio data, animal sound audio data, mechanical sound audio data, white noise audio data, and effect sound audio data.
[0016] In a second aspect, an embodiment of the present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute any control method provided by the first aspect and various possible implementations of the first aspect.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing instructions executed by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, for executing the instructions stored in the memory to implement any one of the control methods provided by the first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1a A schematic diagram showing an interaction scenario between a vibration terminal 100 and a control terminal 200 is shown;
[0019] Figure 1b A schematic diagram showing an interaction scenario between a handle 100 and a computer 200 is shown;
[0020] Figure 2 According to an embodiment of the present application, a schematic diagram of a method for issuing a vibration instruction is shown;
[0021] Figure 3 According to an embodiment of the present application, a flow chart of a control method is shown;
[0022] Figure 4 According to an embodiment of the present application, a flow chart of a vibration instruction generating method is shown;
[0023] Figure 5 According to an embodiment of the present application, a schematic diagram of an interaction scenario between a handle 100 and a computer 200 is shown;
[0024] Figure 6 According to an embodiment of the present application, a structural schematic diagram of an electronic device 1300 is shown. DETAILED DESCRIPTION
[0025] With the development of VR and AR technologies, users have higher and higher requirements for immersive experience. Figure 1a The interactive scene of the vibration terminal 100 and the control terminal 200 shown in the figure can enhance the user's immersive experience by combining auditory information and tactile information, so that the user feels immersive. Among them, the vibration terminal 100 can be any one of a computer, a mobile phone, a drone, a robot, a tablet computer, and a personal computer, and the control terminal 200 can be any one of a handle, a virtual reality headset, an augmented reality glasses, a tactile glove, a smart bracelet, and a smart watch.
[0026] The following takes the vibration terminal 100 as the handle 100 and the control terminal 200 as the computer 200 as an example. Figure 1b The interaction scenario between the vibration terminal 100 and the control terminal 200 is introduced. Figure 1b As shown, when a user uses the handle 100 to play a game, the handle 100 can be first connected to the computer 200 by wire, such as a universal serial bus (USB), or wirelessly, such as a Bluetooth connection. During the running of the game, when the computer 200 detects that the current scene is a vibration scene, for example, when it is detected that the user presses the forward button 101 to control the game character to drive the vehicle on a bumpy mountain road, the computer 200 can send a vibration instruction to the handle 100 to simulate the bumpy feeling of the road surface during the vehicle driving on the mountain road, so as to drive the handle 100 to vibrate. For another example, when the computer 200 detects that the user presses the X button 102 to control the game character to shoot, it can send a vibration instruction to the handle 100 to simulate the impact of firing a bullet, so as to drive the handle 100 to vibrate.
[0027] When generating vibration instructions for vibration scenes in the game, in some embodiments, the control terminal, such as computer 200, can pre-identify all vibration scenes in the game during the design or development stage of the game, and then pre-make corresponding vibration instructions for each vibration scene respectively, and load them into the control terminal. During the running process of the game, the control terminal can send vibration instructions for the corresponding scene to the vibration terminal, such as handle 100, when each vibration scene appears. After receiving the vibration instruction, the vibration terminal can parse the vibration instruction and generate a corresponding driving waveform, thereby driving the motor to vibrate according to the driving waveform to generate a vibration sensation corresponding to the vibration scene.
[0028] However, the workload of pre-making vibration commands for vibration scenes is large, and the vibration commands for the same vibration scene also need to be adapted to games of different game manufacturers, resulting in poor versatility.
[0029] In order to solve the problem of how to enable tactile feedback products to generate corresponding vibrations in vibration scenarios, an embodiment of the present application provides a control method, which includes: a control terminal detects target audio data in real time, and when it is determined that the amplitude of the target audio data is greater than the vibration threshold, determines that the scenario corresponding to the target audio data is a vibration scenario, extracts the audio feature information of the target audio data, and performs vibration mapping based on the audio feature information to obtain a vibration instruction corresponding to the target audio data. The control terminal determines whether the vibration terminal is in the target state; if the judgment result is yes, the vibration terminal is driven to vibrate based on the vibration instruction, wherein the target state indicates that: at least one button of the vibration terminal is in use; or, all buttons of the vibration terminal are in an idle state, and the idle time is less than the preset target time.
[0030] It is understandable that the vibration scene in the game that needs to generate a vibration feeling on the handle is usually when the user presses the button on the handle, or after the user presses the button on the handle. When the scene corresponding to the target audio data is determined to be a vibration scene and the user is using a vibration terminal, the application controls the vibration terminal to vibrate to avoid false triggering.
[0031] It can be understood that the vibration durations of vibration instructions corresponding to different vibration scenes are not exactly the same. For example, if the audio signal in the vibration scene includes more high-frequency components, the vibration duration of the corresponding vibration instruction is shorter; if the audio signal in the vibration scene includes more low-frequency components, the vibration duration of the corresponding vibration instruction is longer. For example, in a scene where the user is walking or driving a vehicle on the road, the audio signal has more high-frequency components, and the vibration duration of the corresponding vibration instruction is shorter. For another example, in a scene where the user uses a handle to control the game character to shoot a gun, the audio signal has more low-frequency components, and the vibration duration of the corresponding vibration instruction is longer.
[0032] When the idle time of all buttons of the vibration terminal is less than the preset target time, the present application still drives the vibration terminal to vibrate based on the vibration instruction, thereby ensuring that the vibration instruction with a longer vibration time can be fully executed, thereby ensuring the user experience.
[0033] In the embodiment of the present application, it is detected that the vibration terminal collects pressing information of the pressing operation, and it is determined that the vibration terminal is in use; it is detected that the vibration terminal does not collect pressing information of the pressing operation, and it is determined that the vibration terminal is in idle state.
[0034] In an embodiment of the present application, the control terminal can poll the state of the vibration terminal in a polling cycle during the running of the game. For example, after generating a vibration instruction, the control terminal can determine whether the vibration terminal is in the target state. If the judgment result is yes, the vibration instruction is sent to the vibration terminal to drive the vibration terminal to vibrate based on the vibration instruction; if the judgment result is no, the vibration instruction is not sent to the vibration terminal, and the amplitude of the target audio data of the next frame of the current frame is detected. Among them, the polling cycle can be any positive number, for example, the polling cycle can be any one of 2s, 3s, 5s, etc.
[0035] For example, Figure 2 As shown, the vibration terminal is a handle, at least one button of the handle is detected to be pressed in frame t1, and all buttons of the handle are detected to be released in frame t2; at least one button of the handle is detected to be pressed in frame t4, and all buttons of the handle are detected to be released in frame t5; at least one button of the handle is detected to be pressed in frame t6, and all buttons of the handle are detected to be released in frame t7.
[0036] From frame t1 to frame t2, at least one button of the controller is in use, that is, from frame t1 to frame t2, the controller is in the target state. From frame t2 to frame t4, all buttons of the controller are in an idle state, and the idle duration is calculated starting from frame t2. When the idle duration of all buttons of the controller being in an idle state is less than the preset target duration, the controller is determined to be in the target state. If the preset target duration is T, and the period from frame t2 to frame t3 is less than T, the controller is determined to be in the target state from frame t2 to frame t3. If the period from frame t3 to frame t4 is greater than T, the controller is determined not to be in the target state from frame t3 to frame t4.
[0037] From frame t4 to frame t5, at least one button of the controller is in use, that is, from frame t4 to frame t5, the controller is in the target state. From frame t5 to frame t6, all buttons of the controller are in an idle state, and the idle duration is calculated starting from frame t5. When the idle duration of all buttons of the controller being in an idle state is less than the preset target duration, the controller is determined to be in the target state. If the preset target duration is T, and the period from frame t5 to frame t8 is less than T, the controller is determined to be in the target state from frame t5 to frame t8.
[0038] Among them, from frame t6 to frame t7 within the time period from frame t5 to frame t8, at least one button of the handle is in use, that is, from frame t6 to frame t7, the handle is in the target state. From frame t7 to frame t10, all buttons of the handle are in an idle state, and the idle duration is calculated starting from frame t7. When the idle duration of all buttons of the handle in the idle state is less than the preset target duration, the handle is determined to be in the target state. If the preset target duration is T, and the time period from frame t7 to frame t9 is less than T, then the handle is determined to be in the target state from frame t7 to frame t9. If the time period from frame t9 to frame t10 is greater than T, then it is determined that the handle is not in the target state from frame t9 to frame t10.
[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the following takes the game running scenario as an example and combines the accompanying drawings to clearly and comprehensively describe the technical solutions in the embodiments of the present application.
[0040] Figure 3 According to an embodiment of the present application, a schematic diagram of a control method is shown. It can be understood that Figure 3 The execution subjects of the schematic diagram shown are all control terminals. Figure 3 As shown, the control method includes:
[0041] 101: The control terminal establishes a connection with the vibration terminal.
[0042] In an embodiment of the present application, the control terminal may be connected to the vibration terminal in communication, for example, the control terminal is connected to the vibration terminal by wire or wirelessly. In an embodiment of the present application, the control terminal is a computer, and the vibration terminal is a handle, and the handle includes at least one button. In other embodiments, the control terminal may be any one of a mobile phone, a drone, a robot, a tablet computer, a personal computer, etc., and the vibration terminal may be any one of a virtual reality (VR) head-mounted device, an augmented reality (AR) glasses, a tactile glove, a smart bracelet, a smart watch, etc.
[0043] 102: Control the terminal to obtain and play the target audio data.
[0044] In an embodiment of the present application, the control terminal can obtain the audio data of the currently running game as the target audio data. In other embodiments, the target audio data can be any one of the audio data of a movie, the audio data of a TV program, the audio data of music, the audio data of a podcast, the audio data of an audio book, etc.
[0045] In some embodiments, the method for obtaining audio data of the currently running game includes: loading the audio effect algorithm library into the system audio effect link through the audio effect framework of the control terminal, and then obtaining the target audio data through the system audio effect link, so that the audio data played by the system passes through the system audio effect link, and the audio effect framework loaded in the system audio effect link processes the target audio data based on the rhythm algorithm. Among them, the audio effect framework can be a framework used by the control terminal to process and control audio effects, such as the AudioEffect framework.
[0046] 103: When the control terminal detects that the amplitude of the target audio data is greater than the vibration threshold during the process of playing the target audio data, the control terminal generates a corresponding vibration instruction based on the target audio data.
[0047] In an embodiment of the present application, the amplitude of the target audio data of the current frame can be detected. If it is determined that the amplitude of the target audio data of the current frame is greater than the vibration threshold, a corresponding vibration instruction is generated based on the target audio data of the current frame; if it is determined that the amplitude of the target audio data of the current frame is less than or equal to the vibration threshold, the amplitude of the target audio data of the next frame of the current frame is detected through a rhythm algorithm.
[0048] In an embodiment of the present application, a target time period can be determined, and a vibration threshold can be determined based on the amplitude mean and amplitude variance within the target time period, wherein the target time period includes any one of N frames before the current frame, N frames after the current frame, P frames before the current frame, and Q frames after the current frame, N>0, P>0, Q>0, and P+Q=N.
[0049] For example, the vibration threshold can be determined based on the amplitude mean and amplitude variance of the N frames before the current frame of the target audio data, or the vibration threshold can be determined based on the amplitude mean M and amplitude variance V of the (i-1-N)th frame to the (i-1)th frame of the target audio data. Wherein, the i-th frame represents the current frame, N>0, the amplitude mean Amplitude variance M x Indicates the amplitude of the x-th frame of audio data.
[0050] In other embodiments, the vibration threshold can be determined based on the amplitude mean and amplitude variance of the N frames before the current frame of the target audio data, or the vibration threshold can be determined based on the amplitude mean M and amplitude variance V of the 1st to Nth frames of the target audio data. Amplitude variance M x Indicates the amplitude of the x-th frame of audio data.
[0051] It can be understood that the vibration threshold is determined based on the amplitude mean and amplitude variance of the target audio data N frames before the current frame, or based on the amplitude mean and amplitude variance of the target audio data N frames before the current frame; when the amplitude of the target audio data of the current frame is greater than the vibration threshold, it indicates that the amplitude of the target audio data of the current frame is too different from the amplitude of the audio data of the N frames before the current frame, or the amplitude of the target audio data of the current frame is too different from the amplitude of the audio data of the N frames before the target audio data, that is, the scene corresponding to the target audio data of the current frame is a vibration scene.
[0052] That is, the present application generates vibration instructions based on the audio data of the current frame only when it determines that the scene corresponding to the target audio data of the current frame is a vibration scene, thereby reducing the workload of generating vibration instructions and realizing the generation of vibration instructions without the need for pre-production and adaptation by game developers.
[0053] It can be understood that the vibration scenes in the game include scenes where the user is walking or driving on the road, scenes where the user uses the handle to control the game character to walk on different terrains (such as snow, sand, and mud), scenes where the user uses the handle to control the game character to drive a vehicle on different roads (such as asphalt roads, grass), scenes where the game character controlled by the user using the handle hits an object, scenes where the game character controlled by the user using the handle shoots a firearm, scenes where the game character controlled by the user using the handle attacks the enemy with a melee weapon (such as a sword or an axe), scenes where the game character controlled by the user using the handle jumps and lands or falls from a height, scenes where the game character controlled by the user using the handle pushes heavy objects or opens doors in the game, scenes where the airplane controlled by the user using the handle experiences turbulence, and the like.
[0054] In an embodiment of the present application, generating a corresponding vibration instruction based on target audio data includes: extracting audio features from the target audio data to obtain audio feature information; performing vibration mapping based on the audio feature information to obtain a vibration instruction corresponding to the target audio data. The audio feature information includes audio amplitude and spectral centroid, and the vibration instruction corresponding to the target audio data includes motor vibration intensity and motor vibration frequency.
[0055] It can be understood that audio amplitude refers to the amplitude of audio data, which is usually used to describe the intensity of audio data. Spectral centroid is one of the physical parameters used to describe the frequency distribution and energy distribution of audio data. The spectral centroid represents the center of gravity of the frequency component, which is the frequency value obtained by energy-weighted averaging of the frequency within a certain frequency range, in Hertz (Hz). The spectral centroid reflects the brightness of the sound. If the spectral centroid is high, it means that the spectrum energy is mainly concentrated in the high-frequency part (or high spectral centroid), and the signal sounds sharper and brighter; if the spectral centroid is low, it means that the spectrum energy is mainly concentrated in the low-frequency part (or low spectral centroid), and the signal sounds low and heavy.
[0056] In an embodiment of the present application, audio feature extraction can be performed directly on the target audio data to obtain audio feature information to ensure extraction efficiency. In other embodiments, noise of the target audio data, such as ambient sound audio data and human voice audio data, can also be removed first, and then audio feature extraction can be performed on the target audio data to obtain audio feature information to ensure extraction accuracy.
[0057] In an embodiment of the present application, vibration mapping is performed based on audio feature information to obtain vibration instructions corresponding to the target audio data, including: mapping the audio amplitude to obtain the corresponding motor vibration intensity; mapping the spectral centroid to obtain the corresponding motor vibration frequency.
[0058] In other embodiments, the audio feature information also includes short-time energy, zero-crossing rate, fundamental frequency, Mel-frequency cepstral coefficients, spectral contrast, and loudness; the above-mentioned vibration mapping based on the audio feature information obtains the vibration instruction corresponding to the target audio data, including: determining the motor vibration intensity based on short-time energy and / or loudness, and determining the motor vibration frequency based on the zero-crossing rate and / or fundamental frequency.
[0059] It can be understood that the motor vibration intensity and the motor vibration frequency are used to limit the vibration instructions corresponding to the target audio data to ensure the accuracy of the vibration instructions corresponding to the target audio data.
[0060] In some embodiments, the target audio data includes multiple different types of sub-audio data, and the different types of sub-audio data include at least two of ambient sound audio data, human voice audio data, animal sound audio data, mechanical sound audio data, white noise audio data, and effect sound audio data.
[0061] For example, target audio data in a shooting game includes ambient sound audio data and mechanical sound audio data. The ambient sound audio data includes audio data of bullets hitting the target, and the mechanical sound audio data includes audio data of a game character operated by a user firing a gun.
[0062] In an embodiment of the present application, it is detected that the amplitude of the target audio data is greater than the vibration threshold, and a corresponding vibration instruction is generated based on the target audio data, including: detecting that the amplitudes of at least two sub-audio data are greater than the vibration threshold; determining the sub-audio data with the largest amplitude among the at least two sub-audio data; and generating a vibration instruction based on the sub-audio data with the largest amplitude.
[0063] For example, the target audio data in a shooting game includes audio data of a bullet hitting a target and audio data of a game character operated by a user firing a gun. If the amplitude of the audio data of the game character operated by the user firing a gun is greater than the amplitude of the audio data of the bullet hitting the target, a vibration instruction is generated based on the audio data of the game character operated by the user firing a gun.
[0064] In this way, by generating vibration instructions only based on the sub-audio data with the maximum amplitude, the number of vibration instructions generated can be reduced, thereby reducing power consumption.
[0065] 104: The control terminal determines whether the vibration terminal is in a target state; wherein the target state indicates that: at least one key of the vibration terminal is in use; or, all keys of the vibration terminal are in an idle state for an idle time shorter than a preset target time.
[0066] If the judgment result is yes, go to 105: the control terminal sends a vibration instruction to the vibration terminal to drive the vibration terminal to vibrate; if the judgment result is no, go to 106: the control terminal does not send the vibration instruction to the vibration terminal.
[0067] In the embodiment of the present application, it is detected that the vibration terminal collects pressing information of the pressing operation, and it is determined that the vibration terminal is in use; it is detected that the vibration terminal does not collect pressing information of the pressing operation, and it is determined that the vibration terminal is in idle state.
[0068] For example, if it is detected that a key of the vibration terminal is pressed, it is determined that the key of the vibration terminal is in use; if it is detected that a key of the vibration terminal is not pressed, it is determined that the key of the vibration terminal is in idle state.
[0069] For example, Figure 2As shown, the vibration terminal is a handle, at least one button of the handle is detected to be pressed in frame t1, and all buttons of the handle are detected to be released in frame t2; at least one button of the handle is detected to be pressed in frame t4, and all buttons of the handle are detected to be released in frame t5; at least one button of the handle is detected to be pressed in frame t6, and all buttons of the handle are detected to be released in frame t7.
[0070] From frame t1 to frame t2, at least one button of the controller is in use, that is, from frame t1 to frame t2, the controller is in the target state. From frame t2 to frame t4, all buttons of the controller are in an idle state, and the idle duration is calculated starting from frame t2. When the idle duration of all buttons of the controller being in an idle state is less than the preset target duration, the controller is determined to be in the target state. If the preset target duration is T, and the period from frame t2 to frame t3 is less than T, the controller is determined to be in the target state from frame t2 to frame t3. If the period from frame t3 to frame t4 is greater than T, the controller is determined not to be in the target state from frame t3 to frame t4.
[0071] From frame t4 to frame t5, at least one button of the controller is in use, that is, from frame t4 to frame t5, the controller is in the target state. From frame t5 to frame t6, all buttons of the controller are in an idle state, and the idle duration is calculated starting from frame t5. When the idle duration of all buttons of the controller being in an idle state is less than the preset target duration, the controller is determined to be in the target state. If the preset target duration is T, and the period from frame t5 to frame t8 is less than T, the controller is determined to be in the target state from frame t5 to frame t8.
[0072] Among them, from frame t6 to frame t7 within the time period from frame t5 to frame t8, at least one button of the handle is in use, that is, from frame t6 to frame t7, the handle is in the target state. From frame t7 to frame t10, all buttons of the handle are in an idle state, and the idle duration is calculated starting from frame t7. When the idle duration of all buttons of the handle in the idle state is less than the preset target duration, the handle is determined to be in the target state. If the preset target duration is T, and the time period from frame t7 to frame t9 is less than T, then the handle is determined to be in the target state from frame t7 to frame t9. If the time period from frame t9 to frame t10 is greater than T, then it is determined that the handle is not in the target state from frame t9 to frame t10.
[0073] In the embodiment of the present application, the handle may include any one of a game handle, a racing handle, a flight handle, a fighting handle, a portable handle, a multi-function handle, a wireless handle, and an elite handle. Figure 1bThe handle 100 shown is suitable for action games and adventure games; the racing handle usually imitates the design of a car steering wheel and is suitable for racing games; the flight handle can simulate the joystick of an airplane, usually includes multiple buttons and switches, and is suitable for flight simulation games; the fighting handle is suitable for fighting games; the portable handle is suitable for games on mobile devices; the multi-function handle is suitable for users with customization options; the wireless handle is suitable for games that work with TV game consoles; and the elite handle is suitable for professional players and e-sports players.
[0074] It can be understood that the preset target duration can be determined based on user input or based on experience.
[0075] 105: The control terminal sends a vibration instruction to the vibration terminal to drive the vibration terminal to vibrate.
[0076] 106: The control terminal does not send a vibration instruction to the vibration terminal.
[0077] In an embodiment of the present application, the control terminal can poll the state of the vibration terminal in a polling cycle during the running of the game. When the control terminal detects that the vibration terminal is in the target state, it sends a vibration instruction to the vibration terminal to drive the vibration terminal to vibrate based on the vibration instruction; when the control terminal detects that the vibration terminal is not in the target state, it does not send a vibration instruction to the vibration terminal, and detects the amplitude of the target audio data of the next frame of the current frame. Among them, the polling cycle can be any positive number, for example, the polling cycle can be any one of 2s, 3s, 5s, etc.
[0078] After generating a vibration instruction, the embodiment of the present application determines whether to drive the vibration terminal to vibrate based on the vibration instruction in conjunction with the pressed state of the vibration terminal button, so as to ensure that the vibration instruction corresponds to the user operation and avoid the vibration instruction being obtained based on the noise of the target audio data, such as the vibration instruction being obtained based on the ambient sound audio data or the human voice audio data, thereby reducing the possibility of false vibration.
[0079] Combine the following Figure 4 The flowchart shown in FIG. 1 introduces a method for detecting that the amplitude of the target audio data is greater than the vibration threshold and generating a corresponding vibration instruction based on the target audio data, such as Figure 4 As shown, including:
[0080] 201: Get the audio frame stream.
[0081] In an embodiment of the present application, the control terminal can obtain the audio data of the currently running game as an audio frame stream. In other embodiments, the audio frame stream can be any one of the audio data of a movie, the audio data of a TV program, the audio data of music, the audio data of a podcast, the audio data of an audio book, etc.
[0082] 202: Determine the amplitude A of the current frame audio data according to the audio frame stream.
[0083] In an embodiment of the present application, the audio data of the current frame in the audio frame stream may be preprocessed first, and then the amplitude A of the audio data of the current frame may be determined according to the audio frame stream.
[0084] It can be understood that by preprocessing the current frame audio data in the audio frame stream, the interference of background noise in the current frame audio data can be removed or reduced, the clarity of the current frame audio data can be improved, a more reliable data basis can be provided for feature extraction of the current frame audio data, and the processing efficiency of the audio data can be improved. The preprocessing includes at least one of noise removal, data cleaning, noise reduction, filtering, sound enhancement, speech enhancement, audio repair, etc.
[0085] In an embodiment of the present application, a method for determining an amplitude A of audio data of a current frame according to an audio frame stream includes: reading the amplitude A of audio data of a current frame in the audio frame stream through audio processing software or a programming library. In other embodiments, the amplitude A of audio data of a current frame can be determined by performing audio processing on the audio of the current frame, such as any one of noise elimination, sound enhancement, sound synthesis, audio compression, echo suppression, spectrum analysis, audio equalizer, audio time domain processing, speech recognition, and audio effect processing.
[0086] 203: Determine an amplitude mean M of the first N frames of audio data according to the audio frame stream.
[0087] In the embodiment of the present application, the amplitude mean M can be determined based on the amplitude of the N frames before the current frame (or the i-th frame) of the target audio data, or based on the amplitude of the target audio data (current frame-1-N) frame to (current frame-1) frame to determine the amplitude mean M, the amplitude mean Where N>0, M x Indicates the amplitude of the x-th frame of audio data.
[0088] In some embodiments, the amplitude mean M can be determined based on the amplitudes of the target audio data from the 1st frame to the Nth frame. Where N>0, M x Indicates the amplitude of the x-th frame of audio data.
[0089] 204: Determine the amplitude variance V of the first N frames of audio data according to the audio frame stream.
[0090] In the embodiment of the present application, the amplitude variance V can be determined based on the amplitude of the target audio data N frames before the current frame, or based on the amplitude of the target audio data (current frame i-1-N) frame to (current frame-1) frame. The amplitude variance V is Where N is a positive integer greater than 0, M xrepresents the amplitude of the x-th frame of audio data, M represents the mean amplitude, and N is a positive integer greater than 0.
[0091] In the embodiment of the present application, the amplitude variance V can be determined based on the amplitudes of the target audio data from the first frame to the Nth frame. Among them, M x represents the amplitude of the x-th frame of audio data, M represents the mean amplitude, and N is a positive integer greater than 0.
[0092] 205: Determine whether A>(k*V+b)*M, where A is the amplitude of the current frame audio data, V is the amplitude variance of the previous N frames of audio data, M is the amplitude mean of the previous N frames of audio data, k is the adjustment coefficient of the amplitude variance V, and b is the adjustment parameter of the amplitude variance V. If the determination result is yes, go to 206: determine the spectral centroid of the current frame audio data; if the determination result is no, go to 201: obtain the audio frame stream.
[0093] It can be understood that in the embodiment of the present application, the vibration threshold = (k*V+b)*M. Corresponding to A>(k*V+b)*M, that is, the amplitude of the current frame audio data is greater than the vibration threshold, it is determined that the current scene is a vibration scene, and the process goes to 206: determining the spectral centroid of the current frame audio data; corresponding to A≤(k*V+b)*M, that is, the amplitude of the current frame audio data is less than or equal to the vibration threshold, it is determined that the current scene is not a vibration scene, and the process goes to 201: acquiring the audio frame stream.
[0094] 206: Determine the spectral centroid of the current frame audio data.
[0095] In an embodiment of the present application, a method for determining the spectral centroid of the current frame audio data includes: calculating the spectral centroid of the current frame audio data by a preset formula. For example, Wherein, f(i) represents the signal frequency of the i-th frame audio data, and E(i) represents the spectral energy of the corresponding frequency after short-time Fourier transform of the i-th frame audio data.
[0096] In other embodiments, a method for determining the spectral centroid of the current frame audio data includes: calculating the spectral centroid of the current frame audio data through an audio processing library, such as librosa.
[0097] It can be understood that the value of the spectral centroid can reflect the brightness of the sound. The higher the spectral centroid, the brighter the audio data sounds; the lower the spectral centroid, the darker the audio data sounds.
[0098] 207: Perform intensity mapping based on the amplitude of the current frame audio data.
[0099] In an embodiment of the present application, a method for intensity mapping based on the amplitude of the current frame audio data includes: converting the amplitude of the current frame audio data into a decibel value to represent the loudness of the current frame audio data through a decibel value.
[0100] 208: Perform frequency mapping based on the spectral centroid of the current frame audio data.
[0101] It can be understood that the spectral centroid refers to the "center of gravity" or "center position" of the spectrum. The spectral centroid indicates the center of gravity of the frequency component, which is the frequency value obtained by energy-weighted averaging of the frequencies within a certain frequency range, and the unit is Hertz (Hz). The spectral centroid reflects the brightness of the sound. If the spectral centroid is high, it means that the spectrum energy is mainly concentrated in the high-frequency part (or high spectral centroid), and the signal sounds sharper and brighter; if the spectral centroid is low, it means that the spectrum energy is mainly concentrated in the low-frequency part (or low spectral centroid), and the signal sounds low and heavy.
[0102] In an embodiment of the present application, intensity mapping can be performed based on the amplitude of the current frame audio data to obtain the motor vibration intensity; frequency mapping can be performed based on the spectral centroid of the current frame audio data to obtain the motor vibration frequency; and vibration instructions can be obtained based on the motor vibration intensity and the motor vibration frequency.
[0103] The embodiment of the present application detects the current frame audio data through a rhythm detection algorithm, and extracts audio feature information when it is determined that the amplitude of the current frame audio data is greater than the vibration threshold, so as to filter out the audio data that does not need to generate vibration instructions, thereby reducing the vibration instructions that need to be generated.
[0104] Combine the following Figure 5 The control method of the embodiment of the present application is introduced. Figure 5 As shown, in the process of a user using a handle 100 and a computer 200 to play a game, the generation module of the computer 200 can obtain the audio frame stream of the game, detect the current frame audio data in real time, and poll the use status of the handle 100. When it is determined based on the rhythm algorithm that the amplitude of the current frame audio data is greater than the amplitude threshold, a vibration instruction is generated based on the current frame audio data, and the vibration instruction is sent to the microcontroller unit (MCU) of the computer 200. After receiving the vibration instruction, the MCU queries the state of the handle 100, and when it is determined that the handle 100 is in use, the vibration instruction is sent to the handle 100, so that the handle 100 vibrates based on the vibration instruction. Among them, the target state indicates that: at least one button of the vibration terminal is in use; or, the idle time of all buttons of the vibration terminal being in an idle state is less than the preset target time.
[0105] It can be understood that the method provided in the embodiment of the present application can be applied to any electronic device, including but not limited to a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The embodiment of the present application does not limit the specific form of the electronic device.
[0106] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the above control method.
[0107] An embodiment of the present application also provides an electronic device, including a memory for storing instructions executed by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, for executing the instructions stored in the memory to implement the above-mentioned control method.
[0108] According to an embodiment of the present application, Figure 6 FIG. 1 is a block diagram of an electronic device 1300 based on a system on chip (SOC). Figure 6 In the FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SOCs. Figure 6In the embodiment, the electronic device 1300 includes: an interconnect unit 1350, which is coupled to the processor 1315; a system agent unit 1370; a bus controller unit 1380; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random-access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a special-purpose processor, such as, for example, a network or communication processor, a compression engine, or an embedded processor.
[0109] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to floppy disks, optical disks, optical disks, read-only memories (compact disc-read only memories, CD-ROMs), magneto-optical disks, read-only memories (read only memory, ROM), random access memories (random access memory, RAM), erasable programmable read-only memories (erasable programmable read-only memories, EPROM), electrically erasable programmable read-only memories (electrically erasable programmable read-only memories, EEPROM), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
[0110] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.
[0111] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.
[0112] It should be noted that in the examples and description of this patent, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises one" does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0113] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.
Claims
1. A control method, characterized in that: Applied to a control terminal, wherein the control method comprises: The control terminal establishes a connection with the vibration terminal; The control terminal acquires and plays the target audio data; When the control terminal detects that the amplitude of the target audio data is greater than the vibration threshold during the process of playing the target audio data, the control terminal generates a corresponding vibration instruction based on the target audio data; and The control terminal determines whether the vibration terminal is in a target state, wherein the target state is: at least one button of the vibration terminal is in use; or all buttons of the vibration terminal are in an idle state and the idle time is less than a preset target time; If the judgment result is yes, the control terminal sends the vibration instruction to the vibration terminal to drive the vibration terminal to vibrate.
2. The control method according to claim 1, characterized in that: include: Detecting that the vibration terminal collects pressing information of a pressing operation, and determining that the vibration terminal is in use; It is detected that the vibration terminal does not collect the pressing information of the pressing operation, and it is determined that the vibration terminal is in an idle state.
3. The control method according to claim 1, characterized in that: The vibration threshold is determined by the following method: Determine the target period; The vibration threshold is determined based on the amplitude mean and amplitude variance of the target audio data within the target time period, wherein the target time period includes any one of N frames before the current frame, N frames after the current frame, P frames before the current frame, and Q frames after the current frame, N>0, P>0, Q>0, and P+Q=N.
4. The control method according to claim 1, characterized in that: The method further comprises: If the judgment result is no, the control terminal does not send the vibration instruction to the vibration terminal.
5. The control method according to claim 1, characterized in that: The control terminal is any one of a computer, a mobile phone, a drone, a robot, a tablet computer, and a personal computer, and the vibration terminal is any one of a handle, a virtual reality headset, an augmented reality glasses, a tactile glove, a smart bracelet, and a smart watch.
6. The control method according to claim 1, characterized in that: The generating a corresponding vibration instruction based on the target audio data comprises: Extracting audio features from the target audio data to obtain audio feature information; Vibration mapping is performed based on the audio feature information to obtain a vibration instruction corresponding to the target audio data.
7. The control method according to claim 6, characterized in that: The audio feature information includes audio amplitude and spectrum centroid, and the vibration instruction includes motor vibration intensity and motor vibration frequency. The performing vibration mapping based on the audio feature information to obtain a vibration instruction corresponding to the target audio data includes: Mapping the audio amplitude to obtain the corresponding motor vibration intensity; The spectral centroid is mapped to obtain the corresponding motor vibration frequency.
8. The control method according to claim 1, characterized in that: The target audio data includes a plurality of sub-audio data of different types; The detecting that the amplitude of the target audio data is greater than a vibration threshold, and generating a corresponding vibration instruction based on the target audio data, comprises: When the amplitude of at least two of the sub-audio data is detected to be greater than a vibration threshold; Determine the sub-audio data with the maximum amplitude among the at least two sub-audio data; The vibration instruction is generated based on the sub audio data of the maximum amplitude.
9. The control method according to claim 8, characterized in that: The different types of sub-audio data include: at least two of: ambient sound audio data, human voice audio data, animal sound audio data, mechanical sound audio data, white noise audio data, and effect sound audio data.
10. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the control method as claimed in any one of claims 1 to 9.
11. An electronic device, characterized in that: It includes a memory for storing instructions executed by one or more processors of an electronic device; and a processor, which is one of the processors of the electronic device, for executing the instructions stored in the memory to implement the control method described in any one of claims 1 to 9.