Audio and vibration coordinated control method and device

Abstract

The present disclosure provides a method and device for coordinated control of audio and vibration, including, in the process of processing audio data and vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitude of audio data and vibration data based on the volume adjustment event, driving a speaker to work based on the audio data after the amplitude adjustment, and driving a vibration motor to work based on the vibration data after the amplitude adjustment. In the embodiment of the present disclosure, synchronous adjustment of audio playback and vibration output intensity can be achieved based on a single volume adjustment event, without entering the setting page to separately adjust the motor vibration intensity, which greatly facilitates the user's adjustment operation and improves the user's operation experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic equipment, and in particular to a method and device for coordinated control of audio and vibration. Background Art

[0002] Nowadays, there are more and more scenarios where audio and touch are coordinated in electronic devices. For example, when music is playing, the device vibrates to the rhythm of the music; in shooting games, the device vibrates to the sound of gunfire, etc. Currently, more and more manufacturers are conducting research on the coordinated control of device audio and touch to improve user experience.

[0003] However, in the related art, the signal pathways for audio and vibration effects are independent of each other, and the output intensities of the two cannot be adjusted synchronously, resulting in a poor user experience. Summary of the invention

[0004] In order to achieve synchronous control of the output intensity of audio and vibration effects, the embodiments of the present disclosure provide a method, device, electronic device and storage medium for coordinated control of audio and vibration.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for coordinated control of audio and vibration, comprising:

[0006] In the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitudes of the audio data and the vibration data based on the volume adjustment event;

[0007] The speaker is driven based on the audio data with the amplitude adjusted, and the vibration motor is driven based on the vibration data with the amplitude adjusted.

[0008] In some implementations, in the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitudes of the audio data and the vibration data based on the volume adjustment event includes:

[0009] The audio processing module receives the sound and vibration effect data sent by the audio hardware abstraction layer, wherein the sound and vibration effect data includes the audio data and the vibration data;

[0010] In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event;

[0011] The audio processing module adjusts the amplitude of the sound vibration effect data based on the first adjustment parameter.

[0012] In some implementations, the step of driving a speaker based on the audio data after adjusting the amplitude, and driving a vibration motor based on the vibration data after adjusting the amplitude, comprises:

[0013] The audio processing module performs channel separation on the sound and vibration effect data after adjusting the amplitude to obtain target audio data and target vibration data;

[0014] The speaker is driven to operate based on the target audio data, and the vibration motor is driven to operate based on the target vibration data.

[0015] In some implementations, in the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitudes of the audio data and the vibration data based on the volume adjustment event includes:

[0016] In the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, the application framework layer adjusts the amplitude of the vibration data based on the volume adjustment event;

[0017] The audio hardware abstraction layer sends an audio adjustment parameter to the audio processing module based on the volume adjustment event;

[0018] The audio processing module adjusts the amplitude of the audio data based on the audio adjustment parameter.

[0019] In some implementations, the step of driving a speaker based on the audio data after adjusting the amplitude, and driving a vibration motor based on the vibration data after adjusting the amplitude, comprises:

[0020] The vibration hardware abstraction layer drives the vibration motor to operate based on the vibration data with adjusted amplitude, and the audio processing module drives the speaker to operate based on the audio data with adjusted amplitude.

[0021] In some implementations, the application framework layer adjusts the amplitude of the vibration data based on the volume adjustment event, including:

[0022] The application framework layer determines a vibration adjustment parameter based on the volume adjustment event, and adjusts the amplitude of the vibration data based on the vibration adjustment parameter.

[0023] In some implementations, the sound and vibration effect data includes tactile OGG data.

[0024] In a second aspect, an embodiment of the present disclosure provides an audio and vibration coordinated control device, comprising:

[0025] an amplitude adjustment module, configured to synchronously adjust the amplitudes of the audio data and the vibration data based on the volume adjustment event in response to detecting the volume adjustment event during the processing of the audio data and the vibration effect data;

[0026] The driving control module is configured to drive the speaker to work based on the audio data with adjusted amplitude, and to drive the vibration motor to work based on the vibration data with adjusted amplitude.

[0027] In some implementations, the amplitude adjustment module is configured to:

[0028] The audio processing module receives the sound and vibration effect data sent by the audio hardware abstraction layer, wherein the sound and vibration effect data includes the audio data and the vibration data;

[0029] In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event;

[0030] The audio processing module adjusts the amplitude of the sound vibration effect data based on the first adjustment parameter.

[0031] In some embodiments, the drive control module is configured to:

[0032] The audio processing module performs channel separation on the sound and vibration effect data after adjusting the amplitude to obtain target audio data and target vibration data;

[0033] The speaker is driven to operate based on the target audio data, and the vibration motor is driven to operate based on the target vibration data.

[0034] In some implementations, the amplitude adjustment module is configured to:

[0035] In the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, the application framework layer adjusts the amplitude of the vibration data based on the volume adjustment event;

[0036] The audio hardware abstraction layer sends an audio adjustment parameter to the audio processing module based on the volume adjustment event;

[0037] The audio processing module adjusts the amplitude of the audio data based on the audio adjustment parameter.

[0038] In some embodiments, the drive control module is configured to:

[0039] The vibration hardware abstraction layer drives the vibration motor to operate based on the vibration data with adjusted amplitude, and the audio processing module drives the speaker to operate based on the audio data with adjusted amplitude.

[0040] In some implementations, the amplitude adjustment module is configured to:

[0041] The application framework layer determines a vibration adjustment parameter based on the volume adjustment event, and adjusts the amplitude of the vibration data based on the vibration adjustment parameter.

[0042] In some implementations, the sound and vibration effect data includes tactile OGG data.

[0043] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0044] speaker;

[0045] Vibration motor; and

[0046] A controller comprises a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method according to any embodiment of the first aspect.

[0047] In a fourth aspect, an embodiment of the present disclosure provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any embodiment of the first aspect.

[0048] The audio and vibration coordinated control method of the disclosed embodiment includes, in the process of processing audio data and vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitude of the audio data and the vibration data based on the volume adjustment event, driving the speaker to work based on the audio data after the amplitude adjustment, and driving the vibration motor to work based on the vibration data after the amplitude adjustment. In the disclosed embodiment, the synchronous adjustment of audio playback and vibration effect output intensity can be achieved based on a single volume adjustment event, without entering the setting page to separately adjust the motor vibration intensity, which greatly facilitates the user's adjustment operation and improves the user's operation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a structural block diagram of an electronic device according to some implementations of the present disclosure.

[0051] Figure 2 is a system architecture diagram of an electronic device in some implementations of the present disclosure.

[0052] Figure 3 It is a flowchart of a method for coordinated control of audio and vibration according to some embodiments of the present disclosure.

[0053] Figure 4 It is a flowchart of a method for coordinated control of audio and vibration according to some embodiments of the present disclosure.

[0054] Figure 5 It is a schematic diagram of a method for coordinated control of audio and vibration according to some embodiments of the present disclosure.

[0055] Figure 6 It is a flowchart of a method for coordinated control of audio and vibration according to some embodiments of the present disclosure.

[0056] Figure 7 It is a schematic diagram of a method for coordinated control of audio and vibration according to some embodiments of the present disclosure.

[0057] Figure 8 It is a structural block diagram of an audio and vibration coordinated control device according to some embodiments of the present disclosure.

[0058] Fig. 9 It is a structural block diagram of an electronic device according to some implementations of the present disclosure. DETAILED DESCRIPTION

[0059] The technical solution of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described implementation is a part of the implementation of the present disclosure, rather than all the implementations. Based on the implementation in the present disclosure, all other implementations obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. In addition, the technical features involved in the different implementations of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0060] Nowadays, there are more and more scenarios in which audio and touch work together in electronic devices, and users can get a better sensory experience in the audio-touch collaboration scenarios.

[0061] For example, in a shooting game scenario, through audio tactile synergy, the body of the device can vibrate synchronously with the sound of the gun when the user shoots, thus simulating a more realistic shooting effect. Another example is music with a strong sense of rhythm. Through audio tactile synergy, the body of the device can vibrate synchronously with the rhythm of the music, such as the drum beats, so that the user can have an immersive music experience.

[0062] Currently, more and more manufacturers have designed solutions for controlling the synchronous output of audio and vibration effects. However, in related technical solutions, the audio signal and vibration effect signal paths are independent of each other, and the user's single volume adjustment cannot synchronously adjust the vibration effect.

[0063] For example, in a scenario where a user plays music using the synergy of audio and vibration, when the user wants to increase the volume and vibration effects to get a stronger audio-visual experience, the user can only adjust the volume and vibration intensity separately. For example, the user can control the volume with the volume button, and then enter the settings page to adjust the vibration intensity.

[0064] For example, in another example scenario, the user sets the vibration effect when the phone rings. When a call comes in, the user can use the volume button to turn down the ringtone volume, but the vibration intensity cannot be reduced synchronously.

[0065] It can be seen that in the related technology, in the scenario where audio and vibration effects are coordinated, the output intensity of audio and vibration cannot be adjusted synchronously, which makes the operation complicated when adjusting the audio and vibration effects separately, and reduces the user experience.

[0066] Based on this, the embodiments of the present disclosure provide a method, device, electronic device and storage medium for coordinated control of audio and vibration, aiming to achieve synchronous adjustment of audio and vibration output intensity. Users only need to perform a single volume adjustment operation to synchronously adjust the vibration intensity.

[0067] The audio and vibration coordinated control method of the embodiment of the present disclosure can be applied to any electronic device with a speaker and a vibration motor. For example, the electronic device can be a smart phone, a tablet computer, a smart watch, etc., and the present disclosure does not limit this.

[0068] For example Figure 1 An example structural diagram of an electronic device 100 in some embodiments of the present disclosure is shown below. Figure 1 Provide explanation.

[0069] like Figure 1 As shown, in some embodiments, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0070] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0071] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0072] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0073] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0074] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0075] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] In an embodiment of the present disclosure, the processor 110 enables the electronic device to execute the audio and vibration coordinated control method of the present disclosure by running instructions stored in the internal memory 121 .

[0077] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the plurality of microphones 170C, the headphone jack 170D, and the application processor.

[0078] The audio module 170 is used to convert digital audio data into analog audio electrical signals for output, and also to convert analog audio electrical signals into digital audio data. For example, the audio module 170 is used to convert analog audio electrical signals output by the microphone 170C into digital audio data.

[0079] The audio module 170 may further include an audio processing module. The audio processing module is used to perform audio processing on digital audio data in the video recording mode to generate audio. The audio module 170 may also be used to encode and decode audio data.

[0080] In some implementations, the audio module 170 may be disposed in the processor 110 , or some functional modules of the audio module 170 may be disposed in the processor 110 .

[0081] Speaker 170A, also called "horn", is used to convert analog audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A. Receiver 170B, also called "earpiece", is used to convert analog audio electrical signals into sound signals. When the electronic device 100 answers a call or voice message, the voice can be answered by placing the receiver 170B close to the human ear. Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into analog audio electrical signals. When making a call or sending a voice message, the user can speak by placing the human mouth close to the microphone 170C to input the sound signal into the microphone 170C.

[0082] It is understood that the structure illustrated in the embodiments of the present disclosure does not constitute a specific limitation on the electronic device 100. In other embodiments of the present disclosure, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0083] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present disclosure takes the Android system of a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0084] Figure 2A structural block diagram of the software system of the electronic device 100 in some embodiments of the present disclosure is shown. The Android architecture divides the system into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces.

[0085] In some implementations, the Android system is divided into four layers, from top to bottom, namely, the application layer (Applications), the application framework layer (Application Framework), the hardware abstraction layer (HAL, Hardware Abstraction Layer) and the kernel layer (Linux Kernel).

[0086] The application layer is the layer that interacts with the user in the Android system. The application layer includes various applications of the mobile phone, such as system applications such as settings, contacts, text messages, calls, gallery, calendar, browser, or third-party applications such as WeChat. The application layer can access the services provided by the application framework layer according to different applications. Generally speaking, applications are developed using the Java language and are completed by calling the application programming interface (API) provided by the application framework layer.

[0087] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example Figure 2 As shown, the application framework layer may include media services (Media Server), audio services (Audio Server), camera services (Camera Server), system services (System Server), etc.

[0088] The HAL layer is an interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: the audio hardware abstraction layer (Audio HAL) and the vibration hardware abstraction layer (Vibrator HAL). Among them, the Audio HAL is used to process the audio stream, for example, to adjust the volume of the audio stream, and the Vibrator HAL is used to process the vibration effect stream.

[0089] The kernel layer provides the core system services of the operating system, such as security, memory management, process management, network protocol stack and driver model, all of which are based on the Linux kernel. The Linux kernel also serves as an abstraction layer between the hardware and software stacks. This layer has many drivers related to electronic devices, the main drivers are: display driver; Linux-based frame buffer driver; keyboard driver as input device; Flash driver based on memory technology device; camera driver; audio driver; motor driver; Bluetooth driver; Wi-Fi driver, etc.

[0090] Based on the understanding of the above system architecture, Figure 3 The method for implementing the coordinated control of audio and vibration disclosed in the present invention is described.

[0091] like Figure 3 As shown, in some embodiments, the audio and vibration coordinated control method of the example of the present disclosure includes:

[0092] S310 . In a process of processing audio data and vibration data, in response to detecting a volume adjustment event, synchronously adjusting the amplitudes of the audio data and the vibration data based on the volume adjustment event.

[0093] S320: Drive the speaker based on the audio data with adjusted amplitude, and drive the motor based on the vibration data with adjusted amplitude.

[0094] In the disclosed embodiments, the electronic device can process audio data and vibration data based on the aforementioned system architecture. The audio data can drive the speaker of the electronic device to make sound, while the vibration data can drive the vibration motor of the electronic device to vibrate.

[0095] In the embodiment of the present disclosure, when the speaker and the vibration motor are working, in response to detecting a volume adjustment event, it indicates that the user desires to adjust the audio loudness (ie, volume), such as increasing or decreasing the volume of the audio playback.

[0096] In this case, in the embodiments of the present disclosure, the audio data is not only adjusted based on the event, but the vibration data is also adjusted synchronously based on the event, thereby realizing a single volume adjustment event, and the audio playback loudness and vibration intensity can be adjusted synchronously.

[0097] It can be understood that the loudness of audio playback is also the volume, and the volume is essentially the vibration amplitude of the speaker diaphragm, so the amplitude of the audio data also represents the volume, and adjusting the volume of audio playback is also adjusting the amplitude of the audio data. Similarly, the intensity of motor vibration is essentially the amplitude of the motor vibration, so the amplitude of the vibration data also represents the intensity of the vibration, and adjusting the vibration intensity of the motor vibration is also adjusting the amplitude of the vibration data.

[0098] A volume adjustment event refers to an event triggered by a user to adjust the audio playback volume. For example, taking a smartphone as an example, a user can trigger a volume adjustment event by pressing a volume button on the side of the phone, or by touching the phone screen.

[0099] Volume adjustment events generally include turning up the volume and turning down the volume. In the embodiment of the present disclosure, the user can achieve synchronous adjustment of audio playback and vibration intensity through different volume adjustment events.

[0100] In an embodiment of the present disclosure, when a volume adjustment event is detected, an adjustment parameter can be first determined based on the volume adjustment event, where the adjustment parameter represents a coefficient for adjusting the amplitude of the audio data and the vibration data. The amplitude of the audio data and the vibration data can then be synchronously adjusted based on the adjustment parameter, and then the speaker and the vibration motor can be driven to operate based on the adjusted audio data and vibration data, thereby completing the output intensity adjustment of the audio playback and the vibration effect.

[0101] It is understandable that in the audio and vibration effect coordinated control scheme in the related art, the volume of audio playback and the intensity of the vibration effect are controlled independently. Therefore, when the user wants to adjust the volume and vibration size at the same time, he first needs to use the volume button to adjust the volume to the appropriate level, and then needs to enter the settings page to further adjust the motor vibration intensity. The operation is very cumbersome and the adjustment efficiency is low.

[0102] In the embodiments of the present disclosure, synchronous adjustment of audio playback and vibration output intensity can be achieved based on a single volume adjustment event. For example, when the user lowers the volume, the intensity of the motor vibration can be simultaneously lowered, and when the user raises the volume, the intensity of the motor vibration can be simultaneously increased. There is no need to enter the settings page to separately adjust the motor vibration intensity, which greatly facilitates user adjustment operations and improves user operating experience.

[0103] It is worth mentioning that there are two main ways to coordinate audio and vibration in the relevant technology:

[0104] 1) Encapsulate the audio data and vibration effect data in the same audio and vibration effect file.

[0105] For example, an OGG file is a file that includes both audio data and vibration effect data. An OGG file may include multiple channels. For example, a conventional OGG music file includes three channels, two of which are audio data and the other is vibration effect data.

[0106] 2) Audio data and vibration effect data are two independent files.

[0107] For example, audio data can be packaged into a file in an audio format, such as a file with a suffix of .MP3, and vibration effect data can be packaged into a file in a vibration effect format, generally a file with a suffix of .bin.

[0108] In the embodiments of the present disclosure, different system control modes are set for the above two situations to achieve synchronous adjustment of audio data and vibration effect data, which are explained below respectively.

[0109] like Figure 4 and Figure 5 As shown, in some embodiments, the audio and vibration coordinated control method of the example of the present disclosure includes:

[0110] S410: The audio processing module receives the sound vibration effect data sent by the audio hardware abstraction layer.

[0111] In the embodiments of the present disclosure, the sound and vibration effect data refers to a data file that contains both audio data and vibration data. For example, in one example, the sound and vibration effect data may be tactile OGG data, which is the aforementioned OGG data file. The OGG data file includes 3 channels, 2 of which are audio data and 1 is vibration data. Those skilled in the art can understand this, and the present disclosure will not elaborate on it.

[0112] Combination Figure 5 As shown, the sound vibration effect data generated by the application layer (APP) is sent to the application framework layer (Framework), and the application framework layer can call the API interfaces of the media service (Media Server) and the audio service (AudioServer) to perform corresponding data processing on the sound vibration effect data.

[0113] The audio hardware abstraction layer (Audio HAL) included in the hardware abstraction layer (HAL) serves as an interface layer between the system and the hardware circuit, and can send the received sound vibration effect data to the audio processing module ADSP.

[0114] ADSP, also known as Audio DSP (Digital Signal Process), is a hardware for audio data processing, i.e., the audio processing module described in the present disclosure. In the embodiments of the present disclosure, the audio processing module ADSP can be used to adjust the amplitude of audio data and vibration data.

[0115] When ADSP processes the audio and vibration effect data, it can separate the channels of the audio and vibration effect data, thereby separating the audio data and vibration data in the audio and vibration effect data. Then, ADSP can send the audio data and vibration data to the audio driver and motor driver of the kernel layer respectively, and the audio driver and motor driver drive the speakers and vibration motors of the hardware layer to achieve synchronous generation of audio playback and vibration effects.

[0116] S420: In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event.

[0117] Combination Figure 5 As shown, in the embodiment of the present disclosure, when the user triggers a volume adjustment event, for example, the user triggers the volume adjustment event by pressing the volume key, or for example, the user triggers the volume adjustment event by a touch operation, at this time, the application framework layer (Framework) can detect the volume adjustment event, and then send the volume adjustment event to the audio hardware abstraction layer (Audio HAL).

[0118] The audio hardware abstraction layer (Audio HAL) can determine a first adjustment parameter k according to the volume adjustment event, where the first adjustment parameter k represents a coefficient for adjusting the amplitude of the sound vibration effect data. Then, the audio hardware abstraction layer (Audio HAL) can send the first adjustment parameter k to the processing module of the audio processing module ADSP by means of parameter setting.

[0119] In some embodiments, the first adjustment parameter k may be positively correlated with the volume value, that is, the higher the volume value corresponding to the volume adjustment event, the larger the value of the first adjustment parameter k; conversely, the lower the volume value corresponding to the volume adjustment event, the smaller the value of the first adjustment parameter k.

[0120] S430: The audio processing module adjusts the amplitude of the sound vibration effect data based on the first adjustment parameter.

[0121] Combination Figure 5 As shown, after the audio processing module ADSP receives the first adjustment parameter k, the processing module can adjust the amplitude of the current sound vibration effect data by the first adjustment parameter k. For example, in one example, the amplitude parameter of the current sound vibration effect data can be multiplied by the first adjustment parameter k to achieve the amplitude adjustment of the sound vibration effect data.

[0122] It can be understood that since the sound vibration effect data at this time includes both audio data and vibration effect data, when the amplitude is adjusted based on the first adjustment parameter k, the amplitude of the audio data and the vibration effect data can be adjusted at the same time, thereby realizing a single volume adjustment operation to synchronously adjust the audio playback volume and vibration intensity.

[0123] S440, the volume processing module performs channel separation on the sound and vibration effect data after adjusting the amplitude to obtain target audio data and target vibration data.

[0124] Combined with the above, it can be known that after the audio processing module ADSP adjusts the amplitude of the sound and vibration effect data in combination with the first adjustment parameter k, the sound and vibration effect data is still a package file containing both audio data and vibration effect data. Then, the audio processing module ADSP needs to perform channel separation on the sound and vibration effect data to obtain the audio data and vibration data after the amplitude is adjusted, that is, the target audio data and target vibration data described in the present disclosure.

[0125] S450: driving the speaker to operate based on the target audio data, and driving the vibration motor to operate based on the target vibration data.

[0126] Combination Figure 5 As shown, the audio processing module ADSP sends the target audio data and the target vibration data to the corresponding driver of the kernel layer (Kernel) respectively.

[0127] For example, in one example, the target audio data is sent to the audio driver (PA Driver) of the kernel layer through the TDM Port interface, and then the speaker of the hardware layer (Hardware) can be driven to make sound, so as to adjust the volume of the audio playback. At the same time, the target vibration data is sent to the motor driver (Haptic Driver) of the kernel layer through the SWR Port interface, and then the vibration motor of the hardware layer (Hardware) can be driven to vibrate, so as to adjust the vibration intensity.

[0128] From the above, it can be seen that in the embodiments of the present disclosure, synchronous adjustment of audio playback and vibration output intensity can be achieved based on a single volume adjustment event. For example, when a user plays a shooting game, the vibration intensity will also be stronger while adjusting the volume. There is no need to enter the settings page to separately adjust the motor vibration intensity, which greatly facilitates user adjustment operations and improves user operating experience.

[0129] like Figure 6 and Figure 7 As shown, in some embodiments, the audio and vibration coordinated control method of the present disclosure example includes:

[0130] S610: During the processing of audio data and vibration data, in response to detecting a volume adjustment event, the application framework layer adjusts the amplitude of the vibration data based on the volume adjustment event.

[0131] Combination Figure 7 As shown, different from Figure 4 , Figure 5 Implementation method, in Figure 6and Figure 7 In the implementation manner, the audio data and the vibration data are two data files respectively. For example, in one example, the audio data may be an audio file with a suffix of .MP3, and the vibration data may be a vibration effect file with a suffix of .bin.

[0132] For vibration data, since the data stream of vibration data does not pass through the audio processing module ADSP, the amplitude adjustment of vibration data cannot be implemented in ADSP. Therefore, in the embodiment of the present disclosure, the amplitude adjustment of vibration data can be performed in the application framework layer (Framework).

[0133] For example Figure 7 In an implementation manner, when the application framework layer detects a volume adjustment event, the amplitude of the vibration data may be adjusted according to the volume adjustment event by an adjustment module located in the application framework layer.

[0134] In some embodiments, the adjustment module can determine the vibration adjustment parameter j based on the volume adjustment event, where the vibration adjustment parameter j represents the coefficient that needs to be used to adjust the amplitude of the vibration data. The vibration adjustment parameter j can be positively correlated with the volume value, that is, the higher the volume value corresponding to the volume adjustment event, the larger the value of the vibration adjustment parameter j. Conversely, the lower the volume value corresponding to the volume adjustment event, the smaller the value of the vibration adjustment parameter j.

[0135] After the vibration adjustment parameter j is determined, the amplitude of the vibration data can be adjusted according to the vibration adjustment parameter j. For example, in one example, the amplitude parameter of the current vibration data can be multiplied by the vibration adjustment parameter j to achieve the amplitude adjustment of the vibration data.

[0136] S620: The audio hardware abstraction layer sends an audio adjustment parameter to the audio processing module based on the volume adjustment event.

[0137] S630: The audio processing module adjusts the amplitude of the audio data based on the audio adjustment parameter.

[0138] Combination Figure 7 As shown, for audio data, the general processing flow is the same as the above Figure 4 , Figure 5 The implementation methods are the same, except that since the audio data no longer contains vibration data, when a volume adjustment event is detected, the audio hardware abstraction layer (Audio HAL) sends the audio adjustment parameters to the audio processing module ADSP for amplitude adjustment based on the volume adjustment event. There is no need to perform channel separation processing, but directly send the audio data after the amplitude adjustment to the kernel layer (Kernel). Those skilled in the art can understand this in combination with the foregoing, and this disclosure will not elaborate on it.

[0139] S640: The vibration hardware abstraction layer drives the vibration motor to operate based on the vibration data with adjusted amplitude, and the audio processing module drives the speaker to operate based on the audio data with adjusted amplitude.

[0140] Combination Figure 7 As shown, for vibration data, after the application framework layer (Framework) adjusts the amplitude of the vibration data, the adjusted vibration data can be sent to the vibration hardware abstraction layer (Vibrator HAL). The vibration hardware abstraction layer (Vibrator HAL) serves as an interface layer between the system and the hardware circuit, and can send the received vibration data to the motor driver (Haptic Driver) of the kernel layer (Kernel), and then drive the vibration motor of the hardware layer (Hardware) to vibrate, thereby adjusting the vibration intensity.

[0141] For audio data, after the audio processing module ADSP adjusts the amplitude of the audio data, the adjusted audio data can be sent to the audio hardware abstraction layer (Audio HAL). The audio hardware abstraction layer (Audio HAL) serves as an interface layer between the system and the hardware circuit. It can send the received audio data to the audio driver (PADriver) of the kernel layer (Kernel), and then drive the speakers of the hardware layer (Hardware) to make sound, thereby adjusting the volume.

[0142] From the above, it can be seen that in the embodiments of the present disclosure, synchronous adjustment of audio playback and vibration output intensity can be achieved based on a single volume adjustment event. For example, when a user plays a shooting game, the vibration intensity will also be stronger while adjusting the volume. There is no need to enter the settings page to separately adjust the motor vibration intensity, which greatly facilitates user adjustment operations and improves user operating experience.

[0143] In some embodiments, the present disclosure provides an audio and vibration coordinated control device, which can be applied to any electronic device with a speaker and a vibration motor. For example, the electronic device can be a smart phone, a tablet computer, a smart watch, etc., and the present disclosure does not limit this.

[0144] like Figure 8 As shown, in some embodiments, the audio and vibration coordinated control device of the example of the present disclosure includes:

[0145] The amplitude adjustment module 10 is configured to synchronously adjust the amplitudes of the audio data and the vibration data based on the volume adjustment event in response to detecting the volume adjustment event during the processing of the audio data and the vibration effect data;

[0146] The driving control module 20 is configured to drive the speaker to work based on the audio data with adjusted amplitude, and to drive the vibration motor to work based on the vibration data with adjusted amplitude.

[0147] In some implementations, the amplitude adjustment module 10 is configured to:

[0148] The audio processing module receives the sound and vibration effect data sent by the audio hardware abstraction layer, wherein the sound and vibration effect data includes the audio data and the vibration data;

[0149] In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event;

[0150] The audio processing module adjusts the amplitude of the sound vibration effect data based on the first adjustment parameter.

[0151] In some embodiments, the drive control module 20 is configured to:

[0152] The audio processing module performs channel separation on the sound and vibration effect data after adjusting the amplitude to obtain target audio data and target vibration data;

[0153] The speaker is driven to operate based on the target audio data, and the vibration motor is driven to operate based on the target vibration data.

[0154] In some implementations, the amplitude adjustment module 10 is configured to:

[0155] In the process of processing the audio data and the vibration data, in response to detecting a volume adjustment event, the application framework layer adjusts the amplitude of the vibration data based on the volume adjustment event;

[0156] The audio hardware abstraction layer sends an audio adjustment parameter to the audio processing module based on the volume adjustment event;

[0157] The audio processing module adjusts the amplitude of the audio data based on the audio adjustment parameter.

[0158] In some embodiments, the drive control module 20 is configured to:

[0159] The vibration hardware abstraction layer drives the vibration motor to operate based on the vibration data with adjusted amplitude, and the audio processing module drives the speaker to operate based on the audio data with adjusted amplitude.

[0160] In some implementations, the amplitude adjustment module 10 is configured to:

[0161] The application framework layer determines a vibration adjustment parameter based on the volume adjustment event, and adjusts the amplitude of the vibration data based on the vibration adjustment parameter.

[0162] In some implementations, the sound and vibration effect data includes tactile OGG data.

[0163] From the above, it can be seen that in the embodiments of the present disclosure, synchronous adjustment of audio playback and vibration output intensity can be achieved based on a single volume adjustment event. For example, when a user plays a shooting game, the vibration intensity will also be stronger while adjusting the volume. There is no need to enter the settings page to separately adjust the motor vibration intensity, which greatly facilitates user adjustment operations and improves user operating experience.

[0164] In some embodiments, the present disclosure provides an electronic device, including:

[0165] speaker;

[0166] Vibration motor; and

[0167] The controller includes a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method described in any of the above embodiments.

[0168] In some embodiments, the present disclosure provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any of the above embodiments.

[0169] Specifically, Fig. 9 A schematic diagram of the structure of a device 600 suitable for implementing the method disclosed in the present invention is shown. Fig. 9 The device shown can realize the corresponding functions of the above-mentioned processor and storage medium.

[0170] like Fig. 9 As shown, the device 600 includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the memory 602 or the program loaded from the storage part 608 into the memory 602. In the memory 602, various programs and data required for the operation of the device 600 are also stored. The processor 601 and the memory 602 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0171] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0172] In particular, according to an embodiment of the present disclosure, the above method process can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611.

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for coordinated control of audio and vibration, characterized in that: include: The audio processing module receives the sound and vibration effect data sent by the audio hardware abstraction layer, wherein the sound and vibration effect data includes the audio data and the vibration data; In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event; wherein the first adjustment parameter is positively correlated with the adjusted volume value corresponding to the volume adjustment event; The audio processing module adjusts the amplitude of the sound vibration effect data based on the first adjustment parameter; The audio processing module performs channel separation on the sound and vibration effect data after adjusting the amplitude to obtain target audio data and target vibration data; The speaker is driven to operate based on the target audio data, and the vibration motor is driven to operate based on the target vibration data.

2. The method according to claim 1, characterized in that The sound and vibration effect data includes tactile OGG data.

3. A method for coordinated control of audio and vibration, characterized in that: include: In the process of processing audio data and vibration data, in response to detecting a volume adjustment event, the application framework layer determines a vibration adjustment parameter based on the volume adjustment event, and adjusts the amplitude of the vibration data based on the vibration adjustment parameter, wherein the vibration adjustment parameter is positively correlated with the adjusted volume value corresponding to the volume adjustment event; The audio hardware abstraction layer sends an audio adjustment parameter to the audio processing module based on the volume adjustment event, and the audio processing module adjusts the amplitude of the audio data based on the audio adjustment parameter, wherein the audio adjustment parameter is positively correlated with the adjusted volume value corresponding to the volume adjustment event; The vibration hardware abstraction layer drives the vibration motor to operate based on the vibration data with the amplitude adjusted, and the audio processing module drives the speaker to operate based on the audio data with the amplitude adjusted.

4. An audio and vibration coordinated control device, characterized in that: include: Audio Hardware Abstraction Layer; An audio processing module, configured to receive the sound and vibration effect data sent by the audio hardware abstraction layer, wherein the sound and vibration effect data includes the audio data and the vibration data; In response to detecting a volume adjustment event, the audio hardware abstraction layer sends a first adjustment parameter to the audio processing module based on the volume adjustment event; wherein the first adjustment parameter is positively correlated with the adjusted volume value corresponding to the volume adjustment event; The audio processing module adjusts the amplitude of the sound and vibration effect data based on the first adjustment parameter; and performs channel separation on the sound and vibration effect data after the amplitude adjustment to obtain target audio data and target vibration data; drives the speaker to operate based on the target audio data, and drives the vibration motor to operate based on the target vibration data.

5. An electronic device, characterized in that: include: speaker; Vibration motor; as well as A controller comprises a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 3.

Images