Method, electronic device, medium and program product for performing cabin linkage

Abstract

Embodiments of the present disclosure provide a method for performing cabin linkage. The method includes obtaining first audio from a vehicle entertainment system or a device in a vehicle. The method further comprises the steps of comparing the first audio with a pre-established vehicle sound effect library, and executing cabin linkage corresponding to a first sound effect when the first audio is matched with the first sound effect in the vehicle sound effect library. By using the method, the vehicle control cabin linkage can be effectively triggered in time in the operation process of the user, and accurate and efficient immersive cabin experience is brought to the user.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of vehicles, and more particularly to methods, electronic devices, media, and computer program products for performing cockpit linkage. Background Art

[0002] With the development of smart cockpit chips, in-car entertainment features are proliferating. Car computers can now include an increasing number of built-in games. Leveraging the computing power and ecosystem of the car or the mobile phone, these devices can access hardware and software resources such as the in-car display and audio system to provide users with a visual and auditory entertainment experience.

[0003] In-car entertainment features can transform a vehicle from a simple means of transportation into a mobile entertainment space. This immersive gaming experience immerses users in the game scene, creating a richer and more realistic gaming experience through visual, auditory, and even tactile stimulation. Summary of the Invention

[0004] Embodiments of the present disclosure are methods, electronic devices, media, and computer program products for executing cockpit linkage.

[0005] According to a first aspect of the present disclosure, a method for performing cockpit interaction is provided. The method includes obtaining a first audio signal from a vehicle entertainment system or a device in the vehicle. The method also includes comparing the first audio signal with a pre-established vehicle sound effect library, and, in response to the first audio signal matching a first sound effect in the vehicle sound effect library, performing cockpit interaction corresponding to the first sound effect.

[0006] According to a second aspect of the present disclosure, an electronic device for performing cockpit linkage is provided, which includes a processor and a memory, which is coupled to the processor and stores instructions. When these instructions are executed by the processor, the device performs the steps of the method for performing cockpit linkage according to an embodiment of the present disclosure.

[0007] In a third aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When these computer-executable instructions are executed, the computer executes the steps of the method for executing cockpit linkage according to an embodiment of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the steps of the method for performing cockpit linkage according to an embodiment of the present disclosure.

[0009] Please note that the present invention summary is provided to introduce some concepts in a simplified form, which will be further described in the following detailed description. The present invention summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0011] Figure 1 is a schematic diagram of an example application scenario of a method for executing cockpit linkage according to an embodiment of the present disclosure;

[0012] Figure 2 A flowchart of a method for executing cockpit linkage according to an embodiment of the present disclosure is shown;

[0013] Figure 3 A schematic diagram illustrating a method for determining audio or sound effect features according to an embodiment of the present disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating another method for determining audio or sound effect features according to an embodiment of the present disclosure is shown;

[0015] Figure 5 A schematic diagram illustrating a method for establishing a vehicle sound effect library according to an embodiment of the present disclosure is shown;

[0016] Figure 6 A flowchart illustrating a method for sound effect matching according to an implementation method of the present disclosure is shown;

[0017] Figure 7 Another flowchart illustrating a method for sound effect matching according to an implementation method of the present disclosure; and

[0018] Figure 8 Schematic blocks of devices that can be used to implement embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0020] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0021] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". Although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" and similar terms include all combinations of any one, multiple and all of the associated listed items.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0024] When users interact with the vehicle's cockpit, it's currently impossible to instantly and accurately identify when a user triggers an interaction event, such as an event within a game. Some approaches might involve negotiating with game developers to provide programming interfaces, but these methods are costly.

[0025] To at least address the aforementioned and other potential issues, embodiments of the present disclosure provide a method for implementing cockpit linkage. The method includes acquiring a first audio signal from a vehicle entertainment system or a device within the vehicle. The method also includes comparing the first audio signal with a pre-established vehicle sound effect library, and when the first audio signal matches a first sound effect within the library, executing a cockpit linkage corresponding to the first sound effect. This method enables timely and effective triggering of vehicle-control-cockpit linkage during user operation, providing users with a precise, efficient, and immersive cockpit experience.

[0026] The basic principles and several exemplary implementations of the present disclosure are described below with reference to the following figures. It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0027] Figure 1 FIG is a schematic diagram of an example application scenario of a method for executing cockpit linkage according to an embodiment of the present disclosure. Figure 1 As shown, the example environment 100 includes a vehicle 110, on which elements or components such as a display screen 120 of a vehicle controller, a smart seat 130, a headlight 140, and an air-conditioning vent 150 are arranged. It should be understood that Figure 1 The elements shown in FIG. 1 are merely exemplary, and the vehicle 110 may include more or fewer elements or components, and the present disclosure is not limited in this regard.

[0028] When a user plays a game in vehicle 110 via a smartphone or display 120, the vehicle controller's sound detector can detect and capture the game audio from the smartphone or display 120. The vehicle controller can then compare the captured game audio with the sound effects stored in its sound effects library. Once a matching sound effect is accurately identified, the corresponding in-cabin interaction is performed. According to embodiments of the present disclosure, in-cabin interaction can include the vehicle controller controlling the vehicle speakers to play or amplify matching game sound effects. This provides the user with a more powerful audio experience through the vehicle's high-quality speaker system, thereby enhancing the user's immersive gaming experience. In some embodiments, the vehicle speakers can also play other sound effects or music associated with the captured sound effects. For example, in some embodiments, the vehicle speakers can select a related sound effect or music that best matches the captured primary sound effect, for example, based on factors such as the emotional atmosphere of the game scene or the type of sound effect, and play that sound effect or music. In some embodiments, other sound effects or music can also be mixed with the captured primary sound effect to create a richer auditory experience for the user.

[0029] In some embodiments, the vehicle controller can also turn on or off the vehicle's 110 lights 140 based on the currently determined game sound effects. According to embodiments of the present disclosure, the vehicle controller can also control the flashing pattern and duration of the lights 140. In some embodiments, the lights 140 can also include interior lighting within the vehicle 110. According to embodiments of the present disclosure, the vehicle controller can also change the color of the vehicle's 110 lights 140 based on the current game sound effects, thereby creating a suitable gaming atmosphere for the user.

[0030] For example, in some embodiments, when an explosion occurs in a game, the vehicle's interior lights can instantly turn red, simulating the effect of an explosion. In some embodiments, when a user is traveling through a forest in an exploration game, a sound detector detects the rustling of leaves as the user walks through the jungle, and the vehicle's interior lights can turn green while playing matching natural sounds, such as birdsong or flowing water, to enhance the natural atmosphere and the feeling of jungle exploration. According to embodiments of the present disclosure, when a user is playing a space exploration game, the vehicle's interior lights can use dark blue or white to simulate the vastness of space.

[0031] In some embodiments, the vehicle controller can also adjust the wind speed or temperature of the vehicle's air-conditioning outlet 150 based on the current game sound effects. For example, when the sound speaker detects the sound made by the user passing through the desert in the game, the air-conditioning temperature can be increased to simulate the high temperature of the desert. When the sound speaker detects the sound made by the user in the winter scene in the game, the air-conditioning temperature can be reduced and a lower temperature wind can be provided to simulate winter. In some embodiments of the present disclosure, different vehicle fragrance release schemes can also be preset based on in-game events. For example, when the user passes through a garden in the game, the scent of flowers can be released. When simulating a beach scene, the smell of the sea can be released. In some embodiments, the fragrance can also be changed according to the driving situation, such as releasing a scent to remind the spirit when driving at night.

[0032] In some embodiments, the vehicle controller can also change the state of the seat 130 of the vehicle 110 based on the current game sound effects. For example, vibration effects can be applied to simulate in-game actions or environmental influences. In some embodiments, the vibration applied to the seat 130 can be subtle to simulate the vibration feeling of driving on an uneven road. In some embodiments, the vibration applied to the seat 130 can be more intense to simulate explosions or impact effects in the game. In some embodiments, when the sound effects of a high-speed car chase game are detected, the vehicle seat 130 can also simulate the feeling of being pushed back by the car as it accelerates.

[0033] The above describes a schematic diagram of a method for executing cockpit linkage according to an embodiment of the present disclosure to create an immersive gaming space environment for users. It should be understood that the above examples are not intended to be limiting, but are merely illustrative for the purpose of helping understanding, and the embodiments of the present disclosure are not limited to the above examples. Figure 2 1 and 2 are flowcharts of a method 200 for executing cockpit linkage according to an embodiment of the present disclosure.

[0034] At box 202, a first audio is obtained from a vehicle entertainment system or a device in the vehicle. According to an embodiment of the present disclosure, the vehicle's sound detector can detect sound audio emitted by a user device or a vehicle speaker device. These sound audios can be acquired or sampled at intervals of several seconds and buffered in a buffer, and then fed into the vehicle controller. As an example, the first audio can be the sound made by the user during a game. Additionally or alternatively, in some embodiments, the first audio can also be audio emitted by software or hardware in the vehicle, such as a message control system, office software application, etc. According to an embodiment of the present disclosure, the vehicle's entertainment system can be a device in the vehicle, such as a smart phone, laptop computer, tablet computer, mobile game console, and other devices.

[0035] At block 204, the first audio is compared with a pre-established vehicle sound library. The vehicle controller may compare the acquired first audio with the vehicle sound library stored in the vehicle memory. According to an embodiment of the present disclosure, the vehicle controller may first determine or extract a first feature set of the first audio. For example, the first feature set may include factors such as audio energy, audio frequency, amplitude, rhythm, etc. of the first audio. The specific method for determining the audio feature set will be described in detail below. Figure 3 and Figure 4 Provide a description.

[0036] The vehicle controller can then use its audio processing capabilities to compare the extracted first feature set with the second feature set stored in the vehicle sound effect library. According to an embodiment of the present disclosure, the second feature set can be pre-recorded and analyzed features representing various confirmed sound effects. Similarly, the second feature set can include factors such as audio energy, audio frequency, amplitude, rhythm, etc. For example, the second feature set can be sound effect features related to racing cars in a racing game, such as the sounds of the car in different situations, including engine sounds, brake sounds, collision sounds, etc.

[0037] According to an embodiment of the present disclosure, the vehicle controller may compare the similarity between the first feature set and the second feature set, such as the similarity or degree of matching of audio energy or audio frequency. In some embodiments, the similarity between the first feature set and the second feature set may also be determined by determining the Euclidean distance, cosine similarity, or other methods between the features.

[0038] At block 206 , in response to the first audio matching a first sound effect in the vehicle sound effect library, cockpit interaction corresponding to the first sound effect is executed. According to embodiments of the present disclosure, two audios are considered sufficiently similar when a similarity score between them is determined to be above a predetermined threshold. In some embodiments, the threshold can be dynamically adjusted based on actual application requirements or game type to balance false positives and false negatives.

[0039] According to an embodiment of the present disclosure, the first sound effect can correspond to a specific event in the game. The vehicle controller can perform cabin interaction based on the specific event identifier. For example, when the first sound effect corresponds to a loud explosion, the vibration function of the vehicle seat can be activated, the interior vehicle lighting can be adjusted to a red color that matches the explosion, and the air conditioning temperature can be increased to simulate the impact and heat of the explosion. In some embodiments, when a sound effect corresponding to the sound of an accelerating engine is determined, the interior vehicle ambient lighting can be switched to a dynamic mode to simulate the feeling of acceleration, and the seat can be adjusted to tilt back to simulate the inertia effect during acceleration.

[0040] Figure 3 The diagram illustrates a method 300 for determining audio or sound effect characteristics according to an embodiment of the present disclosure. According to some embodiments of the present disclosure, an audio processing module in a vehicle controller may divide audio from a buffer into multiple audio frames, such as audio frame 302, audio frame 304, and audio frame 306. Dashed lines within a frame may represent different frequencies or frequency bands. For example, the dashed lines between dashed lines 311 and 313 may represent different frequency bands from high to low.

[0041] For example, multiple audio frames can represent audio at different times. For example, audio frame 302 can represent audio at time t, audio frame 304 can represent audio at time t+1, and audio frame 306 can represent audio at time t+2. Multiple audio frames enable the audio processing module to process the audio signal in each small period of time separately, thereby enabling detailed analysis of the temporal characteristics of the audio signal, such as sound changes, rhythm, and the instant when specific events occur.

[0042] According to an embodiment of the present disclosure, the audio processing module can identify energy values ​​and energy peaks within multiple audio frames. According to an embodiment of the present disclosure, the energy value can be a quantification of the audio intensity within the time frame. The energy value can be determined by calculating the sum of squares of all samples within the frame or other similar methods. The magnitude of the energy value can be characterized by the loudness or volume level of the sound.

[0043] In some embodiments, an energy peak may indicate the maximum energy value within an audio frame within a local frequency band. For example, energy peaks F1t 308, F2t 310, and F3t 312 at time t 302, and energy peaks F4(t+2) 314, F5(t+2) 316, F6(t+2) 318, and F7(t+2) 320 at time t+2 306, etc. According to embodiments of the present disclosure, energy peaks may indicate intra-band characteristics or intensity changes of important events in the acquired audio signal, such as musical beats, speech emphasis, or specific sound events such as explosions.

[0044] In some embodiments, the audio processing module may use the identified energy peaks F1t 308, F2t 310, and F3t 312 as a feature set. For example, the feature set may include elements such as a first feature [F1t, delta_t, F4(t+2)], a second feature [F1t, delta_t, F5(t+2)], and a third feature [F1t, delta_t, F6(t+2)], where delta_t represents a time increment. The feature set may represent the change in energy peak value of the audio from time t to time t+2, thereby reflecting the temporal correlation between energy peaks at different time points.

[0045] In some embodiments, the audio processing module may compare the determined feature set with a feature set of sound effects in a vehicle sound effect library of the vehicle, for example, by comparing the similarity or matching degree of energy peaks in the same frequency or frequency band between the two feature sets, such as by using methods such as cosine similarity, Euclidean distance, and Manhattan distance. When it is determined that the matching degree between the two is close to a predetermined threshold, the audio processing module may determine that the vehicle sound effect library contains a sound effect corresponding to the feature set.

[0046] The vehicle controller can then perform cockpit interaction based on the identifier of the game event corresponding to the sound effect, such as adjusting the vehicle lights, activating seat vibration, or adjusting the audio system output, thereby simulating or reacting to specific events in the game.

[0047] Figure 4The figure illustrates another method 400 for determining audio or sound effect features according to an embodiment of the present disclosure. According to some embodiments of the present disclosure, the audio processing module in the vehicle controller can divide the audio from the buffer into multiple audio frames, such as audio frame 402, audio frame 404, and audio frame 406. As an example, the multiple audio frames can represent audio at different times. For example, audio frame 402 can represent audio at time t, audio frame 404 can represent audio at time t+1, and audio frame 406 can represent audio at time t+2.

[0048] In some embodiments, the audio processing module may identify the spectral centroid for each frequency band, such as 408 and 412. In some embodiments, the audio processing module may further identify the spectral centroid for all frequency bands within each audio frame, such as 414, 416, and 418, by using a Fourier transform or other method. According to an embodiment of the present disclosure, the spectral centroid may be the center of gravity of a frequency component, which is used to reflect the center of gravity position of the sound spectrum energy distribution.

[0049] According to an embodiment of the present disclosure, the spectral centroid can be determined by energy-weighted averaging within a certain frequency range, thereby representing the "center point" of the audio energy distribution within a specific frequency band and the frequency distribution characteristics of the audio signal within the frequency band. In some cases, for example, when the spectral centroid is higher, the sound sounds "brighter" or "sharper"; if the spectral centroid is lower, the sound sounds "darker" or "muffled". For example, in some embodiments, the climax of game music may have a higher spectral centroid value, while the character dialogue portion of the game may have a lower spectral centroid value.

[0050] According to an embodiment of the present disclosure, the full-band spectrum centroids 414, 416, and 418 can represent the center of gravity of the overall frequency distribution of audio frames 402, 404, and 406. The audio processing module can connect these full-band spectrum centroids 414, 416, and 418 to form a full-band spectrum centroid graph and use it as a feature set for the acquired audio. For example, in some embodiments, the audio processing module can connect the determined full-band spectrum centroids 414, 416, and 418 in chronological order to form a continuous curve graph as a feature set.

[0051] This feature set reflects how the spectral centroid of an audio signal changes over time. For example, in some embodiments, the full-band spectral centroid plot may show a noticeable rise when an audio melody transitions from a low pitch to a climax, reflecting an increase in the "brightness" or "sharpness" of the audio signal. In some embodiments, the feature set based on the full-band spectral centroid can also be used to identify emotions in game characters.

[0052] In some embodiments, the audio processing module may further use each spectral centroid of the spectral centroid for each frequency band (e.g., 408 and 412, etc.) as an element in a per-band spectral centroid matrix. The audio processing module may then use the spectral centroid matrix as a feature set of the acquired audio. For example, according to an example of the present disclosure, the spectral centroid matrix may be expressed as follows:

[0053]

[0054] In matrix (1), each row can represent a frequency band, each column can represent an audio frame, and the corresponding element value (such as H11, H12) can represent the spectral centroid value of the frequency band. The matrix can represent the frequency distribution characteristics of the audio signal acquired in a continuous time period in each frequency band.

[0055] According to the method implemented in the embodiments of the present disclosure, the audio processing module can capture and analyze the frequency characteristics of audio signals in a structured and efficient manner, thereby supporting more complex and sophisticated audio processing applications in games. Additionally or alternatively, in some embodiments, the acquired audio can also be analyzed based on factors such as the audio's pitch, volume, timbre, time domain characteristics (such as the fluctuation of the sound), and spectral characteristics.

[0056] Figure 5 The figure illustrates a schematic diagram of a method 500 for establishing a vehicle sound effects library according to an embodiment of the present disclosure. According to some embodiments of the present disclosure, the audio processing module in the vehicle controller can divide music or sound effects uploaded by the vehicle manufacturer or user into multiple audio frames, such as audio frame 502, audio frame 504, and audio frame 506. As an example, the multiple audio frames can represent audio at different times. For example, audio frame 502 can represent audio at time t, audio frame 504 can represent audio at time t+1, and audio frame 506 can represent audio at time t+2.

[0057] In some embodiments, the audio processing module may then utilize the reference Figure 3 The method described in the identifies the energy peaks within each frequency band in each frame of the spectrum of the uploaded music or sound effects. In some embodiments, the audio processing module can then use the reference Figure 4 The method described in is used to identify elements such as the spectrum centroid within each frequency band and the full frequency band centroid in each frame of the spectrum.

[0058] Additionally or alternatively, in some embodiments, the audio processing module may further determine the frequency center value F of each frequency band of each frame. For example, the spectral centroid may be calculated for each frame of audio, and the centroid may be considered as the center value of the frequency distribution of the audio of that frame. The audio processing module may design the matching system for sound effect matching as a sound effect slot model based on different slot width ranges of high and low frequency bands. According to an embodiment of the present disclosure, each slot range may be F+ / -alpha_f, where alpha_f may represent the redundancy of the allowed frequency deviation.

[0059] According to an embodiment of the present disclosure, since the high frequency band is less affected by background noise, alpha_f can be set to a smaller value, making the slot range more compact. In some embodiments, for the low frequency band, since it is more affected by background noise, alpha_f can be set to a larger value, thereby providing a larger fault tolerance space. In some embodiments, a redundancy sequence [alpha_f0, alpha_fn] can be set, which can grow exponentially to adapt to changes in the sensitivity of different frequencies to background noise. Depending on the game application scenario, parameters such as the redundancy growth factor, the number and range of frequency bands can be adjusted accordingly to achieve the best effect. In this way, according to an embodiment of the present disclosure, the noise impact of the game background music can be shielded, and the robustness of the sound slot model can be enhanced.

[0060] According to an embodiment of the present disclosure, the sound effect slot model used for matching or comparison can utilize one or more features such as the audio brush model 508 based on energy peak, full-band spectrum centroid, spectrum centroid, etc. to extract the characteristics of the target sound effect or the real-time audio stream of the game and design the slot. According to an embodiment of the present disclosure, the audio brush model 508 can be brushed on a specific slot model, leaving behind a model that can match or characterize the audio features. For example, in some embodiments, each target sound effect (1-3 seconds) can correspond to multiple groups of "sound brushes". Each "sound brush" can be a feature set. The audio data of the target sound effect is preprocessed, and then converted from time-frequency data to spectrum through short-time Fourier transform, and the "sound brush" is further extracted.

[0061] For example, in some embodiments, the audio brush in the audio brush model 508 may represent audio frequency characteristics, such as spectral centroid, frequency band energy, the occurrence of specific frequencies, etc. In some embodiments, the audio brush model 508 may represent audio temporal characteristics, such as the duration of a sound, the time interval between sound events, etc. In some embodiments, the audio brush model 508 may represent audio statistical characteristics, such as the average amplitude and variance of the audio signal, etc.

[0062] According to embodiments of the present disclosure, each slot can represent one or more of these characteristics. The sound effect slot model can then compare and match the characteristics in the real-time audio stream with the characteristics of the target sound effect in the sound effect card. When the pin combination of the audio brush can match the current slot, it means that the current slot triggers the target sound effect.

[0063] For example, in some embodiments, the sound effect slot model can compare the energy peak slots in the real-time audio stream with the corresponding slots of the target sound effect. The degree of match can be determined based on the similarity of the energy peaks and their position in the frequency band. In some embodiments, the sound effect slot model can also analyze the structural similarity between the full-band spectrum centroid connection diagram of the real-time audio stream and the corresponding full-band spectrum centroid connection diagram of the target sound effect. In some embodiments, the sound effect slot model can also analyze the similarity between the spectrum centroid matrix and the matrix of the target sound effect.

[0064] In some embodiments, these features can be considered separately. In some embodiments, these features can also be considered comprehensively through weighted averaging or setting priority rules. When the degree of match between the slot of the real-time audio stream and the slot of a target sound effect reaches a predetermined threshold, it can be determined that the match is consistent. In some embodiments, the sound effect slot model can be based on a machine learning model such as a deep learning model.

[0065] Figure 6 FIG. 6 shows a flowchart 600 for sound effect matching according to a method implemented in the present disclosure. Figure 6 As shown, in some embodiments, the car manufacturer and / or user can use the audio brush model 604 to extract key features of the target sound effect or music 602. In some embodiments, the target sound effect or music 602 may include, but is not limited to, specific vehicle prompt sounds, environmental sounds, or specific game-related sound effects. The key features may include one or more feature data such as a spectrum centroid sequence or spectrogram information. These feature data can form a unique sound fingerprint for subsequent sound effect matching and identification.

[0066] The car manufacturer and / or user can then construct a sound effect library 606 based on the extracted features. For example, the extracted features can be matched to sound effect names, and then the sound effect names can be mapped to event identifiers in the game, defining a clear context for each sound effect. This ensures that cockpit interaction can be triggered based on the sound of specific events occurring in the game. In some embodiments, constructing the sound effect library can include collecting or creating sound effects suitable for in-game events from various resources.

[0067] For example, these sound effects can be existing audio clips or custom-made by a sound designer. In some embodiments, the collected sound effects can also be categorized. For example, they can be categorized into categories such as ambient sound, character action sound, and game prompt sound based on their purpose, type, or context associated with specific events in the game. In some embodiments, one or more associated game event identifiers can be defined for each sound effect, thereby establishing a mapping relationship between sound effects and game events. In this way, when a specific event occurs in the game, the vehicle controller can quickly find the corresponding sound effect to play or trigger a linkage.

[0068] In some embodiments, the vehicle's sound detector can detect the real-time game audio stream 612 and cache it in a buffer 616. According to an embodiment of the present disclosure, the buffer 616 can cache 2 seconds of audio data and continuously update, for example, 8K data at a time, to ensure that sufficient data is collected for subsequent analysis. According to an embodiment of the present disclosure, the vehicle controller can splice multiple audio frames in the audio data 614 and extract features 618 from the audio frames using the sound card slot model 610.

[0069] The vehicle controller can match 608 the features 618 in the audio frame with the features in the sound effect library. The matching strategy can be based on Figure 3 and Figure 4 The methods described, such as the matching degree of the energy peak frequency band slots, the similarity of the connection diagram structure of the full-band spectrum centroid, and the similarity of the sub-band spectrum centroid matrix, etc. When a match is detected, the corresponding cockpit linkage can be triggered, thereby improving the user's gaming experience. For example, the output of the vehicle's audio system is dynamically adjusted so that the sound comes from different directions and provides the user with an immersive auditory experience. In some embodiments, visual effects or information related to the game can also be displayed on the vehicle display to increase the interactivity and information richness of the game. In some embodiments, driving safety-related prompts can also be displayed on the vehicle display, such as using specific sound effects to remind the driver to pay attention to the surrounding environment or driving status, so that the gaming experience can be combined with actual driving safety.

[0070] Figure 7 Another flowchart 700 for sound effect matching according to a method implemented in the present disclosure is shown. Figure 7 The method described in Figure 6 In some embodiments, Figure 7 Some steps of the method described in Figure 6 The steps of the method described in are the same or similar.

[0071] like Figure 7As shown, in some embodiments, the car manufacturer and / or user can extract 704 key features of a target game sound effect or music clip 702. In some embodiments, the car manufacturer and / or user can then build a sound effect library 706 based on the extracted features, for example, by matching the extracted features with the sound effect name. In some embodiments, audio pre-processing 710 can be performed for the target game sound effect, and the audio can be converted to a spectrum. The frequency band and time interval of the brightest part of the region in the spectrogram are extracted as the sound effect features to build the game sound effect library 706.

[0072] In some embodiments, a combination of local high-energy time-frequency segments based on a sound effect spectrogram can be used as a sound effect feature. This not only requires little computation, but can also effectively reflect the personalized characteristics of the sound effect in real time. For example, a fast Fourier transform or short-time Fourier transform can be performed on the target sound effect to convert the audio signal from the time domain to the frequency domain to generate a spectrogram of the audio signal. On the spectrogram, the brightest areas are analyzed and identified. These areas can represent the frequency components with the highest energy in the sound effect, such as a single bright spot or a bright line that lasts for a period of time. These areas can represent the characteristic frequency and duration of the sound effect.

[0073] In some embodiments, the vehicle's sound detector can detect the real-time game audio stream 708 and pre-process the detected music 710, such as by real-time decoding, resampling, or streaming, to extract features from the detected audio 712. For example, in some embodiments, the sampling rate of the audio signal can be changed by increasing or decreasing the number of samples to match the vehicle's processing requirements or equipment standards. In some embodiments, the extracted feature data can also be uploaded to a cloud server associated with the vehicle, leveraging the more powerful processing power and large-scale data storage capabilities of cloud computing to improve the efficiency and accuracy of sound effect matching, reduce the burden on the vehicle's hardware, and save the vehicle's computing resources and storage space.

[0074] According to an embodiment of the present disclosure, at 714, a frequency band feature match can be performed between the target game sound effect features in the sound effect library and the features of the real-time audio stream 708. At 716, a determination is made as to whether the degree of match between the two exceeds a predetermined threshold. If the degree of match exceeds the predetermined threshold, at 718, the vehicle controller can output a corresponding game event identifier or event ID. If the degree of match is less than the predetermined threshold, at 720, the vehicle controller can wait for the next sound effect match.

[0075] At 722 , the vehicle controller can trigger the corresponding vehicle-control-cockpit linkage based on the game event identifier. The method implemented in this embodiment of the present disclosure can accurately identify game sound effects based on in-band feature matching, obtain the game event ID, and trigger the in-cockpit vehicle-control linkage, providing users with a precise and efficient immersive cockpit experience.

[0076] Figure 8 A schematic block diagram of an example device 800 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The vehicle device in the embodiment can be implemented using device 800. As shown in the figure, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or computer program instructions loaded from storage unit 808 into random access memory (RAM) 803. Various programs and data required for the operation of device 800 can also be stored in RAM 803. CPU 801, ROM 802, and RAM 803 are connected to each other via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0077] Various components in device 800 are connected to I / O interface 805, including: an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a memory 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0078] The various processes and procedures described above, such as method 200, may be performed by processing unit 801. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of method 200 and process 300 described above may be performed.

[0079] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0080] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0081] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0082] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0083] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0084] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0085] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0086] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0087] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for performing cockpit linkage, comprising: Obtaining first audio from a vehicle entertainment system or a device in the vehicle; comparing the first audio with a pre-established library of vehicle sound effects; as well as In response to the first audio being matched with a first sound effect in the vehicle sound effect library, cockpit linkage corresponding to the first sound effect is performed.

2. The method according to claim 1, wherein executing the cockpit linkage corresponding to the first sound effect comprises one or more of the following: Playing a second sound effect or music determined to be associated with the first sound effect; turning the lights of said vehicle on or off; changing the color of the light of the vehicle; Adjusting the air conditioning speed or temperature of the vehicle; or Changing a seat state of the vehicle.

3. The method of claim 1 , wherein comparing the first audio to a vehicle sound effects library comprises: determining a first feature set for the first audio, wherein the first feature set is associated with one or more of audio energy and audio frequency; as well as The first feature set is compared with a second feature set stored in a vehicle sound effect library.

4. The method of claim 3, wherein determining the first feature set comprises: determining an audio energy peak of the audio energy for a plurality of time frames of the first audio; as well as The first set of features is determined based on the audio energy peak.

5. The method of claim 3, wherein determining the first feature set comprises: determining a full-frequency centroid for the audio frequencies for each of a plurality of time frames of the first audio; as well as The first feature set is determined based on the full-frequency centroid.

6. The method of claim 3, wherein determining the first feature set further comprises: determining a frequency centroid for each of the audio frequencies for each of the plurality of time frames of the first audio; determining a frequency centroid matrix based on the frequency centroid for each of the audio frequencies; as well as The first feature set is determined based on the frequency centroid matrix. 7 . The method of claim 3 , wherein the second feature set is determined based on audio energy and audio frequency of sound effects stored in the vehicle sound effect library.

8. The method of claim 3, wherein comparing the first feature set with the second feature set stored in the vehicle sound effects library comprises: comparing one or more of an audio energy peak, a full frequency centroid, and a frequency centroid matrix of the first feature set with the second feature set to determine similarity; as well as In response to the similarity being within a threshold range, it is determined that the first audio and the first sound effect associated with the second feature set are consistent.

9. The method according to claim 1, wherein obtaining the first audio comprises: receiving a plurality of audio frames within a predetermined time period; as well as The multiple audio frames are combined to form the first audio.

10. An electronic device for performing cockpit linkage, comprising: processor, and A memory is coupled to the processor and stores instructions, which, when executed by the processor, cause the device to perform the method according to any one of claims 1 to 9. 11 . A computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed, the computer is caused to perform the method according to claim 1 .

12. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions which, when executed, cause a machine to perform the method according to any one of claims 1 to 9.

Images