Model training method, vehicle audio parameter prediction method, device and medium

Through the knowledge transfer learning method of the student-teacher framework, combined with multimodal data of on-board and non-board music scenes, the problem of the neglected sound quality influence in the existing technology is solved, more accurate sound parameter adjustment is achieved, and user experience is improved personalization and sound quality.

CN118175478BActive Publication Date: 2024-11-22CHONGQING SELIS PHOENIX INTELLIGENT INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410230654.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-29
Publication Date
2024-11-22
Estimated Expiration
2044-02-29

AI Technical Summary

Technical Problem

The existing vehicle audio parameter adjustment technology focuses on user preferences and ignores the impact of sound quality, resulting in the inability to provide a better auditory experience.

Method used

Using the knowledge transfer learning method of the student-teacher framework, use multimodal sample data of non-vehicle music scenes to train the teacher network, learn high-quality audio parameter adjustment knowledge, and transfer it to the car music scene, and combine it with car music scenes through the student network. The data can realize the modeling and adaptation of users’ personalized music preferences, taking into account user preferences and sound quality.

Benefits of technology

It realizes more accurate and personalized audio parameter adjustment in different driving scenarios and at different time points, improving user pleasure and comfort, and providing a more personalized, rich and diverse music experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118175478B_ABST
    Figure CN118175478B_ABST
Patent Text Reader

Abstract

The present application relates to a model training method, a vehicle audio parameter prediction method, a device and a medium. The method comprises: obtaining multimodal sample data in a vehicle music scene; inputting the multimodal sample data into a teacher network and a student network to be trained that share a partial network structure, respectively, wherein the teacher network is pre-trained to a convergence state through multimodal sample data in a non-vehicle music scene; obtaining the audio parameter prediction information outputted by the teacher network and the student network respectively, recorded as the first audio parameter prediction information and the second audio parameter prediction information; training the student network according to the first audio parameter prediction information and the second audio parameter; obtaining a vehicle audio parameter prediction model based on the trained student network. The embodiments of the present application can take into account the influence of user preferences and audio quality on user experience, and recommend to users audio parameters that both meet user preferences and improve audio quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of in-vehicle acoustics, and in particular to a model training method, a vehicle audio parameter prediction method, a device and a medium. Background Art

[0002] The quality of in-car audio is directly related to the driving experience of drivers and passengers. By adjusting the sound parameters reasonably, we can not only improve the sound quality and create a more realistic and clear sound effect, but also create an immersive auditory atmosphere that suits different scenes and music types. This not only helps to improve the comfort and pleasure of driving, but also creates a more personalized and pleasant driving space, allowing drivers to feel a more pleasant and relaxed driving experience in the car. Therefore, adjusting the parameters of in-car audio is not only a pursuit of technological innovation, but also a key link in providing users with a comprehensive and personalized driving experience.

[0003] At present, the relevant technologies for adjusting the parameters of in-vehicle audio are mainly based on the real-time environmental information monitored in the car (such as the noise situation in the car, whether the passengers in the car are talking, etc.) and the user's personal information (such as the user's preference for various types of music, etc.) to recommend audio parameter adjustment plans to users. The relevant technologies can recommend audio parameters that are more in line with the user's preferences. However, the relevant technologies still have some shortcomings, which are mainly reflected in the fact that they focus on the impact of user preferences on user experience, while ignoring the impact of audio quality on user experience. Therefore, the audio parameters recommended to users may be in line with the user's personal preferences, but the audio parameters may not bring better audio quality, making it impossible for users to enjoy a better listening experience. Summary of the invention

[0004] In response to the above-mentioned deficiencies or shortcomings, the present application provides a model training method, a vehicle audio parameter prediction method, a device and a medium. The embodiments of the present application can take into account the impact of user preferences and audio quality on user experience, and recommend to users audio parameters that both meet user preferences and improve audio quality.

[0005] The present application provides a model training method according to a first aspect. In some embodiments, the method includes:

[0006] Obtain multimodal sample data in the car music scene;

[0007] The multimodal sample data are respectively input into a teacher network and a student network to be trained that share a partial network structure, wherein the teacher network is pre-trained to a convergence state using multimodal sample data in non-car music scenes; the number of non-car music scenes is one or more, and the multimodal sample data in the non-car music scenes includes scene information, audio layout information, and audio parameter setting information of all non-car music scenes;

[0008] Acquire the sound parameter prediction information outputted by the teacher network and the student network respectively, recorded as first sound parameter prediction information and second sound parameter prediction information;

[0009] training a student network based on the first acoustic parameter prediction information and the second acoustic parameter;

[0010] The vehicle audio parameter prediction model is obtained based on the trained student network.

[0011] In some embodiments, training the student network according to the first sound parameter prediction information and the second sound parameter includes:

[0012] Calculating a squared difference distance between the first acoustic parameter prediction information and the second acoustic parameter as model loss information;

[0013] The student network is trained based on the model loss information.

[0014] In some embodiments, after the multimodal sample data is respectively input into the teacher network and the student network to be trained that share a partial network structure, the method further includes:

[0015] Performing feature extraction processing and sound parameter prediction processing on the multimodal sample data through a teacher network to obtain first multimodal feature information and first sound parameter prediction information;

[0016] The multimodal sample data is subjected to feature extraction processing and sound parameter prediction processing through the student network to obtain second multimodal feature information and second sound parameter prediction information.

[0017] In some embodiments, training the student network according to the first sound parameter prediction information and the second sound parameter includes:

[0018] Calculating the similarity between the first multimodal feature information and the second multimodal feature information as first loss information;

[0019] calculating a square difference distance between the first acoustic parameter prediction information and the second acoustic parameter as second loss information;

[0020] The student network is trained according to the first loss information and the second loss information.

[0021] In some embodiments, training the student network according to the first loss information and the second loss information includes:

[0022] Adding the first loss information and the second loss information to obtain total loss information;

[0023] The student network is trained based on the total loss information, and the training goal is to reduce the total loss information until the preset conditions are met.

[0024] In some embodiments, the multimodal sample data includes in-vehicle audio information, vehicle environment related information, and user historical music preference information;

[0025] The teacher network includes a multimodal self-attention network for performing feature extraction processing on the above-mentioned multimodal sample data to obtain first multimodal feature information, and the student network includes an on-board multimodal self-attention network for performing feature extraction processing on the above-mentioned multimodal sample data to obtain second multimodal feature information; the above-mentioned multimodal self-attention network and the on-board multimodal self-attention network share some network weights.

[0026] According to a second aspect, the present application provides a method for predicting vehicle audio parameters. In some embodiments, the method includes:

[0027] Obtain real-time multimodal data related to the target car audio system;

[0028] Inputting the real-time multimodal data into a vehicle audio parameter prediction model trained by the training method provided by any of the above embodiments, and obtaining the audio parameter prediction information output by the vehicle audio parameter prediction model;

[0029] The sound parameters of the target vehicle audio system are adjusted according to the sound parameter prediction information.

[0030] In some embodiments, the above method further comprises:

[0031] The layout of application icons and / or the activation status of specific functions in the display interface of the vehicle screen are dynamically adjusted according to the audio parameter prediction information.

[0032] In some embodiments, adjusting the sound parameters of the target vehicle audio system according to the sound parameter prediction information includes:

[0033] Real-time monitoring of the vehicle environment where the target vehicle audio system is located;

[0034] Based on the monitoring results, the acoustic parameter prediction information is dynamically adjusted through the domain control mechanism;

[0035] The sound parameter of the target vehicle audio system is adjusted based on the adjusted sound parameter prediction information.

[0036] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the model training method or the vehicle audio parameter prediction method provided in any of the above embodiments are implemented.

[0037] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method or the vehicle audio parameter prediction method provided in any of the above embodiments are implemented.

[0038] In the above-mentioned embodiment of the present application, the high-quality audio parameter adjustment knowledge in other scenarios is transferred to the in-vehicle music scenario through the knowledge transfer learning method of the student-teacher framework, wherein the multimodal sample data in the non-in-vehicle music scenario, that is, the scene information, audio layout information and audio parameter setting information of all non-in-vehicle music scenarios are used to train the teacher network, so that the teacher network learns the high-quality audio parameter adjustment knowledge in other scenarios, and the multimodal sample data in the in-vehicle music scenario is used to train the student network, so that the student network learns the audio parameter adjustment knowledge of the in-vehicle music scenario, and realizes the modeling and adaptation of the user's personalized music preferences, so that the user's audio parameter requirements in different driving scenarios and at different time points can be more accurately and personalized. The student network and the teacher network share part of the network structure, and the output of the trained teacher network is used to train the student network to transfer the audio parameter adjustment knowledge learned by the teacher network in other scenarios to the student network across scenarios, so that the student network can learn from the tuning experience of other scenarios, thereby taking into account the impact of user preferences and audio quality on user experience, and recommending to users audio parameters that both meet user preferences and improve audio quality.

[0039] Furthermore, the above embodiment also introduces a self-attention mechanism to establish associations in multimodal data, which can not only learn the internal relationships of music features, but also understand the dependencies between different audio parameters, so that the teacher network and the student network can better learn the knowledge of audio parameter adjustment and perform knowledge transfer, and ultimately achieve better sound performance in the in-car music scene.

[0040] In addition, the above-mentioned embodiment also utilizes features extracted based on the self-attention mechanism, namely, the first multimodal feature information and the second multimodal feature information to perform feature alignment, so as to make the feature representations of the teacher network and the student network as similar as possible, ensuring that the teacher network and the student network are also consistent in terms of music feature representation, so that the student network can better learn the knowledge transmitted by the teacher network, which helps to improve the performance and adaptability of the student network in the in-vehicle music scene, and thus better meet the user's music needs, and provide the user with a more personalized, richer and more diverse music experience, thereby enhancing the user's pleasure and comfort throughout the driving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a model training method provided by the present application according to one or more embodiments;

[0042] Figure 2 A schematic diagram of a process for training a student network according to one or more embodiments of the present application;

[0043] Figure 3 A schematic diagram of a student-teacher framework provided by the present application according to one or more embodiments;

[0044] Figure 4 A schematic diagram of a flow chart of a method for predicting vehicle audio parameters provided by the present application according to one or more embodiments;

[0045] Figure 5 This is a diagram of the internal structure of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0047] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0048] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0049] In view of the shortcomings or defects of the relevant technology, the present application provides a model training method, which is used to train a vehicle audio parameter prediction model, and the model can generate vehicle audio parameters for the vehicle audio system, wherein the method transfers high-quality audio parameter adjustment knowledge in other scenarios to the vehicle music scene through the knowledge transfer learning method of the student-teacher framework, specifically using multimodal sample data in non-vehicle music scenes to train a teacher network, so that the teacher network learns high-quality audio parameter adjustment knowledge in other scenes, and using multimodal sample data in the vehicle music scene to train a student network, so that the student network learns the vehicle music scene. The audio parameter adjustment knowledge is used to model and adapt to the user's personalized music preferences, so that the user's audio parameter needs in different driving scenarios and at different time points can be met more accurately and closely to the individual. In addition, the student network and the teacher network share part of the network structure, and then the output of the trained teacher network is used to train the student network. The audio parameter adjustment knowledge learned by the teacher network in other scenarios can be passed to the student network, so that the student network can draw on the tuning experience of other scenarios, thereby taking into account the impact of user preferences and audio quality on user experience, and recommending to users audio parameters that both meet user preferences and improve audio quality.

[0050] In some embodiments, the model training method includes: Figure 1 The steps shown are described below.

[0051] S110: Acquire multimodal sample data in a car music scene.

[0052] The multimodal sample data in the in-car music scene refers to the multimodal data in the in-car music scene used to train the student network.

[0053] After clearly informing the user about the purpose and storage method of multimodal data, if the user authorizes, the multimodal data in the in-car music scene will be stored in a manner that complies with relevant laws and regulations. The multimodal data in the in-car music scene includes audio signal characteristics collected at multiple times, environmental noise, vehicle driving information, user's historical music preferences, vehicle equipment type, and other related information related to the in-car audio, the in-car and out-of-car environment, and the user's personal preferences. This information can be subdivided into multiple categories of information, such as in-car audio information, vehicle environment related information, and user's historical music preference information.

[0054] Among them, the in-car audio information includes the voice information of the passengers in the car.

[0055] The vehicle environment-related information includes vehicle information and external environment information. The vehicle information may include driving-related information such as vehicle speed, acceleration, and temperature inside the vehicle. It may also include vehicle audio equipment type and its layout information inside the vehicle (such as the setting positions of each speaker of the car audio), vehicle seating space distribution information (such as the distribution information of each seat in the car), and other vehicle audio-related information. The external environment information may include weather type (such as sunny, cloudy, heavy rain, rainstorm, light rain, etc.), external scene type (such as highway scene, urban traffic scene, suburban traffic scene, etc.), and other information.

[0056] The user's historical music preference information may be historical music play records and / or music collection records, wherein the historical music play records include the play time, play count, music type, etc. corresponding to each piece of music played by the user within a preset time period in the past, and the music collection records include each piece of music in the user's collection list.

[0057] The above-mentioned multimodal data can be collected using professional sensors such as microphones, cameras, and vehicle sensors. It can also be connected to the Internet to obtain information such as weather information and record it synchronously to ensure the temporal consistency of the data.

[0058] By using multi-source multimodal data as training data, the trained student network can more comprehensively understand the current driving environment and user needs, realize real-time dynamic adjustment of audio parameters, and improve the adaptability of the vehicle audio system, so that it can flexibly respond to different driving scenarios, including urban driving, high-speed driving, etc. It also helps the trained student network to comprehensively understand all aspects of audio adjustment needs, including voice interaction, music preferences, vehicle information, and environmental noise, so as to meet user needs more comprehensively and accurately.

[0059] S120: Input the above multimodal sample data into the teacher network that shares a partial network structure and the student network to be trained respectively, and the teacher network is pre-trained to a convergence state using multimodal sample data in a non-in-vehicle music scene.

[0060] This embodiment transfers high-quality audio parameter adjustment knowledge from other scenarios to the in-car music scenario through the knowledge transfer learning method of the student-teacher framework. The teacher network can be a deep neural network pre-trained in other scenarios, and the student network can also be a deep neural network structure. The teacher network and the student network realize knowledge transfer by sharing some network layers.

[0061] The inputs of the teacher network and the student network are all multimodal data, and the outputs are all audio parameter information. Among them, the multimodal data input to the student network can refer to the above description of the multimodal sample data, which will not be repeated here. The multimodal data input to the teacher network is specifically multimodal data from non-car music scenes. The number of non-car music scenes is one or more. The multimodal sample data in the non-car music scene includes scene information, audio layout information, audio codec information and audio parameter setting information of all non-car music scenes. Non-car music scenes can be, for example, concert hall scenes, cinema scenes, concert scenes and other scenes that are different from car music scenes. The multimodal data of non-car music scenes can be information on factors related to audio quality, such as scene information, audio layout information, audio parameter setting information, etc. of each scene. Among them, scene information refers to the basic information of the scene, which may include the venue information and environmental noise level information of the non-car music scene. The venue information may include one or more information such as venue type, number of people accommodated in the scene, and scene space. Understandably, the venue types of different non-car music scenes are different. For example, the venue types of concert hall scenes may include symphony halls, chamber music halls, recital halls, solo halls, chorus halls, etc. according to the different performance contents. For another example, the venue types of concert scenes may include livehouses, theaters, indoor gymnasiums, outdoor venues, outdoor stadiums, etc. according to the number of people they can accommodate. The audio layout information refers to the layout position of the audio in the scene. In non-car music scenes, in order to obtain better sound effects and allow the audience to have a better listening experience, the appropriate audio layout form is usually selected based on the venue size, audio requirements, audience position and actual acoustic characteristics of the venue. After the audio layout form is determined, the audio is laid out in the venue. The audio layout forms may include, but are not limited to, stereo sound layout, surround sound layout, surround sound plus high-effect audio layout, omnidirectional sound layout, etc. When different audio layout forms are adopted, the specific types and quantities of audio used may change accordingly. For example, when a stereo sound layout is adopted, a left front speaker and a right front speaker are usually selected and arranged at the left front and right front of the stage respectively; when a surround sound layout is adopted, a left front speaker, a right front speaker, a left rear speaker and a right rear speaker are selected and arranged at the left front and right front of the stage, the left front and the right front of the audience area respectively; if a surround sound plus high-effect audio layout is adopted, then on the basis of the surround sound layout, a speaker is set on the ceiling or the top of the venue to play high-effect audio; if an omnidirectional sound layout is adopted, when arranging the audio, multiple speakers are distributed around the entire venue, including the front, left and right sides, the rear and the top. Furthermore, for scenes where pre-recorded audio needs to be played, such as non-car music scenes such as concerts and cinemas, the corresponding multimodal data may further include audio codec information.Audio codec information is a general term for various data related to audio codec, including information about technologies used in related scenarios to encode and decode audio, such as compression algorithms for audio encoding, performance information and setting information of decoders, etc., which will affect the sound quality, stereo effect, channel separation and other parameters of the audio. Selecting relevant information of a variety of non-vehicle scenarios to train the teacher network allows the teacher network to learn the correlation between different non-vehicle scenarios and audio parameters, and can more accurately predict the optimal audio parameter setting information for various non-vehicle scenarios. For the relevant information of non-vehicle scenarios, the specific selection of scene information, audio layout information, and audio parameter setting information allows the teacher network to learn the correlation between information at different levels and angles and audio parameters, which helps to improve the performance and generalization ability of the teacher network.

[0062] Among them, in order to help the teacher network and the student network better understand and process multimodal data and improve the prediction accuracy, prediction efficiency and robustness of the trained teacher network and the student network, the multimodal data can be preprocessed and then used as input data.

[0063] Exemplarily, for the in-car audio information, signal processing technology can be used to pre-process it, such as using noise reduction, filtering and other operations to remove environmental noise and extract clear audio signals related to user experience. Feature extraction can be performed on vehicle information and weather information to extract feature information such as vehicle speed, acceleration, temperature, weather type, etc. These feature information help the model to better understand the current driving scene, so as to more accurately adjust the sound parameters. Audio feature extraction methods such as Mel frequency cepstral coefficients (MFCC) can be used to extract in-car voice features, and then by analyzing the user's voice interaction in the car, the user's speech features in different situations can be obtained, such as communication or voice commands while driving. The Transformer model can also be used to model the user's historical music preferences. The Transformer model performs well in processing time series data and can learn the time series changes of music preferences. Therefore, the user's historical music play records and other historical music preference information can be input into the Transformer model as time series music preference features, so that the Transformer model learns the user's preferences for different music types, artists or time periods, so as to better understand the user's personalized trends. In some implementations, features from different sources can also be combined into a feature vector, for example, the above-extracted in-car voice features, time-series music preference features, and environmental features can be integrated into a multimodal feature vector and then input into the network. This helps the network to comprehensively consider information from different sources, and thus more comprehensively understand the user's in-car music experience needs.

[0064] The data types of multimodal data in non-car music scenes are richer, the data volume is larger, and the quality is better than that of multimodal data in car music scenes. By training the teacher network with multimodal data in non-car music scenes, the trained teacher network can have better generalization and guiding significance. The student network to be trained is pre-trained with multimodal data of car music scenes. Due to the relatively small amount of data and relatively low quality, its performance is relatively weak. For this reason, it is further trained with the trained teacher network to improve its performance in the car music scene, while at the same time learning the relevant knowledge migrated from the teacher network for improving the sound quality. The following further explains the details of transfer learning through steps S130 and S140.

[0065] S130: Acquire the acoustic parameter prediction information outputted by the teacher network and the student network respectively, and record them as first acoustic parameter prediction information and second acoustic parameter prediction information.

[0066] For the input multimodal data, both the teacher network and the student network perform feature extraction processing and sound parameter prediction processing. The feature extraction processing can extract high-order semantic features (which can be called multimodal feature information) from the multimodal data, and the sound parameter prediction processing can predict and output the sound parameter information based on the extracted high-order semantic features. The sound parameter prediction information can include sound parameters such as audio balance, sound localization, volume, and sound effects.

[0067] In some embodiments, after the multimodal sample data are respectively input into a teacher network and a student network to be trained that share a partial network structure, the method further includes: performing feature extraction processing and sound parameter prediction processing on the multimodal sample data through the teacher network to obtain first multimodal feature information and first sound parameter prediction information; and performing feature extraction processing and sound parameter prediction processing on the multimodal sample data through the student network to obtain second multimodal feature information and second sound parameter prediction information.

[0068] As described above, both the teacher network and the student network perform feature extraction processing and sound parameter prediction processing on the input multimodal data, and obtain high-order semantic feature information and sound parameter prediction information in turn. For the convenience of distinction, in this embodiment, the high-order semantic feature information extracted by the teacher network and the sound parameter prediction information output are respectively referred to as first multimodal feature information and first sound parameter prediction information, and the high-order semantic feature information extracted by the student network and the sound parameter prediction information output are respectively referred to as second multimodal feature information and second sound parameter prediction information.

[0069] S140: Training the student network according to the first sound parameter prediction information and the second sound parameter.

[0070] In some embodiments, the operation of training the student network according to the first sound parameter prediction information and the second sound parameter includes: calculating the square difference distance between the first sound parameter prediction information and the second sound parameter as model loss information; and training the student network based on the model loss information.

[0071] This embodiment performs label alignment on the teacher network and the student network to train the student network. The above label alignment is to ensure the consistency of the outputs of the two networks. This embodiment believes that the sound parameters output by the trained teacher network can well improve the sound quality. In order to make the sound parameters output by the student network also improve the sound quality, it is necessary to make the outputs of the student network and the teacher network as consistent as possible, so as to ensure that the high-quality sound parameter adjustment knowledge learned by the teacher network can be successfully transferred to the student network, thereby improving the performance of the trained student network in the car music scene, while retaining the generalization and guiding significance of the knowledge learned by the teacher network from the non-car music scene.

[0072] Specifically, the square difference distance between the first sound parameter prediction information and the second sound parameter is calculated as the model loss information, and based on the loss information, the back propagation algorithm is used to adjust the network parameters of the student network. After adjusting the network parameters, it is determined whether the student network has reached a convergence state or whether other preset training end conditions (such as the total number of training times reaches a preset threshold) are met. If so, the current student network is used as the trained student network. If not, other multimodal sample data are called to execute the above steps S110-S140 to continue training the student network.

[0073] Among them, the above squared difference distance can be calculated by the following loss function:

[0074]

[0075] in, represents the i-th output in the car music scenario, represents the i-th output in the non-car music scenario, L MSE Represents the squared difference distance between the N outputs of the teacher network and the student network. The goal of training the student network is to minimize the loss L MSE , to ensure that the teacher network and the student network are as consistent as possible in terms of acoustic parameter prediction.

[0076] In some embodiments, Figure 2 As shown, the student network is trained according to the first sound parameter prediction information and the second sound parameter, including:

[0077] S141: Calculate the similarity between the first multimodal feature information and the second multimodal feature information as first loss information;

[0078] S142: Calculate the square difference between the first acoustic parameter prediction information and the second acoustic parameter as second loss information;

[0079] S143: Train the student network according to the first loss information and the second loss information.

[0080] Accordingly, in some embodiments, the teacher network includes a multimodal self-attention network for performing feature extraction processing on the above-mentioned multimodal sample data to obtain first multimodal feature information, and the student network includes an on-board multimodal self-attention network for performing feature extraction processing on the above-mentioned multimodal sample data to obtain second multimodal feature information; the above-mentioned multimodal self-attention network and the on-board multimodal self-attention network share some network weights.

[0081] See also Figure 3 In each training of the student network, in addition to the label alignment provided in the above embodiment, feature domain alignment is also performed, that is, on the basis of calculating the square difference distance between the first acoustic parameter prediction information and the second acoustic parameter as the loss information (i.e., the first loss information), the similarity between the first multimodal feature information and the second multimodal feature information is also calculated as the loss information (i.e., the second loss information). Finally, the two parts of the loss information are combined, for example, by adding them together to obtain the total loss information, and then the total loss information is used to train the student network. In this embodiment, the training goal is to reduce the total loss information as much as possible until the preset condition is met. For example, when the total loss information is less than the preset threshold, it can be determined that the preset condition is met and the training is terminated.

[0082] This embodiment introduces feature domain alignment to ensure consistency in music feature representation between the teacher network and the student network, thereby helping the student network to better learn the knowledge transferred from the teacher network. This consistency helps to improve the performance and adaptability of the student network in the in-car music scenario, enabling it to better meet the user's music needs.

[0083] Among them, cosine similarity can be used as the similarity between the first multimodal feature information and the second multimodal feature information.

[0084] The cosine similarity can be calculated using the following formula:

[0085]

[0086] Among them, <φ t (x),φ s (x)> represents the inner product of vectors, ||φ s (x)|| represents the norm of the vector, L alignRepresents the cosine similarity between the high-order semantic features extracted by the teacher network and the student network. The closer the cosine similarity value is to 1, the more similar the two high-order semantic features are. By adding the feature domain alignment loss to the overall loss function and then using the back propagation algorithm to optimize the network parameters of the student network, the feature domain alignment loss L is minimized. align , which can make the feature representations of the teacher network and the student network as similar as possible.

[0087] S150: Obtain a vehicle audio parameter prediction model based on the trained student network.

[0088] The vehicle audio parameter prediction model can be obtained by saving the network parameters of the trained student network (such as the weights and biases of each layer) and exporting the trained student network in a specified model file format. The vehicle audio parameter prediction model is more lightweight than the trained teacher network or teacher model. At the same time, it can take into account the impact of user preferences and audio quality on user experience, and generate audio parameters that both meet user preferences and improve audio quality.

[0089] Based on the same inventive concept, the present application also provides a method for predicting vehicle audio parameters. In some embodiments, Figure 4 As shown, the above method includes:

[0090] S210: Acquire real-time multimodal data related to the target vehicle audio system;

[0091] S220: inputting the real-time multimodal data into the vehicle audio parameter prediction model to obtain the audio parameter prediction information output by the vehicle audio parameter prediction model; wherein the vehicle audio parameter prediction model is trained based on the model training method provided in any of the above embodiments.

[0092] S230: Adjusting the audio parameters of the target vehicle audio system according to the audio parameter prediction information.

[0093] The method provided in this embodiment can be applied to a cloud server that has deployed a vehicle audio parameter prediction model. For example, the vehicle system of a vehicle equipped with a target vehicle audio system transmits real-time multimodal data to the cloud server via a wireless network, and the cloud server predicts based on the real-time multimodal data to obtain audio parameter prediction information, which is then sent to the vehicle system. The vehicle system adjusts the audio parameters of the target vehicle audio system based on the audio parameter prediction information. The target vehicle audio system refers to the vehicle audio system whose audio parameters need to be adjusted.

[0094] In order to improve the real-time adjustment of audio parameters, the vehicle audio parameter prediction model can be deployed in the vehicle computer system, and then the vehicle computer system can directly input real-time multimodal data into the vehicle audio parameter prediction model, and adjust the audio parameters of the target vehicle audio based on the audio parameter prediction information output by the model.

[0095] In some embodiments, the above method further comprises:

[0096] The layout of application icons and / or the activation status of specific functions in the display interface of the vehicle screen are dynamically adjusted according to the audio parameter prediction information.

[0097] In this embodiment, the vehicle system can also dynamically adjust the layout of the display interface (such as rearranging application icons, changing the home screen layout, etc.) and the enabled status of specific functions based on the model prediction results, thereby improving the convenience of users in using applications and specific functions.

[0098] In some embodiments, adjusting the sound parameters of the target vehicle audio system according to the sound parameter prediction information includes:

[0099] Real-time monitoring of the vehicle environment where the target vehicle audio system is located;

[0100] Based on the monitoring results, the acoustic parameter prediction information is dynamically adjusted through the domain control mechanism;

[0101] The sound parameter of the target vehicle audio system is adjusted based on the adjusted sound parameter prediction information.

[0102] This embodiment introduces a domain control mechanism to ensure knowledge transfer while taking into account the differences between different scenarios. Specifically, in real-time adjustment, the vehicle system monitors the current environment in real time, and based on the real-time monitoring results, dynamically adjusts the parameters predicted by the vehicle audio parameter prediction model through the domain control mechanism to adapt to the audio adjustment needs of different scenarios. This helps the vehicle audio system maintain high efficiency in different driving scenarios.

[0103] For the specific limitations of the vehicle audio parameter prediction method, please refer to the limitations of the model training method above, which will not be repeated here.

[0104] It should be noted that, with respect to the various steps included in the model training method provided in any of the above embodiments, unless otherwise clearly stated herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of these steps may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0105] In some embodiments, the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the model training method or the vehicle audio parameter prediction method provided in any of the above embodiments can be implemented.

[0106] In some embodiments, the computer device may be mounted on a vehicle-mounted terminal 110 in the vehicle 100, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. In addition to a processor and a memory connected via a system bus, the computer device also includes a network interface and a database, etc. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as multimodal data, and the specific stored data can also be found in the definitions in the above method embodiments. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model training method or a vehicle audio parameter prediction method is implemented.

[0107] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In some embodiments, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method or the vehicle audio parameter prediction method provided in any of the above embodiments are implemented.

[0109] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0110] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A model training method, characterized in that: The method comprises: Obtain multimodal sample data in the car music scene; The multimodal sample data are respectively input into a teacher network and a student network to be trained that share a partial network structure, wherein the teacher network is pre-trained to a convergence state through multimodal sample data in non-car music scenes; the number of the non-car music scenes is one or more, and the multimodal sample data in the non-car music scenes includes scene information, audio layout information, and audio parameter setting information of all non-car music scenes; Acquiring the sound parameter prediction information outputted by the teacher network and the student network respectively, recorded as first sound parameter prediction information and second sound parameter prediction information; training the student network according to the first sound parameter prediction information and the second sound parameter prediction information; Based on the trained student network, a prediction model for vehicle audio parameters is obtained; Training the student network according to the first sound parameter prediction information and the second sound parameter prediction information includes: calculating the squared difference distance between the first acoustic parameter prediction information and the second acoustic parameter prediction information as model loss information, and training the student network based on the model loss information; or, The similarity between the first multimodal feature information and the second multimodal feature information extracted by the teacher network and the student network respectively is calculated as the first loss information, the square difference distance between the first sound parameter prediction information and the second sound parameter prediction information is calculated as the second loss information, and the student network is trained according to the first loss information and the second loss information.

2. The method according to claim 1, characterized in that After the multimodal sample data are respectively input into a teacher network and a student network to be trained that share a partial network structure, the method further includes: Performing feature extraction processing and sound parameter prediction processing on the multimodal sample data through the teacher network to obtain the first multimodal feature information and the first sound parameter prediction information; The student network is used to perform feature extraction processing and sound parameter prediction processing on the multimodal sample data to obtain the second multimodal feature information and the second sound parameter prediction information.

3. The method according to claim 1, characterized in that Training the student network according to the first loss information and the second loss information includes: Adding the first loss information and the second loss information to obtain total loss information; The student network is trained according to the total loss information, and the goal of the training is to reduce the total loss information until a preset condition is met.

4. The method according to claim 3, characterized in that The multimodal sample data includes in-car audio information, vehicle environment related information and user historical music preference information; The teacher network includes a multimodal self-attention network for performing feature extraction processing on the multimodal sample data to obtain the first multimodal feature information, and the student network includes an on-board multimodal self-attention network for performing feature extraction processing on the multimodal sample data to obtain the second multimodal feature information; the multimodal self-attention network and the on-board multimodal self-attention network share some network weights.

5. A method for predicting vehicle audio parameters, characterized in that: The method comprises: Obtain real-time multimodal data related to the target car audio system; Inputting the real-time multimodal data into a vehicle audio parameter prediction model trained based on the method according to any one of claims 1 to 4, and obtaining audio parameter prediction information output by the vehicle audio parameter prediction model; The sound parameter of the target vehicle audio system is adjusted according to the sound parameter prediction information.

6. The method according to claim 5, characterized in that The method further comprises: The layout of application icons and / or the activation status of specific functions in the display interface of the vehicle screen are dynamically adjusted according to the audio parameter prediction information.

7. The method according to claim 5, characterized in that Adjusting the sound parameters of the target vehicle audio system according to the sound parameter prediction information includes: Performing real-time monitoring of the vehicle environment in which the target vehicle audio system is located; Based on the monitoring results, dynamically adjusting the acoustic parameter prediction information through a domain control mechanism; The sound parameter of the target vehicle audio system is adjusted based on the adjusted sound parameter prediction information.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.