Audio description text prediction model training method, text prediction method and device

Abstract

The invention provides an audio description text prediction model training method, and a text prediction method and device. The training method comprises the following steps: acquiring an audio sample and an audio description text sample corresponding to the audio sample, and constructing a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair comprises an audio sample and an audio description text sample which are matched with each other, and the negative sample pair comprises an audio sample and an audio description text sample which are not matched with each other; performing iterative training on a to-be-trained audio description text prediction model based on the positive sample pair and the negative sample pair to obtain a trained audio description text prediction model; the audio description text prediction model is used for predicting an audio description text corresponding to the audio data according to an audio vector corresponding to the audio data. According to the training method, the generation efficiency and objective accuracy of the audio description text are improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio data processing, and in particular to an audio description text prediction model training method, a text prediction method and a device. Background Art

[0002] As a mobile means of transportation, the demand for in-car entertainment is gradually being taken seriously. Human-car interaction is the core of user experience. The functional area of ​​the traditional car cockpit cannot meet the needs of human-car interaction. The emergence of smart cockpits solves some of the needs of users. Specifically, the smart cockpit can automatically adjust the seat orientation, vibration frequency, amplitude, and release corresponding odors according to the content of the audio and video played in the car.

[0003] In order to realize the above intelligent control, the intelligent cockpit control system needs to obtain the audio description text information in the audio file (for example, the tone of the program (sad, cheerful), strong vibration, rock music with strong rhythm, etc.), and then generate the cockpit control instructions for controlling the cockpit according to the cockpit metadata generated by the audio description text information. In the related art, the audio file is usually obtained by compressing the audio description text corresponding to the audio stream data in the audio stream, and the intelligent control signal for the cockpit is directly generated according to the audio description text in the file when rendering the audio file.

[0004] However, currently most audio streams do not generate audio description text synchronously when they are generated. It is necessary to manually add text description to the audio stream in the audio file to generate the corresponding audio description text. The above-mentioned method of manually generating audio description text has the problems of low efficiency, strong subjectivity, and differences between the texts generated by different people. Summary of the invention

[0005] The present application provides an audio description text prediction model training method, device, electronic device and storage medium, which are used to solve the problems of strong subjectivity and low efficiency in the prior art when manually generating audio description text. By using a pre-trained neural network model to automatically generate audio description text, the problem of subjective errors caused by manually generating audio description text is reduced, and the generation efficiency and objective accuracy of audio description text are improved.

[0006] In a first aspect, the present application provides a method for training an audio description text prediction model, comprising:

[0007] Acquire an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair includes a matching audio sample and an audio description text sample, and the negative sample pair includes an unmatched audio sample and an audio description text sample;

[0008] The audio description text prediction model to be trained is iteratively trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model; the audio description text prediction model is used to predict the audio description text corresponding to the audio data according to the audio vector corresponding to the audio data.

[0009] Optionally, the training objectives of the iterative training of the audio description text prediction model are: minimizing the difference between the predicted audio description text corresponding to the audio sample in the positive sample pair and the audio description text sample; maximizing the difference between the predicted audio description text corresponding to the audio sample in the negative sample pair and the audio description text sample.

[0010] Optionally, the audio description text prediction model includes an audio encoding sub-model and a text prediction sub-model;

[0011] Accordingly, the iterative training of the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model includes:

[0012] The following training process is iterated until the training completion condition is met to obtain the trained audio encoding sub-model and text prediction sub-model:

[0013] Inputting the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector;

[0014] Inputting the audio vector into the text prediction sub-model to be trained to perform text prediction processing to obtain a predicted audio description text;

[0015] A model loss function is generated based on the predicted audio description text and the audio description text sample, and parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained are adjusted based on the model loss function.

[0016] Optionally, the model loss function includes:

[0017]

[0018] Where Loss represents the model loss function; N represents the number of audio samples / audio description text samples; M represents the dimension of the text code table; c represents the i-th dimension text vector; y ic Indicates that the true category of input sample i is equal to c and takes 1, otherwise it takes 0; p ic It represents the predicted probability that sample i belongs to category c.

[0019] Optionally, adjusting parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained respectively based on the model loss function includes:

[0020] Obtaining a first model parameter of the audio coding model to be trained and a second model parameter of the text prediction sub-model to be trained;

[0021] Determining a first model adjustment parameter of the audio coding sub-model to be trained based on the model loss function and the first model parameter, and performing parameter adjustment on the first model parameter based on the first model parameter;

[0022] Determine the second model adjustment parameters of the text prediction sub-model to be trained based on the model loss function and the second model parameters, and perform parameter adjustment on the second model parameters based on the second model parameters.

[0023] Optionally, the step of inputting the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector includes:

[0024] Performing data preprocessing on the audio sample to obtain a processed audio sample, and inputting the processed audio sample into a preset spectral coding model to perform spectral coding processing to obtain spectral coding data;

[0025] Inputting the spectral coding data into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector;

[0026] Among them, the model loss function generated based on the audio vector is also used to synchronously adjust the third model parameter of the spectral coding model to be trained.

[0027] In a second aspect, the present application provides an audio description text prediction method, the method comprising:

[0028] Acquire vehicle audio data, and input the vehicle audio data into a pre-trained audio description text prediction model for audio encoding processing to obtain an audio vector;

[0029] Based on the audio vector, text prediction processing is performed in the audio description text prediction model to obtain the audio description text corresponding to the vehicle audio data; the audio description text prediction model is trained based on the audio description text prediction model training method described in any one of the first aspects.

[0030] Optionally, the method further includes:

[0031] Acquire a pre-trained text classification model, perform text classification processing on the audio description text based on the text classification model, and obtain a text classification result corresponding to the audio description text; the text classification result includes vehicle cockpit metadata of the vehicle where the on-board audio data is located; the vehicle cockpit metadata is used to generate a vehicle cockpit control signal corresponding to the on-board audio data.

[0032] In a third aspect, the present application provides an audio description text prediction model training device, comprising:

[0033] A sample pair acquisition module, used to acquire an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair includes a matching audio sample and an audio description text sample, and the negative sample pair includes an unmatched audio sample and an audio description text sample;

[0034] The model training module is used to iteratively train the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model; the audio description text prediction model is used to predict the audio description text corresponding to the audio data based on the audio vector corresponding to the audio data.

[0035] In a fourth aspect, the present application provides an audio description text prediction device, comprising:

[0036] An audio vector acquisition module is used to acquire vehicle audio data. The vehicle audio data is input into a pre-trained audio description text prediction model for audio encoding processing to obtain an audio vector;

[0037] A text prediction module is used to perform text prediction processing in the audio description text prediction model based on the audio vector to obtain the audio description text corresponding to the vehicle audio data; the audio description text prediction model is trained based on the audio description text prediction model training method described in any one of the first aspects.

[0038] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0039] The memory stores computer-executable instructions;

[0040] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0041] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0042] In a seventh aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0043] The technical solution provided by the present application jointly trains the audio coding model and the audio description text prediction model in an iterative manner to obtain a trained audio coding model and an audio description text prediction model, thereby achieving approximate equivalence between the audio vector output by the audio coding model and the text vector output by the audio description text prediction model; further, the audio description text prediction model is trained based on the text vector to obtain a trained audio description text prediction model; and then text prediction is performed on the audio data to obtain the corresponding audio description text, thereby reducing the problem of subjective errors caused by the manual generation of audio description texts in the prior art and improving the generation efficiency and objective accuracy of the audio description text. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 An application scenario diagram of the audio description text prediction model training method provided in this application;

[0046] Figure 2 A flowchart of a method for training an audio description text prediction model provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of another audio description text prediction model training method provided in an embodiment of the present application;

[0048] Figure 4 A flowchart of another audio description text prediction model training method provided in an embodiment of the present application;

[0049] Figure 5 A flowchart of an audio description text prediction method provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of an audio description text prediction model training device provided in an embodiment of the present application;

[0051] Figure 7A schematic diagram of the structure of an audio description text prediction device provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0053] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] In practical applications, with the development of intelligent technology, consumers are not only satisfied with the safety performance of vehicles when buying cars, but also have increasing demands for the comfort of vehicles. At present, the functional areas of traditional car cockpits can no longer meet the needs of human-vehicle interaction, and the emergence of smart cockpits solves some of the needs of users. For example, in order to enable car users to have a better viewing experience in the car. The smart cockpit can automatically adjust the seat orientation, vibration frequency, amplitude, release response smell and other intelligent controls according to the audio and video content played in the current display screen. In order to achieve the above-mentioned intelligent control, the control system of the vehicle needs to obtain the audio description text information in the audio file (for example, the tone of the program (sad, cheerful), strong vibration, strong rhythm rock music, etc.), and then generate cockpit control instructions for controlling the cockpit according to the cockpit metadata generated by the audio description text information. In the related art, the audio file is usually obtained by compressing the audio description text corresponding to the audio stream data and the audio stream, and the intelligent control signal for the cockpit is directly generated according to the audio description text in the file when rendering the audio file.

[0057] However, currently most audio streams do not generate audio description text synchronously when they are generated. It is necessary to manually add text description to the audio stream in the audio file to generate the corresponding audio description text. The above-mentioned method of manually generating audio description text has the problems of low efficiency, strong subjectivity, and differences between the texts generated by different people.

[0058] The audio description text prediction model training method provided in the present application is intended to solve the above technical problems of the prior art. Specifically, a large amount of audio data is obtained in advance as sample data, and the audio description text and audio file corresponding to each audio data are obtained to construct a sample pair, and then the audio description text prediction model is iteratively trained according to the sample pair to obtain a trained audio description text prediction model, and then the audio data is directly processed to obtain the audio description text corresponding to the audio data, which reduces the problem of subjective errors caused by the manual generation of audio description text in the prior art and improves the generation efficiency and objective accuracy of the audio description text.

[0059] Figure 1 This is an application scenario diagram of the audio description text prediction model training method provided in this application. Figure 1 The application scenarios to which the embodiments of the present application are applicable are described. Figure 1 , including a data end and a training end; the data end is specifically used to obtain audio samples and audio description samples for model training, and construct sample pairs based on the above sample data; and then transmit the sample pairs to the training end; specifically, the training end iteratively trains the prediction model to be trained, and obtains the trained prediction model, so as to predict the audio description text corresponding to the audio data according to the audio vector corresponding to the audio data.

[0060] It should be noted that the above-mentioned data end and training end can be different modules located in the same device, or modules located in different devices, and no specific limitation is made to this.

[0061] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0062] Figure 2 A flowchart of a method for training an audio description text prediction model is provided in an embodiment of the present application. The method can be executed by an audio description text prediction model training device, which can be a server or an electronic device. The following description is made using an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method includes the following steps.

[0063] S210: Obtain an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample.

[0064] In the technology of the present application, an audio sample can be a data sample containing any audio content. For example, it can be song data, or dubbing data of a film or TV series, etc. The audio description text is a text that describes the audio content in the audio data. For example, if the audio content in the audio data is audio of continuous thunder, the corresponding audio content audio description text is the rumbling sound of thunder; for another example, according to different audio data, the audio description text can also be the children's loud and small voices coming from the front, the whistling sound of explosions, the sound of gentle flowing water, and the rustling sound of leaves, etc.

[0065] It should be noted that in order to improve the learning ability in the subsequent model training process, the number of audio samples and audio description text samples obtained can reach hundreds of thousands.

[0066] On the basis of obtaining the above audio samples and the corresponding audio description text samples, the above samples can be annotated to obtain sample pairs for subsequent model iteration training. Specifically, a set of sample pairs contains an audio sample and an audio description text sample. The positive sample pairs in the sample pairs include matching audio samples and audio description text samples, and the negative sample pairs include mismatching audio samples and audio description text samples. Specifically, if the audio samples and audio description text samples in a set of sample pairs are matching samples, it means that the sample pair is a positive sample pair; otherwise, if they are mismatching samples, it means that the sample pair is a negative sample pair.

[0067] S220 , iteratively train the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model.

[0068] In practical applications, when training a target network model based on contrastive learning, data augmentation is generally performed on the sample data, and the enhanced data after enhancement is used as the positive data of the sample data and put into the model for training, and other data except the sample data and its enhanced data are used as the negative data of the sample data and put into the model for training. During the training process, the model continuously learns that any sample data and the corresponding enhanced data are similar data, and that the sample data and other enhanced data are dissimilar data, thereby improving the data processing capability of the target network model.

[0069] On this basis, the technical solution of the present application improves the existing iterative method, and then obtains the audio description text prediction model in the technical solution of the present application based on the improved method training, thereby improving the training efficiency of the model during the training process and the model performance of the obtained model.

[0070] Specifically, the technical solution of the embodiment of the present application, based on the obtained audio sample, describes any audio sample, and uses the obtained audio description text sample as the similar data of the audio sample, that is, a positive sample pair is composed based on the matched audio data and the audio description text, and the audio description text samples corresponding to other audio samples other than the audio sample are used as the dissimilar data of the audio sample, that is, a negative sample pair is composed based on the mismatched audio data and the audio description text. Furthermore, the audio description text prediction model to be trained is iteratively trained based on the above positive samples and negative samples to obtain a trained audio description text prediction model.

[0071] It should be noted that the trained audio description text prediction model is used to predict the audio description text corresponding to the audio data according to the audio vector corresponding to the audio data. It should also be noted that the training process of the above-mentioned model to be trained will be described in detail in subsequent embodiments and will not be described in detail here.

[0072] In the above technical scheme, a large amount of audio data is obtained as audio samples, descriptions are added to each audio data to obtain a corresponding audio description file, and positive sample pairs and negative sample pairs are constructed based on the above large amount of audio files and audio description texts; then, the audio description text model to be trained is iteratively trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model, and then text prediction of the audio data is implemented to obtain the corresponding audio description text, which reduces the problem of subjective errors caused by the manual generation of audio description texts in the prior art and improves the generation efficiency and objective accuracy of the audio description text.

[0073] Figure 3 A flowchart of another audio description text prediction model training method provided in an embodiment of the present application. This embodiment can be understood as a specific introduction embodiment of the steps mentioned in the method described in the above embodiment, see Figure 3 , the method may specifically include:

[0074] S310: Obtain an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample.

[0075] Specifically, the understanding and examples of the technical means, technical effects, and technical terms in step S310 may refer to the explanation of step S210 in the above embodiment.

[0076] S320 , iteratively train the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model.

[0077] Specifically, the understanding and examples of the technical means, technical effects, and technical terms in step S320 may refer to the explanation of step S220 in the above embodiment.

[0078] In an embodiment of the present application, the training objectives of iterative training of the audio description text prediction model to be trained are: minimizing the difference between the predicted audio description text and the audio description text sample corresponding to the audio sample in the positive sample pair; maximizing the difference between the predicted audio description text and the audio description text sample corresponding to the audio sample in the negative sample pair.

[0079] Specifically, during the training process, the sample pair can be input into the audio description text prediction model; in the model, the audio sample is first encoded to obtain an audio vector; and then the audio vector is subjected to text prediction processing to obtain the predicted audio description text corresponding to the audio sample. Furthermore, if the current sample pair is a positive sample pair, the prediction model can make the obtained predicted audio description text infinitely close to the audio description text sample during the learning process, thereby minimizing the difference between the predicted audio description text corresponding to the audio sample in the positive sample pair and the audio description text sample; conversely, if the current sample pair is a negative sample pair, the prediction model can make the obtained predicted audio description text infinitely far away from the audio description text sample during the learning process, thereby maximizing the difference between the predicted audio description text corresponding to the audio sample in the positive sample pair and the audio description text sample.

[0080] It should be noted that the audio description text prediction model in the present application includes an audio coding sub-model and a text prediction sub-model; based on the above implementation, the technical solution of this embodiment can iteratively perform the following training process until the training completion condition is met to obtain the trained audio coding sub-model and text prediction sub-model. Specifically, the training process may include:

[0081] S321. Input the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain audio vectors.

[0082] In an embodiment of the present application, for any audio sample in a positive sample pair and a negative sample pair, the audio sample is input into an audio encoding sub-model in an audio description text prediction model, the audio encoding sub-model encodes the audio sample and outputs an audio vector obtained after the processing.

[0083] Optionally, the process of encoding audio samples by an audio coding model may include: performing data preprocessing on the audio samples to obtain processed audio samples, and inputting the processed audio samples into a preset spectral coding model for spectral coding processing to obtain spectral coding data; inputting the spectral coding data into the audio coding model to be trained for audio coding processing to obtain an audio vector.

[0084] Specifically, the audio coding process in the present application can be understood as a two-layer coding process, that is, the audio samples are pre-input into the first-layer spectral coding model to encode the data in the form of sampling points into spectral coding data in frames, and then the spectral coding data is subjected to a second-layer coding process, that is, it is input into the above-mentioned audio coding model for vectorization processing to obtain an audio vector.

[0085] S322: Input the audio vector into the text prediction sub-model to be trained for text prediction processing to obtain a predicted audio description text.

[0086] In an embodiment of the present application, the audio vector obtained as described above is input into the audio coding sub-model in the audio description text prediction model; the audio coding sub-model then performs text prediction processing on the audio vector and outputs the predicted audio description text obtained after the processing.

[0087] S323. Generate a model loss function based on the predicted audio description text and the audio description text sample, and adjust parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained based on the model loss function.

[0088] In an embodiment of the present application, based on the above-mentioned predicted audio description text, a function calculation is performed on the predicted audio description text and the corresponding audio description text sample based on a preset loss function expression to generate a model loss function of the model during the training process.

[0089] Exemplarily, the calculation expression for generating the model loss function may include:

[0090]

[0091] Where Loss represents the model loss function; N represents the number of audio samples / audio description text samples; M represents the dimension of the text code table; c represents the i-th dimension text vector; y ic Indicates that the true category of input sample i is equal to c and takes 1, otherwise it takes 0; p ic It represents the predicted probability that sample i belongs to category c.

[0092] Furthermore, parameters of the audio coding sub-model to be trained and the text prediction sub-model to be trained are adjusted based on the generated model loss function. Optionally, if the audio coding sub-model and the text prediction sub-model after parameter adjustment meet the iteration stop condition, the training is stopped to obtain the trained audio coding sub-model and the text prediction sub-model, i.e., the audio description prediction model; otherwise, if the condition is not met, the iterative training is continued based on the above training process until the training is stopped when the stop condition is met.

[0093] Optionally, the process of adjusting parameters of the audio coding sub-model to be trained and the text prediction sub-model to be trained based on the model loss function specifically includes: obtaining first model parameters of the audio coding sub-model to be trained and second model parameters of the text prediction sub-model to be trained; determining first model adjustment parameters of the audio coding sub-model to be trained based on the model loss function and the first model parameters, and adjusting the first model parameters based on the first model parameters; determining second model adjustment parameters of the text prediction sub-model to be trained based on the model loss function and the second model parameters, and adjusting the second model parameters based on the second model parameters.

[0094] Specifically, the process of adjusting parameters for the audio coding sub-model may be: respectively calculating the first loss partial differential result of the model loss function and the first model partial differential result of the first model parameter of the audio coding sub-model, and calculating the ratio between the first loss partial differential result and the first model partial differential result, using the ratio as the first model adjustment parameter of the audio coding sub-model, and then subtracting the ratio calculated above from the first model parameter, and the difference obtained is the adjusted first model parameter.

[0095] Optionally, the parameters of the text prediction sub-model are adjusted based on the same method as above. The specific adjustment process includes: respectively calculating the first loss partial differential result of the model loss function and the second model partial differential result of the second model parameter of the text prediction sub-model, and calculating the ratio between the first loss partial differential result and the second model partial differential result, and using the ratio as the second model adjustment parameter of the text prediction sub-model, and then subtracting the above calculated ratio from the second model parameter, and the difference obtained is the adjusted second model parameter.

[0096] It should be noted that the spectral coding model mentioned above in the embodiment of the present application can be a pre-trained model; of course, in order to improve the training performance in the subsequent audio coding sub-model training process, the spectral coding model and the audio coding sub-model can be trained simultaneously. In other words, the model loss function generated by the audio coding sub-model during the training process synchronously adjusts the spectral coding model, so that the spectral coding data output by the trained spectral coding model can be more easily encoded by the trained audio coding model, thereby improving the encoding quality.

[0097] Specifically, during the training process of the spectral coding model, on the basis of obtaining the model loss function, the third model parameters of the spectral coding model are synchronously obtained, and based on the above-mentioned implementation method, the third model adjustment parameters of the spectral coding model are determined based on the model loss function and the third model parameters, and the third model parameters are parameter adjusted based on the third model parameters.

[0098] It should also be noted that the loss function calculation method and parameter adjustment method in the above-mentioned iterative process are only illustrative examples of the technical solution of the present application. The present application can also adopt other unmentioned methods to perform the above-mentioned function calculation and parameter conditions, and the present application does not make specific limitations on this.

[0099] In the above technical scheme, by obtaining audio samples and audio description text samples corresponding to the audio samples, and constructing positive sample pairs and negative sample pairs based on the above samples, the audio coding sub-model and text prediction sub-model to be trained are iteratively trained according to the positive sample pairs and the negative sample pairs, so as to obtain the trained audio coding sub-model and text prediction sub-model, that is, the audio description text prediction model; it is possible to predict the audio description text corresponding to the audio data according to the audio vector corresponding to the audio data, thereby reducing the problem of subjective errors caused by the manual generation of audio description texts in the prior art, and improving the generation efficiency and objective accuracy of the audio description text.

[0100] On the basis of the above implementation modes, the technical solution of the present application further provides a preferred embodiment, and then based on the preferred embodiment, the audio description text prediction model training method provided by the present application is specifically introduced. Figure 4 A flowchart of another audio description text prediction model training method provided in an embodiment of the present application. Figure 4 , the specific training method in the preferred embodiment may include:

[0101] First, the audio encoding sub-model and text prediction sub-model in the training network are trained. Specifically, in the training network, multiple stereo audios are obtained as audio data training samples. Specifically, the stereo audio is normalized and resampled to 24kHz; the above stereo audio is divided into 10 seconds / segment. Optionally, if the stereo audio or the last segment of stereo audio after segmentation is less than ten seconds, the length of 0-10 seconds is filled according to the length of the stereo audio segment, and then each segment of stereo audio is recorded as (1,2,240000), where 2 is the number of audio channels and 240000 is the number of samples in 10 seconds.

[0102] The processed stereo audio is spectrally encoded. For example, STFT (short-time Fourier transform, or short-term Fourier transform) can be used to convert the audio into an STFT spectrum (1, 2, 1002, 1025) with 2048 sample points as one frame, and Resnet is used as the STFT spectrum encoder to encode the STFT spectrum into (1, 2048) spectrum encoding data.

[0103] The spectral coded data is vectorized by using an audio coding sub-model, for example, a multi-layer perceptron network is used to encode the spectral coded data into a 10320-dimensional audio vector.

[0104] The text prediction sub-model is used to perform text prediction processing on the above audio vector. For example, the predicted audio description text corresponding to the audio data is predicted through the T5Tokenizer.decoder network.

[0105] The loss function is calculated by using a preset loss function calculation expression on the audio description text sample corresponding to the predicted audio description text and the audio sample to obtain a loss function; illustratively, the loss function calculation expression may include:

[0106]

[0107] Where Loss represents the model loss function; N represents the number of audio samples / audio description text samples; M represents the dimension of the text code table; c represents the i-th dimension text vector; y ic Indicates that the true category of input sample i is equal to c and takes 1, otherwise it takes 0; p ic It represents the predicted probability that sample i belongs to category c.

[0108] According to the loss function as the objective function, the model parameters (STFT spectral encoder + multi-layer perceptron + T5 network) are adjusted using the gradient descent method, that is: The model parameters are adjusted to obtain the trained spectral coding model (STFT spectral encoder), audio coding sub-model (multi-layer perceptron) and text prediction sub-model (T5 network).

[0109] Figure 5 The present invention provides a flowchart of an audio description text prediction method according to an embodiment of the present invention. The method can be executed by an audio description text prediction device, which can be a server or an electronic device. The following description is made using an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 5 As shown, the method includes the following steps.

[0110] S410: Acquire vehicle audio data, and input the vehicle audio data into a pre-trained audio description text prediction model for audio encoding processing to obtain an audio vector.

[0111] S420: Perform text prediction processing in an audio description text prediction model based on the audio vector to obtain an audio description text corresponding to the vehicle-mounted audio data.

[0112] In an embodiment of the present application, the audio description text prediction model includes an audio encoding sub-model and a text prediction sub-model. Specifically, the acquired vehicle audio data is input into the audio encoding sub-model for encoding processing to obtain an audio vector output by the model. Further, the above audio vector is input into the text prediction sub-model for text prediction processing to obtain an audio description text output by the model.

[0113] Based on the above implementation methods, the technical solution of the present application also includes: obtaining a pre-trained text classification model, performing text classification processing on the audio description text based on the text classification model, and obtaining a text classification result corresponding to the audio description text; the text classification result includes vehicle cockpit metadata of the vehicle where the vehicle-mounted audio data is located; the vehicle cockpit metadata is used to generate a vehicle cockpit control signal corresponding to the vehicle-mounted audio data.

[0114] In an embodiment of the present application, the vehicle cabin metadata may include but is not limited to metadata such as ambient light data, vibration data, fragrance data, air conditioning data, and video data.

[0115] Specifically, based on the above-mentioned implementation modes, if the audio description text obtained is "the loud laughter of children came from the front", the text is input into a pre-trained text classification model, and the model output result is: atmosphere: cheerful; direction: front; for another example, if the audio description text obtained is "the whistling sound of explosion", the text is input into a pre-trained text classification model, and the model output result is: atmosphere: tense; direction: from far to near; for another example, if the audio description text obtained is "the sound of flowing water and rustling leaves", the text is input into a pre-trained text classification model, and the model output result is: atmosphere: quiet; direction: surrounded on all sides.

[0116] Furthermore, drive control processing is performed based on the cockpit metadata obtained above, and control signals corresponding to various components in the cockpit are generated respectively.

[0117] In the above scheme, vehicle-mounted audio data is obtained, and the audio data is input into a pre-trained audio encoding sub-model for audio encoding processing to obtain an audio vector; the audio vector is input into a pre-trained text prediction sub-model for text prediction processing to obtain an audio description text corresponding to the vehicle-mounted audio data; and the method of automatically generating audio description text can be used to replace the method of manually processing audio data to generate audio description text in the prior art, thereby reducing the problem of subjective errors caused by manually generating audio description text in the prior art, and further improving the generation efficiency and objective accuracy of the audio description text.

[0118] Figure 6 A schematic diagram of the structure of an audio description text prediction model training device provided in an embodiment of the present application. Figure 6 The device includes: a sample pair acquisition module 510 and a model training module 520; wherein,

[0119] A sample pair acquisition module 510 is used to acquire an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair includes a matching audio sample and an audio description text sample, and the negative sample pair includes an unmatched audio sample and an audio description text sample;

[0120] The model training module 520 is used to iteratively train the audio description text prediction model to be trained based on positive sample pairs and negative sample pairs to obtain a trained audio description text prediction model; the audio description text prediction model is used to predict the audio description text corresponding to the audio data based on the audio vector corresponding to the audio data.

[0121] Optionally, the training objectives of the iterative training of the audio description text prediction model are: minimizing the difference between the predicted audio description text and the audio description text sample corresponding to the audio sample in the positive sample pair; maximizing the difference between the predicted audio description text and the audio description text sample corresponding to the audio sample in the negative sample pair.

[0122] Optionally, the audio description text prediction model includes an audio encoding sub-model and a text prediction sub-model;

[0123] Accordingly, the model training module 520 includes:

[0124] The training unit includes: iteratively executing the following training process until the training completion condition is met to obtain the trained audio coding sub-model and text prediction sub-model:

[0125] Inputting the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector;

[0126] Input the audio vector into the text prediction sub-model to be trained for text prediction processing to obtain the predicted audio description text;

[0127] A model loss function is generated based on the predicted audio description text and the audio description text sample, and parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained are adjusted based on the model loss function.

[0128] Optionally, the model loss function includes:

[0129]

[0130] Where Loss represents the model loss function; N represents the number of audio samples / audio description text samples; M represents the dimension of the text code table; c represents the i-th dimension text vector; y ic Indicates that the true category of input sample i is equal to c and takes 1, otherwise it takes 0; p ic It represents the predicted probability that sample i belongs to category c.

[0131] Optional training modules include:

[0132] A model parameter acquisition subunit, used to acquire a first model parameter of the audio coding model to be trained and a second model parameter of the text prediction submodel to be trained;

[0133] A first parameter adjustment subunit, used to determine a first model adjustment parameter of the audio coding submodel to be trained based on the model loss function and the first model parameter, and perform parameter adjustment on the first model parameter based on the first model parameter;

[0134] The second parameter adjustment subunit is used to determine the second model adjustment parameters of the text prediction submodel to be trained based on the model loss function and the second model parameters, and to adjust the second model parameters based on the second model parameters.

[0135] Optional training modules include:

[0136] The spectral coding data acquisition subunit is used to perform data preprocessing on the audio sample to obtain the processed audio sample, and input the processed audio sample into a preset spectral coding model to perform spectral coding processing to obtain spectral coding data;

[0137] The audio vector acquisition subunit is used to input the spectral coding data into the audio coding submodel to be trained for audio coding processing to obtain an audio vector; wherein the model loss function generated based on the audio vector is also used to synchronously adjust the third model parameters of the spectral coding model to be trained.

[0138] Figure 7A schematic diagram of the structure of an audio description text prediction device provided in an embodiment of the present application. Figure 7 , the device includes: an audio vector acquisition module 610 and a text prediction module 620; wherein,

[0139] The audio vector acquisition module 610 is used to acquire the vehicle audio data, and the vehicle audio data is input into the pre-trained audio description text prediction model for audio encoding processing to obtain the audio vector;

[0140] The text prediction module 620 is used to perform text prediction processing in the audio description text prediction model based on the audio vector to obtain the audio description text corresponding to the vehicle audio data; the audio description text prediction model is trained based on the audio description text prediction model training method of any of the above embodiments.

[0141] Optionally, the device further comprises:

[0142] The vehicle cockpit metadata acquisition module is used to obtain a pre-trained text classification model, perform text classification processing on the audio description text based on the text classification model, and obtain a text classification result corresponding to the audio description text; the text classification result includes the vehicle cockpit metadata of the vehicle where the on-board audio data is located; the vehicle cockpit metadata is used to generate a vehicle cockpit control signal corresponding to the on-board audio data.

[0143] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device of this embodiment may include:

[0144] At least one processor 710; and a memory 720 communicatively connected to the at least one processor;

[0145] The memory 720 stores instructions that can be executed by at least one processor 710, and the instructions are executed by at least one processor 710 to enable the server to execute a method as described in any of the above embodiments.

[0146] Optionally, the memory 720 may be independent or integrated with the processor 710 .

[0147] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0148] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0149] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0151] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0152] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0153] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0154] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a server or a main control device.

[0155] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 , device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0156] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0157] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0158] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0159] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0160] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0161] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0162] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0163] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0164] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0165] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0166] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned split-screen processing method of the electronic device.

[0167] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0168] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training an audio description text prediction model, characterized in that: The method comprises: Acquire an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair includes a matching audio sample and an audio description text sample, and the negative sample pair includes an unmatched audio sample and an audio description text sample; The audio description text prediction model to be trained is iteratively trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model; the audio description text prediction model is used to predict the audio description text corresponding to the audio data according to the audio vector corresponding to the audio data.

2. The method according to claim 1, characterized in that The training objectives of the iterative training of the audio description text prediction model are: minimizing the difference between the predicted audio description text corresponding to the audio sample in the positive sample pair and the audio description text sample; maximizing the difference between the predicted audio description text corresponding to the audio sample in the negative sample pair and the audio description text sample.

3. The method according to claim 1, characterized in that The audio description text prediction model includes an audio encoding sub-model and a text prediction sub-model; Accordingly, the iterative training of the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model includes: The following training process is iterated until the training completion condition is met to obtain the trained audio encoding sub-model and text prediction sub-model: Inputting the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector; Inputting the audio vector into the text prediction sub-model to be trained to perform text prediction processing to obtain a predicted audio description text; A model loss function is generated based on the predicted audio description text and the audio description text sample, and parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained are adjusted based on the model loss function.

4. The method according to claim 3, characterized in that The model loss function includes: Where Loss represents the model loss function; N represents the number of audio samples / audio description text samples; M represents the dimension of the text code table; c represents the i-th dimension text vector; y ic Indicates that the true category of input sample i is equal to c and takes 1, otherwise it takes 0; p ic It represents the predicted probability that sample i belongs to category c.

5. The method according to claim 3, characterized in that: Based on the model loss function, parameters of the audio encoding sub-model to be trained and the text prediction sub-model to be trained are respectively adjusted, including: Obtaining a first model parameter of the audio coding model to be trained and a second model parameter of the text prediction sub-model to be trained; Determining a first model adjustment parameter of the audio coding sub-model to be trained based on the model loss function and the first model parameter, and performing parameter adjustment on the first model parameter based on the first model parameter; Determine the second model adjustment parameters of the text prediction sub-model to be trained based on the model loss function and the second model parameters, and perform parameter adjustment on the second model parameters based on the second model parameters.

6. The method according to claim 3, characterized in that The step of inputting the audio samples in the positive sample pair and the negative sample pair into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector includes: Performing data preprocessing on the audio sample to obtain a processed audio sample, and inputting the processed audio sample into a preset spectral coding model to perform spectral coding processing to obtain spectral coding data; Inputting the spectral coding data into the audio coding sub-model to be trained for audio coding processing to obtain an audio vector; Among them, the model loss function generated based on the audio vector is also used to synchronously adjust the third model parameter of the spectral coding model to be trained.

7. A method for predicting audio description text, characterized in that: The method comprises: Acquire vehicle audio data, and input the vehicle audio data into a pre-trained audio description text prediction model for audio encoding processing to obtain an audio vector; Based on the audio vector, text prediction processing is performed in the audio description text prediction model to obtain the audio description text corresponding to the vehicle audio data; the audio description text prediction model is trained based on the audio description text prediction model training method according to any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The method further comprises: Acquire a pre-trained text classification model, perform text classification processing on the audio description text based on the text classification model, and obtain a text classification result corresponding to the audio description text; the text classification result includes vehicle cockpit metadata of the vehicle where the on-board audio data is located; the vehicle cockpit metadata is used to generate a vehicle cockpit control signal corresponding to the on-board audio data.

9. An audio description text prediction model training device, characterized in that: The device comprises: A sample pair acquisition module, used to acquire an audio sample and an audio description text sample corresponding to the audio sample, and construct a positive sample pair and a negative sample pair based on the audio sample and the audio description text sample; wherein the positive sample pair includes a matching audio sample and an audio description text sample, and the negative sample pair includes an unmatched audio sample and an audio description text sample; The model training module is used to iteratively train the audio description text prediction model to be trained based on the positive sample pairs and the negative sample pairs to obtain a trained audio description text prediction model; the audio description text prediction model is used to predict the audio description text corresponding to the audio data based on the audio vector corresponding to the audio data.

10. An audio description text prediction device, characterized in that: The device comprises: An audio vector acquisition module is used to acquire vehicle audio data. The vehicle audio data is input into a pre-trained audio description text prediction model for audio encoding processing to obtain an audio vector; A text prediction module is used to perform text prediction processing in the audio description text prediction model based on the audio vector to obtain the audio description text corresponding to the vehicle audio data; the audio description text prediction model is trained based on the audio description text prediction model training method described in any one of claims 1 to 6.

11. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When executing the computer-executable instructions, the processor is used to implement the audio description text prediction model training method as described in any one of claims 1 to 6, and / or the audio description text prediction method as described in any one of claims 7 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the audio description text prediction model training method as described in any one of claims 1 to 6, and / or the audio description text prediction method as described in any one of claims 7 to 8.

Images