Multimedia recognition method, device, storage medium and electronic device

Abstract

The present disclosure relates to a method, device, storage medium and electronic device for multimedia recognition, the method comprising: collecting multimedia data through a collection component; obtaining data features of the multimedia data; inputting the data features into a multimedia recognition model to obtain a recognition result of the data features; wherein the multimedia recognition model is a model obtained by training a preset training model according to a multimedia sample data set, a first loss function and an objective function, the objective function comprising a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of a plurality of multimedia sample data at different amplitudes; the second loss function is used to make the weights and sample data features of the plurality of multimedia sample data approach a preset value. Thus, the number of multimedia sample data required for training is reduced, the storage space occupied by the multimedia sample data is reduced, and the accuracy of multimedia recognition is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method, device, storage medium and electronic device for multimedia recognition. Background Art

[0002] With the development of artificial intelligence technology, people have a deeper understanding of multimedia recognition such as speech recognition and image recognition, and at the same time, the accuracy requirements for multimedia recognition are getting higher and higher. In the relevant technologies of multimedia recognition, it is generally necessary to train the preset training model with a large amount of multimedia sample data to obtain a multimedia recognition model that meets the recognition requirements. However, when there is less multimedia sample data used for training, the recognition accuracy of the trained multimedia recognition model will be low. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method, device, storage medium and electronic device for multimedia recognition.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for multimedia recognition is provided, the method comprising:

[0005] Collect multimedia data through a collection component;

[0006] Acquiring data features of the multimedia data;

[0007] Inputting the data feature into a multimedia recognition model to obtain a recognition result of the data feature, wherein the recognition result is used to characterize target information corresponding to the multimedia data;

[0008] Among them, the multimedia recognition model is a model obtained after training a preset training model according to a multimedia sample data set, a first loss function and an objective function, wherein the multimedia sample data set includes multiple multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the objective function includes a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of multiple multimedia sample data at different amplitudes; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model for multiple multimedia sample data and the target recognition results; the second loss function is used to make the weight sum of the sample data features of the multiple multimedia sample data approach a preset value, and the weight characterizes the degree of influence of the sample data features on the recognition result output by the multimedia recognition model.

[0009] By adopting the above method, the embodiment of the present disclosure can use a sample random function to characterize the different amplitudes of the sample data features in the training of the multimedia recognition model, and / or use a second loss function to reduce the impact of the different amplitudes of the sample data features, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training; the multimedia recognition model obtained by this training method can realize accurate recognition of multimedia data with different amplitudes, thereby improving the accuracy of multimedia recognition.

[0010] Optionally, the multimedia recognition model is trained in the following manner:

[0011] Acquire the multimedia sample data set, where the multimedia sample data set includes a plurality of the multimedia sample data and a target recognition result corresponding to each of the multimedia sample data;

[0012] Acquire sample data features corresponding to each of the multimedia sample data;

[0013] Determine the first loss function according to the training recognition result and the target recognition result;

[0014] The preset training model is trained according to the sample data characteristics, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model.

[0015] Using the above method, a sample random function is used to characterize the different amplitudes of the sample data characteristics, and / or a second loss function is used to reduce the impact of different amplitudes of the sample data characteristics, thereby reducing the amount of multimedia sample data required for training and reducing the storage space occupied by the multimedia sample data.

[0016] Optionally, when the objective function includes a sample random function, training the preset training model according to the sample data feature, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model includes:

[0017] Calculate and obtain new sample data features according to the sample data features and the sample random function;

[0018] The new sample data feature is used as the input of a preset training model, the target recognition result is used as the output of the preset training model, and the preset training model is trained according to the first loss function to obtain the multimedia recognition model.

[0019] In the above manner, the sample random function is used to characterize the different amplitudes of the sample data characteristics, thereby reducing the storage space occupied by the multimedia sample data.

[0020] Optionally, when the objective function includes a second loss function, training the preset training model according to the sample data feature, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model includes:

[0021] Determining a second loss function according to the weight of the sample data feature;

[0022] The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

[0023] In the above manner, the second loss function is used to reduce the influence of different amplitudes of the sample data features, thereby reducing the amount of multimedia sample data required for training and reducing the storage space occupied by the multimedia sample data.

[0024] Optionally, the acquiring the sample data feature corresponding to each of the multimedia sample data includes:

[0025] Performing feature extraction on each of the multimedia sample data, and obtaining a temporary feature of the sample data corresponding to each of the multimedia sample data;

[0026] Obtaining a standard deviation of the temporary feature of the sample data, wherein the standard deviation represents a degree of dispersion of the temporary feature of the sample data;

[0027] Performing standardization processing on each of the temporary features of the sample data according to the standard deviation to obtain the sample data feature;

[0028] Determining the second loss function according to the weight of the sample data feature includes:

[0029] A second loss function is determined according to the weight of the sample data feature and the standard deviation.

[0030] In the above manner, when the sample features are standardized, the influence of different amplitudes of the sample data features can be reduced by determining the second loss function using the weights and standard deviations of the sample features.

[0031] Optionally, the multimedia data includes voice data or image data.

[0032] By adopting the above method, multimedia recognition such as voice recognition, voice wake-up, and image recognition can be realized.

[0033] According to a second aspect of an embodiment of the present disclosure, a device for multimedia recognition is provided, the device comprising:

[0034] A data acquisition module, configured to collect multimedia data through a collection component;

[0035] A feature acquisition module, configured to acquire data features of the multimedia data;

[0036] A data recognition module is configured to input the data feature into a multimedia recognition model to obtain a recognition result of the data feature, wherein the recognition result is used to characterize target information corresponding to the multimedia data;

[0037] Among them, the multimedia recognition model is a model obtained after training a preset training model according to a multimedia sample data set, a first loss function and an objective function, wherein the multimedia sample data set includes multiple multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the objective function includes a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of multiple multimedia sample data at different amplitudes; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model for multiple multimedia sample data and the target recognition results; the second loss function is used to make the weight sum of the sample data features of the multiple multimedia sample data approach a preset value, and the weight characterizes the degree of influence of the sample data features on the recognition result output by the multimedia recognition model.

[0038] Optionally, the device further includes a model training module, and the model training module includes:

[0039] A sample data set acquisition submodule is configured to acquire the multimedia sample data set, wherein the multimedia sample data set includes a plurality of the multimedia sample data and a target recognition result corresponding to each of the multimedia sample data;

[0040] A sample feature acquisition submodule, configured to acquire sample data features corresponding to each of the multimedia sample data;

[0041] A first loss function determination submodule, configured to determine the first loss function according to the training recognition result and the target recognition result;

[0042] The first training submodule is configured to train the preset training model according to the sample data characteristics, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model.

[0043] Optionally, when the objective function includes a sample random function, the first training submodule is configured as follows:

[0044] Calculate and obtain new sample data features according to the sample data features and the sample random function;

[0045] The new sample data feature is used as the input of a preset training model, the target recognition result is used as the output of the preset training model, and the preset training model is trained according to the first loss function to obtain the multimedia recognition model.

[0046] Optionally, when the objective function includes a second loss function, the first training submodule is configured as follows:

[0047] Determining a second loss function according to the weight of the sample data feature;

[0048] The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

[0049] Optionally, the sample feature acquisition submodule is configured as follows:

[0050] Performing feature extraction on each of the multimedia sample data, and obtaining a temporary feature of the sample data corresponding to each of the multimedia sample data;

[0051] Obtaining a standard deviation of the temporary feature of the sample data, wherein the standard deviation represents a degree of dispersion of the temporary feature of the sample data;

[0052] Performing standardization processing on each of the temporary features of the sample data according to the standard deviation to obtain the sample data feature;

[0053] The first training submodule is configured as follows:

[0054] Determining a second loss function according to the weight of the sample data feature and the standard deviation;

[0055] The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

[0056] Optionally, the multimedia data includes voice data or image data.

[0057] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect of the present disclosure are implemented.

[0058] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0059] a memory having a computer program stored thereon;

[0060] A processor is used to execute the computer program in the memory to implement the steps of the method described in the first aspect of the present disclosure.

[0061] The technical solution provided by the embodiments of the present disclosure can at least achieve the following beneficial effects:

[0062] In the training of the multimedia recognition model, a sample random function is used to characterize the different amplitudes of the sample data features, and / or a second loss function is used to reduce the impact of the different amplitudes of the sample data features, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training. The multimedia recognition model obtained by this training method can accurately recognize multimedia data with different amplitudes, thereby improving the accuracy of multimedia data recognition.

[0063] Other features and advantages of the present disclosure will be described in detail in the following detailed description.

[0064] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0066] Figure 1 The present invention is a flowchart of a multimedia recognition method according to an exemplary embodiment.

[0067] Figure 2 The figure is a flowchart of a method for training a multimedia recognition model according to an exemplary embodiment.

[0068] Figure 3 The figure is a schematic diagram showing the structure of a multimedia recognition device according to an exemplary embodiment.

[0069] Figure 4 It is a schematic structural diagram of another multimedia recognition device according to an exemplary embodiment.

[0070] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0071] Figure 6is a block diagram of another electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0073] First, the application scenario of the present disclosure is described. The present disclosure can be applied to artificial intelligence scenarios, especially multimedia recognition scenarios, such as image recognition, speech recognition, voice wake-up and other scenarios. In the related art, when there is less multimedia sample data for training, the recognition accuracy of the trained multimedia recognition model will be low. The inventors found that the amplitude of the multimedia data (that is, the brightness of the image, the volume of the audio) has a great influence on the accuracy of the multimedia recognition model. In order to improve the accuracy of the multimedia recognition model, it is necessary to obtain a large amount of multimedia sample data with different amplitudes for model training, but the acquisition efficiency of multimedia sample data with different amplitudes through manual collection is relatively low, and the acquired multimedia sample data also needs to occupy a large amount of storage space.

[0074] In the related art, data augmentation is generally used to obtain multimedia sample data of different amplitudes for model training, that is, before model training, the original multimedia sample data is amplitude-transformed to obtain a large amount of multimedia sample data of different amplitudes and store them, and then the stored multimedia sample data of different amplitudes are used for model training. However, this method still requires a large amount of storage space and reduces the efficiency of model training.

[0075] In order to solve the above problems, the present disclosure provides a method, device, storage medium and electronic device for multimedia recognition. In the training of the multimedia recognition model, a sample random function is used to characterize the different amplitudes of the sample data characteristics, and / or a second loss function is used to reduce the impact of the different amplitudes of the sample data characteristics, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training. Moreover, the multimedia recognition model obtained by the training method can realize accurate recognition of multimedia data with different amplitudes.

[0076] It should be noted that the multimedia recognition involved in the embodiments of the present disclosure may include multimedia recognition such as image recognition, speech recognition, and speech wake-up; the multimedia recognition model involved in the embodiments of the present disclosure may be a model that has certain capabilities after learning through samples, for example, it may be a neural network model, such as a CNN (Convolutional Neural Networks) model, an RNN (Recurrent Neural Networks) model, etc. Of course, the multimedia recognition model may also adopt other types of machine learning models, such as SVM (Support Vector Machine).

[0077] The present disclosure is described below in conjunction with specific embodiments.

[0078] Figure 1 is a flowchart of a multimedia recognition method according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0079] S101, collecting multimedia data through a collection component.

[0080] The multimedia data may be voice data or image data, for example, voice data sent by a user received through a microphone, or image data collected through a camera.

[0081] S102: Acquire data features of the multimedia data.

[0082] In this step, there are many ways to obtain data features. For example, when the multimedia data is voice data, the voice data can be feature extracted by the FilterBank method to obtain data features. When the multimedia data is an image, the pixels of the image can be obtained as data features by scanning the image. Through this step, the multimedia data can be converted into data features that can be processed by a computer program for multimedia recognition.

[0083] S103: Input the data feature into a multimedia recognition model to obtain a recognition result of the data feature.

[0084] The recognition result of the data feature is used to characterize the target information corresponding to the multimedia data. When the multimedia data is voice data, the target information may be a target text, a target action, or a target wake-up word, etc. When the multimedia data is image data, the target information may be a target text, a target action, a target emotion, a target animal, or a target plant, etc.

[0085] The above-mentioned multimedia recognition model is a model obtained after training a preset training model according to a multimedia sample data set, a first loss function and an objective function. The multimedia sample data set includes multiple multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model of the multiple multimedia sample data and the target recognition result; the objective function includes a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of the multiple multimedia sample data at different amplitudes; the second loss function is used to make the weight sum of the sample data features of the multiple multimedia sample data approach a preset value, and the weight characterizes the degree of influence of the sample data features on the recognition result output by the multimedia recognition model.

[0086] By adopting the above method, in the training of the multimedia recognition model, a sample random function is used to characterize the different amplitudes of the sample data features, and / or a second loss function is used to reduce the influence of the different amplitudes of the sample data features, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training; the multimedia recognition model obtained by the training method can accurately recognize multimedia data of different amplitudes, thereby improving the accuracy of multimedia recognition. For example, taking the voice wake-up scenario as an example, the method can significantly improve the voice wake-up rate in a far-field low-volume environment.

[0087] The method for obtaining a multimedia recognition model by training a preset training model according to a multimedia sample data set, a first loss function and an objective function is as follows: Figure 2 As shown, Figure 2 The present invention is a flowchart of a method for training a multimedia recognition model according to an exemplary embodiment. The method may include the following steps:

[0088] S201: Obtain a multimedia sample data set.

[0089] The multimedia sample data set includes a plurality of multimedia sample data and a target recognition result corresponding to each multimedia sample data, and can be used to train a preset training model, or can also be used to correct a multimedia recognition model.

[0090] The multimedia sample data set can be obtained by manual collection and annotation, or by using an existing multimedia training data set. For example, the multimedia sample data set can be a voice sample data set or an image sample data set. For example, the multimedia sample data set can include multiple voice sample data and a target recognition result corresponding to each voice sample data. The target recognition result can be a target text, a target action, or a target wake-up word, etc. For another example, the multimedia sample data set can also include multiple image sample data and a target recognition result corresponding to each image sample data. The target recognition result can be a target text, a target action, a target animal, or a target plant, etc.

[0091] S202: Obtain sample data features corresponding to each multimedia sample data.

[0092] In the case where the multimedia sample data is speech sample data, each speech sample data can be subjected to signal processing and feature extraction to obtain the corresponding sample data features. For example, the speech sample data can contain both the speech signal emitted by the speaker and the external noise signal. First, the speech sample data needs to be subjected to signal processing, such as noise reduction, reverberation, etc. After the signal processing, a relatively pure speech signal can be obtained; then the speech signal is subjected to feature extraction to obtain the sample data features corresponding to the speech sample data. The feature extraction method can adopt the FilterBank method, and the feature extraction steps include: first, the speech sample data is subjected to framing, windowing, DC component removal and pre-emphasis processing, and then after FFT (Fast Fourier Transform), it passes through the Mel filter group, and finally the speech sample features are obtained after modulo operation, square operation and logarithm operation.

[0093] In the case where the multimedia sample data is image sample data, pixels in the image sample data may be acquired as sample data features through image scanning.

[0094] S203: Determine a first loss function according to the training recognition result and the target recognition result.

[0095] The training recognition result represents the result obtained by the preset training model of the multimedia sample data, the target recognition result is obtained from the above multimedia sample data set, and the first loss function is used to represent the gap between the training recognition result and the target recognition result. The goal of model training is to make the value of the first loss function approach the first target loss value, which can be 0 or a preset value.

[0096] S204: Train the preset training model according to the sample data characteristics, the target recognition result, the first loss function and the target function to obtain a multimedia recognition model.

[0097] Among them, the objective function includes a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of multimedia sample data under different feature amplitudes; the second loss function is used to make the weight sum of sample data features of multiple multimedia sample data approach a preset value, and the weight represents the degree of influence of the sample data features on the recognition result output by the multimedia recognition model.

[0098] There are three ways to train in this step:

[0099] Training method one: when the objective function includes a sample random function, new sample data features of multiple multimedia sample data at different feature amplitudes can be first obtained according to the sample random function; then the new sample data features are used as the input of the preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function to obtain a multimedia recognition model.

[0100] According to whether the multimedia sample data is speech sample data or image sample data, the sample random function may have different expressions.

[0101] In the case where the multimedia sample data is speech sample data, the expression of the sample random function may be 2ln(rand()), where rand() represents obtaining a random variable greater than 0. Further, the following first feature update formula may be used to calculate a new sample data feature:

[0102] x2=x1+2ln(rand())

[0103] Among them, x1 represents the sample data feature obtained by feature extraction, 2ln(rand()) represents a random variable greater than 0 obtained by the sample random function, and x2 represents the new sample data feature after calculation.

[0104] It should be noted that in the related art, the amplitude augmentation method of multimedia sample data used for training can multiply the multimedia sample data by multiple preset multiples to obtain multiple new multimedia sample data, and store the multiple new multimedia sample data as input for model training. When the preset multiple is greater than 0 and less than 1, it means that the amplitude of the multimedia sample data is reduced to obtain new multimedia sample data; and when the preset multiple is greater than 1, it means that the amplitude of the multimedia sample data is increased to obtain new multimedia sample data. In order to improve the effect of amplitude augmentation, it is necessary to set multiple preset multiples. For example, the multiple preset multiples can be 0.1, 0.5, 0.8, 2, 5, 10 and other multiple values. The multimedia sample data obtained in this way needs to occupy a large amount of storage space.

[0105] The inventors have found that, in the case where the multimedia sample data is speech sample data, in the method of obtaining the sample data features in step S202, framing, windowing, DC component removal, pre-emphasis, FFT and Mel filtering are all linear transformations, and the final modulo operation, square operation and logarithm operation are nonlinear transformations. Therefore, the above-mentioned method of amplitude augmentation by multiplying by a preset multiple can be replaced by data obtained by adding a preset variable to the sample data features after feature extraction, and the preset variable can be twice the logarithm of the preset multiple. In the case where the preset multiple is a random variable obtained by a sample random function, the method of amplitude augmentation by the above-mentioned first feature update formula can be obtained.

[0106] Furthermore, the above-mentioned training of the preset training model can perform multiple rounds of training for each input sample data feature. In order to reflect the impact of different amplitudes, each sample data feature can obtain the corresponding new sample data feature through the above-mentioned first feature update formula before being used for model training each time, and then use the new sample data feature as the input of the preset training model.

[0107] Similarly, when the multimedia sample data is image sample data, and the image sample data is a pixel, the pixel acquisition of the image sample data is also linear, so the sample data feature can also be amplitude-enhanced by adding random values. The sample random function can be y1+rand(), where y1 represents the value of the pixel in the image sample data, and rand() represents the acquired random variable, which can be positive or negative. Further, the new sample data feature can be calculated by the following second feature update formula:

[0108] y2=y1+rand()

[0109] Among them, x2 represents the new sample data features after calculation, x1 represents the sample data features obtained by feature extraction, y1 represents the value of the pixel in the image sample data, and rand() represents the obtained random variable, which can be positive or negative.

[0110] In this way, by adopting the above-mentioned training method 1, a new sample data feature representing different amplitudes can be obtained by using a sample random function after feature extraction, thereby reducing the storage space occupied and improving the efficiency of obtaining sample data features. The multimedia recognition model obtained by training using the new sample data features can realize accurate recognition of multimedia data with different amplitudes.

[0111] Training method two: When the objective function includes a second loss function, the second loss function can be first determined based on the weight of the sample data features; then the sample data features are used as the input of the preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain a multimedia recognition model.

[0112] The second loss function is used to make the sum of weights of the sample data features of the multiple multimedia sample data approach a preset value, and the weight represents the degree of influence of the sample data feature on the recognition result output by the multimedia recognition model.

[0113] In the case of using a neural network to train the multimedia recognition model, the output of the first hidden layer is: wx1+b, where x1 represents the sample data feature, w represents the weight learned by the model, and b represents the bias learned by the model. According to the description of the first training method, a random variable can be obtained through a sample random function, and the sample data feature is added to the random variable to obtain a new sample data feature for training. In this case, the output of the first hidden layer becomes wx2+b=w(x1+2ln(β))+b, that is, wx1+b+2ln(β)*wE, where β represents a random variable obtained through a sample random function, which is used to characterize the amplitude, and E represents a unit vector. Although the random variable is generated in a random form, by making the weight sum of the sample data feature approach 0, that is, wE approaches 0, the influence of the variable β representing the amplitude on the multimedia recognition model can be eliminated, that is, the influence of the amplitude on the recognition result of the multimedia recognition model can be eliminated. In particular, when wE is 0, the influence of the amplitude on the recognition result of the multimedia recognition model is completely eliminated.

[0114] It should be noted that when training a multimedia recognition model, since the range of the acquired sample data features is often large and the discreteness is high, the acquired sample data features can be standardized, thereby improving the convergence speed of the model parameters during the training process and improving the efficiency of model training. The acquired sample data features and the standardized processing of the acquired sample data features can be performed through the following steps:

[0115] First, feature extraction is performed on each multimedia sample data, and a temporary feature of the sample data corresponding to each multimedia sample data is obtained.

[0116] Secondly, the standard deviation of the temporary characteristics of the sample data is obtained. The standard deviation represents the arithmetic square root of the arithmetic mean of the square of the mean deviation, which characterizes the degree of dispersion of the temporary characteristics of all sample data.

[0117] Finally, each temporary feature of sample data is standardized according to the standard deviation to obtain the sample data feature.

[0118] Of course, when training the model, the acquired sample data features may not be standardized. The following describes how to obtain the second loss function based on whether the sample data features are standardized when training the multimedia recognition model:

[0119] Method 1 for obtaining the second loss function: When the multimedia recognition model is trained, without standardizing the sample data features, the second loss function can be determined according to the weights of the sample data features. The expression of the second loss function can include:

[0120] or,

[0121] Among them, L1 represents the value of the second loss function, w represents the weight of the sample data feature, M represents the length of the convolution kernel of the first hidden layer of the multimedia recognition model, the first hidden layer is a hidden layer directly connected to the input layer of the multimedia recognition model, N represents the width of the convolution kernel, C represents the number of convolution kernels, and M, N and C together represent multiple convolution kernels for segmenting the sample data features.

[0122] Method 2 for obtaining the second loss function: When the multimedia recognition model is trained, the sample data features are standardized, and the second loss function can be determined according to the weight of the sample data features and the above standard deviation. The expression of the second loss function can include:

[0123] or,

[0124] Among them, L1 represents the value of the second loss function, w represents the weight of the sample data feature, v represents the standard deviation, and C represents the number of convolution kernels of the first hidden layer of the multimedia recognition model. The first hidden layer is a hidden layer directly connected to the input layer of the multimedia recognition model.

[0125] In this way, by adopting the above-mentioned training method 2, the influence of amplitude on the multimedia recognition model is eliminated through the second loss function, so that a relatively accurate multimedia recognition model can be trained without adding multimedia sample data of different amplitudes, thereby reducing the storage space occupied by the multimedia sample data used for training, and the multimedia recognition model obtained by training can realize accurate recognition of multimedia data of different amplitudes.

[0126] Training method three: When the objective function includes a sample random function and a second loss function, new sample data features of multiple multimedia sample data at different feature amplitudes can be first obtained according to the sample random function; then the new sample data features are used as the input of the preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain a multimedia recognition model.

[0127] This method three combines the sample random function in the training method one and the second loss function in the training method two. The specific implementation method can refer to the description in the training method one and the training method two, which will not be repeated here.

[0128] By adopting the above-mentioned training method three, the recognition accuracy of the trained multimedia recognition model can be further improved.

[0129] Figure 3 is a schematic diagram showing the structure of a multimedia recognition device according to an exemplary embodiment. Figure 3 As shown, the device includes a data acquisition module 301, a feature acquisition module 302, and a data identification module 303:

[0130] The data acquisition module 301 is configured to collect multimedia data through a collection component;

[0131] The feature acquisition module 302 is configured to acquire data features of the multimedia data;

[0132] The data recognition module 303 is configured to input the data feature into the multimedia recognition model to obtain a recognition result of the data feature, and the recognition result is used to characterize the target information corresponding to the multimedia data;

[0133] Among them, the multimedia recognition model is a model obtained after training a preset training model according to a multimedia sample data set, a first loss function and an objective function, and the multimedia sample data set includes multiple multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the objective function includes a sample random function and / or a second loss function; the sample random function is used to obtain sample data features of multiple multimedia sample data at different amplitudes; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model for multiple multimedia sample data and the target recognition results; the second loss function is used to make the weight sum of the sample data features of the multiple multimedia sample data approach a preset value, and the weight characterizes the degree of influence of the sample data features on the recognition result output by the multimedia recognition model.

[0134] Through the above-mentioned device, in the training of the multimedia recognition model, a sample random function is used to characterize the different amplitudes of the sample data features, and / or a second loss function is used to reduce the impact of the different amplitudes of the sample data features, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training; the multimedia recognition model obtained by this training method can realize accurate recognition of multimedia data with different amplitudes, thereby improving the accuracy of multimedia recognition.

[0135] Optionally, Figure 4 is a schematic diagram showing the structure of a multimedia recognition device according to another exemplary embodiment. Figure 4 As shown, the device also includes a model training module 401, and the model training module 401 includes:

[0136] The sample data set acquisition submodule 4011 is configured to acquire the multimedia sample data set, where the multimedia sample data set includes a plurality of multimedia sample data and a target recognition result corresponding to each multimedia sample data;

[0137] The sample feature acquisition submodule 4012 is configured to acquire sample data features corresponding to each multimedia sample data;

[0138] A first loss function determination submodule 4013 is configured to determine the first loss function according to the training recognition result and the target recognition result;

[0139] The first training submodule 4014 is configured to train the preset training model according to the sample data characteristics, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model.

[0140] Optionally, when the objective function includes a sample random function, the first training submodule 4014 is configured as follows:

[0141] According to the sample data feature and the sample random function, a new sample data feature is calculated;

[0142] The new sample data feature is used as the input of the preset training model, the target recognition result is used as the output of the preset training model, and the preset training model is trained according to the first loss function to obtain the multimedia recognition model.

[0143] Optionally, when the objective function includes a second loss function, the first training submodule 4014 is configured as follows:

[0144] Determine a second loss function according to the weight of the sample data feature;

[0145] The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

[0146] Optionally, the sample feature acquisition submodule 4012 is configured as follows:

[0147] Perform feature extraction on each of the multimedia sample data, and obtain temporary features of the sample data corresponding to each of the multimedia sample data;

[0148] Obtaining the standard deviation of the temporary feature of the sample data, where the standard deviation represents the degree of dispersion of the temporary feature of the sample data;

[0149] Standardize each temporary feature of the sample data according to the standard deviation to obtain the sample data feature;

[0150] The first training submodule 4014 is configured to:

[0151] Determine a second loss function according to the weight of the sample data feature and the standard deviation;

[0152] The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

[0153] Optionally, the multimedia data includes voice data or image data.

[0154] Through the above-mentioned device, in the training of the multimedia recognition model, a sample random function is used to characterize the different amplitudes of the sample data features, and / or a second loss function is used to reduce the impact of the different amplitudes of the sample data features, thereby reducing the number of multimedia sample data required for training, reducing the storage space occupied by the multimedia sample data, and improving the efficiency of model training; the multimedia recognition model obtained by this training method can realize accurate recognition of multimedia data with different amplitudes, thereby improving the accuracy of multimedia recognition.

[0155] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0156] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the multimedia recognition method provided by the present disclosure are implemented.

[0157] Figure 5 1 is a block diagram of an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0158] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0159] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned multimedia recognition method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0160] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0161] The power component 506 provides power to the various components of the electronic device 500. The power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0162] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0164] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0165] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0166] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0167] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned multimedia recognition method.

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to complete the above-mentioned multimedia recognition method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above multimedia recognition method when executed by the programmable device.

[0170] Figure 6 600 is a block diagram of another electronic device according to an exemplary embodiment. For example, the device 600 may be provided as a server. Figure 6 The apparatus 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions executable by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the above-mentioned multimedia recognition method.

[0171] The device 600 may also include a power supply component 626 configured to perform power management of the device 600, a wired or wireless network interface 650 configured to connect the device 600 to a network, and an input / output (I / O) interface 658. The device 600 may operate based on an operating system stored in the memory 632, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0172] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A multimedia recognition method, characterized in that: The method comprises: Collect multimedia data through a collection component; Acquiring data features of the multimedia data; Inputting the data feature into a multimedia recognition model to obtain a recognition result of the data feature, wherein the recognition result is used to characterize target information corresponding to the multimedia data; The multimedia recognition model is a model obtained by training a preset training model according to a multimedia sample data set, a first loss function and an objective function, wherein the multimedia sample data set includes a plurality of multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the objective function includes a sample random function; the sample random function is used to obtain sample data features of the plurality of multimedia sample data at different amplitudes; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model for the plurality of multimedia sample data and the target recognition results; The multimedia recognition model is trained in the following way: Acquire the multimedia sample data set, where the multimedia sample data set includes a plurality of the multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; Acquire sample data features corresponding to each of the multimedia sample data; Determine the first loss function according to the training recognition result and the target recognition result; The preset training model is trained according to the sample data feature, the target recognition result, the first loss function and the target function to obtain the multimedia recognition model; In the case where the objective function includes a sample random function, training the preset training model according to the sample data feature, the target recognition result, the first loss function and the objective function to obtain the multimedia recognition model includes: According to the sample data feature and the sample random function, new sample data features representing different amplitudes are calculated; The new sample data feature is used as the input of a preset training model, the target recognition result is used as the output of the preset training model, and the preset training model is trained according to the first loss function to obtain the multimedia recognition model.

2. The method according to claim 1, characterized in that The objective function also includes a second loss function, and the second loss function is used to make the weight sum of the sample data features of the plurality of multimedia sample data approach a preset value, and the weight represents the influence of the sample data features on the recognition result output by the multimedia recognition model; when the objective function includes the second loss function, the multimedia recognition model is obtained by training the preset training model according to the sample data features, the target recognition result, the first loss function and the objective function, including: Determining a second loss function according to the weight of the sample data feature; The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

3. The method according to claim 2, characterized in that The acquiring of sample data features corresponding to each of the multimedia sample data comprises: Performing feature extraction on each of the multimedia sample data, and obtaining a temporary feature of the sample data corresponding to each of the multimedia sample data; Obtaining a standard deviation of the temporary feature of the sample data, wherein the standard deviation represents a degree of dispersion of the temporary feature of the sample data; Performing standardization processing on each of the temporary features of the sample data according to the standard deviation to obtain the sample data feature; Determining the second loss function according to the weight of the sample data feature includes: A second loss function is determined according to the weight of the sample data feature and the standard deviation.

4. The method according to any one of claims 1 to 3, characterized in that: The multimedia data includes voice data or image data.

5. A multimedia recognition device, characterized in that: The device comprises: A data acquisition module, configured to collect multimedia data through a collection component; A feature acquisition module, configured to acquire data features of the multimedia data; A data recognition module is configured to input the data feature into a multimedia recognition model to obtain a recognition result of the data feature, wherein the recognition result is used to characterize target information corresponding to the multimedia data; The multimedia recognition model is a model obtained by training a preset training model according to a multimedia sample data set, a first loss function and an objective function, wherein the multimedia sample data set includes a plurality of multimedia sample data and a target recognition result corresponding to each of the multimedia sample data; the objective function includes a sample random function; the sample random function is used to obtain sample data features of the plurality of multimedia sample data at different amplitudes; the first loss function is used to characterize the gap between the training recognition results obtained by the preset training model for the plurality of multimedia sample data and the target recognition results; The device also includes a model training module, and the model training module includes: A sample data set acquisition submodule is configured to acquire the multimedia sample data set, wherein the multimedia sample data set includes a plurality of multimedia sample data and a target recognition result corresponding to each multimedia sample data; A sample feature acquisition submodule, configured to acquire sample data features corresponding to each of the multimedia sample data; A first loss function determination submodule, configured to determine the first loss function according to the training recognition result and the target recognition result; A first training submodule is configured to train the preset training model according to the sample data feature, the target recognition result, the first loss function and the target function to obtain the multimedia recognition model; In the case where the objective function includes a sample random function, the first training submodule is configured as follows: According to the sample data feature and the sample random function, new sample data features representing different amplitudes are calculated; The new sample data feature is used as the input of a preset training model, the target recognition result is used as the output of the preset training model, and the preset training model is trained according to the first loss function to obtain the multimedia recognition model.

6. The device according to claim 5, characterized in that The objective function further includes a second loss function, which is used to make the sum of weights of sample data features of the plurality of multimedia sample data approach a preset value, wherein the weight represents the degree of influence of the sample data features on the recognition result output by the multimedia recognition model; when the objective function includes the second loss function, the first training submodule is configured as follows: Determining a second loss function according to the weight of the sample data feature; The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

7. The device according to claim 6, characterized in that The sample feature acquisition submodule is configured as follows: Performing feature extraction on each of the multimedia sample data, and obtaining a temporary feature of the sample data corresponding to each of the multimedia sample data; Obtaining a standard deviation of the temporary feature of the sample data, wherein the standard deviation represents a degree of dispersion of the temporary feature of the sample data; Performing standardization processing on each of the temporary features of the sample data according to the standard deviation to obtain the sample data feature; The first training submodule is configured as follows: Determining a second loss function according to the weight of the sample data feature and the standard deviation; The sample data feature is used as the input of a preset training model, and the target recognition result is used as the output of the preset training model. The preset training model is trained according to the first loss function and the second loss function to obtain the multimedia recognition model.

8. The device according to any one of claims 5 to 7, characterized in that The multimedia data includes voice data or image data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 4.

Images