A model training and data analysis method, device, storage medium and equipment

Abstract

The specification discloses a model training and data analysis method, device, storage medium and equipment. The model training method comprises the following steps: inputting sample data into an analysis model to be trained, determining data features of each type of data through a feature extraction layer corresponding to each type of data, obtaining analysis results corresponding to each type of data according to the data features of each type of data, determining gradient information corresponding to each type of data during training of the analysis model, fusing the data features of each type of data through a feature fusion layer to obtain fused features, and then obtaining a comprehensive analysis result, determining fused gradient information generated by all types of data during training of the analysis model according to a deviation between the comprehensive analysis result and a label corresponding to the sample data, taking the fused gradient information as a gradient label, and training the analysis model with the minimum deviation between the gradient information corresponding to each type of data and the gradient label as an optimization target.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and device for model training and data analysis. Background Technology

[0002] With the development of artificial intelligence, sentiment analysis technology has been widely used in various fields. For example, in human-computer interaction scenarios of intelligent robots, image acquisition devices (such as cameras) are usually used to collect user image data to extract information such as facial expressions and body movements. At the same time, the user's voice tone is also collected, and the content of the dialogue is extracted. Based on the above information, the user's current emotions and feelings are analyzed, and then corresponding interaction strategies are executed to interact with the user based on the analysis results.

[0003] However, in practical applications, there are often situations where the emotions expressed by some types of data in the data to be analyzed are contradictory. In such cases, current analysis methods cannot obtain accurate analysis results. For example, when the data to be analyzed is relatively calm content spoken by a user in an agitated tone, existing methods have difficulty in determining whether the user's expressed emotion is positive, negative, or neutral.

[0004] Therefore, how to accurately obtain the analysis results of the data to be analyzed based on the different types of data contained therein is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a method, apparatus, storage medium, and device for model training and data analysis, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This manual provides a method for model training, including:

[0008] Obtain sample data, wherein the sample data contains at least two types of data;

[0009] The sample data is input into the analysis model to be trained. For each type of data contained in the sample data, the data features of the data type are determined through the feature extraction layer corresponding to the data type. Based on the data features of the data type, the analysis result corresponding to the data type is obtained. Based on the deviation between the analysis result corresponding to the data type and the label corresponding to the sample data, the gradient information generated by the data type during the training of the analysis model is determined as the gradient information corresponding to the data type.

[0010] The feature fusion layer in the analysis model fuses the data features of various types of data to obtain fused features. Based on the fused features, a comprehensive analysis result is obtained. Based on the deviation between the comprehensive analysis result and the label corresponding to the sample data, the gradient information generated by all types of data in the sample data during the training of the analysis model is determined as fused gradient information.

[0011] The fused gradient information is used as the gradient label, and the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient label.

[0012] Optionally, the feature fusion layer in the analysis model fuses the data features of various types of data to obtain fused features, specifically including:

[0013] Select a specified type of data from the various types of data included in the sample data as the first type of data, and take the other types of data in the sample data other than the first type of data as the second type of data;

[0014] For each type of second data, the feature transformation parameters corresponding to the second type of data are determined based on the data features corresponding to the first type of data and the data features corresponding to the second type of data.

[0015] The data features corresponding to the second type of data are transformed using the feature transformation parameters to obtain the transformed features corresponding to the second type of data.

[0016] The fusion feature is determined based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second data.

[0017] Optionally, the fusion features are determined based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second data, specifically including:

[0018] The transformed features corresponding to each type of second-type data are fused to obtain the fused features of each type of second-type data.

[0019] Based on the data features corresponding to the first type of data and the fusion features of each second type of data, the weights corresponding to the fusion features of each second type of data are determined.

[0020] Based on the weights, the data features corresponding to the first type of data are fused with the fusion features of each second type of data to determine the fusion features obtained after fusing the data features of each type of data.

[0021] Optionally, the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient label, specifically including:

[0022] For each type of data, the gradient information is fused to minimize the deviation between the gradient information corresponding to that type of data and the label gradient information. The gradient information corresponding to that type of data is then adjusted to obtain the adjusted gradient information corresponding to that type of data.

[0023] Based on the adjusted gradient information corresponding to each type of data, the feature extraction layer corresponding to each type of data in the analysis model is trained.

[0024] Optionally, the method further includes:

[0025] The fused gradient information is adjusted by adjusting the gradient information corresponding to each type of data to obtain the adjusted fused gradient information;

[0026] The feature fusion layer is trained based on the adjusted fusion gradient information.

[0027] This manual provides a data analysis method, including:

[0028] Acquire the data to be analyzed, wherein the data to be analyzed contains at least two types of data;

[0029] The data to be analyzed is input into a pre-trained analysis model. For each type of data contained in the data to be analyzed, the data features corresponding to that type of data are determined through the feature extraction layer corresponding to that type of data. The analysis model is trained using the model training method described above.

[0030] Through the feature fusion layer in the analysis model, the fusion features corresponding to the data to be analyzed are determined according to the data features corresponding to each type of data;

[0031] Based on the fusion features, the analysis results of the data to be analyzed are determined.

[0032] This specification provides a model training apparatus, comprising:

[0033] The acquisition module acquires sample data, wherein the sample data contains at least two types of data;

[0034] The input module inputs the sample data into the analysis model to be trained. For each type of data contained in the sample data, the feature extraction layer corresponding to that type of data is used to determine the data features of that type of data. Based on the data features of that type of data, the analysis result corresponding to that type of data is obtained. Based on the deviation between the analysis result corresponding to that type of data and the label corresponding to the sample data, the gradient information generated by that type of data during the training of the analysis model is determined as the gradient information corresponding to that type of data.

[0035] The fusion module fuses the data features of various types of data through the feature fusion layer in the analysis model to obtain fused features. Based on the fused features, it obtains a comprehensive analysis result. Based on the deviation between the comprehensive analysis result and the labels corresponding to the sample data, it determines the gradient information generated by all types of data in the sample data during the training of the analysis model, which is used as fused gradient information.

[0036] The training module uses the fused gradient information as gradient labels and trains the analysis model with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient labels.

[0037] This specification provides a data analysis apparatus, comprising:

[0038] The acquisition module acquires the data to be analyzed, wherein the data to be analyzed contains at least two types of data;

[0039] The input module inputs the data to be analyzed into a pre-trained analysis model. For each type of data contained in the data to be analyzed, the data features corresponding to that type of data are determined through the feature extraction layer corresponding to that type of data. The analysis model is trained using the model training method described above.

[0040] The determination module, through the feature fusion layer in the analysis model, determines the fusion features corresponding to the data to be analyzed based on the data features corresponding to each type of data;

[0041] The analysis module determines the analysis results of the data to be analyzed based on the fusion features.

[0042] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described methods for model training and data analysis.

[0043] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for model training and data analysis.

[0044] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0045] In the model training method provided in this specification, the acquired sample data is first input into the analysis model to be trained. Then, through the feature extraction layer corresponding to each type of data, the data features of each type of data are determined, and the analysis results corresponding to each type of data are obtained. At the same time, based on the deviation between the analysis results corresponding to each type of data and the labels corresponding to the sample data, the gradient information corresponding to each type of data is determined. Then, through the feature fusion layer in the analysis model, the data features of each type of data are fused to obtain fused features. Based on the fused features, a comprehensive analysis result is obtained. Based on the deviation between the comprehensive analysis result and the labels corresponding to the sample data, the fused gradient information generated by all types of data contained in the sample data during the training of the analysis model is determined. Then, the fused gradient information is used as the gradient label, and the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to the data type and the gradient label.

[0046] As can be seen from the above method, this scheme can determine the gradient information corresponding to each type of data based on the analysis results corresponding to each type of data during the training process of the analysis model, and determine the fusion gradient information based on the comprehensive analysis results obtained from the fusion features. Thus, the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to each type of data and the gradient label. In this process, the problem of conflicting information expressed by different types of data can be transformed to the level of model gradient. By adjusting the model gradient, the problem of inaccurate data analysis results caused by the contradiction between the information expressed by each type of data can be solved, further improving the accuracy of the trained analysis model in analyzing data. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart illustrating a model training method provided in this specification.

[0049] Figure 2This is a flowchart illustrating a data analysis method provided in this specification;

[0050] Figure 3 This is a schematic diagram of a model training apparatus provided in this specification;

[0051] Figure 4 This is a schematic diagram of a data analysis device provided in this specification;

[0052] Figure 5 The one provided in this specification corresponds to Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0054] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart illustrating a model training method provided in this specification, including the following steps:

[0056] S101: Obtain sample data, wherein the sample data contains at least two types of data;

[0057] In fields such as intelligent customer service, human-computer interaction, and content moderation, it is often necessary to analyze different types of data (such as text data, image data, and audio data) to execute corresponding business operations based on the analysis results. For example, in the application scenario of intelligent chatbots, the robot's interaction system typically collects facial images, body movements, speech tone, speech rate, and extracts text content contained in the dialogue. By analyzing this data, it determines the actual emotions or feelings of the user currently interacting, and then, based on the analysis results, enables the intelligent chatbot to perform corresponding interactive actions or conduct corresponding voice dialogues.

[0058] For example, in the content review scenario of short video platforms, it is usually necessary to review the content posted by users. In this process, analysis models can be used to conduct data analysis on the content posted by users (such as videos, text, images, voice, etc.) to deal with overly negative or negative content (such as deleting or blocking it).

[0059] In practical applications, appropriate model algorithms are typically used to analyze data and obtain results. However, in certain scenarios, situations often arise where the emotions expressed by different types of data within the data being analyzed contradict each other. For example, a user might express relatively calm content in an agitated tone, or negative remarks in a normal tone. The model may struggle to determine the user's true emotions. Furthermore, when a trained model pays high attention to a particular type of data, the emotion expressed by that data can even influence the final analysis results. Therefore, the model's training method and process significantly impact the accuracy of its data analysis capabilities.

[0060] Based on this, this specification provides a model training method that uses the fused gradient information generated by all types of data in the sample data during model training to correct the gradient information corresponding to each type of data. This reduces the impact of a single type of data on the overall analysis results of the sample data, avoids the impact of contradictions between the information corresponding to different types of data on the final analysis results of the analysis model, and further improves the accuracy of data analysis.

[0061] This requires acquiring sample data, which contains two or more data types. In practical applications, this sample data can be samples specific to a particular business scenario. For example, in a human-computer interaction scenario, the sample data could include collected user text data (such as extracted dialogue content), audio data, and image data. Similarly, in a content moderation scenario, the sample data could include user-posted text, video, and audio.

[0062] It should be noted that the sample data mentioned above can be user data collected from a single user at the same time, or user data collected from multiple time points. Of course, the sample data may also include other types of data from other business scenarios, which this specification does not specifically limit.

[0063] In this specification, the executing entity used to implement the model training method and the data analysis method can refer to a specified device such as a server set up on the business platform. For ease of description, this specification will only use the server as the executing entity as an example to illustrate one model training method provided in this specification.

[0064] S102: Input the sample data into the analysis model to be trained, and for each type of data contained in the sample data, determine the data features of the data type through the feature extraction layer corresponding to the data type, and obtain the analysis result corresponding to the data type based on the data features of the data type, and determine the gradient information generated by the data type during the training of the analysis model based on the deviation between the analysis result corresponding to the data type and the label corresponding to the sample data, as the gradient information corresponding to the data type.

[0065] The above analysis results can be the actual emotional type expressed by each type of data, such as positive emotions, negative emotions, and neutral emotions, or specific emotional types, such as happiness, sadness, anger, indifference, and peace. Of course, they can also be other types of emotions or other types of analysis results. This manual does not make any specific limitations on this.

[0066] After the server acquires the sample data, it can input the sample data into the analysis model to be trained. This analysis model includes a feature extraction layer and a feature fusion layer. The feature extraction layer is used to extract the data features corresponding to each type of data in the sample data. The feature fusion layer is used to fuse the data features corresponding to each type of data to obtain the fused features corresponding to the sample data. The feature fusion process will be described in detail below, and will not be elaborated on here.

[0067] It should be noted that this analysis model can contain multiple feature extraction layers. Each feature extraction layer's feature extraction method corresponds to a specific data type. After each data type is input into its corresponding feature extraction layer, each layer extracts the data features for that type of data. Alternatively, the analysis model can contain only one feature extraction layer. This layer contains feature extraction methods corresponding to each data type. After each data type is input into this layer, its corresponding feature extraction methods are used to extract features for each data type.

[0068] Specifically, the server can first extract the data features corresponding to each type of data in the sample data through the feature extraction layer of each type of data in the analysis model. Since the feature dimensions corresponding to each type of feature are different, different methods can be used in the feature extraction layer to extract the features corresponding to different types of data.

[0069] For example, for image data, tools like Facet can be used to extract features such as facial movements, facial landmarks, head pose, gaze position, and histogram of oriented gradients (HOG). For text data, appropriate transcription tools can be used to transcribe video or audio into text data, followed by manual correction, and then feature extraction can be performed on the transcribed text data using tools such as text embedding feature extractors. For audio data, tools like COVAREP can be used to extract features such as the first three formants, spectral tilt or peaks of the wavelet response.

[0070] Of course, the feature extraction layer in this specification can also extract data features corresponding to various types of data through other tools or methods, and this specification does not make specific limitations on this.

[0071] Then, for each type of data, the server can obtain the corresponding analysis results based on the data characteristics of that type of data. Furthermore, based on the deviation between the analysis results and the labels of the sample data, and the data characteristics of that type of data, the server determines the gradient information generated by that type of data during model training. This gradient information is then used as the gradient information for that type of data. The gradient information for each type can be expressed by the following formula:

[0072]

[0073] Where y represents the analysis result corresponding to type m data, T represents the data feature corresponding to text data, A represents the data feature corresponding to audio data, and V represents the data feature corresponding to image data. For the gradient information corresponding to data type m, L m (θ) represents the loss value of the loss function of the analysis model determined based on the analysis results corresponding to each type of data and the actual labels of the sample data, E. (m，y)～B Let B be the expected value of the sample data label, and let B be the sample data.

[0074] It should be noted that the gradient information determined by the above formula is the gradient information corresponding to each type. In other words, the above text type, audio type and video type each have a corresponding gradient information.

[0075] S103: Through the feature fusion layer in the analysis model, the data features of various types of data are fused to obtain fused features. Based on the fused features, a comprehensive analysis result is obtained. Based on the deviation between the comprehensive analysis result and the label corresponding to the sample data, the gradient information generated by all types of data contained in the sample data during the training of the analysis model is determined as fused gradient information.

[0076] After extracting the data features corresponding to each type of data contained in the sample data, the server can input these features into the feature fusion layer to obtain the fused features of the sample data.

[0077] In this specification, a pre-trained language representation model (Bidirectional Encoder Representation from Transformers, BERT) can be used as the encoding tool for the feature extraction layer, and a feature fusion layer can be added after the first encoding layer.

[0078] Specifically, after obtaining the data features corresponding to each type of data, the server can input the data features corresponding to each type of data into the feature fusion layer, select the specified type of data from the various types of data contained in the sample data as the first type of data, and take the other types of data in the sample data other than the first type of data as the second type of data. Then, for each second type of data, the feature transformation parameters (such as the gating vector) corresponding to the second type of data are determined according to the data features corresponding to the first type of data and the data features corresponding to the second type of data.

[0079] Let T be the data feature corresponding to the text data. i The data feature corresponding to the audio data is A. i The data feature corresponding to the image data is V. i For example, typically, text data is selected as the primary data type, meaning text data is chosen as the first data type. In this case, A... i and V i For the second type of data, the data features corresponding to audio data are A. i The corresponding gating vector can be represented by the following formula:

[0080]

[0081] in, Data characteristics A of audio data i The corresponding gate vector (feature transformation parameter), R is a non-linear function, W ga To determine the data characteristics A of the audio data iThe preset parameters for the corresponding gating vector, b a This is the corresponding bias.

[0082] Data characteristics of video data V i The corresponding gating vector can be represented by the following formula:

[0083]

[0084] in, V, the data feature of video data i The corresponding gate vector, R is a nonlinear function, W gv To determine the data characteristics V of video data i The preset parameters for the corresponding gating vector, b v This is the corresponding bias.

[0085] Then, the server can use the gating vectors corresponding to each type of data to transform the data features corresponding to each type of data, obtain the transformed features corresponding to each type of data, and then determine the fusion features of each type of data based on the data features corresponding to the first type of data and the transformed features corresponding to each type of data.

[0086] Furthermore, after obtaining the transformed features corresponding to each type of second-type data, the server can fuse the transformed features corresponding to each type of second-type data to obtain the fused features of each type of second-type data. The fused features of each type of second-type data can be represented by the following formula:

[0087]

[0088] Among them, H i For the fusion characteristics of each type of second-class data, W a For data feature A i The corresponding preset parameter, W v For data feature V i The corresponding preset parameter, b H This is the corresponding bias.

[0089] Typically, the data features corresponding to audio data and image data are larger than those corresponding to text data. However, if the text data is too small, the text features may be unable to express the final analysis results, thus losing their meaning. Therefore, to avoid this situation, the server can determine the weight of each second type of data based on the data features corresponding to the first type of data and the fusion features of each second type of data. This weight can be expressed by the following formula:

[0090]

[0091] Where α is the weight corresponding to each second type of data, and β is the corresponding adjustment factor. When the fusion feature of each of the second types of data is greater than the feature of the corresponding data of the first type of data, then... At this point, the larger the initial fusion feature, the smaller the weight. In other words, the larger the initial fusion feature is than the data feature corresponding to the first type of data, the smaller the weight.

[0092] When the feature value of the initial fusion feature is less than the feature value of the corresponding data feature of the first type of data, the weight is 1. In other words, the maximum value of the weight is 1.

[0093] The server then uses the aforementioned weights to fuse the data features corresponding to the first type of data with the fusion features of each second type of data to determine the target fusion feature corresponding to the sample data. This fusion feature can be represented by the following formula:

[0094]

[0095] in, These are the fusion features corresponding to the sample data, and also the output of the feature fusion layer.

[0096] After determining the fusion characteristics of the sample data, the server can determine the comprehensive analysis result of the model based on these characteristics. Then, based on the deviation between the comprehensive analysis result and the labels corresponding to the sample data, it can determine the gradient information generated by all types of data in the sample data during model training, which is then used as the fusion gradient information. This fusion gradient information can be expressed by the following formula:

[0097]

[0098] In this context, the data feature corresponding to text data is T, the data feature corresponding to audio data is A, and the data feature corresponding to image data is V. For the fusion of gradient information in the model, L M (θ) represents the loss value of the loss function of the analysis model, determined based on the deviation between the comprehensive analysis results and the labels corresponding to the sample data. (T,A,V,y)～B Let B be the expected value of the model's expected analysis results, and let B be the sample data.

[0099] S104: The fused gradient information is used as the gradient label, and the analysis model is trained with the goal of minimizing the deviation between the gradient information corresponding to this type of data and the gradient label.

[0100] Specifically, the server can adjust the gradient information for each data type based on the fused gradient information, aiming to minimize the deviation between the gradient information for that data type and the label gradient information. This results in adjusted gradient information for that data type, making the gradient information for each data type closer to the fused gradient information. This avoids contradictions or conflicts between the sentiment expressed by some data types and the sentiment expressed by other data types, thus making the data features extracted by the feature extraction layer for each data type more reliable and accurate. The adjusted gradient information for each data type can be represented by the following formula:

[0101]

[0102] in, For the adjusted gradient information corresponding to type i, g i This represents the original gradient information corresponding to type i. To fuse gradient information.

[0103] It should be noted that the gradient information corresponding to each type can be applied to the feature extraction layer of the analysis model. After obtaining the gradient information corresponding to each type, the server can train the feature extraction layer of the analysis model based on the gradient information, thereby updating the parameters of the feature extraction layer and making the features extracted by the feature extraction layer after training more accurate.

[0104] Furthermore, the server can adjust the fused gradient information based on the adjusted gradient information corresponding to each type, thus obtaining adjusted fused gradient information, which makes the information expressed by the fused feature more comprehensive. This adjusted fused gradient information can be represented by the following formula:

[0105]

[0106] in, This is to adjust the fused gradient information.

[0107] The aforementioned fusion gradient information can be applied to the feature fusion layer. The server can use this adjusted fusion gradient information to train the feature fusion layer of the analysis model, thereby updating the parameters of the feature fusion layer so that the trained feature fusion layer obtains richer and more comprehensive information.

[0108] In this specification, in addition to the feature extraction layer and the feature fusion layer, the model may also include a processing layer to obtain the final analysis result based on the fused features. During the model training process, the processing layer can also be trained based on the fused gradient information to update the parameters corresponding to the processing layer.

[0109] The server can then train the analysis model using the sample data until the training objective is met. This training objective can be: the analysis model converges to a preset threshold range, or a preset number of training iterations, to ensure that the analysis results obtained by the model accurately reflect the information actually expressed by the sample data (such as the actual emotion type). The preset threshold and preset number of training iterations can be set according to actual needs; this manual does not impose specific limitations.

[0110] Once the analysis model has been trained, it can be deployed to determine the analysis results of the data to be analyzed and to execute corresponding business operations based on the analysis results.

[0111] The above describes the model training method provided in this manual from the perspective of model training. The following will describe the data analysis method provided in this manual from the perspective of the practical application of the model.

[0112] Figure 2 This specification provides a data analysis method.

[0113] S201: Obtain the data to be analyzed, wherein the data to be analyzed contains at least two types of data.

[0114] In practical applications, the data to be analyzed can be data actually collected during interaction in different scenarios, and the execution subject of the data analysis method can be a server. Of course, it can also be an interactive system deployed in a terminal device in a real human-computer interaction scenario. For ease of description, this manual will only use the server as the execution subject as an example to explain the data analysis method provided in this manual.

[0115] S202: Input the data to be analyzed into a pre-trained analysis model. For each type of data contained in the data to be analyzed, determine the data features corresponding to that type of data through the feature extraction layer corresponding to that type of data. The analysis model is trained using the model training method described above.

[0116] After the server obtains the data to be analyzed, it can input the data into the pre-trained analysis model, and then extract the data features corresponding to each type of data through the feature extraction layer corresponding to each type of data.

[0117] S203: Through the feature fusion layer in the analysis model, the fusion features corresponding to the data to be analyzed are determined according to the data features corresponding to each type of data.

[0118] After determining the data features corresponding to each type of data, the server can further input these features into the feature fusion layer, thereby fusing the data features corresponding to each type of data through the feature fusion layer to obtain the fused features corresponding to the data to be analyzed.

[0119] S204: Determine the analysis result of the data to be analyzed based on the fusion features.

[0120] The server can then input the aforementioned fused features into the feature extraction layer of the model, thereby obtaining the analysis results of the data to be analyzed based on the fused features. After determining the analysis results, the server can perform corresponding business processing based on the analysis results. For example, it can determine the type of emotion expressed by the user and the content of the dialogue based on the analysis results of the model, and interact with the user based on the type of emotion and the content of the dialogue (such as sending voice or text replies to the user).

[0121] Of course, in this specification, the analysis results of the sample data can also be determined based on the data characteristics corresponding to each type of data, or the final analysis results can be obtained based on the data characteristics corresponding to each type of data and the fused data characteristics (such as combining the analysis results determined based on the data characteristics corresponding to each type of data and the analysis results determined based on the fused characteristics).

[0122] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the data protection laws and policies of the country where the application is located, and with authorization from the owner of the relevant device.

[0123] As can be seen from the above method, this scheme can determine the gradient information corresponding to each type of data based on the analysis results corresponding to each type of data during the training process of the analysis model, and determine the fusion gradient information based on the comprehensive analysis results obtained from the fusion features. Thus, the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to each type of data and the gradient label. In this process, the problem of conflicting information expressed by different types of data can be transformed to the level of model gradient. By adjusting the model gradient, the problem of inaccurate data analysis results caused by the contradiction between the information expressed by each type of data can be solved, further improving the accuracy of the trained analysis model in analyzing data.

[0124] The above describes one or more methods for implementing model training and data analysis as outlined in this manual. Based on the same approach, this manual also provides corresponding apparatus for model training and data analysis, such as... Figure 3 or Figure 4 As shown.

[0125] Figure 3 A schematic diagram of a model training apparatus provided in this specification includes:

[0126] The acquisition module 301 is used to acquire sample data, wherein the sample data contains at least two types of data;

[0127] The input module 302 is used to input the sample data into the analysis model to be trained, so as to determine the data features of each type of data in the sample data through the feature extraction layer corresponding to the data type, and obtain the analysis result corresponding to the data type based on the data features of the data type, and determine the gradient information generated by the data type during the training of the analysis model based on the deviation between the analysis result corresponding to the data type and the label corresponding to the sample data, and use it as the gradient information corresponding to the data type.

[0128] The fusion module 303 is used to fuse the data features of various types of data through the feature fusion layer in the analysis model to obtain fused features, and to obtain a comprehensive analysis result based on the fused features. Based on the deviation between the comprehensive analysis result and the label corresponding to the sample data, the gradient information generated by all types of data in the sample data during the training of the analysis model is determined as fused gradient information.

[0129] The training module 304 is used to train the analysis model by using the fused gradient information as gradient labels and minimizing the deviation between the gradient information corresponding to this type of data and the gradient labels as the optimization objective.

[0130] Optionally, the fusion module 303 is specifically configured to: select a specified type of data from the various types of data included in the sample data as the first type of data, and take the other types of data in the sample data besides the first type of data as the second type of data; for each type of second type of data, determine the feature transformation parameter corresponding to the second type of data based on the data features corresponding to the first type of data and the data features corresponding to the second type of data; transform the data features corresponding to the second type of data using the feature transformation parameter to obtain the transformed features corresponding to the second type of data; and determine the fused feature based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second type of data.

[0131] Optionally, the fusion module 303 is specifically used to: fuse the transformed features corresponding to each type of second type of data to obtain fusion features of each type of second type of data; determine the weights corresponding to the fusion features of each type of second type of data based on the data features corresponding to the first type of data and the fusion features of each type of second type of data; and fuse the data features corresponding to the first type of data with the fusion features of each type of second type of data based on the weights to determine the fusion features obtained after fusing the data features of each type of data.

[0132] Optionally, the training module 304 is specifically used to, for each type of data, adjust the gradient information corresponding to that type of data by means of the fused gradient information with the goal of minimizing the deviation between the gradient information corresponding to that type of data and the label gradient information, to obtain the adjusted gradient information corresponding to that type of data; and train the feature extraction layer corresponding to each type of data in the analysis model according to the adjusted gradient information corresponding to each type of data.

[0133] Optionally, the training module 304 is further configured to adjust the fused gradient information using the adjusted gradient information corresponding to each type of data to obtain adjusted fused gradient information; and to train the feature fusion layer based on the adjusted fused gradient information.

[0134] Figure 4 This is a schematic diagram illustrating a data analysis method provided in this specification, including:

[0135] The acquisition module 401 acquires the data to be analyzed, wherein the data to be analyzed contains at least two types of data;

[0136] The input module 402 inputs the data to be analyzed into a pre-trained analysis model. For each type of data contained in the data to be analyzed, the data features corresponding to that type of data are determined through the feature extraction layer corresponding to that type of data. The analysis model is trained by the above-mentioned model training method.

[0137] The determination module 403 determines the fusion features corresponding to the data to be analyzed based on the data features corresponding to each type of data through the feature fusion layer in the analysis model.

[0138] The analysis module 404 determines the analysis result of the data to be analyzed based on the fusion features.

[0139] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 or Figure 2This provides a method for model training and data analysis.

[0140] This instruction manual also provides Figure 5 The one shown corresponds to Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 or Figure 2 The methods for model training and data analysis described herein. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0141] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0142] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0143] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0144] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0145] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0150] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0151] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0152] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0153] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0156] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for training a model, characterized in that, include: Obtain sample data, wherein the sample data contains at least two types of data, and the types of sample data include at least text data, image data, and audio data; The sample data is input into the analysis model to be trained. For each type of data contained in the sample data, the data features of the data type are determined through the feature extraction layer corresponding to the data type. Based on the data features of the data type, the analysis result corresponding to the data type is obtained. Based on the deviation between the analysis result corresponding to the data type and the label corresponding to the sample data, the gradient information generated by the data type during the training of the analysis model is determined as the gradient information corresponding to the data type. The feature fusion layer in the analysis model fuses the data features of various types of data to obtain fused features. Based on the fused features, a comprehensive analysis result is obtained. Based on the deviation between the comprehensive analysis result and the label corresponding to the sample data, the gradient information generated by all types of data in the sample data during the training of the analysis model is determined as fused gradient information. The fused gradient information is used as the gradient label, and the analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient label. The feature fusion layer in the analysis model fuses the data features of each type of data to obtain fused features, specifically including: Select a specified type of data from the various types of data included in the sample data as the first type of data, and take the other types of data in the sample data other than the first type of data as the second type of data; For each type of second data, the feature transformation parameters corresponding to the second type of data are determined based on the data features corresponding to the first type of data and the data features corresponding to the second type of data. The data features corresponding to the second type of data are transformed using the feature transformation parameters to obtain the transformed features corresponding to the second type of data. The fusion feature is determined based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second data.

2. The method as described in claim 1, characterized in that, Based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second data, the fusion features are determined, specifically including: The transformed features corresponding to each type of second-type data are fused to obtain the fused features of each type of second-type data. Based on the data features corresponding to the first type of data and the fusion features of each second type of data, the weights corresponding to the fusion features of each second type of data are determined. Based on the weights, the data features corresponding to the first type of data are fused with the fusion features of each second type of data to determine the fusion features obtained after fusing the data features of each type of data.

3. The method as described in claim 1, characterized in that, The analysis model is trained with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient label, specifically including: For each type of data, the gradient information is fused to minimize the deviation between the gradient information corresponding to that type of data and the gradient label. The gradient information corresponding to that type of data is then adjusted to obtain the adjusted gradient information corresponding to that type of data. Based on the adjusted gradient information corresponding to each type of data, the feature extraction layer corresponding to each type of data in the analysis model is trained.

4. The method as described in claim 3, characterized in that, The method further includes: The fused gradient information is adjusted by adjusting the gradient information corresponding to each type of data to obtain the adjusted fused gradient information; The feature fusion layer is trained based on the adjusted fusion gradient information.

5. A data analysis method, characterized in that, include: Acquire the data to be analyzed, wherein the data to be analyzed contains at least two types of data; The data to be analyzed is input into a pre-trained analysis model. For each type of data contained in the data to be analyzed, the data features corresponding to that type of data are determined through the feature extraction layer corresponding to that type of data. The analysis model is trained by the model training method described in any one of claims 1 to 4. Through the feature fusion layer in the analysis model, the fusion features corresponding to the data to be analyzed are determined according to the data features corresponding to each type of data; Based on the fusion features, the analysis results of the data to be analyzed are determined.

6. A device for model training, characterized in that, include: The acquisition module acquires sample data, wherein the sample data contains at least two types of data, and the types of sample data include at least text data, image data, and audio data; The input module inputs the sample data into the analysis model to be trained. For each type of data contained in the sample data, the feature extraction layer corresponding to that type of data is used to determine the data features of that type of data. Based on the data features of that type of data, the analysis result corresponding to that type of data is obtained. Based on the deviation between the analysis result corresponding to that type of data and the label corresponding to the sample data, the gradient information generated by that type of data during the training of the analysis model is determined as the gradient information corresponding to that type of data. The fusion module fuses the data features of various types of data through the feature fusion layer in the analysis model to obtain fused features. Based on the fused features, it obtains a comprehensive analysis result. Based on the deviation between the comprehensive analysis result and the labels corresponding to the sample data, it determines the gradient information generated by all types of data in the sample data during the training of the analysis model, which is used as fused gradient information. The training module uses the fused gradient information as gradient labels and trains the analysis model with the optimization objective of minimizing the deviation between the gradient information corresponding to this type of data and the gradient labels. The feature fusion layer in the analysis model fuses the data features of various types of data to obtain fused features, specifically including: Select a specified type of data from the various types of data included in the sample data as the first type of data, and take the other types of data in the sample data other than the first type of data as the second type of data; For each type of second data, the feature transformation parameters corresponding to the second type of data are determined based on the data features corresponding to the first type of data and the data features corresponding to the second type of data. The data features corresponding to the second type of data are transformed using the feature transformation parameters to obtain the transformed features corresponding to the second type of data. The fusion feature is determined based on the data features corresponding to the first type of data and the transformed features corresponding to each type of second data.

7. A data analysis apparatus, characterized in that, include: The acquisition module acquires the data to be analyzed, wherein the data to be analyzed contains at least two types of data; The input module inputs the data to be analyzed into a pre-trained analysis model. For each type of data contained in the data to be analyzed, the data features corresponding to that type of data are determined through the feature extraction layer corresponding to that type of data. The analysis model is trained by the model training method described in any one of claims 1 to 4. The determination module, through the feature fusion layer in the analysis model, determines the fusion features corresponding to the data to be analyzed based on the data features corresponding to each type of data; The analysis module determines the analysis results of the data to be analyzed based on the fusion features.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 5.

Images