Neural network model fusion monitoring method and device

Abstract

The present invention provides a neural network model fusion monitoring method, including: processing a first training sample set and a first test sample set respectively to determine corresponding sample processing results; calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing results; testing the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the data set characteristics that match the first training sample set and the first test sample set, and obtaining the model fusion test processing results of the target neural network model set; the present invention also provides a neural network model fusion monitoring device, an electronic device and a storage medium. The present invention can automatically perform model fusion testing on the target neural network model set without considering artificial fluctuations and additional data costs, thereby improving the accuracy and robustness of the model.

Description

Technical Field

[0001] The present invention relates to information processing technology, and in particular to a training method, device, electronic device and server for a target neural network model set. Background Art

[0002] The key technologies of speech technology include automatic speech recognition technology (ASR Automatic Speech Recognition), text-to-speech conversion technology (TTS Test-to-speech) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech becomes one of the most convenient ways of human-computer interaction. Due to the complexity of the speech processing environment, a set of neural network models can usually be provided, and different neural network models can be used to process speech information in complex environments. At the same time, better task processing effects can be obtained through the fusion of neural network models. However, there has been a lack of automated solutions for the circumstances under which model fusion processing is performed. Summary of the invention

[0003] In view of this, the embodiment of the present invention provides a neural network model fusion monitoring method, device, electronic device and server. The technical solution of the embodiment of the present invention is implemented as follows:

[0004] The embodiment of the present invention generally provides a neural network model fusion monitoring method, the method comprising:

[0005] According to task information in a use environment of a target neural network model set, obtaining a first training sample set and a first test sample set that match the target neural network model set, wherein the target neural network model set includes at least two different target neural network models;

[0006] Performing adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set;

[0007] Processing the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results;

[0008] Calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing results;

[0009] Based on the correlation characteristics of each target neural network model in the target neural network model set and the characteristics of the data set that matches the first training sample set and the first test sample set, the model fusion of the target neural network model set is tested and processed to obtain the model fusion test processing result of the target neural network model set;

[0010] Based on the model fusion test processing results of the target neural network model set, the task information in the usage environment is processed through the target neural network model set.

[0011] The embodiment of the present invention further provides a neural network model fusion monitoring device, the device comprising:

[0012] An information transmission module, configured to obtain a first training sample set and a first test sample set matching the target neural network model set according to task information in a use environment of the target neural network model set, wherein the target neural network model set includes at least two different target neural network models;

[0013] An information processing module, configured to perform adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set;

[0014] The information processing module is used to process the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results;

[0015] The information processing module is used to calculate the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result;

[0016] The information processing module is used to test the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the data set characteristics that match the first training sample set and the first test sample set, so as to obtain the model fusion test processing result of the target neural network model set;

[0017] The information processing module is used to process the task information in the use environment through the target neural network model set based on the model fusion test processing results of the target neural network model set.

[0018] In the above scheme,

[0019] The information processing module is used to rewrite the sample label of the first training sample set to 0, so as to obtain a first training sample set with a sample label of 0;

[0020] The information processing module is used to rewrite the sample label of the first test sample set to 1, so as to obtain a first test sample set with the sample label of 1;

[0021] The information processing module is used to perform data fusion processing on the first training sample set whose sample label is 0 and the first test sample set whose local label is 1, and divide the data fusion processing result to obtain a second training sample set and a second test sample set;

[0022] The information processing module is used to calculate the data set features that match the first training sample set and the first testing sample set based on the second training sample set and the second testing sample set.

[0023] In the above scheme,

[0024] The information processing module is used to process the second test sample set through the extreme gradient boosting model to obtain the AUC value corresponding to the second test sample set;

[0025] The information processing module is used to calculate the single feature vectors corresponding to the second training sample set and the second test sample set respectively in response to the AUC value corresponding to the second test sample set;

[0026] The information processing module is used to determine the association relationship between the feature vectors of different dimensions respectively corresponding to the second training sample set and the second test sample set based on the single feature vectors respectively corresponding to the second training sample set and the second test sample set;

[0027] The information processing module is used to calculate the data set features that match the first training sample set and the first test sample set based on the single feature vectors corresponding to the second training sample set and the second test sample set respectively, and the correlation relationship between the feature vectors of different dimensions corresponding to the second training sample set and the second test sample set respectively.

[0028] In the above scheme,

[0029] The information processing module is used to obtain the sample processing results of the base model in the target neural network model set and the sample processing results of any target neural network model in the target neural network model set;

[0030] The information processing module is used to calculate the Pearson correlation coefficient of each target neural network model in the target neural network model set according to the sample processing result of the base model and the sample processing result of any target neural network model;

[0031] The information processing module is used to determine the model diversity parameter based on the number of different types of target neural network models in the target neural network model set and the Pearson correlation coefficient.

[0032] In the above scheme,

[0033] The information processing module is used for, when the task information in the use environment of the target neural network model set is a classification task,

[0034] The information processing module is used to calculate the information entropy of each target neural network model in the target neural network model set according to the sample processing result;

[0035] The information processing module is used to calculate the standard deviation of the corresponding sample processing results of different target neural network models in the target neural network model set according to the sample processing results when the task information in the use environment of the target neural network model set is a regression task;

[0036] The information processing module is used to determine the deterministic parameters of each target neural network model in the target neural network model set based on the information entropy of each target neural network model, or the standard deviation of the corresponding sample processing results of the different target neural network models.

[0037] In the above scheme,

[0038] The information processing module is used to divide the data points corresponding to the sample processing result according to the sample processing result;

[0039] The information processing module is used to determine the divergence parameter corresponding to the sample processing result based on the corresponding data points of the sample processing result;

[0040] The information processing module is used to calculate the predicted boundary clarity parameter of each target neural network model in the target neural network model set through the divergence parameter corresponding to the sample processing result.

[0041] In the above scheme,

[0042] The information processing module is used to perform model fusion test processing on different target neural network models in the target neural network model set based on the diversity parameters, deterministic parameters, prediction boundary clarity parameters and data set characteristics of each target neural network model in the target neural network model set, and obtain the model fusion test processing results of the target neural network model set.

[0043] In the above scheme,

[0044] The information processing module is used to output the sample processing results of the base model in the target neural network model set when the model fusion test processing result is to close the model fusion; or

[0045] The information processing module is used to trigger the corresponding target neural network model fusion strategy when the model fusion test processing result is to start model fusion, and to perform fusion processing on different target neural network models in the target neural network model set.

[0046] In the above scheme,

[0047] The information processing module is used for, when the target neural network model set is a fusion speech information processing model set,

[0048] The information processing module is used to obtain a third training sample set, wherein the third training sample set is audio samples in the video information collected by the terminal;

[0049] The information processing module is used to add noise to the third training sample set to form a corresponding fourth training sample set;

[0050] The information processing module is used to process the fourth training sample set through the fusion speech information processing model to determine the initial parameters of the fusion speech information processing model;

[0051] The information processing module is used to process the fourth training sample set through the fusion speech information processing model in response to the initial parameters of the fusion speech information processing model to determine the update parameters of the fusion speech information processing model;

[0052] The information processing module is used to iteratively update the network parameters of the fusion speech information processing model through the fourth training sample set according to the update parameters of the fusion speech information processing model.

[0053] In the above scheme,

[0054] The information processing module is used to determine the dynamic noise type that matches the use environment of the fusion speech information processing model;

[0055] The information processing module is used to add noise to the third training sample set according to the dynamic noise type to change the background noise, volume or sampling rate of the audio samples in the third training sample set to form a corresponding fourth training sample set.

[0056] In the above scheme,

[0057] The information processing module is used to substitute different audio samples in the fourth training sample set into the loss function corresponding to the ternary loss function layer network of the fusion speech information processing model;

[0058] The information processing module is used to determine the parameters of the ternary loss function layer network in the fusion speech information processing model when the loss function meets the corresponding convergence condition;

[0059] The information processing module is used to use the parameters of the ternary loss function layer network as update parameters of the fusion speech information processing model.

[0060] In the above scheme,

[0061] The information processing module is used to determine the convergence condition matching the ternary loss function layer network in the fusion speech information processing model;

[0062] The information processing module is used to iteratively update the network parameters of the ternary loss function layer network until the loss function corresponding to the ternary loss function layer network meets the corresponding convergence condition.

[0063] An embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0064] A memory for storing executable instructions;

[0065] The processor is used to implement the preceding neural network model fusion monitoring method when running the executable instructions stored in the memory.

[0066] An embodiment of the present invention also provides a computer-readable storage medium storing executable instructions, characterized in that when the executable instructions are executed by a processor, the preceding neural network model fusion monitoring method is implemented.

[0067] The embodiments of the present invention have the following beneficial effects:

[0068] The present invention obtains a first training sample set and a first test sample set matching the target neural network model set according to task information in the use environment of the target neural network model set, wherein the target neural network model set includes at least two different target neural network models; performs adversarial verification processing on the first training sample set and the first test sample set to determine data set features matching the first training sample set and the first test sample set; processes the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results; calculates the correlation features of each target neural network model in the target neural network model set according to the sample processing results; performs test processing on the model fusion of the target neural network model set based on the correlation features of each target neural network model in the target neural network model set and the data set features matching the first training sample set and the first test sample set to obtain the model fusion test processing results of the target neural network model set; based on the model fusion test processing results of the target neural network model set, the task information in the use environment is processed through the target neural network model set, thereby automatically performing model fusion testing on the target neural network model set, and automatically fitting decisions are made to replace the method of empirical judgment. This approach does not need to consider artificial fluctuations and additional data costs, thereby improving the accuracy and robustness of the model and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a use scenario of the neural network model fusion monitoring method provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of the structure of a neural network model fusion monitoring device provided in an embodiment of the present invention;

[0071] Figure 3 It is a schematic diagram of the process of neural network model fusion in the prior art;

[0072] Figure 4 An optional flowchart of a neural network model fusion monitoring method provided in an embodiment of the present invention;

[0073] Figure 5 An optional flowchart of a neural network model fusion monitoring method provided in an embodiment of the present invention;

[0074] Figure 6 An optional flowchart of a neural network model fusion monitoring method provided in an embodiment of the present invention;

[0075] Figure 7 An optional flowchart of a neural network model fusion monitoring method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0077] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0078] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0079] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0080] 2) Model parameters: It is a quantity that uses universal variables to establish the relationship between functions and variables. In artificial neural networks, model parameters are usually real number matrices.

[0081] 3) Speech Translation: Also known as automatic speech semantic understanding, it is a technology that uses computers to understand the speech semantics of one natural language into text or speech of another natural language. It generally consists of two stages: semantic understanding and machine semantic understanding.

[0082] 4) Convolutional Neural Networks (CNN) is a type of feed-forward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure.

[0083] 5) Model training, multi-classification learning of image datasets. The model can be built using deep learning frameworks such as Tensor Flow and torch, and uses multiple layers of neural network layers such as CNN to form a multi-classification model. The input of the model is a three-channel or original channel matrix formed by reading the image through tools such as openCV. The model output is a multi-classification probability, and the web page category is finally output through algorithms such as softmax. During training, the model approaches the correct trend through objective functions such as cross entropy.

[0084] 6) Neural Network (NN): Artificial Neural Network (ANN), referred to as neural network or quasi-neural network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the field of machine learning and cognitive science, and is used to estimate or approximate functions.

[0085] 7) Machine Translation (MT): It belongs to the field of computational linguistics. Its research is to translate text or speech from one natural language into another natural language by computer programs. Neural Machine Translation (NMT) is a technology that uses neural network technology for machine translation.

[0086] 8) Encoder-decoder structure: A network structure commonly used in machine translation technology. It consists of two parts: an encoder and a decoder. The encoder converts the input text into a series of context vectors that can express the characteristics of the input text. The decoder receives the output of the encoder as its own input and outputs the corresponding text sequence in another language.

[0087] 8) Base model: In the embodiments provided in this application, the base model refers to a single neural network model trained to solve actual tasks.

[0088] 9) Fusion model: refers to a model that is formed by fusing multiple base models in the target neural network model set through model fusion methods such as bagging, voting, stacking, etc. For example, stacking model fusion can be expressed as a way of learning a new learner by using the initial training data to learn several base learners and then using the prediction results of these learners as new training sets.

[0089] 10) KL distance: measures the difference between two probability distributions in the same event space and is not symmetric.

[0090] 11)JS divergence: It is symmetric and is a smoothing method relative to KL distance.

[0091] 12) Wasserstein distance: measures the distance between two distributions. Since its derivation process can be vividly explained by digging and filling, it is also called the bulldozer distance.

[0092] 13) Euclidean distance: This is a commonly used distance definition, which refers to the actual distance between two points in m-dimensional space. The Euclidean distance in two-dimensional and three-dimensional space is the actual distance between two points.

[0093] 14) Information entropy: In information theory, entropy is the average amount of information contained in each received message, also known as information entropy, source entropy, and average self-information. In the embodiments of the present application, it can be used to measure the certainty of the model for the prediction results. The higher the information entropy value, the greater the uncertainty of the model.

[0094] Figure 1 A schematic diagram of a use scenario of a neural network model training method for a speech information processing function provided by an embodiment of the present invention, referring to Figure 1 The terminal (including the terminal 10-1 and the terminal 10-2) is provided with a client of the voice recognition software. The user can input the corresponding sentence to be voice recognized through the provided voice recognition software client. The chat client can also receive the corresponding voice recognition result and display the received voice recognition result to the user, or execute the task matching the voice command. The terminal is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission.

[0095] As an example, server 200 is used to deploy the neural network model of the voice information processing function and train the neural network model of the voice information processing function, and deploy the trained neural network model of the voice information processing function in the corresponding terminal, and use the deployed neural network model of the voice information processing function to process voice information in various complex usage environments through the terminal (terminal 10-1 and / or terminal 10-2).

[0096] As an example, a client of semantic understanding software is set on the terminal (including terminal 10-1 and terminal 10-2), and the user can input the corresponding sentence to be semantically understood through the set semantic understanding software client, and the artificial intelligence client (such as the voice assistant applet in the instant messaging client) can also receive the corresponding semantic understanding results and display the received semantic understanding results to the user; the terminal is connected to the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission, wherein the voice information recognition method provided by the present invention can be used as a cloud service to serve various types of customers (packaged in a vehicle-mounted terminal or in different mobile electronic devices), and the present application does not make specific restrictions, wherein it is provided to corporate customers as a cloud service to help them train the neural network model of the voice information processing function according to different equipment usage environments.

[0097] Of course, before the speech information in the use environment is processed by the neural network model of the speech information processing function to generate the corresponding speech recognition result, it is also necessary to judge whether the neural network model of the speech information processing function is to be model fused, which specifically includes: according to the task information in the use environment of the target neural network model set, a first training sample set and a first test sample set matching the target neural network model set are obtained, wherein the target neural network model set includes at least two different target neural network models; the first training sample set and the first test sample set are subjected to adversarial verification processing to determine the data set features matching the first training sample set and the first test sample set; the first training sample set and the first test sample set are processed respectively by different target neural networks in the target neural network model set to determine the corresponding sample processing results; according to the sample processing results, the correlation features of each target neural network model in the target neural network model set are calculated; based on the correlation features of each target neural network model in the target neural network model set and the data set features matching the first training sample set and the first test sample set, the model fusion of the target neural network model set is tested and processed to obtain the model fusion test processing results of the target neural network model set.

[0098] Among them, the smart device wake-up method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0099] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0100] In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned speech processing technology and machine learning. For example, it may involve speech recognition technology (Automatic Speech Recognition, ASR) in speech technology (Speech Technology), including speech signal preprocessing (Speech signal preprocessing), speech signal frequency domain analysis (Speech signal frequency analyzing), speech signal feature extraction (Speech signal feature extraction), speech signal feature matching / recognition (Speech signal feature matching / recognition), speech training (Speech training), etc.

[0101] For example, it may involve machine learning (ML), which is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning usually includes technologies such as deep learning. Deep learning includes artificial neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc.

[0102] It can be understood that the neural network model training method and voice processing function of the voice information processing function provided in the present application can be applied to intelligent devices (Intelligent device), and the intelligent device can be any device with voice command recognition function, for example, it can be a smart terminal, smart home device (such as smart speakers, smart washing machines, etc.), smart wearable devices (such as smart watches), in-vehicle intelligent central control system (using voice commands to wake up small programs in the terminal to perform different tasks) or AI smart medical equipment (wake-up trigger through voice commands), etc.

[0103] The structure of the neural network model fusion monitoring device of the embodiment of the present invention is described in detail below. The neural network model fusion monitoring device can be implemented in various forms, such as a dedicated terminal with a neural network model training function with a voice information processing function, or a server with a neural network model training function with a voice information processing function, such as the preamble Figure 1 Server 200 in. Figure 2 The schematic diagram of the structure of the neural network model fusion monitoring device provided by the embodiment of the present invention can be understood as follows: Figure 2 Only the exemplary structure of the neural network model fusion monitoring device is shown, not the entire structure, and can be implemented as needed. Figure 2 Partial or complete structure shown.

[0104] The neural network model fusion monitoring device provided in the embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the neural network model fusion monitoring device are coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 205 .

[0105] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0106] It is understood that the memory 202 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The memory 202 in the embodiment of the present invention can store data to support the operation of the terminal (such as 10-1). Examples of these data include: any computer program for operating on the terminal (such as 10-1), such as an operating system and an application. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application can include various applications.

[0107] In some embodiments, the neural network model fusion monitoring device provided in the embodiments of the present invention can be implemented in a combination of software and hardware. As an example, the target neural network model set task information processing method device provided in the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the neural network model fusion monitoring method provided in the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array) or other electronic components.

[0108] As an example of a neural network model fusion monitoring device provided in an embodiment of the present invention being implemented by a combination of software and hardware, the neural network model fusion monitoring device provided in an embodiment of the present invention can be directly embodied as a combination of software modules executed by processor 201, and the software module can be located in a storage medium, and the storage medium is located in memory 202. Processor 201 reads the executable instructions included in the software module in memory 202, and combines with necessary hardware (for example, including processor 201 and other components connected to bus 205) to complete the neural network model fusion monitoring method provided in an embodiment of the present invention.

[0109] As an example, the processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0110] As an example of hardware implementation of the neural network model fusion monitoring device provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed using a processor 201 in the form of a hardware decoding processor, for example, one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components to implement the neural network model fusion monitoring method provided in an embodiment of the present invention.

[0111] The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the neural network model fusion monitoring device. Examples of these data include: any executable instructions for operating on the neural network model fusion monitoring device, such as executable instructions, and the program for implementing the neural network model fusion monitoring method of the embodiment of the present invention can be included in the executable instructions.

[0112] In other embodiments, the neural network model fusion monitoring device provided by the embodiment of the present invention can be implemented in software. Figure 2The neural network model fusion monitoring device stored in the memory 202 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules. As an example of a program stored in the memory 202, a neural network model fusion monitoring device can be included, and the neural network model fusion monitoring device includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the neural network model fusion monitoring device are read into the RAM by the processor 201 and executed, the neural network model fusion monitoring method provided by the embodiment of the present invention will be implemented. The functions of each software module in the neural network model fusion monitoring device will be described below, among which,

[0113] An information transmission module 2081 is used to obtain a first training sample set and a first test sample set that match the target neural network model set according to task information in the use environment of the target neural network model set, wherein the target neural network model set includes at least two different target neural network models;

[0114] An information processing module 2082 is used to perform adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set;

[0115] The information processing module 2082 is used to process the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results;

[0116] The information processing module 2082 is used to calculate the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result;

[0117] The information processing module 2082 is used to test the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the data set characteristics that match the first training sample set and the first test sample set, and obtain the model fusion test processing result of the target neural network model set;

[0118] The information processing module 2082 is used to process the task information in the use environment through the target neural network model set based on the model fusion test processing results of the target neural network model set.

[0119] according to Figure 2The electronic device shown, in one aspect of the present application, the present application also provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the different embodiments and combinations of embodiments provided in various optional implementations of the above-mentioned neural network model fusion monitoring method.

[0120] Combination Figure 2 The neural network model fusion monitoring device shown illustrates the neural network model fusion monitoring method provided by the embodiment of the present invention. Before introducing the neural network model fusion monitoring method provided by the embodiment of the present invention, the related art is first introduced, in which a traditional target neural network model set is used to generate a corresponding answer sentence according to a question text. Figure 3 It is a schematic diagram of the process of neural network model fusion in the prior art, where: Figure 3 In the target neural network model fusion judgment process shown in the figure, the model fusion part is used as a supplement to the base model. Generally, the decision maker initially decides whether to add the fusion module based on his experience, and decides whether to cancel this part by comparing the performance of the fusion model with the base model. Therefore, the premise for achieving better results consists of the following two aspects:

[0121] 1) The decision maker is experienced and makes the right decision. However, the drawback of this solution is that the model strategy relies too much on human factors, which will lead to greater volatility, unquantifiable and low objective interpretation.

[0122] 2) There is sufficient test data to compare the performance of the base model and the fusion model. However, the drawback of this solution is that in real industrial environments, sufficient and accurately labeled data is often too expensive, or even impossible to obtain due to limited conditions, which is not conducive to the promotion and use of model fusion.

[0123] To address the deficiencies in this related art, see Figure 4 , Figure 4 An optional flow chart of the neural network model fusion monitoring method provided by the embodiment of the present invention is as follows: Figure 4 The steps shown can be performed by various electronic devices running the target neural network model set task information processing method device, such as a dedicated terminal with a model training function, a server with a target neural network model set training function, or a server cluster. Figure 4 The steps shown are explained.

[0124] Step 401: The task information processing device obtains a first training sample set and a first test sample set that match the target neural network model set based on task information in the use environment of the target neural network model set.

[0125] Wherein, the target neural network model set includes at least two different target neural network models.

[0126] Step 402: The task information processing device performs adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set.

[0127] In some embodiments of the present invention, determining the data set features that match the first training sample set and the first test sample set may be achieved by:

[0128] The sample labels of the first training sample set are rewritten as 0 to obtain a first training sample set with a sample label of 0; the sample labels of the first test sample set are rewritten as 1 to obtain a first test sample set with a sample label of 1; data fusion processing is performed on the first training sample set with a sample label of 0 and the first test sample set with a sample label of 1, and the data fusion processing result is divided to obtain a second training sample set and a second test sample set; based on the second training sample set and the second test sample set, the data set features that match the first training sample set and the first test sample set are calculated.

[0129] Continue to refer Figure 5 , Figure 6 An optional flow chart of the neural network model fusion monitoring method provided by an embodiment of the present invention includes the following steps:

[0130] Step 501: The task information processing device processes the second test sample set through the extreme gradient boosting model to obtain the AUC value corresponding to the second test sample set.

[0131] Among them, AUC (Area Under Curve) can characterize the area under the ROC curve. In the embodiment of the present application, the AUC value is used as the evaluation criterion of the model because the ROC curve in the test phase cannot clearly explain which neural network model's classifier has a better processing effect, and as a numerical value, the classifier with a larger AUC has a better effect. Since the full name of the ROC curve is the receiver operating characteristic curve, it is a curve drawn based on a series of different binary classification methods (cutoff value or decision threshold) with the true positive rate (sensitivity) as the ordinate and the false positive rate (1-specificity) as the abscissa. Therefore, in the test phase, the AUC value is between 0 and 1, and approaching 1 indicates that there is a huge difference between the first training sample set and the first test sample set. A more ideal state is that the AUC value is around 0.5, indicating that there is no obvious difference in the data of the first training sample set and the first test sample set.

[0132] When processing the sample information in the collection, the extreme gradient boosting model (XGBoost, Xtreme Gradient Boosting) can be used for calculation.

[0133] The extreme gradient boosting model XGBoost involved in the present invention is introduced below.

[0134] The extreme gradient boosting model XGBoost is an improved algorithm of the iterative decision tree algorithm (GBDT GradientBoosting Decision Tree) in the field of machine learning. For a given sample size n and a dataset D with variable dimension m, it can be written as:

[0135] D={(X i ,y i )|X i ∈R m ,y i ∈R,i,j=1,2,...n}

[0136] Fitting the data with an additive ensemble tree model can be expressed as follows:

[0137]

[0138] where f k A function representing the function space Γ represents a tree model, including the specific tree structure and leaf nodes.

[0139] Then the objective function is minimized, that is:

[0140]

[0141] Among them, Ω(f k ) is the regularization term, expressed as M k is the number of leaf nodes, w kj is the leaf node coefficient, γ represents the difficulty of node segmentation, and λ represents the L2 regularization coefficient. is the loss function, which means With y i The deviation between .

[0142] Because we want to obtain a set of all tree models, but we cannot obtain them all at once, we can take a method that is: fix the model obtained in the last (t-1) time first, and then in the next training, that is, the tth training, we will train based on the previously fixed results to obtain the corresponding tth tree.

[0143] And so on, training is carried out in sequence. The prediction result of the tth time is expressed as:

[0144]

[0145] Then the objective function is:

[0146]

[0147] Using the second-order Taylor expansion,

[0148]

[0149] in

[0150] After removing the constant term, we get:

[0151]

[0152] Definition I j ={i|q(X j )=j} is the jth leaf node, and Ω(f t ) is expanded to obtain:

[0153]

[0154] From this we can get:

[0155] The final objective function is:

[0156]

[0157] The objective function Obj t*It can be used to measure the quality of the tree structure. The smaller the value, the better the structure. The optimal split point is selected through this objective function to construct a classification and regression tree (CARTC lassification and regression tree).

[0158] Step 502: The task information processing device calculates single feature vectors corresponding to the second training sample set and the second test sample set respectively in response to the AUC value corresponding to the second test sample set.

[0159] Among them, each sample in the second training sample set and the second test sample set can correspond to a single feature vector. When calculating the single feature vector, the statistics of each sample can be calculated, for example, including at least one of the following: mean, extreme value, range, variance, etc., so as to convert each sample in the second training sample set and the second test sample set into a corresponding feature vector, and calculate using a specific distance. As an embodiment, in this calculation process, the Euclidean distance can be optionally used to characterize the change of a single feature. The larger the distance parameter, the larger the deviation of each sample in the second training sample set and the second test sample set in the single feature dimension.

[0160] Step 503: The task information processing device determines the association relationship between the feature vectors of different dimensions respectively corresponding to the second training sample set and the second test sample set based on the single feature vectors respectively corresponding to the second training sample set and the second test sample set.

[0161] In some embodiments of the present invention, the correlation between the two data sets can be characterized by calculating the covariance matrix of each sample in the second training sample set and the second test sample set and stretching the covariance matrix into a one-dimensional vector, wherein the distribution difference can be calculated using the Wasserstein distance for comparing distribution changes, so that the correlation between the feature vectors of different dimensions corresponding to the second training sample set and the second test sample set can be represented by the Wasserstein distance.

[0162] Step 504: The task information processing device calculates the data set features that match the first training sample set and the first test sample set based on the single feature vectors corresponding to the second training sample set and the second test sample set respectively, and the correlation relationship between the feature vectors of different dimensions corresponding to the second training sample set and the second test sample set respectively.

[0163] In some embodiments of the present invention, in order to more clearly reflect the calculation method of the distance characterization part, the second training sample set and the second test sample set are assigned values ​​to illustrate the specific process of feature extraction in terms of the association relationship between a single feature and a feature vector.

[0164] For example, there are 3 data from the second training sample set, [1, 2, 4, 4], [3, 2, 5, 4], [1, 1, 3, 3], and 4 data from the second test sample set, [10, 4, 9, 8], [1, 8, 7, 6], [4, 4, 4, 4], [2, 4, 6, 8]. The distance between these two sets is characterized. Calculation of a single feature: For two sets of data with a feature dimension of 4, the mean is calculated for each dimension of the feature, and the second training sample set and the second test sample set are converted into two means:

[0165] Eigenvector A:

[0166] Eigenvector B: Then, the distance between feature vectors A and B is calculated. Here, Euclidean distance is used. The calculation method is shown in Formula 1:

[0167]

[0168] Calculation of the eigenvector association relationship: Calculate the corresponding covariance matrix for the data of A and the second test sample set respectively to obtain two 4*4 eigencovariance matrices.

[0169] Feature covariance matrix A:

[0170] Feature covariance matrix B: Take the covariance numbers of the triangles in the upper half of the diagonal of the two 4*4 symmetric covariance matrices and stretch them into two 1*8 vectors, namely:

[0171] A: [1.33333333, 0.33333333, 1, 0.33333333, 0.33333333, 0.5, 0.333333331, 0.5, 0.33333333]

[0172] B: [16.25, -4.33333333, 4.83333333, 2.5, 4, 0.66666667, -0.66666667, 4.33333333, 3, 3.0.66666667]

[0173] Then, the Wasserstein distance is used for calculation. Since the actual calculation method of the Wasserstein distance is relatively complicated, the scipy.stats.wasserstein_distance interface in python is called for calculation, and the distance value is 3.9583333339999998, that is, the Wasserstein distance is used to characterize the association relationship between the feature vectors of different dimensions corresponding to the second training sample set and the second test sample set.

[0174] After determining the association relationship between the feature vectors of different dimensions respectively corresponding to the second training sample set and the second test sample set, continue to execute step 403.

[0175] Step 403: The task information processing device processes the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results.

[0176] Step 404: The task information processing device calculates the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing results.

[0177] In some embodiments of the present invention, the correlation characteristics of each target neural network model in the target neural network model set may include at least three types, namely, model diversity parameters, certainty parameters and prediction boundary clarity parameters. The calculation processes of the model diversity parameters, certainty parameters and prediction boundary clarity parameters are respectively described below.

[0178] In some embodiments of the present invention, when calculating the model diversity parameter, the sample processing results of the base model in the target neural network model set and the sample processing results of any target neural network model in the target neural network model set can be obtained; the Pearson correlation coefficient of each target neural network model in the target neural network model set is calculated according to the sample processing results of the base model and the sample processing results of any target neural network model; the model diversity parameter is determined based on the number of different types of target neural network models in the target neural network model set and the Pearson correlation coefficient. Among them, the difference between the models can be reflected by calculating the number of different structural models and the change in the correlation of the prediction results of each model. The Pearson correlation coefficient is used when calculating the correlation parameter, and the change in correlation is reflected by the standard deviation. For example, for a model fusion problem of a regression problem, the correlation between the prediction results of each single model and the prediction results of the base model can be calculated, and the standard deviation of all the correlation coefficients obtained can be calculated to represent the diversity parameter of the neural network model.

[0179] In some embodiments of the present invention, when determining the deterministic parameters of the target neural network model, classification processing can be performed according to different tasks. When the task information in the use environment of the target neural network model set is a classification task, the information entropy of each target neural network model in the target neural network model set is calculated according to the sample processing results; when the task information in the use environment of the target neural network model set is a regression task, the standard deviation of the corresponding sample processing results of different target neural network models in the target neural network model set is calculated according to the sample processing results; based on the information entropy of each target neural network model, or the standard deviation of the corresponding sample processing results of the different target neural network models, the deterministic parameters of each target neural network model in the target neural network model set are determined. Specifically, for the classification model, the information entropy can be calculated by reporting loss:

[0180]

[0181] Where X is a random variable, P(X i ) is X=X i For a classification model under a three-classification task, the model's prediction result for a certain data is a three-dimensional probability vector, such as [0.1, 0.2, 0.7]. The information entropy calculation method at this time is E = -0.1*ln(0.1)-0.2*ln(0.2)-0.7*ln(0.7). The more uncertain the model is about the current prediction result, the larger the corresponding information entropy value; for the regression model, the standard deviation between the prediction results of multiple models can be calculated to reflect the certainty of the current prediction of multiple models. The smaller the standard deviation, the more similar the prediction results of multiple models in the target neural network model set are, and the more confident the current prediction result is.

[0182] In some embodiments of the present invention, when determining the predicted boundary clarity parameter of the target neural network model, it can be achieved in the following manner:

[0183] According to the sample processing results, the corresponding data points of the sample processing results are divided; based on the corresponding data points of the sample processing results, the divergence parameters corresponding to the sample processing results are determined; through the divergence parameters corresponding to the sample processing results, the prediction boundary clarity parameters of each target neural network model in the target neural network model set are calculated. Among them, when the sample processing result is binary data, the closer the two types of data points are divided by the classification model, the farther the distance is, and the clearer the boundary is. For the tasks to be processed by the target neural network model, the clearer the boundary of the prediction effect of the fusion model is compared with the prediction effect of the single base model, the better the effect. In the part of processing boundary clarity, JS divergence can be used to characterize the distribution of different categories of predictions, and finally predict the boundary clarity parameters. Specifically, we can first divide the data points based on the prediction results of the model, and then calculate the mean, standard deviation, and extreme value of different categories based on the data points in the current category. Based on the mean, standard deviation, maximum, and minimum, we can simulate the distribution under the assumption that the distribution is normal. Finally, we can solve the JS divergence of the simulated distribution pairwise, and take the minimum, maximum, and average values ​​of the final JS divergence as the corresponding features.

[0184] Step 405: The task information processing device performs test processing on the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the data set characteristics that match the first training sample set and the first test sample set, and obtains the model fusion test processing result of the target neural network model set.

[0185] In some embodiments of the present invention, obtaining the model fusion test processing result of the target neural network model set can be achieved by:

[0186] Based on the diversity parameters, deterministic parameters, prediction boundary clarity parameters and data set characteristics of each target neural network model in the target neural network model set, model fusion test processing is performed on different target neural network models in the target neural network model set to obtain the model fusion test processing result of the target neural network model set. When the model fusion test processing result is to close model fusion, the sample processing result of the base model in the target neural network model set is output; or when the model fusion test processing result is to open model fusion, the corresponding target neural network model fusion strategy is triggered to perform fusion processing on different target neural network models in the target neural network model set.

[0187] Step 406: The task information processing device processes the task information in the use environment through the target neural network model set based on the model fusion test processing results of the target neural network model set.

[0188] In some embodiments of the present invention, Figure 1 Taking the implementation environment shown as an example, a semantic understanding software client is set on the terminal (including terminal 10-1 and terminal 10-2). The semantic understanding software client can collect corresponding oral test sentences in the oral test environment, and the artificial intelligence client (such as the voice assistant applet in the instant messaging client) can score the oral test sentences through different target neural network models in the target neural network model set to obtain corresponding results. When it is determined that model fusion is required, the fused speech information processing model can be trained and deployed, and then combined with Figure 2 The electronic device 20 shown illustrates the voice information processing method provided by the embodiment of the present invention, see Figure 6 , Figure 6 An optional flow chart of the voice information processing method provided by the embodiment of the present invention is provided. It can be understood that: Figure 6 The steps shown can be performed by various electronic devices running a voice information processing device, such as a terminal, a server or a server cluster with a voice information processing function. The trained fusion voice information processing model can be deployed in the server to score the uploaded oral test information. Of course, before deploying the fusion voice information processing model, the fusion voice information processing model needs to be trained, which specifically includes the following steps:

[0189] Step 601: Acquire a third training sample set, wherein the third training sample set is audio samples in video information collected by a terminal.

[0190] Step 602: Add noise to the third training sample set to form a corresponding fourth training sample set.

[0191] In some embodiments of the present invention, adding noise to the third training sample set to form a corresponding fourth training sample set can be achieved in the following manner:

[0192] Determine the dynamic noise type that matches the use environment of the speech information processing model; according to the dynamic noise type, add noise to the third training sample set to change the background noise, volume or sampling rate of the audio samples in the third training sample set to form a corresponding fourth training sample set. Wherein, due to the complex use environment of audio information, for example: the audio frequency of spoken information changes, the video speed of spoken information is different, so in the process of constructing the training sample set, an audio enhancement data set can be made according to the audio speech speed of common spoken information. Common audio enhancement forms include: voice change, adding background noise, volume change, sampling rate change, sound quality change, etc., and different enhanced audio can be obtained by setting different parameters. It should be noted that in some embodiments of the present invention, the speech information processing model trained by the complex training sample set can be adapted to different types of oral examinations, for example: it can be applicable to the scoring of speech information in children's oral examinations with slower speech speeds, and it can also be applicable to the scoring of speech information in an interpretation environment with faster speech speeds.

[0193] A training sample set is created based on the audio enhancement data. For example, one original audio corresponds to 20 spoken test audios. Here, each spoken test audio has the same duration as the original audio and has no frame shift (that is, the audios at the corresponding time points are the same). The audio duration is dur, and the step is 0.3s. Then, each group of audios (original audio + corresponding spoken test audio) will generate dur / 0.3 labels, and the labels at the same time points are the same.

[0194] Step 603: Process the fourth training sample set through a speech information processing model to determine initial parameters of the speech information processing model.

[0195] Step 604: In response to the initial parameters of the speech information processing model, the fourth training sample set is processed by the speech information processing model to determine the updated parameters of the speech information processing model.

[0196] In some embodiments of the present invention, in response to the initial parameters of the speech information processing model, the fourth training sample set is processed by the speech information processing model to determine the updated parameters of the speech information processing model, which can be achieved in the following manner:

[0197] Substitute different audio samples in the fourth training sample set into the loss function corresponding to the ternary loss function layer network of the speech information processing model; determine that when the loss function meets the corresponding convergence conditions, the parameters of the ternary loss function layer network in the speech information processing model are used as update parameters of the speech information processing model.

[0198] Step 605: According to the update parameters of the speech information processing model, the network parameters of the speech information processing model are iteratively updated through the fourth training sample set.

[0199] Specifically, the convergence condition that matches the ternary loss function layer network in the speech information processing model can be determined; the network parameters of the ternary loss function layer network are iteratively updated until the loss function corresponding to the ternary loss function layer network meets the corresponding convergence condition. In the training phase, the 128-dimensional vector obtained by the VGGish network is input into the ternary loss function network (triplet-loss layer) of the speech information processing model for training, and similar audio is finally achieved to obtain similar embedding output results. The formula of triplet loss refers to formula 3:

[0200] L=max(d(a,p)-d(a,n)+margin,0)Formula 3

[0201] Where L represents the ternary loss function, a is a sample, p represents similar samples, n represents samples of different categories, d(a, p) is the distance between a and p in the vector space, and d(a, n) is similar. By minimizing the above loss function, the distinction between similar samples and different samples can be learned.

[0202] In order to better illustrate the neural network model fusion monitoring method provided by the present application, the scoring of oral samples in an oral test environment is continued as an example to illustrate the neural network model fusion monitoring method provided by the present application. Figure 7 , Figure 7 An optional flow chart of the voice information processing method provided by the embodiment of the present invention is provided. It can be understood that: Figure 7 The steps shown can be performed by various electronic devices running a voice information processing device, and specifically include the following steps:

[0203] Step 701: The speech information processing device obtains a spoken test sample.

[0204] Step 702: triggering the acoustic processing model to process the spoken test sample into a first training sample set and a first test sample set.

[0205] Step 703: Perform adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set.

[0206] Step 704: Obtain sample processing results of each target neural network model in the speech processing neural network model set, and calculate the correlation characteristics of each target neural network model in the target neural network model set.

[0207] Step 705: Determine whether model fusion processing is required, and obtain a fusion speech processing neural network model.

[0208] Step 706: Determine that model fusion processing is not required, and output the sample processing results of the base model in the speech processing neural network model set.

[0209] Beneficial technical effects:

[0210] The present invention obtains a first training sample set and a first test sample set matching the target neural network model set according to task information in the use environment of the target neural network model set, wherein the target neural network model set includes at least two different target neural network models; performs adversarial verification processing on the first training sample set and the first test sample set to determine data set features matching the first training sample set and the first test sample set; processes the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results; calculates the correlation features of each target neural network model in the target neural network model set according to the sample processing results; performs test processing on the model fusion of the target neural network model set based on the correlation features of each target neural network model in the target neural network model set and the data set features matching the first training sample set and the first test sample set to obtain the model fusion test processing results of the target neural network model set; based on the model fusion test processing results of the target neural network model set, the task information in the use environment is processed through the target neural network model set, thereby automatically performing model fusion testing on the target neural network model set, and automatically fitting decisions are made to replace the method of empirical judgment. This approach does not need to consider artificial fluctuations and additional data costs, thereby improving the accuracy and robustness of the model and enhancing the user experience.

[0211] The above descriptions are merely embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network model fusion monitoring method, characterized in that: The method comprises: According to task information in a use environment of a target neural network model set, obtaining a first training sample set and a first test sample set that match the target neural network model set, wherein the target neural network model set includes at least two different target neural network models; Performing adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set; Processing the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results; Calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing results; Based on the correlation characteristics of each target neural network model in the target neural network model set and the characteristics of the data set that matches the first training sample set and the first test sample set, the model fusion of the target neural network model set is tested and processed to obtain the model fusion test processing result of the target neural network model set; Based on the model fusion test processing results of the target neural network model set, the task information in the usage environment is processed through the target neural network model set.

2. The method according to claim 1, characterized in that The performing adversarial verification processing on the first training sample set and the first test sample set to determine the data set features matching the first training sample set and the first test sample set includes: Rewrite the sample label of the first training sample set to 0 to obtain a first training sample set with a sample label of 0; Rewrite the sample label of the first test sample set to 1 to obtain a first test sample set with a sample label of 1; Performing data fusion processing on the first training sample set whose sample label is 0 and the first test sample set whose sample label is 1, and performing partition processing on the data fusion processing result to obtain a second training sample set and a second test sample set; According to the second training sample set and the second test sample set, data set features matching the first training sample set and the first test sample set are calculated.

3. The method according to claim 2, characterized in that The step of calculating, based on the second training sample set and the second test sample set, the data set features that match the first training sample set and the first test sample set comprises: Processing the second test sample set by using an extreme gradient boosting model to obtain an AUC value corresponding to the second test sample set; In response to the AUC value corresponding to the second test sample set, calculating the single feature vectors corresponding to the second training sample set and the second test sample set respectively; Based on the single feature vectors respectively corresponding to the second training sample set and the second test sample set, determining the association relationship between the feature vectors of different dimensions respectively corresponding to the second training sample set and the second test sample set; Based on the single feature vectors corresponding to the second training sample set and the second test sample set respectively, and the correlation relationship between the feature vectors of different dimensions corresponding to the second training sample set and the second test sample set respectively, the data set features matching the first training sample set and the first test sample set are calculated.

4. The method according to claim 1, characterized in that: The step of calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result includes: Obtaining sample processing results of a base model in the target neural network model set and sample processing results of any target neural network model in the target neural network model set; Calculate the Pearson correlation coefficient of each target neural network model in the target neural network model set according to the sample processing result of the base model and the sample processing result of any target neural network model; A model diversity parameter is determined based on the number of different types of target neural network models in the target neural network model set and the Pearson correlation coefficient.

5. The method according to claim 1, characterized in that: The step of calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result includes: When the task information in the use environment of the target neural network model set is a classification task, Calculating the information entropy of each target neural network model in the target neural network model set according to the sample processing result; When the task information in the use environment of the target neural network model set is a regression task, Calculate the standard deviation of the sample processing results corresponding to different target neural network models in the target neural network model set according to the sample processing results; Based on the information entropy of each target neural network model, or the standard deviation of the corresponding sample processing results of the different target neural network models, the deterministic parameters of each target neural network model in the target neural network model set are determined.

6. The method according to claim 1, characterized in that The step of calculating the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result includes: According to the sample processing result, the data points corresponding to the sample processing result are divided; Determining a divergence parameter corresponding to the sample processing result based on a corresponding data point of the sample processing result; The predicted boundary clarity parameter of each target neural network model in the target neural network model set is calculated using the divergence parameter corresponding to the sample processing result.

7. The method according to claim 1, characterized in that The method of testing the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the characteristics of the data set that matches the first training sample set and the first test sample set to obtain the model fusion test processing result of the target neural network model set includes: Based on the diversity parameter, certainty parameter, prediction boundary clarity parameter and data set characteristics of each target neural network model in the target neural network model set, Model fusion test processing is performed on different target neural network models in the target neural network model set to obtain a model fusion test processing result of the target neural network model set.

8. The method according to claim 7, characterized in that The model fusion test processing result based on the target neural network model set, processing the task information in the use environment through the target neural network model set, includes: When the model fusion test processing result is to close the model fusion, output the sample processing result of the base model in the target neural network model set; or When the model fusion test processing result is to turn on model fusion, the corresponding target neural network model fusion strategy is triggered to perform fusion processing on different target neural network models in the target neural network model set.

9. The method according to claim 8, characterized in that The method further comprises: When the target neural network model set is a fusion speech information processing model set, Acquire a third training sample set, wherein the third training sample set is audio samples in video information collected by the terminal; adding noise to the third training sample set to form a corresponding fourth training sample set; Processing the fourth training sample set by the fusion speech information processing model to determine initial parameters of the fusion speech information processing model; In response to the initial parameters of the fusion speech information processing model, the fourth training sample set is processed by the fusion speech information processing model to determine the update parameters of the fusion speech information processing model; According to the update parameters of the fusion speech information processing model, the network parameters of the fusion speech information processing model are iteratively updated through the fourth training sample set.

10. The method according to claim 9, characterized in that The adding noise to the third training sample set to form a corresponding fourth training sample set includes: Determining a dynamic noise type that matches the use environment of the fusion speech information processing model; According to the dynamic noise type, noise is added to the third training sample set to change the background noise, volume or sampling rate of the audio samples in the third training sample set to form a corresponding fourth training sample set.

11. The method according to claim 9, characterized in that The step of processing the fourth training sample set by the fusion speech information processing model in response to the initial parameters of the fusion speech information processing model to determine the update parameters of the fusion speech information processing model includes: Substituting different audio samples in the fourth training sample set into the loss function corresponding to the ternary loss function layer network of the fusion speech information processing model; Determine the parameters of the ternary loss function layer network in the fusion speech information processing model when the loss function satisfies the corresponding convergence condition; The parameters of the ternary loss function layer network are used as update parameters of the fusion speech information processing model.

12. The method according to claim 9, characterized in that The iterative updating of the network parameters of the fusion speech information processing model through the fourth training sample set according to the update parameters of the fusion speech information processing model comprises: Determining a convergence condition that matches a ternary loss function layer network in the fusion speech information processing model; The network parameters of the ternary loss function layer network are iteratively updated until the loss function corresponding to the ternary loss function layer network meets the corresponding convergence condition.

13. A neural network model fusion monitoring device, characterized in that: The device comprises: An information transmission module, configured to obtain a first training sample set and a first test sample set matching the target neural network model set according to task information in a use environment of the target neural network model set, wherein the target neural network model set includes at least two different target neural network models; An information processing module, configured to perform adversarial verification processing on the first training sample set and the first test sample set to determine data set features that match the first training sample set and the first test sample set; The information processing module is used to process the first training sample set and the first test sample set respectively through different target neural networks in the target neural network model set to determine corresponding sample processing results; The information processing module is used to calculate the correlation characteristics of each target neural network model in the target neural network model set according to the sample processing result; The information processing module is used to test the model fusion of the target neural network model set based on the correlation characteristics of each target neural network model in the target neural network model set and the data set characteristics that match the first training sample set and the first test sample set, so as to obtain the model fusion test processing result of the target neural network model set; The information processing module is used to process the task information in the use environment through the target neural network model set based on the model fusion test processing results of the target neural network model set.

14. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor is used to implement the neural network model fusion monitoring method described in any one of claims 1 to 9 when running the executable instructions stored in the memory.

15. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the neural network model fusion monitoring method described in any one of claims 1 to 8 is implemented.

Images