Fault identification method based on dynamic multi-modal fusion network

Abstract

The invention relates to a fault identification method based on a dynamic multi-mode fusion network, and belongs to the field of fault identification. The method comprises the following steps: constructing a multi-modal fusion network based on different granularity level fault identification; and training and optimizing a multi-modal fusion network based on different granularity level fault identification to realize different granularity level fault identification. According to the method, a forward path depending on data is generated in the reasoning training process, the problem of resource waste caused by the fact that a traditional static multi-modal fusion method is difficult to balance different calculation requirements of different modals is solved, the model calculation efficiency is improved, and the calculation cost is saved; for the fault identification problem of different granularity levels, two DynMM algorithms based on a gating network are provided, so that simple fault identification tasks such as dichotomy and the like can be solved, the multi-modal fusion fault identification problem which is similar to semantic segmentation and has higher difficulty can be completed, and the universality and the identification precision of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of fault identification, and in particular relates to a fault identification method based on a dynamic multi-modal fusion network. Background Art

[0002] In recent years, with the optimization of power systems and the explosive growth of a large number of new energy systems, power equipment, as large-scale industrial equipment, is expensive and its health condition gradually deteriorates over time. In addition, the difficulty and high cost of collecting fault samples have brought challenges to the research in the field of power equipment fault identification. At the same time, the complexity and correlation of power equipment faults are high, and even local faults can cause global abnormalities. Therefore, how to accurately identify and locate faults using the multimodal data generated by power equipment when it is online has become a difficult problem that needs to be solved in the current field of power equipment fault identification. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that single modal data cannot achieve accurate power equipment fault identification, and to provide a fault identification method based on a dynamic multimodal fusion network to improve the model calculation efficiency, save calculation costs, and improve the versatility and identification accuracy of the model.

[0004] To achieve the above object, the technical solution of the present invention is: a fault identification method based on a dynamic multi-modal fusion network, comprising:

[0005] Construct a multi-modal fusion network based on fault identification at different granularity levels;

[0006] The training and optimization are based on the multimodal fusion network of fault identification at different granularity levels to realize fault identification at different granularity levels.

[0007] In one embodiment of the present invention, a multi-modal fusion network based on fault identification at different granularity levels includes a modal-level decision DynMM unit and a fusion-level decision DynMM unit.

[0008] In one embodiment of the present invention, the modal level decision DynMM unit is specifically implemented as follows:

[0009] Assume that the input data has M modes, and the input is represented by x=(x1,x2,...,x M );

[0010] Following the expert mixture MoE framework, a set of expert networks are designed, each of which studies a subset of the M modalities;

[0011] A gating network G(x) is proposed to determine the expert network to be activated. The gating network G(x) uses multimodal input x to form a global view, and then generates a B-dimensional sparse vector g as output. The final output y is in the form of:

[0012]

[0013] Among them, B is the number of selected expert networks, g i is a sparse vector, x i represents the modal subset that the i-th expert takes as input, E i is the i-th expert network; in the above formula, the sparse vector g outputs a one-hot encoding, that is, only one branch needs to be selected for each fault instance. Therefore, each time the network is passed forward, only one branch is selected for calculation.

[0014] In one embodiment of the present invention, the input of the gating network G(x) includes input-level features and intermediate features of each modality.

[0015] In one embodiment of the present invention, the fusion-level decision DynMM unit is specifically implemented as follows:

[0016] Constructing the fusion unit: Assume that the input data has M modes, that is, x = (x1, x2, ..., x M ), a set of fusion operations is represented as {O i},O i The fusion operation implemented as any function to fuse multimodal features; the output h of the fusion unit is expressed as:

[0017]

[0018] Among them, B is the total number of operations, O i is the i-th fusion operation; the separate gating network G(x) in each fusion unit accepts multimodal input and generates a B-dimensional sparse vector g that determines the execution of the corresponding fusion operation. The sparse vector g outputs a one-hot encoding;

[0019] By stacking fusion units and interleaving static feature extraction blocks, a fusion-level decision DynMM unit is constructed. At the same time, the separate gating network G(x) in each fusion unit is replaced by a global gating network

[0020] In one embodiment of the present invention, the fusion unit has two input modes: x=(x1, x2)) and three operations: O1=x1, O2=x1+x2, O3=w1x1+w2x2, where w1 and w2 are learnable parameters.

[0021] In one embodiment of the present invention, a resource-aware loss function is introduced in the multimodal fusion network training for fault identification at different granularity levels:

[0022]

[0023] Among them, C(E i ) represents the execution expert network E i The computational cost, L task represents the task loss and λ is a hyperparameter that controls the relative importance of the two loss terms.

[0024] In one embodiment of the present invention, a resource-aware loss function is introduced in the multimodal fusion network training for fault identification at different granularity levels:

[0025]

[0026] Among them, C(O i ,j) represents the computational cost of the i-th fusion operation in the j-th unit, L task Indicates mission loss, represents the decision vector given by the jth fusion unit, F is the number of fusion units, and λ is a hyperparameter that controls the relative importance of the two loss terms.

[0027] In one embodiment of the present invention, training and optimization of a multimodal fusion network based on fault identification at different granularity levels is specifically implemented as follows:

[0028] Step 1: Pre-training: For the modality-level decision DynMM unit, fully train each expert network of the modality-level decision DynMM unit; for the fusion-level decision DynMM unit, adopt random decision for each fusion unit of the fusion-level decision DynMM unit;

[0029] Step 2: Fine-tuning: During the training process, two training techniques are used: First, soft Instead of hard g, to achieve back propagation; second, follow the straight-through technique, use hard g in forward propagation and soft g in backward propagation The gradient is approximately The updating of network parameters is accomplished through two training techniques.

[0030] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0031] Compared with the prior art, the present invention has the following beneficial effects: the present invention generates a forward path that depends on data during the inference training process, solves the problem of resource waste caused by the difficulty of traditional static multimodal fusion methods in balancing the different computing requirements of different modalities, improves the model computing efficiency, and saves computing costs; for fault identification problems at different granularity levels, two gated network-based DynMM algorithms are proposed, which can not only solve simple fault identification tasks such as binary classification, but also can be used to complete more difficult multimodal fusion fault identification problems similar to semantic segmentation, and improve the versatility and identification accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of a modal-level DynMM unit containing two modal input data according to the present invention.

[0033] Figure 2 The present invention is a fusion-level DynMM unit design containing two modal input data.

[0034] Figure 3 A dynamic multimodal architecture with stacked fusion units. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0036] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] The present invention provides a fault identification method based on a dynamic multi-modal fusion network, comprising:

[0039] Construct a multi-modal fusion network based on fault identification at different granularity levels;

[0040] The training and optimization are based on the multimodal fusion network of fault identification at different granularity levels to realize fault identification at different granularity levels.

[0041] The following is the specific implementation process of the present invention.

[0042] like Figure 1 As shown, this embodiment provides a fault identification method based on a dynamic multimodal fusion network, including: first, introducing a decision-making scheme based on a gating network, taking both modal level (coarse level) and fusion level (fine level) decisions into account, so that DynMM can generate a data-dependent forward path during the inference training process; second, proposing a training loss function that considers resource budget, thereby reducing the decision bias during DynMM training; finally, implementing non-differentiable gating network optimization, so that DynMM can back propagate, thereby completing end-to-end training. The specific steps are as follows:

[0043] Step 1: Modal-level decision DynMM.

[0044] First, suppose the input data has M modes, then the input is represented as x=(x1,x2,...,x M ). Following the classic Mixture-of-Experts (MoE) framework, a set of expert networks is designed, each of which specializes in a subset of the M modalities. For example, if M = 3, there can be at most 7 expert networks, represented as E1(x1), E2(x2), E3(x3), E4(x1,x2), E5(x2,x3), E6(x1,x2), E7(x1,x2,x3). In practical applications, the scope of candidate expert networks can be narrowed down by domain expertise.

[0045] Here we propose a gating network G(x) to decide which expert network should be activated. The gating network uses multimodal input x to form a global view and then generates a B-dimensional sparse vector g as output. The final output y is in the form of:

[0046]

[0047] Where B is the number of selected expert networks. i is a sparse vector. i represents the modal subset that the i-th expert takes as input, E i For expert networks.

[0048] Unlike traditional MoE, where the output of the traditional MoE is the weighted sum of the expert network and each branch needs to be executed, in the above formula, the g output of the gated network is one-hot encoding, that is, only one branch needs to be selected for each fault instance. Therefore, each time the network is passed forward, only one branch is selected for calculation, thereby saving the amount of calculation to the greatest extent. Figure 1 is a modality-level DynMM with two modal input data. The dotted line indicates that the expert network is not selected for activation, and the solid line indicates that it is activated.

[0049] The gating network G(x) is not limited to obtaining input-level features. It can also use the intermediate features of each modality as network input, so that the modality-level DynMM can be inserted into any part of the multimodal network, saving the amount of computation after the gating network.

[0050] Step 2: Fusion-level decision DynMM.

[0051] First, construct the fusion unit. Assume that the input data has M modes, that is, x = (x1, x2, ..., x M ). A set of fusion operations is represented as {O i}. i Can be implemented as any function to fuse multimodal features, such as a simple identity mapping (i.e. O i = x1), addition (i.e. O i =x1+x2+...+x M ), series connection (ie O i =[x1, x2, ..., x M ]) and self-attention. Figure 2 An example design of a fusion unit is shown with two input patterns (i.e., x = (x1, x2)) and three operations (i.e., O1 = x1, O2 = x1+x2, O3 = w1x1+w2x2), where w1 and w2 are learnable parameters.

[0052] In practice, more complex fusion operations can be used in each unit to expand the representation capability. The output of unit h can be expressed as:

[0053]

[0054] Where B is the total number of operations. i is the i-th fusion operation. The gated network G(x) accepts multimodal input and generates a B-dimensional vector g that determines which operation to perform. According to Step 1, g is used as a one-hot code to improve computational efficiency.

[0055] The fusion-level DynMM unit builds a dynamic network by stacking fusion units, which supports making operational decisions in a finer granularity and in a more flexible manner. Figure 3 (a) provides an example architecture for two modal inputs. The network consists of four fusion blocks and a global gating network, which allows it to flexibly control the degree of fusion in a sample-by-sample manner. The blocks marked in yellow and green are static feature extraction blocks, which are interleaved with fusion units.

[0056] Figure 3(b) shows the final architecture when the gating network assigns fusion units 1 and 2 to select O2 operations, and assigns fusion units 3 and 4 to select O1 operations, which not only skips the complex fusion operations that are not selected within the fusion unit, but also saves unnecessary computations in the feature extraction layer. Since we only take the features of modality 1 after fusion unit 2, there is no need to process the features of modality 2. Therefore, we can skip the computation in the feature extraction layer of x2 (blocks 3-4 marked in gray). In essence, when combining low-level features from each modality (i.e., fusion in the early stages) is sufficient to achieve good predictions, the fusion-level DynMM can save future fusion and modality operations for certain multimodal inputs. On the other hand, for fault instances that are difficult to analyze, DynMM provides the option of combining multimodal features in each unit with complex fusion operations to obtain maximum expressive power. Here, the four separate gating networks G(x) in each fusion unit are replaced by a global gating network To achieve better integration; Take the multimodal features (x1, x2) as input and decide which fusion operation to use for the four fusion units.

[0057] Step 3: Resource-aware loss function.

[0058] For modality-level and fusion-level DynMM designs, each expert network E i (or operation O i ) are calculated differently. Typically, computationally expensive expert networks (or operations) have stronger expressive power. If the network is trained directly by minimizing the task-specific loss, the gating network will always choose the computationally expensive branch as the solution, which is often a simple solution. To achieve effective reasoning, a resource-aware loss function is introduced in training, as follows:

[0059] (Modal Level)

[0060] (Fusion Level)

[0061] In which, let C(E i ) represents the execution expert network E i The computational cost of C(O i ,j) represents the computational cost of the i-th fusion operation in the j-th unit. task represents the task loss, such as the cross entropy between the network prediction and the true label of the classification. g(j) represents the decision vector given by the jth fusion unit. B is the total number of experts (operations) and F is the number of fusion units. λ is a hyperparameter that controls the relative importance of the two loss terms. The computational cost can be set before training and is a constant term.

[0062] Both modality-level and fusion-level resource-aware loss functions take into account the computational cost of executing each path, enabling DynMM to achieve the desired trade-off between accuracy and efficiency. In addition, the value of λ can be adjusted according to the constraints of computational deployment. For larger λ, DynMM will prioritize lightweight computations for high computational efficiency. For smaller λ, DynMM will explore these computationally expensive paths more frequently, resulting in higher accuracy.

[0063] Step 4: Non-differentiable gating network optimization.

[0064] DynMM needs to be trained in an end-to-end manner. The current gated network provides discrete decisions, so branch selection cannot be directly differentiated relative to the gated network. To this end, Gumbel-softmax and a network parameter update method are introduced during the training process. From Step 1, g represents the required one-hot encoded B-dimensional decision vector generated by the gated network G(x), that is:

[0065] g = one-hot(argmax i G(x) i )

[0066] Therefore, we choose to use a real-valued soft vector To realize network back propagation, its expression is as follows:

[0067]

[0068] where b1, b2, ..., b B are samples independently drawn from Gumbel(0,1), and τ represents the temperature coefficient of the softmax function. When τ is large, The distribution of is more uniform, and when τ is smaller, The distribution of is similar to the categorical distribution. is a continuous, differentiable fit to g.

[0069] During the training process, two training techniques are used: First, soft Instead of hard g, to achieve back propagation. During the training process, we strengthen τ so that Gradually converge to the desired one-hot vector. Second, follow the straight-through technique, using hard g in the forward propagation and soft g in the backward propagation The gradient is approximately In this way, the gate network still outputs discrete decisions during training. In order to make the calculation more advantageous, hard g will always be used during inference. Next, a two-stage training of DynMM is proposed to jointly optimize the multimodal network and the gated network module to complete the update of network parameters. The steps are as follows:

[0070] Step 1: Pre-training. In the early stages of training, following the sparse decisions of the gating network may lead to biased optimization. The branches that are less selected have fewer and smaller weight updates, which leads to poor performance, resulting in fewer selections, and thus it will be difficult to optimize further. The goal of the pre-training stage is to ensure that each branch of the DynMM is fully optimized before the gating module intervenes. For the modality-level DynMM, we fully train each expert network at this stage. For the fusion-level DynMM, random decisions are made for each fusion unit (i.e., an operation is randomly selected from the candidate operation set) so that each path of the dynamic network is uniformly optimized.

[0071] Step 2: Fine-tuning: Update the network parameters using the two training techniques mentioned above, and jointly optimize the dynamic network and the gating network in an end-to-end manner.

[0072] Step 5: Fault identification is realized based on the trained DynMM network.

[0073] After the training of the fault identification model based on DynMM is completed, the obtained multi-modal related data is analyzed through the trained network model to obtain the fault identification results.

[0074] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0075] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0076] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A fault identification method based on a dynamic multi-modal fusion network, characterized in that: include: Construct a multi-modal fusion network based on fault identification at different granularity levels; The training and optimization are based on the multimodal fusion network of fault identification at different granularity levels to realize fault identification at different granularity levels.

2. The fault identification method based on dynamic multi-modal fusion network according to claim 1 is characterized in that: A multi-modal fusion network based on fault identification at different granularity levels includes a modal-level decision-making DynMM unit and a fusion-level decision-making DynMM unit.

3. The fault identification method based on dynamic multi-modal fusion network according to claim 2 is characterized in that: The modal level decision DynMM unit is specifically implemented as follows: Assume that the input data has M modes, and the input is represented by x=(x1,x2,...,x M ); Following the expert mixture MoE framework, a set of expert networks are designed, each of which studies a subset of the M modalities; A gating network G(x) is proposed to determine the expert network to be activated. The gating network G(x) uses multimodal input x to form a global view, and then generates a B-dimensional sparse vector g as output. The final output y is in the form of: Among them, B is the number of selected expert networks, g i is a sparse vector, x i represents the modal subset that the i-th expert takes as input, E i is the i-th expert network; in the above formula, the sparse vector g outputs a one-hot encoding, that is, only one branch needs to be selected for each fault instance. Therefore, each time the network is passed forward, only one branch is selected for calculation.

4. The fault identification method based on dynamic multi-modal fusion network according to claim 3 is characterized in that: The input of the gating network G(x) includes input-level features and intermediate features of each modality.

5. The fault identification method based on dynamic multi-modal fusion network according to claim 2 is characterized in that: The fusion-level decision DynMM unit is specifically implemented as follows: Constructing the fusion unit: Assume that the input data has M modes, that is, x = (x1, x2, ..., x M ), a set of fusion operations is represented as {O i },O i The fusion operation implemented as any function to fuse multimodal features; the output h of the fusion unit is expressed as: Among them, B is the total number of operations, O i is the i-th fusion operation; the separate gating network G(x) in each fusion unit accepts multimodal input and generates a B-dimensional sparse vector g that determines the execution of the corresponding fusion operation. The sparse vector g outputs a one-hot encoding; By stacking fusion units and interleaving static feature extraction blocks, a fusion-level decision DynMM unit is constructed. At the same time, the separate gating network G(x) in each fusion unit is replaced by a global gating network 6. The fault identification method based on dynamic multi-modal fusion network according to claim 5 is characterized in that: The fusion unit has two input modes: x = (x1, x2)) and three operations: O1 = x1, O2 = x1 + x2, O3 = w1x1 + w2x2, where w1 and w2 are learnable parameters.

7. The fault identification method based on dynamic multi-modal fusion network according to claim 3 is characterized in that: The resource-aware loss function is introduced in the multimodal fusion network training for fault identification at different granularity levels: Among them, C(E i ) represents the execution expert network E i The computational cost, L task represents the task loss and λ is a hyperparameter that controls the relative importance of the two loss terms.

8. The fault identification method based on dynamic multi-modal fusion network according to claim 5 is characterized in that: The resource-aware loss function is introduced in the multimodal fusion network training for fault identification at different granularity levels: Among them, C(O i ,j) represents the computational cost of the i-th fusion operation in the j-th unit, L task Indicates mission loss, represents the decision vector given by the jth fusion unit, F is the number of fusion units, and λ is a hyperparameter that controls the relative importance of the two loss terms.

9. The fault identification method based on dynamic multi-modal fusion network according to claim 2 is characterized in that: Training and optimization of multi-modal fusion network based on fault identification at different granularity levels is implemented as follows: Step 1: Pre-training: For the modality-level decision DynMM unit, fully train each expert network of the modality-level decision DynMM unit; for the fusion-level decision DynMM unit, adopt random decision for each fusion unit of the fusion-level decision DynMM unit; Step 2: Fine-tuning: During the training process, two training techniques are used: First, soft Instead of hard g, to achieve back propagation; second, follow the straight-through technique, use hard g in forward propagation and soft g in backward propagation The gradient is approximately The updating of network parameters is accomplished through two training techniques.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images