Fault detection model training method, facility fault detection method and related apparatus, electronic device, storage medium, and program

Abstract

To provide a fault detection model training method, a facility fault detection method, and a related apparatus.SOLUTION: A method specifically includes: sampling a designated facility based on a preset drone fleet formation to obtain a sample sequence; position encoding the sample sequence based on the drone fleet formation to obtain a drone fleet formation encoding result; inputting the sample sequence and the drone fleet formation encoding result into a trained model to obtain a fault detection result output by the trained model; determining a loss value based on the fault detection result and a true value of the fault detection result of the sample sequence; and adjusting model parameters of the trained model based on the loss value to obtain a fault detection model. The trained model includes an encoder and a decoder.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to technical fields such as artificial intelligence, computer vision, and image processing.

Background Art

[0002] In the industrial scene of the spinning process, since there are too many facilities in the spinning workshop, it is necessary to monitor the facilities in the spinning workshop in real time to ensure the quality of chemical fiber products.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The manual monitoring method wastes too much human resources, and there are also areas where manual detection cannot be performed. Therefore, how to automate the monitoring of the spinning workshop is one of the problems faced by the prior art.

Means for Solving the Problems

[0004] The present disclosure provides a method for training a fault detection model, a facility fault detection method and related devices, an electronic device, a storage medium, and a program, which solve or alleviate one or more technical problems in the related art.

[0005] In a first aspect, the present disclosure provides a method for training a fault detection model, the method comprising: sampling a specified facility based on a preset drone fleet formation to obtain a sample sequence; performing position encoding on the sample sequence based on the drone fleet formation to obtain a drone fleet formation encoding result; inputting the sample sequence and the drone fleet formation encoding result into a model to be trained, and obtaining a fault detection result output by the model to be trained; Determining a loss value based on the failure detection result and the true value of the failure detection result of the sample sequence, Adjusting the model parameters of the model to be trained based on the loss value to obtain a failure detection model, The model to be trained includes an encoder and a decoder, The encoder is used to perform feature extraction on each sample image in the sample sequence respectively to obtain the multi-scale features of each sample image, and based on the UAV fleet formation encoding result, fuse the features of the same scale of the sample sequence to obtain the corresponding fused features of each scale, and fuse the fused features of each scale to obtain the target features, The decoder is used to determine a failure detection result including a failure prediction type and a failure prediction box at the same failure position in the failure sample diagram of the specified equipment based on the target features, and obtain a failure sample diagram by splicing the sample images in the sample sequence with reference to the UAV fleet formation.

[0006] In a second aspect, the present disclosure provides a method for detecting equipment failures, which is applied to the failure detection model according to the first aspect, and the method includes: Obtaining an initial image set of a target equipment based on a UAV fleet, where the UAV fleet is for collecting images of the target equipment from multiple viewpoints to obtain the initial image set, Performing noise removal processing on each initial image of the initial image set respectively to obtain a detection target image set, Performing position encoding on the formation of the UAV fleet to obtain a UAV position encoding result, Inputting the detection target image set and the UAV position encoding result into the failure detection model to obtain a failure detection result of the failure detection model for the target equipment. The fault detection result includes the fault prediction type and the fault prediction box at the same fault position in the target diagram, and the target diagram is obtained by connecting the detection target images in the detection target image set with reference to the formation of the drone fleet.

[0007] In a third aspect, the present disclosure provides an apparatus for training a fault detection model, the apparatus including: a sampling module configured to sample facilities specified based on a preset drone fleet formation to obtain a sample sequence; a first encoding module configured to perform position encoding on the sample sequence based on the drone fleet formation to obtain a drone fleet formation encoding result; an input module configured to input the sample sequence and the drone fleet formation encoding result into a model to be trained and obtain a fault detection result output by the model to be trained; a determination module configured to determine a loss value based on the fault detection result and the ground truth of the fault detection result of the sample sequence; an adjustment module configured to adjust model parameters of the model to be trained based on the loss value to obtain a fault detection model. The model to be trained includes an encoder and a decoder. The encoder is configured to perform feature extraction on each sample image in the sample sequence to obtain multi-scale features of each sample image, perform feature fusion on the features of the same scale of the sample sequence based on the drone fleet formation encoding result to obtain corresponding fusion features of each scale, and perform feature fusion on the fusion features of each scale to obtain target features. The decoder is configured to determine a fault detection result including a fault prediction type and a fault prediction box at the same fault position in a fault sample diagram of a specified facility based on the target features, and obtain a fault sample diagram by connecting sample images in the sample sequence with reference to the drone fleet formation.

[0008] In a fourth aspect, the present disclosure provides a facility failure detection device, which is applied to the failure detection model according to the third aspect. The facility failure detection device includes: a collection module for obtaining an initial image set of a target facility based on a drone fleet, where the drone fleet is for collecting images of the target facility from multiple viewpoints to obtain the initial image set; a noise removal module for performing noise removal processing on each initial image of the initial image set to obtain a detection target image set; a second encoding module for performing position encoding on the formation of the drone fleet to obtain a drone position encoding result; and a processing module for inputting the detection target image set and the drone position encoding result into the failure detection model to obtain a failure detection result of the target facility by the failure detection model. The failure detection result includes a failure prediction type and a failure prediction box at the same failure position in the target diagram. The detection target images in the detection target image set are joined together with reference to the formation of the drone fleet to obtain the target diagram.

[0009] In a fifth aspect, the present disclosure provides an electronic device, which includes: at least one processor; a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor. When the instructions are executed by the at least one processor, they cause any one of the methods in the embodiments of the present disclosure to be executed.

[0010] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute any one of the methods in the embodiments of the present disclosure is provided.

[0011] In the seventh aspect, a program is provided, which, when executed by a processor, implements any one of the methods in the embodiments of the present disclosure.

[0012] In the eighth aspect, a program is provided, which, when executed by a processor, implements any one of the methods in the embodiments of the present disclosure.

[0013] In the embodiments of the present disclosure, omnidirectional imaging is performed on the same facility using a drone fleet formation, and based on the position encoding of the drones, the training target model is made to learn the correlation between different images, thereby improving the fault detection efficiency of the model.

[0014] It should be understood that the content described herein is not intended to describe the key points or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. For other features of the present disclosure, understanding is promoted through the following specification.

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the plurality of accompanying drawings indicate the same or similar components or elements. These accompanying drawings are not necessarily drawn to scale. It should be understood that these drawings only show some embodiments provided by the present disclosure and should not be regarded as limiting the scope of the present disclosure.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0017] Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar elements. Also, in the accompanying drawings, various aspects of the embodiments are shown, but these accompanying drawings are not necessarily drawn to scale unless otherwise stated.

[0018] Furthermore, in order to better explain the present disclosure, many specific details are described in the following specific examples. Those skilled in the art should understand that the present disclosure can be implemented similarly even without some details. In some examples, methods, means, components, circuits, etc. well known to those skilled in the art are not described in detail so that the gist of the present disclosure becomes clear.

[0019] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used only for the purpose of explanation and should not be construed as indicating or implying relative importance, nor should they be construed as implying the number of technical features shown. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, "a plurality" means two or more unless otherwise stated.

[0020] In the industrial scenario of the spinning process, since there are too many facilities in the spinning workshop, it is necessary to monitor the facilities in the spinning workshop to ensure the normal production of chemical fiber products.

[0021] On the other hand, the spinning mill has a large area, a long process, and each process adopts a plurality of facilities. Some facilities are relatively large and it is difficult to achieve manual detection. Therefore, when the detection area is large, the types of facilities are many, and the detection environment is complex, an automatic detection technology for improving the efficiency of fault detection is required.

[0022] In addition, the main types of spinning products according to the embodiments of the present disclosure may include one or more of partially oriented yarns (POY), fully drawn yarns (FDY), draw textured yarns (DTY) (or low-elasticity yarns), etc. For example, as the type of yarn, specifically, polyester partially oriented yarns, polyester fully drawn yarns, polyester drawn yarns, polyester draw textured yarns, etc. can be mentioned.

[0023] In order to automatically and accurately detect the related facilities of the spinning mill, the embodiments of the present disclosure provide a fault detection model that expects to control a fleet of drones to collect images and complete fault detection by this fault detection model.

[0024] First, the embodiments of the present disclosure propose a training method for the fault detection model, and expect to train a model whose detection effect meets the requirements through this method. As shown in FIG. 1, the following content is included.

[0025] In S101, sample the equipment specified based on the preset unmanned aerial vehicle (UAV) fleet formation to obtain a sample sequence.

[0026] The sample sequence is numbered according to the UAV fleet formation.

[0027] Here, the specified equipment may be any equipment in the spinning workshop. As shown in FIG. 2, the spinning workshop, and the specified equipment may be equipment such as pipes, flanges, gas storage tanks, and liquid storage tanks.

[0028] Since many pieces of equipment in the spinning mill need to observe the status of the equipment from multiple angles, in the embodiments of the present disclosure, in order to obtain the overall status of the equipment at the same time, an image collection is performed around the equipment using a UAV fleet. For example, a plurality of UAVs can arrange a UAV fleet formation based on the structure of the specified equipment and image the specified equipment at multiple angles simultaneously. The UAV fleet formation is UAV 1, UAV 2,..., UAV n. The image captured by UAV 1 is Image 1, the image captured by UAV 2 is Image 2,..., and the image captured by UAV n is Image n. The sample sequence is, that is, {Image 1; Image 2;...; Image n}.

[0029] In S102, position-encode the sample sequence based on the UAV fleet formation to obtain a UAV fleet formation encoding result.

[0030] By obtaining the relative position information of each UAV, the UAV fleet formation as a whole can be position-encoded to obtain a UAV fleet formation encoding result.

[0031] Performing position encoding for the drone fleet formation is because the sample images collected by each drone can be regarded as partial images in the 3D imaging of the specified facility. Since the failures of the specified facility may vary at different angles, there is a dependency relationship between the image contents of multiple sample images in the same sample sequence. Thus, through the position encoding of the drone fleet formation, based on this dependency relationship, in order to perform fault detection, the training target model can be induced to learn relevant knowledge from the sample sequence.

[0032] In S103, input the sample sequence and the drone fleet formation encoding result into the training target model to obtain the fault detection result output by the training target model.

[0033] In S104, determine the loss value based on the fault detection result and the true value of the fault detection result of the sample sequence.

[0034] In S105, adjust the model parameters of the training target model based on the loss value to obtain the fault detection model.

[0035] Here, the training target model includes an encoder and a decoder. The encoder is used to perform feature extraction on each sample image in the sample sequence to obtain the multi-scale features of each sample image, and based on the drone fleet formation encoding result, fuse the features of the same scale of the sample sequence to obtain the corresponding fusion features of each scale, and fuse the fusion features of each scale to obtain the target features. The decoder is used to determine the fault detection result including the fault prediction type and the fault prediction box at the same fault position in the fault sample diagram of the specified facility based on the target features, and refer to the drone fleet formation to piece together the sample images in the sample sequence to obtain the fault sample diagram.

[0036] Here, the sample sequence and the UAV fleet formation encoding result are input into the schematic diagram of the model to be trained. As shown in FIG. 3, for each image in the sample sequence, feature extraction is performed using the feature extraction model to obtain features of different scales of each image. As shown in FIG. 3, each rectangular box with the same shape represents the image features of one scale. Image 1 corresponds to the features f11 of the first scale, f12 of the second scale, …, f1m of the m-th scale. Image 2 corresponds to the features f21 of the first scale, f22 of the second scale, …, f2m of the m-th scale. Image n corresponds to the features fn1 of the first scale, fn2 of the second scale, …, fnm of the m-th scale. The image features of each image belonging to the first scale are input into the fully connected layer for fusion to obtain the first fusion feature. At the same time, the UAV fleet formation encoding result and the first fusion feature are input into the encoder based on the attention mechanism to obtain the first intermediate feature. The image features of each image belonging to the second scale are input into the fully connected layer for fusion to obtain the second fusion feature. At the same time, the UAV fleet formation encoding result and the second fusion feature are input into the encoder based on the attention mechanism to obtain the second intermediate feature. Based on the existence of image features of m scales, that is, m intermediate features are obtained. The m intermediate features can be fused based on the pyramid model to obtain the target feature. Thus, the encoder completes the encoding operation. The target feature is input into the decoder of the model to be trained to obtain the fault detection result output by the model to be trained. The loss value is determined based on the fault detection result and the true value of the fault detection result of the sample sequence. Based on this loss value, the model parameters of the model to be trained are adjusted, and when the loss value meets the convergence condition, the fault detection model is obtained. This fault detection result includes the fault prediction type and the fault prediction box at the same fault position in the fault sample diagram. If there are multiple fault positions, the fault prediction type and the fault prediction box are detected at each fault position.

[0037] In an embodiment of the present disclosure, fusion is performed based on the image features of the sample sequence collected by the drone fleet to obtain target features. These target features can show the features of the same fault from multiple angles, which is beneficial for comprehensively showing the fault situation and enabling the training target model to accurately perform fault detection. In addition, by performing multi-scale feature extraction and fusion on sample images from different angles, important features of different levels can be included in the target features, thereby improving the efficiency of the model for fault recognition.

[0038] In an embodiment of the present disclosure, in order to obtain a good markup result, the true value of the fault detection result of the sample sequence can be determined based on the following method.

[0039] In step A1, the sample sequence is spliced into a fault sample diagram based on the drone fleet formation. The fault sample diagram shows the state of the specified equipment from the perspectives of multiple drones.

[0040] In step A2, first presentation information is constructed based on the fault sample diagram. This first presentation information includes at least one fault at the fault point in the fault sample diagram, and the position information of the detection boxes at different drone perspectives in the fault sample diagram of the same fault is used as sub-position parameters respectively.

[0041] In step A3, position encoding is performed on multiple sub-position parameters of the same fault to obtain the fault position encoding of the same fault.

[0042] For each fault, the following operations are performed respectively.

[0043] In step A31, the fault point of the fault and the fault position encoding of the fault are input into the segmenting model as second presentation information so that the segmenting model cuts out the fault mask diagram of the fault from the fault sample diagram.

[0044] In step A32, obtain the true class label of the fault mask diagram of the fault, and construct the detection box label of the fault based on the position information in the fault sample diagram of the fault mask diagram to obtain the true value of the fault detection result.

[0045] As shown in FIG. 4, a schematic diagram of inputting a fault sample diagram into a segment-ensembling model is as follows: Input the fault sample diagram and the drone fleet formation encoding result in the second prompt information into an image feature extraction network (this network is constructed based on an attention mechanism) to obtain encoded features. Perform convolution processing on a preset mask using at least one convolutional layer to obtain mask features. Note that this preset mask may also be a mask obtained by object segmenting the fault position based on another pre-trained segment network. After fusing the mask features and this encoded feature, input them into a mask decoder. At the same time, input the fault point in the second prompt information into the mask decoder. After being processed by the mask decoder, a fault mask diagram that cuts out the fault in the fault sample diagram can be obtained from the segment-ensembling model. Input the fault mask diagram into a classification model, obtain the confidence score for each preset category of this fault mask diagram, and use the preset category with the highest score as the true class label of the fault mask diagram. Based on the position information of the fault mask diagram in the fault sample diagram, construct a rectangular box surrounding the fault point, and determine this rectangular box as the detection box label of the fault.

[0046] For example, when the specified equipment includes three fault positions, since this fault position may collect images from any of multiple viewpoints, the same fault position may include multiple detection boxes, and each detection box may have a corresponding category. For example, fault position A includes detection box 1 from the perspective of drone 1, whose fault code is a, and detection box 2 from the perspective of drone 2, whose fault code is also a.

[0047] In an embodiment of the present disclosure, based on the segment enclosing model, the segment enclosing model is used to perform segmentation on the fault sample diagram so as to cut out the fault mask diagram of the fault from the fault sample diagram, and furthermore, the true class label of the fault mask diagram can be obtained. Based on the position information of the fault mask diagram in the fault sample diagram, the detection box label of the fault can be further constructed, thereby building a powerful basis for the model to be trained.

[0048] In some embodiments, in order to optimize the parameters of the fault detection model and improve the fault detection efficiency, the loss function in the embodiments of the present disclosure includes the following loss items.

[0049] 1) The position loss between the fault prediction box and the detection box label.

[0050] In some embodiments, the position loss can be measured from multiple angles. For example, the detection box itself is the fault position obtained by the regression task. In the model training stage, for the same fault position, the detection box predicted by the model will become more accurate as the model parameters are optimized.

[0051] In an embodiment of the present disclosure, the position loss can include a first position loss sub-item, a second position loss sub-item, a third position loss sub-item, a fourth position loss sub-item, and a fifth position loss sub-item. The specific calculation method of each loss sub-item is as follows.

[0052] The first position loss sub-item is used to represent the loss between the fault prediction box in a plurality of sample images in the sample sequence of the same fault position and the center point of the corresponding detection box label.

Equation

[0053] The second position loss item is used to represent the detection box width loss between the fault prediction box and the corresponding detection box label in multiple sample images of the same fault position, and the detection box height loss between the fault prediction box and the corresponding detection box label in multiple sample images of the same fault position.

Number

[0054] The third position loss sub-item is used to represent the confidence loss of the fault prediction boxes in multiple sample images at the same fault position. [Number] Here, f3 represents the third position loss sub-item. C^ i j represents the confidence of the fault prediction box. When the j-th detection box in the i-th sample image is responsible for detecting that fault position, C^ i j = 1; otherwise, C^ i j = 0. C i j represents the confidence of the detection box label. I ij оbj represents whether the j-th candidate box of the i-th sample target contains that fault position. If it contains, it is 1; if not, it is 0. B represents the total number of candidate boxes in the i-th sample image, and S represents the total number of sample images.

[0055] The fourth position loss sub-item is used to represent the overlap rate loss between the important fault prediction boxes in multiple sample images at the same fault position and the detection boxes at the important positions in the corresponding detection box labels. [Number] Here, f4 represents the fourth position loss sub-item. I ij оbj represents whether the j-th candidate box of the i-th sample image contains this fault position. If it contains, it is 1; if not, it is 0. O^ i jrepresents the overlap rate between the failure prediction box and the detection box at an important position in the corresponding detection box label. B represents the total number of candidate boxes in the i-th sample image, and S represents the total number of sample images.

[0056] The fifth position loss item is used to represent the quantity loss between the total number of failure prediction boxes in multiple sample images of the same failure position and the total number of detection boxes in the corresponding detection box label.

Number

[0057] During implementation, since there is a detection error in the failure detection model, there may be detection boxes that are not detected or misdetected detection boxes, and all these detection boxes participate in the calculation.

[0058] During implementation, it is possible to mark the important and unimportant labels in the detection box label. Detection boxes that are not important do not participate in the loss calculation even if they are detected, and detection boxes at important positions that are detected may participate in the loss calculation.

[0059] In the embodiments of the present disclosure, in order to improve the accuracy of the failure detection model, based on information such as the center point, width, and height of the detection box, the loss value of the failure prediction box is comprehensively measured. 2) Classification loss between the failure prediction type and the true class label.

[0060] In some embodiments, the classification loss is determined based on the following formula.

Number

[0061] Here, due to the adjustment of the model parameters, the prediction for the detection box of the same fault position changes with the optimization of the model. Therefore, in Equation (6), the cumulative symbol can be understood as the cumulative value for the position prediction results of the same detection box label in multiple prediction processes. The cumulative amount for each time can be determined as needed.

[0062] Here, the statistic is the average value, the number of groups, or the maximum value. During implementation, since one fault position is included in multiple images, there are multiple detection boxes, and each detection box has one prediction score. Therefore, it is necessary to integrate the prediction scores of multiple detection boxes. The average value of the prediction scores of multiple detection boxes may be selected as the prediction score, or the number of groups or the maximum value may be selected as the prediction score.

[0063] After obtaining the above-mentioned position loss and classification loss, the constructed loss function is expressed as in Equation (7).

Number

[0064] In some embodiments, each sample sequence is one training sample, and a plurality of sample sequences construct a sample set. A part can be taken out from the sample set as the first training data, and another part can be taken out as the second training data. The learnable parameters in the model to be trained are adjusted using the first training data, and the hyperparameters in the model to be trained are adjusted using the second training data.

[0065] In the embodiments of the present disclosure, the hyperparameters that require learning can include λ1, λ2, λ3, λ4, λ5, λ6, and ρ τ and can include. The learnable parameters are model parameters excluding the parameters in the loss function.

[0066] The model to be trained shown in FIG. 1 in the embodiments of the present disclosure may be a pre-trained hyperparameter model. The process for learning hyperparameters can include the following.

[0067] Input the first training data into the model to be trained to obtain a fault detection result. This fault detection result includes a fault prediction type and a fault prediction box. Calculate a first loss (including a position loss and a classification loss) based on the difference between the fault detection result and the true value, calculate a gradient based on this first loss, and further optimize the learnable parameters in the model to be trained based on the gradient direction. Based on the model represented by the learnable parameters after this adjustment, input the second training data into the model to be trained to obtain a fault detection result. This fault detection result includes a fault prediction type and a fault prediction box. Calculate a position loss based on the difference between the fault prediction box and the detection box label, and calculate a classification loss. Further, determine a second loss based on the position loss and the classification loss. Optimize the hyperparameters with the goal of minimizing this second loss. Next, adjust the learnable parameters in the model to be trained based on the first training data and perform sequential iteration cycles until the optimal hyperparameters for minimizing the second loss are obtained. Therefore, determine the hyperparameters, and then continue to optimize the learnable parameters of the model using the method shown in FIG. 1. That is, input a sample sequence into the model to be trained to obtain a fault detection result. This fault detection result includes a fault prediction type and a fault prediction box. Adjust the learnable parameters of the position loss in the loss function based on the difference between the fault prediction box and the detection box, and adjust the learnable parameters of the classification loss in the loss function based on the difference between the fault prediction type and the true class label. When the convergence condition is met, obtain a fault detection model.

[0068] In the embodiments of the present disclosure, by comprehensively considering the position loss and the classification loss, the further designed objective function can provide a powerful basis for the model to be trained.

[0069] Based on the obtained fault detection model above, according to the same technical concept, the embodiments of the present disclosure further include a facility fault detection method, as shown in FIG. 5, including the following.

[0070] In S501, an initial image set of the target facility is obtained based on the drone fleet, and the drone fleet is for collecting images of the target facility from multiple viewpoints to obtain the initial image set.

[0071] In S502, noise removal processing is performed on each initial image of the initial image set to obtain a detection target image set.

[0072] During implementation, for each initial image, a target blur kernel is obtained based on the trajectory of the drone that collected this initial image, and noise removal processing is performed on this initial image based on the target blur kernel, so as to obtain a detection target image set constructed from each initial image after noise removal.

[0073] Taking one initial image as an example, first, k keys need to be obtained from this initial image, and an initial blur kernel needs to be obtained based on the k keys.

[0074] In S503, position encoding is performed on the formation of the drone fleet to obtain a drone position encoding result.

[0075] In S504, the detection target image set and the drone position encoding result are input into a fault detection model to obtain a fault detection result of the target facility by the fault detection model.

[0076] The fault detection result includes a fault prediction type and a fault prediction box at the same fault position in the target diagram, and the target diagram is obtained by stitching together the detection target images in the detection target image set with reference to the formation of the drone fleet.

[0077] In the embodiments of the present disclosure, based on the noise removal process of the initial image set collected by the drone fleet, a detection target image set is obtained. This detection target image set can show the characteristics of the same fault from multiple angles, which is beneficial for describing the overall fault situation and enables the fault detection result to accurately perform fault detection. Based on this method, automated monitoring of the target device is realized, and human resources can be saved.

[0078] In some embodiments, in order to reduce the false detection of the prediction box, the following can also be implemented.

[0079] In step B1, at least one key prediction box is filtered from multiple fault prediction boxes at the same fault position.

[0080] Labels of keys and non-keys can be attached to the fault prediction boxes. The detected non-key detection boxes do not participate in the loss calculation, and the detection boxes at the detected key positions participate in the loss calculation.

[0081] In step B2, based on at least one key prediction box, the detection target images of each key prediction box are separated from the target diagram.

[0082] In step B3, based on the detection target images of each key prediction box, a three-dimensional effect diagram at the same position is constructed and output.

[0083] When an abnormality of an important fault prediction box is detected, based on the position encoding of the formation collected by the drone, a three-dimensional effect diagram is constructed at that position and output to the operator to realize the management of the operator.

[0084] In the embodiments of the present disclosure, at least one key prediction box is filtered from multiple fault prediction boxes at the same fault position. In order to reduce the consumption of computing resources, fault prediction is performed using the key detection box, and the position where the fault is detected is rendered so that the operator can process it in a timely manner.

[0085] Based on the same technical concept, embodiments of the present disclosure provide a training apparatus 600 for a fault detection model, as shown in FIG. 6. The apparatus includes a sampling module 601 configured to sample equipment specified based on a preset unmanned aerial vehicle (UAV) fleet formation to obtain a sample sequence; a first encoding module 602 configured to perform position encoding on the sample sequence based on the UAV fleet formation to obtain a UAV fleet formation encoding result; an input module 603 configured to input the sample sequence and the UAV fleet formation encoding result into a model to be trained, and obtain a fault detection result output by the model to be trained; a determination module 604 configured to determine a loss value based on the fault detection result and the true value of the fault detection result of the sample sequence; an adjustment module 605 configured to adjust model parameters of the model to be trained based on the loss value to obtain a fault detection model. The model to be trained includes an encoder and a decoder. The encoder is used to perform feature extraction on each sample image in the sample sequence to obtain multi-scale features of each sample image, perform feature fusion on features of the same scale of the sample sequence based on the UAV fleet formation encoding result to obtain corresponding fusion features of each scale, and perform feature fusion on the fusion features of each scale to obtain target features. The decoder is used to determine a fault detection result including a fault prediction type and a fault prediction box at the same fault position in a fault sample diagram of the specified equipment based on the target features, and obtain a fault sample diagram by splicing sample images in the sample sequence with reference to the UAV fleet formation.

[0086] In some embodiments, the training apparatus for the fault detection model To connect a sample sequence to a fault sample diagram based on a drone fleet formation, the fault sample diagram shows the state of the specified facility from the perspectives of a plurality of drones, and To construct first presentation information based on the fault sample diagram, the first presentation information includes at least one fault at a fault point in the fault sample diagram, and the position information of the detection boxes at different drone perspectives in the fault sample diagram of the same fault is used as sub-position parameters respectively, and Perform position encoding on multiple sub-position parameters of the same fault to obtain a fault position encoding of the same fault, and For each fault, Input the fault point and the fault position encoding of the fault into the segmenting model as the second presentation information so that the segmenting model cuts out the fault mask diagram of the fault from the fault sample diagram, Obtain the true class label of the fault mask diagram of the fault, and construct the detection box label of the fault based on the position information in the fault sample diagram of the fault mask diagram to obtain the true value of the fault detection result, and execute the operations respectively, and Further includes an acquisition module used for. In some embodiments, the loss function of the model to be trained includes, as loss items, The position loss between the fault prediction box and the detection box label, and The classification loss between the fault prediction type and the true class label, including.

[0087] In some embodiments, the decision module is determined based on the following formula,

Equation

[0088] In some embodiments, the statistic is an average value, a group number, or a maximum value. In some embodiments, the position loss is a first position loss sub-item for representing the loss between the center point of the detection box label corresponding to the fault prediction box in multiple sample images in the sample sequence of the same fault location and the fault prediction box, a second position loss sub-item for representing the detection box width loss between the fault prediction box and the corresponding detection box label in multiple sample images of the same fault location, and the detection box height loss between the fault prediction box and the corresponding detection box label in multiple sample images of the same fault location, a third position loss sub-item for representing the confidence loss of the fault prediction box in multiple sample images of the same fault location, a fourth position loss sub-item for representing the overlap rate loss between the important fault prediction box in multiple sample images of the same fault location and the detection box at the important position in the corresponding detection box label, and a fifth position loss sub-item for representing the quantity loss between the total quantity of the fault prediction boxes in multiple sample images of the same fault location and the total quantity of the detection boxes in the corresponding detection box label.

[0089] Based on the same technical concept, an embodiment of the present disclosure provides a facility fault detection device as shown in FIG. 6, which is applied to the fault detection model obtained in the foregoing embodiment, and the device includes A collection module 701 for obtaining an initial image set of a target facility based on a drone fleet, where the drone fleet is for performing image collection on the target facility from multiple viewpoints to obtain the initial image set, the collection module 701, A noise removal module 702 for performing noise removal processing on each initial image of the initial image set to obtain a detection target image set, A second encoding module 703 for performing position encoding on the formation of the drone fleet to obtain a drone position encoding result, A processing module 704 for inputting the detection target image set and the drone position encoding result into a fault detection model to obtain a fault detection result of the target facility by the fault detection model, and comprising, The fault detection result includes a fault prediction type and a fault prediction box at the same fault position in the target diagram, and the detection target images in the detection target image set are stitched together with reference to the formation of the drone fleet to obtain the target diagram.

[0090] In some embodiments, the apparatus Filters at least one key prediction box from a plurality of fault prediction boxes at the same fault position, Separates the detection target images of each key prediction box from the target diagram based on at least one key prediction box, And further comprises a generation module used for constructing and outputting a three-dimensional effect diagram at the same fault position based on the detection target images of each key prediction box.

[0091] For the specific functions and exemplary descriptions of each module and sub-module of the apparatus according to the embodiments of the present disclosure, reference can be made to the related descriptions of the corresponding steps in the embodiments of the above-described method, and will not be repeated here.

[0092] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0093] FIG. 8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 8, the electronic device includes a memory 810 and a processor 820, and a computer program executable by the processor 820 is stored in the memory 810. The number of the memory 810 and the processor 820 can be one or more. The memory 810 can store one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device is caused to execute the method provided by the embodiment of the above method. The electronic device can further include the following. A communication interface 830 is used for communicating with an external device and performing data interaction and transmission.

[0094] When the memory 810, the processor 820, and the communication interface 830 are independently implemented, the memory 810, the processor 820, and the communication interface 830 are connected to each other via a bus and can communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be classified into an address bus, a data bus, a control bus, etc. For ease of explanation, only a single thick line is shown in FIG. 8, but it does not represent only a single bus or a single type of bus.

[0095] Optionally, in a specific implementation form, when the memory 810, the processor 820, and the communication interface 830 are integrated on one chip, the memory 810, the processor 820, and the communication interface 830 can communicate with each other via an internal interface.

[0096] The above-mentioned processor may be a Central Processing Unit (CPU), and it should be understood that it may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware assemblies, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. Note that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0097] Furthermore, optionally, the memory may include a read-only memory and a random access memory, and may further include a non-volatile random access memory. The memory can be either a volatile memory or a non-volatile memory, or can include both a volatile memory and a non-volatile memory. Here, the non-volatile memory can include ROM (Read-Only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically EPROM), or flash memory. The volatile memory can include a random access memory (Random Access Memory, RAM) that functions as an external cache. By way of example and not limitation, many forms of RAM are available. For example, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0098] In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, the whole or part may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are generated in whole or in part. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, Bluetooth®, microwave, etc.). The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device including a server, a data center, etc. integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy (registered trademark) disk, a hard disk, a magnetic tape), an optical medium (such as a Digital Versatile Disc (DVD)), or a semiconductor medium (such as a Solid State Disk (SSD)), etc. It should be noted that the computer-readable storage medium referred to in the present disclosure may be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0099] Those skilled in the art can understand that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0100] In the description of the embodiments of the present disclosure, the descriptions of reference terms such as "one embodiment", "several embodiments", "example", "specific example", or "several examples", etc. mean that the specific features, structures, materials, or features described in relation to the embodiment or example are included in at least one embodiment or example of the present disclosure. And the specific features, structures, materials, or features described can be combined in any one or more embodiments or examples in a suitable manner. Furthermore, those skilled in the art may combine the different embodiments or examples described in the present disclosure and the features of the different embodiments or examples as long as they do not conflict with each other.

[0101] In the description of the embodiments of the present disclosure, " / " represents the meaning of "or" unless otherwise described. For example, A / B may represent either A or B. The "and / or" in the present disclosure only explains the relationship of related objects and indicates that three types of relationships may exist. For example, A and / or B can indicate the following. There are three situations where A exists alone, A and B exist simultaneously, and B exists alone.

[0102] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used only for the purpose of description and should not be construed as indicating or implying relative importance, nor should they be construed as implying the number of technical features shown. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, "a plurality" means two or more unless otherwise described.

[0103] The above are only exemplary embodiments of the present disclosure, and do not limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle scope of the present disclosure should all be included within the protection scope of the present disclosure.

[0104] In the description of this specification, terms such as "center", "longitudinal direction", "lateral direction", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial direction", "radial direction", "circumferential direction", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only intended to facilitate the description of the present disclosure and simplify the description, and do not indicate or imply that the mentioned device or element must have a specific orientation, be configured and operate in a specific orientation, and therefore should not be construed as a limitation of the present disclosure.

Claims

1. 1. A method for training a fault detection model, comprising: Sampling designated equipment based on a preset drone fleet formation to obtain a sample sequence; position encoding the sample sequence based on the drone fleet formation to obtain a drone fleet formation encoding result; Inputting the sample sequence and the unmanned vehicle fleet formation encoding result into a trained model to obtain a fault detection result output by the trained model; determining a loss value based on the fault detection result and a true value of the fault detection result of the sample sequence; and adjusting model parameters of the trained model based on the loss value to obtain a fault detection model. The trained model includes an encoder and a decoder. The encoder is used for: respectively performing feature extraction on each sample image in the sample sequence to obtain multi-scale features of each sample image; and according to the drone fleet formation encoding result, performing feature fusion of the same-scale features of the sample sequence to obtain corresponding fusion features of each scale; and performing feature fusion of the fusion features of each scale to obtain a target feature; The decoder is used for determining a fault detection result of the specified equipment according to the target feature, the fault detection result including a fault prediction type and a fault prediction box of the same fault location in a fault sample diagram, and stitching together sample images in the sample sequence with reference to the drone fleet formation to obtain the fault sample diagram. How to train a fault detection model.

2. The method for training the fault detection model includes: splicing the sample sequence into the fault sample diagram based on the drone fleet formation, the fault sample diagram showing the state of the specified equipment from the perspective of multiple drones; Constructing first presentation information based on the fault sample diagram, the first presentation information includes a fault point of at least one fault in the fault sample diagram, and position information of detection boxes in the fault sample diagram of the same fault at different drone viewpoints is set as a sub-position parameter; performing location encoding on a plurality of sub-location parameters of a same fault to obtain a fault location encoding of the same fault; For each failure, inputting a fault point of the fault and a fault location encoding of the fault as second presentation information into the segment anything model, such that the segment anything model extracts a fault mask diagram of the fault from the fault sample diagram; respectively performing the operations of obtaining a true class label of a fault mask diagram of the fault, constructing a detection box label of the fault according to position information of the fault mask diagram in the fault sample diagram, and obtaining a true value of the fault detection result; determining a true value of a fault detection result of the sample sequence based on a method including: The method for training a fault detection model according to claim 1 .

3. The loss function of the training target model is as follows: the position loss between the fault prediction box and the detection box labels, Classification loss between fault prediction types and true class labels, Including, The method for training a fault detection model according to claim 1 .

4. The classification loss is determined based on the following formula: [0010] where N is the number of fault locations in the fault sample diagram, and when multiple sample images in the fault sample diagram show the same fault location, the same fault location is counted once in N; a ｉ is the i-th fault location, f (a ｉ , b ｉ ) is the true class label of the i-th fault location, f (a ｉ , b′) is a statistic of the prediction score for the i-th fault location in each sample image when the i-th fault location appears in multiple sample images of the fault sample diagram, ρ τ is a temperature scalar, The method for training a fault detection model according to claim 3.

5. The statistic is an average value, a group number, or a maximum value; The method for training a fault detection model according to claim 4.

6. The position loss is a first position loss subitem for representing a loss between a center point of a fault prediction box and a corresponding detection box label in a plurality of sample images in the sample sequence of the same fault location; a second position loss subitem for representing a detection box width loss between a fault prediction box and a corresponding detection box label in the plurality of sample images of the same fault location, and a detection box height loss between a fault prediction box and a corresponding detection box label in the plurality of sample images of the same fault location; a third location loss subitem for expressing the confidence loss of the fault prediction box in the plurality of sample images of the same fault location; a fourth position loss subitem for expressing an overlap rate loss between the important fault prediction box in the plurality of sample images of the same fault location and the detection box of the important location in the corresponding detection box label; and a fifth position loss subitem for expressing a quantity loss between the total amount of fault prediction boxes in the plurality of sample images of the same fault location and the total amount of detection boxes in the corresponding detection box labels. The method for training a fault detection model according to claim 3.

7. An equipment fault detection method, which is applied to the fault detection model according to claim 1, The equipment fault detection method includes: acquiring an initial set of images of a target facility based on a fleet of drones, the fleet of drones being configured to collect images of the target facility from multiple viewpoints to obtain the initial set of images; performing a noise removal process on each initial image of the initial image set to obtain a detection target image set; performing position encoding on the formation of the drone fleet to obtain drone position encoding results; The detection target image set and the drone position encoding result are input into the fault detection model to obtain a fault detection result for the target equipment of the fault detection model; The fault detection result includes a fault prediction type and a fault prediction box of the same fault location in the target map, and the target map is obtained by stitching together the detection target images in the detection target image set with reference to the formation of the drone fleet; Equipment fault detection methods.

8. The equipment fault detection method includes: filtering at least one key prediction box from the plurality of fault prediction boxes of the same fault location; Separating a detection target image of each key prediction box from the target image based on the at least one key prediction box; According to the detection target image of each key prediction box, construct and output a three-dimensional effect diagram of the same fault location. The equipment fault detection method according to claim 7.

9. 1. An apparatus for training a fault detection model, comprising: a sampling module for sampling designated installations based on a preset drone fleet formation to obtain a sample sequence; a first encoding module for position encoding the sample sequence based on the drone fleet formation to obtain an drone fleet formation encoding result; an input module for inputting the sample sequence and the unmanned vehicle fleet formation encoding result into a trained model to obtain a fault detection result output by the trained model; a determination module for determining a loss value based on the fault detection result and a true value of the fault detection result of the sample sequence; an adjustment module for adjusting model parameters of the trained model based on the loss value to obtain a fault detection model; The trained model includes an encoder and a decoder. The encoder is used for: respectively performing feature extraction on each sample image in the sample sequence to obtain multi-scale features of each sample image; and according to the drone fleet formation encoding result, performing feature fusion of the same-scale features of the sample sequence to obtain corresponding fusion features of each scale; and performing feature fusion of the fusion features of each scale to obtain a target feature; The decoder is used for determining a fault detection result of the specified equipment according to the target feature, the fault detection result including a fault prediction type and a fault prediction box of the same fault location in a fault sample diagram, and stitching together sample images in the sample sequence with reference to the drone fleet formation to obtain the fault sample diagram. A training rig for fault detection models.

10. An equipment fault detection device, which is applied to the fault detection model according to claim 9, The equipment fault detection device includes: a collection module for acquiring an initial set of images of a target facility based on a fleet of drones, the fleet of drones being configured to collect images of the target facility from multiple viewpoints to obtain the initial set of images; a noise removal module for performing a noise removal process on each initial image of the initial image set to obtain a detection target image set; a second encoding module for performing position encoding on the formation of the drone fleet to obtain a drone position encoding result; A processing module for inputting the detection target image set and the drone position encoding result into the fault detection model to obtain a fault detection result for the target equipment of the fault detection model; The fault detection result includes a fault prediction type and a fault prediction box of the same fault location in the target map, and the target map is obtained by stitching together the detection target images in the detection target image set with reference to the formation of the drone fleet; Equipment fault detection device.

11. At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 8. Electronic devices.

12. A non-transitory computer readable storage medium for storing instructions that cause a computer to perform the method of any one of claims 1 to 8.

13. A program for implementing the method according to any one of claims 1 to 8 when executed by a processor in a computer.

Images