Open type disconnecting switch state identification method and device, medium and equipment

Abstract

The invention discloses an open type disconnecting switch state recognition method and device, a medium and equipment, and the method comprises the steps: obtaining a training data set of an open type disconnecting switch, carrying out the LORA fine tuning training of a pre-trained visual question and answer model based on the training data set, and obtaining a trained visual question and answer model, the training data set comprises a training image of the open type disconnecting switch and corresponding training question information; obtaining an initial image and question information of a to-be-recognized open type disconnecting switch, and performing feature recognition on the initial image to obtain a target image of the to-be-recognized open type disconnecting switch; and inputting a target image and question information of the to-be-identified open type disconnecting switch into the trained visual question and answer model to obtain a state identification result of the to-be-identified open type disconnecting switch. The visual question and answer technology is applied to judgment of the opening and closing states of the open type disconnecting switch, and the opening and closing states of the open type disconnecting switch can be quickly and accurately recognized based on the multi-mode deep learning technology.

Description

Technical Field

[0001] The present invention relates to the technical field of isolating switch state identification, and in particular to an open isolating switch state identification method, device, medium and equipment. Background Art

[0002] The open type high-voltage isolating switch is an important power equipment, which is mainly used to isolate and connect equipment in high-voltage circuits. For example, in order to ensure the safe maintenance or replacement of equipment, the equipment needs to be completely isolated from the power supply; different power supplies or loads are switched to meet different power supply requirements. The open type high-voltage isolating switch plays the role of isolation or connection through opening and closing.

[0003] However, open-type high-voltage disconnectors have the abnormal situation that the opening and closing of the switch is not in place. The occurrence of such abnormal situation can easily lead to heating of the switch contacts, arc discharge when operating adjacent disconnectors, and other faults or accidents. Compared with closed-type disconnectors, open-type disconnectors use a control robot arm to perform opening and closing operations. Due to the different working principles of the mechanical structure, especially some disconnectors using horizontal rotating connecting rods or vertical connecting rod robot arms, due to angle limitations, multiple angles are required to jointly determine whether the dynamic contacts of opening and closing are in place. In addition, due to the existence of multiple mechanical interlocking loops and strong logical correlation, multiple standards are required to assist in determining whether there are defective characteristics.

[0004] At present, the judgment of the closed or open state of the open type high-voltage isolation is mainly made on site based on the experience of the staff. This judgment method has subjective factors and is relatively inefficient. Summary of the invention

[0005] In view of this, the present invention provides a method, device, medium and equipment for identifying the state of an open-type disconnector, the main purpose of which is to solve the problem that the existing method for identifying the state of an open-type disconnector has low efficiency and low accuracy in judging the closed or open state of an open-type high-voltage disconnector.

[0006] According to one aspect of the present application, a method for identifying a state of an open-type disconnector is provided, the method comprising:

[0007] Obtain a training data set of an open-type disconnector, and perform LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information;

[0008] Acquire an initial image and question information of the open-type disconnector to be identified, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified;

[0009] The target image of the open-type disconnector to be identified and the question information are input into the trained visual question answering model to obtain a state recognition result of the open-type disconnector to be identified.

[0010] Optionally, performing feature recognition on the initial image to obtain a target image of an open-type disconnector to be identified includes:

[0011] Inputting the initial image of the open-type disconnector to be identified into the trained object detection model to obtain an image with the border and category of the open-type disconnector to be identified marked;

[0012] The image with the frame and category of the open-type disconnector to be identified is cropped to obtain a target image.

[0013] Optionally, performing LORA fine-tuning training on the pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model includes:

[0014] Obtaining a pre-trained visual question answering model, wherein the pre-trained visual question answering model includes a self-attention mechanism module;

[0015] Adding a LORA fine-tuning module to the self-attention mechanism module of the pre-trained visual question answering model to obtain a fine-tuned visual question answering model;

[0016] The fine-tuning visual question answering model is trained based on the training data set to obtain a trained visual question answering model.

[0017] Optionally, the parameters of the LORA fine-tuning module include a dimensionality reduction matrix and a dimensionality increase matrix;

[0018] The fine-tuning and training visual question answering model is trained based on the training data set to obtain a trained visual question answering model, including:

[0019] Initializing the dimension reduction matrix and the dimension increase matrix;

[0020] Based on the initial weight parameters in the self-attention mechanism module, the initialized dimensionality reduction matrix and the initialized dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters, and the fine-tuned weight parameters are used as the initial weight parameters for the next iteration;

[0021] The dimension reduction matrix and the dimension increase matrix are updated to obtain an updated dimension reduction matrix and an updated dimension increase matrix;

[0022] Based on the initial weight parameters in the self-attention mechanism module, the updated dimensionality reduction matrix and the updated dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters;

[0023] The fine-tuned weight parameters are used as the initial weight parameters for the next iteration, and the dimension reduction matrix and the dimension increase matrix are continuously updated to obtain new fine-tuned weight coefficients until the iteration ends, thereby obtaining a trained visual question answering model.

[0024] Optionally, the initial weight parameters of the self-attention mechanism module are fine-tuned using the following method:

[0025] W=W0+BA

[0026] Among them, W0 is the initial weight parameter generated by the attention mechanism module, A is the dimension reduction matrix, B is the dimension increase matrix, and W is the weight parameter generated by the attention mechanism module after fine-tuning.

[0027] Optionally, before inputting the initial image of the open-type disconnector to be identified into the trained target detection model, the open-type disconnector state identification method further includes:

[0028] Acquire multiple original training images of open-type disconnectors, and perform box selection and category marking of the disconnector on each of the original training images;

[0029] Divide the labeled multiple training images into a training data set, a verification data set, and a test data set;

[0030] Training a preset detection model based on the training data set to obtain a preliminarily trained detection model;

[0031] Testing the preliminarily trained detection model based on the verification data set, and if the test fails, adjusting the preliminarily trained detection model and retraining it until the test passes;

[0032] Based on the test data set, the detection model that has passed the detection is tested. After the test passes, a target detection model with a trained model is obtained.

[0033] Optionally, before obtaining the training data set of the open type disconnector, the open type disconnector state recognition method includes:

[0034] Acquire multiple original training images of the open-type disconnector, input each of the original training images into the target detection model to obtain a first training image of the open-type disconnector, and crop the first training image to obtain a second training image;

[0035] Based on the category of the open type disconnector in the second training image of each open type disconnector, obtaining question training information of the open type disconnector;

[0036] A second training image and its corresponding question training information are used as an image-text data pair, and a plurality of image-text data pairs constitute a training data set.

[0037] According to another aspect of the present application, there is provided an open-type isolating switch state identification device, comprising:

[0038] A visual question answering model training module is used to obtain a training data set of an open-type disconnector, and perform LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information;

[0039] A target image acquisition module, used to acquire an initial image of the open-type disconnector to be identified and question information, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified;

[0040] The recognition result acquisition module is used to input the target image and question information of the open-type disconnector to be identified into the trained visual question answering model to obtain the state recognition result of the open-type disconnector to be identified.

[0041] Optionally, the target image acquisition module is further used for:

[0042] Inputting the initial image of the open-type disconnector to be identified into the trained object detection model to obtain an image with the border and category of the open-type disconnector to be identified marked;

[0043] The image with the frame and category of the open-type disconnector to be identified is cropped to obtain a target image.

[0044] Optionally, the visual question answering model training module is further used to:

[0045] Obtaining a pre-trained visual question answering model, wherein the pre-trained visual question answering model includes a self-attention mechanism module;

[0046] Adding a LORA fine-tuning module to the self-attention mechanism module of the pre-trained visual question answering model to obtain a fine-tuned visual question answering model;

[0047] The fine-tuning visual question answering model is trained based on the training data set to obtain a trained visual question answering model.

[0048] Optionally, the parameters of the LORA fine-tuning module include a dimensionality reduction matrix and a dimensionality increase matrix; the visual question answering model training module is also used for:

[0049] Initializing the dimension reduction matrix and the dimension increase matrix;

[0050] Based on the initial weight parameters in the self-attention mechanism module, the initialized dimensionality reduction matrix and the initialized dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters, and the fine-tuned weight parameters are used as the initial weight parameters for the next iteration;

[0051] The dimension reduction matrix and the dimension increase matrix are updated to obtain an updated dimension reduction matrix and an updated dimension increase matrix;

[0052] Based on the initial weight parameters in the self-attention mechanism module, the updated dimensionality reduction matrix and the updated dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters;

[0053] The fine-tuned weight parameters are used as the initial weight parameters for the next iteration, and the dimension reduction matrix and the dimension increase matrix are continuously updated to obtain new fine-tuned weight coefficients until the iteration ends, thereby obtaining a trained visual question answering model.

[0054] Optionally, the initial weight parameters of the self-attention mechanism module are fine-tuned using the following method:

[0055] W=W0+BA

[0056] Among them, W0 is the initial weight parameter generated by the attention mechanism module, A is the dimension reduction matrix, B is the dimension increase matrix, and W is the weight parameter generated by the attention mechanism module after fine-tuning.

[0057] Optionally, the open-type isolating switch state identification device further includes:

[0058] The target detection model training module is used to obtain multiple original training images of open-type disconnectors, and to mark each of the original training images with a box selection and category of the disconnector; to divide the labeled multiple training images into a training data set, a verification data set, and a test data set; to train a preset detection model based on the training data set to obtain a preliminarily trained detection model; to test the preliminarily trained detection model based on the verification data set, and when the test fails, to adjust the preliminarily trained detection model and retrain it until the test passes; to test the detection model that passes the test based on the test data set, and after the test passes, to obtain a trained target detection model.

[0059] Optionally, the open-type isolating switch state identification device further includes:

[0060] A visual question answering model training module is used to obtain multiple original training images of an open-type disconnector, input each of the original training images into the target detection model to obtain a first training image of the open-type disconnector, and crop the first training image to obtain a second training image; based on the category of the open-type disconnector in each of the second training images of the open-type disconnector, obtain question training information of the open-type disconnector; use a second training image and its corresponding question training information as an image-text data pair, and multiple image-text data pairs constitute a training data set.

[0061] According to another aspect of the present application, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned open-type disconnector state identification method.

[0062] According to another aspect of the present application, a computer device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0063] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned open type disconnector state identification method.

[0064] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0065] The present application provides a method, device, equipment and medium for identifying the state of an open-type disconnector, obtains a training data set including training images of the open-type disconnector and corresponding training question information, performs LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, performs feature recognition on the image of the open-type disconnector to be identified, obtains a target image containing only the open-type disconnector, inputs the target image of the open-type disconnector to be identified and the question information into the trained visual question answering model, and the trained visual question answering model outputs the recognition result of the state of the open-type disconnector, applies the visual question answering technology to the judgment of the opening and closing state of the open-type disconnector, and performs targeted training through LORA fine-tuning training, and based on multimodal deep learning technology, can quickly and accurately identify the opening and closing state of the open-type disconnector.

[0066] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0068] Figure 1 A flow chart of a method for identifying the state of an open-type disconnector provided in an embodiment of the present application is shown;

[0069] Figure 2 An image of a method for identifying the state of an open-type disconnector provided in an embodiment of the present application, in which a mechanical structure of the open-type disconnector to be identified is marked;

[0070] Figure 3 A target image of an open-type disconnector state recognition method provided by an embodiment of the present application is shown;

[0071] Figure 4 A structure diagram of a pre-trained visual question answering model for an open-type disconnector state recognition method provided in an embodiment of the present application is shown;

[0072] Figure 5 A schematic diagram of a LORA fine-tuning model of an open-type disconnector state recognition method provided in an embodiment of the present application is shown;

[0073] Figure 6 A partial structural diagram of a visual question answering model for fine-tuning training of an open-type disconnector state recognition method provided in an embodiment of the present application is shown;

[0074] Figure 7 A structural block diagram of an open-type isolating switch state identification device provided in an embodiment of the present application is shown;

[0075] Figure 8 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown.

[0076] in,

[0077] Figure 7 Middle: 702-visual question answering model training module; 704-target image acquisition module; 706-recognition result acquisition module;

[0078] Figure 8 In it: 802 - processor; 804 - communication interface; 806 - memory; 808 - communication bus; 810 - program. DETAILED DESCRIPTION

[0079] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0080] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention application are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0081] In view of the problem that the existing open-type isolating switch state recognition method has low efficiency and low accuracy in judging the closed or open state of the open-type high-voltage isolator, the embodiment of the present application provides an open-type isolating switch state recognition method, such as Figure 1 As shown, the method includes:

[0082] 102: Obtain a training data set of an open-type disconnector, and perform LoRa fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information;

[0083] 104: Acquire an initial image of the open-type disconnector to be identified and question information, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified;

[0084] 106: Input the target image of the open-type disconnector to be identified and the question information into the trained visual question answering model to obtain a state recognition result of the open-type disconnector to be identified.

[0085] Specifically, images of the mechanical structure of the open-type disconnector at the substation site are captured by on-site monitoring equipment and handheld video equipment to obtain a large number of images containing the mechanical structure of the open-type disconnector. The disconnector states of these images should be distributed as evenly as possible, that is, the roughly classified states include the three states of closed, open, and closed / open not in place, which should be as evenly distributed as possible in the number of images. The training data set includes multiple target training images of the open-type disconnector and training question information corresponding to each target training image. The training question information is text designed according to the type of disconnector in the target training image, such as "Is the disconnector closed?", "Is the disconnector not closed?", etc., with rich text in various forms; the training image is an image containing only the disconnector, and the original image of the disconnector is feature-recognized and cropped to obtain an image containing only the disconnector.

[0086] Based on the training data set, the pre-trained visual question answering model is fine-tuned with LoRa. LoRa fine-tuning can help the model better adapt to specific texts. The visual question answering model after LoRa fine-tuning training can accurately determine whether the open-type disconnector is opened / closed in place.

[0087] Obtain relevant question information about whether the open-type disconnector to be identified is open / closed and an initial image of the open-type disconnector to be identified, the initial image containing the disconnector and components and background around the disconnector. Perform feature recognition on the initial image, select the disconnector to be identified and determine its type, and obtain a target image corresponding to the open-type disconnector to be identified.

[0088] The target image and question information corresponding to the open-type disconnector to be identified are passed into the fine-tuned and trained visual question answering model. Based on the image information and the relevant question text about whether the switch is fully closed, the visual question answering model outputs the recognition result, thereby realizing the visual question answering technology to judge whether the open-type disconnector is fully opened / closed.

[0089] The present application provides a method for identifying the state of an open-type disconnector. Compared with the prior art, a training data set including training images of the open-type disconnector and corresponding training question information is obtained, and LORA fine-tuning training is performed on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, and feature recognition is performed on the image of the open-type disconnector to be identified to obtain a target image containing only the open-type disconnector. The target image of the open-type disconnector to be identified and the question information are input into the trained visual question answering model, and the trained visual question answering model outputs the recognition result of the state of the open-type disconnector, and the visual question answering technology is applied to the judgment of the opening and closing state of the open-type disconnector, and targeted training is performed through LORA fine-tuning training. Based on multimodal deep learning technology, the opening and closing state of the open-type disconnector can be quickly and accurately identified.

[0090] In one embodiment, feature recognition is performed on the initial image to obtain a target image of the open-type disconnector to be identified, including:

[0091] Inputting the initial image of the open-type disconnector to be identified into the trained object detection model to obtain an image with the border and category of the open-type disconnector to be identified;

[0092] The image with the border and category of the open-type disconnector to be identified is cropped to obtain the target image.

[0093] Specifically, the initial image of the open-type disconnector to be identified is first sent to the trained target detection model. The target detection model marks the target area of ​​the mechanical structure of the open-type disconnector and determines the category of the disconnector. The initial image is cropped based on the marked area to obtain the target image, so that only the image information of the mechanical structure of the open-type disconnector is retained, while other interfering area information in the image is excluded. Figure 2 The image shows the mechanical structure of the open type disconnector to be identified. Figure 3 Shown is the target image after cropping.

[0094] In one embodiment of the present invention, a pre-trained visual question answering model is fine-tuned using LORA based on a training data set to obtain a trained visual question answering model, including:

[0095] Obtain a pre-trained visual question answering model, wherein the pre-trained visual question answering model includes a self-attention mechanism module;

[0096] Add the LORA fine-tuning module to the self-attention mechanism module of the pre-trained visual question answering model to obtain a fine-tuned visual question answering model;

[0097] The fine-tuning visual question answering model is trained based on the training data set to obtain a trained visual question answering model.

[0098] Specifically, obtain a pre-trained visual question answering model, preferably a BLIP pre-trained model, Figure 4 This is the structural diagram of the pre-trained model. The most important module of the pre-trained model is the self-attention mechanism module, and the mathematical formula of the self-attention mechanism module is as follows.

[0099]

[0100] Among them, Q is the Query matrix, K is the Key matrix, V is the Value matrix, and K = W k I, Q = W q I, V = W v I, where W is the change matrix and I is the input matrix.

[0101] Insert the LORA fine-tuning module into the self-attention mechanism module to obtain a fine-tuned visual question answering model. This application adopts the LORA efficient fine-tuning method, such as Figure 5 The figure shows a schematic diagram of the LORA fine-tuning method. The Q and K transformation matrices of the multi-head self-attention in the Transformer are connected in parallel with a bypass, and the bypass is the LORA fine-tuning module, as shown in Figure 6 As shown in the figure, when fine-tuning the visual question answering model for training, the parameters of the original pre-trained visual question answering model remain fixed, and only the parameters of the LORA fine-tuning module are trained to achieve an efficient fine-tuning training process.

[0102] The pre-trained visual question answering model is trained based on a large sample data set. It has rich understanding capabilities of real-world visual and natural language information, and can understand, integrate, transfer and reason about different visual images and diverse natural language text information.

[0103] In one embodiment of the present invention, the parameters of the LORA fine-tuning module include a dimension reduction matrix and a dimension increase matrix;

[0104] The fine-tuned visual question answering model is trained based on the training data set to obtain a trained visual question answering model, including:

[0105] Initialize the dimension reduction matrix and dimension increase matrix;

[0106] Based on the initial weight parameters in the self-attention mechanism module, the initialized dimensionality reduction matrix and the initialized dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters, and the fine-tuned weight parameters are used as the initial weight parameters for the next iteration;

[0107] The dimension reduction matrix and the dimension increase matrix are updated to obtain an updated dimension reduction matrix and an updated dimension increase matrix;

[0108] Based on the initial weight parameters in the self-attention mechanism module, the updated dimensionality reduction matrix, and the updated dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters;

[0109] The fine-tuned weight parameters are used as the initial weight parameters for the next iteration, and the dimension reduction matrix and the dimension increase matrix are continuously updated to obtain new fine-tuned weight coefficients until the iteration ends, and the trained visual question answering model is obtained.

[0110] Specifically, Figure 6 The figure shows the implementation diagram of LORA fine-tuning on Transformer. The added parallel bypass consists of a dimension reduction matrix A and a dimension increase matrix B, which simulates the update of weight parameters based on low-rank decomposition.

[0111] The process of efficient LORA fine-tuning can be expressed by the following formula:

[0112]

[0113] Among them, W0 is the initial weight parameter generated by the attention mechanism module, A is the dimension reduction matrix, B is the dimension increase matrix, and W is the weight parameter generated by the attention mechanism module after fine-tuning.

[0114] ΔW is the weight parameter for fine-tuning training, which consists of A and B matrices, where the rank r<<min(d,k).

[0115] First, the parallel bypass is initialized, where the A matrix is ​​initialized with random Gaussian noise distribution and the B matrix is ​​initialized with all 0s. Then, the weight parameters generated by the fine-tuned attention mechanism module are calculated based on the initial weight parameters generated by the attention mechanism module and the initialized A matrix and B matrix. The A matrix and the B matrix are updated using an update strategy. Each time an update is performed, the weight parameters generated by the fine-tuned attention mechanism module are recalculated based on the initial weight parameters generated by the attention mechanism module and the updated A matrix and B matrix until the end of the iteration. The weight parameters generated by the attention mechanism module after the last fine-tuning are used as the weight parameters in the trained visual question answering model.

[0116] In one embodiment, before inputting the initial image of the open-type disconnector to be identified into the trained object detection model, the open-type disconnector state recognition method further includes:

[0117] Acquire multiple original training images of open-type disconnectors, and perform box selection and category annotation of the disconnector on each original training image;

[0118] Divide the labeled multiple training images into a training data set, a verification data set, and a test data set;

[0119] The preset detection model is trained based on the training data set to obtain a preliminarily trained detection model;

[0120] The preliminarily trained detection model is tested based on the validation data set. If the test fails, the preliminarily trained detection model is adjusted and retrained until the test passes.

[0121] Based on the test data set, the detection model that has passed the detection is tested. After the test passes, the target detection model with the model training completed is obtained.

[0122] Specifically, a large number of images containing the mechanical structure of open-type disconnectors are obtained, and the disconnector states of these images should be distributed as evenly as possible, that is, the general classification states including closed, open, and closed / open states should be as evenly distributed as possible in the number of images.

[0123] Appropriate data enhancement is performed on the collected images, including spatial changes (rotation, translation, scaling, flipping) of the images, adjustment of the color changes (brightness, contrast, saturation) of the images, adding appropriate surroundings to the images, marking the targets in the images (open type disconnector mechanical structure) and annotating the target categories. The target category is one category, namely, the open type disconnector mechanical structure category.

[0124] Based on the collected images of the mechanical structure of the open-type disconnector on site, we designed question-and-answer texts about whether the disconnector is in place, and tried to make the questions and answers as diverse as possible to form a multimodal dataset of image-text pairs; we segmented the data of the image-text pairs and divided the data into training set, validation set and test set.

[0125] Select a preset detection model, such as the lightweight target detection YOLOv5s model, and perform multiple rounds of target detection model training on the preset detection model based on the training set to obtain a preliminarily trained detection model. Use the validation set to evaluate the model performance and understand the model's performance on unknown data. If the model performance is not ideal, adjust the model parameters or fine-tune the model. Use the test set to test the model's generalization ability. If the model performance meets the requirements, use it as a target detection model to detect the mechanical structure of the open-type disconnector.

[0126] In one embodiment, moderate random noise (Gaussian noise) is added to the image, where the mathematical description of the Gaussian noise added by data enhancement is shown in the following formula:

[0127]

[0128] Here, x is a random variable and σ is the standard deviation.

[0129] In one embodiment, the classification loss function of the target detection model is as shown below, and a binary cross entropy loss is used for the result of each category prediction:

[0130]

[0131] Among them, L claSs is the classification loss, N represents the total number of categories, x i is the predicted value of the current category, y i is the probability of the current category after the activation function, is the true value of the current category;

[0132] The localization loss of this object detection model is as follows:

[0133] L local =(xx * ) 2 +(yy * ) 2 +(ww * ) 2 +(hh * ) 2

[0134] Among them, x, y, w, and h represent the coordinates of the upper left corner of the prediction box and the length and width of the box respectively.

[0135] The target detection model also includes losses of IoU, GIoU, DIoU and CIoU, as shown in the following formula:

[0136] L IoU =1-IoU(B1,B2)

[0137] in, B1 and B2 represent bounding box 1 and bounding box 2 respectively;

[0138]

[0139] Where C is the smallest outer rectangular area of ​​the two bounding boxes.

[0140]

[0141] Among them, b and b gtThey represent the center points of the predicted box and the real box respectively, ρ represents the Euclidean distance between the two center points, and c represents the diagonal length of the minimum circumscribed rectangle that can contain both the predicted box and the real box.

[0142]

[0143] Among them, α is the weight function and v is used to measure the consistency of aspect ratio.

[0144] In one embodiment, the initially trained detection model is tested based on the validation set data, and the performance indicators of the test mainly include accuracy, precision, recall, F1-score, and IoU (Intersection over Union) indicators. The mathematical formulas are as follows:

[0145]

[0146] In the above formula, TP (True Positives) represents the number of correctly detected targets; TN (True Negatives) represents the number of correctly undetected targets; FP (False Positives) represents the number of non-targets mistakenly detected as targets; and FN (False Negatives) represents the number of targets mistakenly not detected as targets.

[0147] In one embodiment, before obtaining a training data set of an open-type disconnector, a method for identifying a state of an open-type disconnector includes:

[0148] Acquire multiple original training images of the open-type disconnector, input each original training image into the target detection model to obtain a first training image of the open-type disconnector, and crop the first training image to obtain a second training image;

[0149] Based on the category of the open type disconnector in the second training image of each open type disconnector, obtaining question training information of the open type disconnector;

[0150] A second training image and its corresponding question training information are used as an image-text data pair, and a plurality of image-text data pairs constitute a training data set.

[0151] Specifically, multiple original training images of the open-type disconnector are input into the target detection model to obtain a first training image of the open-type disconnector, the first training image is cropped to obtain a second training image, the second training image and its corresponding question training information are used as a training sample, and the same method is used to obtain multiple training samples to form a training data set.

[0152] Since the forms of the open-type disconnector not opening and closing properly are different and infinite, it is difficult to effectively enumerate them, and only based on the visual model requires sufficient and accurate prior knowledge, which leads to the fact that the performance of the previous detection methods and models in this detection task is very limited. However, the present application is different from the previous methods and models. It uses the transfer reasoning ability of zero-shot learning of multimodal models to effectively deal with the infinite variety of situations that occur when the open-type disconnector is not in place, and gets rid of the past detection and identification of infinite types of opening and closing that were difficult to effectively deal with based on the visual mode alone.

[0153] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a device for identifying the state of an open type isolating switch, such as Figure 7 As shown, the device comprises:

[0154] The visual question answering model training module 702 is used to obtain a training data set of an open-type disconnector, and perform LoRa fine-tuning training on the pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information;

[0155] The target image acquisition module 704 is used to acquire an initial image of the open-type disconnector to be identified and question information, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified;

[0156] The recognition result acquisition module 706 is used to input the target image of the open-type disconnector to be identified and the question information into the trained visual question answering model to obtain the state recognition result of the open-type disconnector to be identified.

[0157] Compared with the prior art, the device for identifying the state of an open-type disconnector obtains a training data set including training images of the open-type disconnector and corresponding training question information, performs LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, performs feature recognition on the image of the open-type disconnector to be identified, obtains a target image containing only the open-type disconnector, inputs the target image of the open-type disconnector to be identified and the question information into the trained visual question answering model, and the trained visual question answering model outputs the recognition result of the state of the open-type disconnector, applies the visual question answering technology to the judgment of the opening and closing state of the open-type disconnector, and performs targeted training through LORA fine-tuning training, and based on multimodal deep learning technology, can quickly and accurately identify the opening and closing state of the open-type disconnector.

[0158] In one embodiment, the target image acquisition module is further used for:

[0159] Inputting the initial image of the open-type disconnector to be identified into the trained object detection model to obtain an image with the border and category of the open-type disconnector to be identified;

[0160] The image with the border and category of the open-type disconnector to be identified is cropped to obtain the target image.

[0161] In one embodiment, the visual question answering model training module is further used to:

[0162] Obtain a pre-trained visual question answering model, wherein the pre-trained visual question answering model includes a self-attention mechanism module;

[0163] Add the LORA fine-tuning module to the self-attention mechanism module of the pre-trained visual question answering model to obtain a fine-tuned visual question answering model;

[0164] The fine-tuning visual question answering model is trained based on the training data set to obtain a trained visual question answering model.

[0165] In one embodiment, the parameters of the LORA fine-tuning module include a dimensionality reduction matrix and a dimensionality increase matrix; the visual question answering model training module is also used to:

[0166] Initialize the dimension reduction matrix and dimension increase matrix;

[0167] Based on the initial weight parameters in the self-attention mechanism module, the initialized dimensionality reduction matrix and the initialized dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters, and the fine-tuned weight parameters are used as the initial weight parameters for the next iteration;

[0168] The dimension reduction matrix and the dimension increase matrix are updated to obtain an updated dimension reduction matrix and an updated dimension increase matrix;

[0169] Based on the initial weight parameters in the self-attention mechanism module, the updated dimensionality reduction matrix, and the updated dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters;

[0170] The fine-tuned weight parameters are used as the initial weight parameters for the next iteration, and the dimension reduction matrix and the dimension increase matrix are continuously updated to obtain new fine-tuned weight coefficients until the iteration ends, and the trained visual question answering model is obtained.

[0171] In one embodiment, the following method is used to fine-tune the initial weight parameters of the self-attention mechanism module:

[0172] W=W0+BA

[0173] Among them, W0 is the initial weight parameter generated by the attention mechanism module, A is the dimension reduction matrix, B is the dimension increase matrix, and W is the weight parameter generated by the attention mechanism module after fine-tuning.

[0174] In one embodiment, the open type isolating switch state identification device further comprises:

[0175] The target detection model training module is used to obtain multiple original training images of open-type disconnectors, and to mark each original training image with a box selection and category of the disconnector; divide the labeled multiple training images into a training data set, a verification data set, and a test data set; train a preset detection model based on the training data set to obtain a preliminarily trained detection model; test the preliminarily trained detection model based on the verification data set, and when the test fails, adjust the preliminarily trained detection model and retrain it until the test passes; test the detection model that passes the test based on the test data set, and after the test passes, obtain a trained target detection model.

[0176] In one embodiment, the open type isolating switch state identification device further comprises:

[0177] The visual question answering model training module is used to obtain multiple original training images of the open-type disconnector, input each original training image into the target detection model to obtain a first training image of the open-type disconnector, and crop the first training image to obtain a second training image; based on the category of the open-type disconnector in the second training image of each open-type disconnector, obtain question training information of the open-type disconnector; use a second training image and its corresponding question training information as an image-text data pair, and multiple image-text data pairs constitute a training data set.

[0178] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the open-type disconnector state identification method in any of the above method embodiments.

[0179] Figure 8 A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0180] like Figure 8 As shown, the computer device may include: a processor (processor) 802 , a communication interface (Communications Interface) 804 , a memory (memory) 806 , and a communication bus 808 .

[0181] The processor 802 , the communication interface 804 , and the memory 806 communicate with each other via a communication bus 808 .

[0182] The communication interface 804 is used to communicate with other devices such as clients or other servers.

[0183] The processor 802 is used to execute the program 810, and specifically can execute the relevant steps in the above-mentioned open type disconnector state identification method embodiment.

[0184] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0185] The processor 802 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0186] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0187] The program 810 may be specifically configured to enable the processor 802 to perform the following operations:

[0188] Obtain a training data set of an open-type disconnector, and perform LoRa fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information;

[0189] Acquire an initial image and question information of the open-type disconnector to be identified, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified;

[0190] The target image of the open-type disconnector to be identified and the question information are input into the trained visual question answering model to obtain a state recognition result of the open-type disconnector to be identified.

[0191] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. In one embodiment, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0192] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for identifying the state of an open type disconnector, characterized in that: include: Obtain a training data set of an open-type disconnector, and perform LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information; Acquire an initial image and question information of the open-type disconnector to be identified, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified; The target image of the open-type disconnector to be identified and the question information are input into the trained visual question answering model to obtain a state recognition result of the open-type disconnector to be identified.

2. The method for identifying the state of an open-type disconnector according to claim 1, characterized in that: Performing feature recognition on the initial image to obtain a target image of the open-type disconnector to be identified includes: Inputting the initial image of the open-type disconnector to be identified into the trained object detection model to obtain an image with the border and category of the open-type disconnector to be identified marked; The image with the frame and category of the open-type disconnector to be identified is cropped to obtain a target image.

3. The method for identifying the state of an open-type disconnector according to claim 1, characterized in that: Based on the training data set, the pre-trained visual question answering model is fine-tuned using LoRa to obtain a trained visual question answering model, including: Obtaining a pre-trained visual question answering model, wherein the pre-trained visual question answering model includes a self-attention mechanism module; Adding a LORA fine-tuning module to the self-attention mechanism module of the pre-trained visual question answering model to obtain a fine-tuned visual question answering model; The fine-tuning visual question answering model is trained based on the training data set to obtain a trained visual question answering model.

4. The method for identifying the state of an open-type disconnector according to claim 3, characterized in that: The parameters of the LORA fine-tuning module include a dimensionality reduction matrix and a dimensionality increase matrix; The fine-tuning and training visual question answering model is trained based on the training data set to obtain a trained visual question answering model, including: Initializing the dimension reduction matrix and the dimension increase matrix; Based on the initial weight parameters in the self-attention mechanism module, the initialized dimensionality reduction matrix and the initialized dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters, and the fine-tuned weight parameters are used as the initial weight parameters for the next iteration; The dimension reduction matrix and the dimension increase matrix are updated to obtain an updated dimension reduction matrix and an updated dimension increase matrix; Based on the initial weight parameters in the self-attention mechanism module, the updated dimensionality reduction matrix and the updated dimensionality increase matrix, the initial weight parameters of the self-attention mechanism module are fine-tuned to obtain the fine-tuned weight parameters; The fine-tuned weight parameters are used as the initial weight parameters for the next iteration, and the dimension reduction matrix and the dimension increase matrix are continuously updated to obtain new fine-tuned weight coefficients until the iteration ends, thereby obtaining a trained visual question answering model.

5. The method for identifying the state of an open-type disconnector according to claim 4, characterized in that: The initial weight parameters of the self-attention mechanism module are fine-tuned using the following method: W=W0+BA Among them, W0 is the initial weight parameter generated by the attention mechanism module, A is the dimension reduction matrix, B is the dimension increase matrix, and W is the weight parameter generated by the attention mechanism module after fine-tuning.

6. The method for identifying the state of an open-type disconnector according to claim 2, characterized in that: Before inputting the initial image of the open-type disconnector to be identified into the trained target detection model, the open-type disconnector state identification method further includes: Acquire multiple original training images of open-type disconnectors, and perform box selection and category marking of the disconnector on each of the original training images; Divide the labeled multiple training images into a training data set, a verification data set, and a test data set; Training a preset detection model based on the training data set to obtain a preliminarily trained detection model; Testing the preliminarily trained detection model based on the verification data set, and if the test fails, adjusting the preliminarily trained detection model and retraining it until the test passes; Based on the test data set, the detection model that has passed the detection is tested. After the test passes, a target detection model with a trained model is obtained.

7. The method for identifying the state of an open-type disconnector according to claim 6, characterized in that: Before obtaining the training data set of the open type disconnector, the open type disconnector state recognition method includes: Acquire multiple original training images of the open-type disconnector, input each of the original training images into the target detection model to obtain a first training image of the open-type disconnector, and crop the first training image to obtain a second training image; Based on the category of the open type disconnector in the second training image of each open type disconnector, obtaining question training information of the open type disconnector; A second training image and its corresponding question training information are used as an image-text data pair, and a plurality of image-text data pairs constitute a training data set.

8. An open type isolating switch state identification device, characterized in that: include: A visual question answering model training module is used to obtain a training data set of an open-type disconnector, and perform LORA fine-tuning training on a pre-trained visual question answering model based on the training data set to obtain a trained visual question answering model, wherein the training data set includes training images of the open-type disconnector and corresponding training question information; A target image acquisition module, used to acquire an initial image of the open-type disconnector to be identified and question information, perform feature recognition on the initial image, and obtain a target image of the open-type disconnector to be identified; The recognition result acquisition module is used to input the target image and question information of the open-type disconnector to be identified into the trained visual question answering model to obtain the state recognition result of the open-type disconnector to be identified.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, wherein the executable instruction enables a processor to execute an operation corresponding to the open-type disconnector state identification method according to any one of claims 1 to 7.

10. A computer device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the open-type disconnector state identification method according to any one of claims 1-7.

Images