Electric power image target detection method and system based on image-text model pre-training

Abstract

The invention discloses an electric power image target detection method and system based on image-text model pre-training, and belongs to the technical field of electric power vision and target detection, and the method specifically comprises the steps: carrying out electric power vision model pre-training based on an image-text model; performing fine tuning on the pre-trained power vision model in the power target detection scene to obtain a power vision model with optimal performance; and inputting the electric power scene image into the obtained electric power visual model to realize defect and component identification. According to the method, the model generalization capability can be improved, the marking cost is reduced, the small target detection capability is enhanced, the model robustness is improved, fine defects of equipment can be accurately identified, and the safety and reliability of a power system are improved.

Description

Background Art

[0002] In the early stage of deep learning, object detection tasks in the field of computer vision mainly relied on convolutional neural networks (CNNs). Two-stage models represented by the R-CNN series of models (including Fast R-CNN and Faster R-CNN) combined deep convolutional networks with region proposal networks (RPNs) to achieve an end-to-end object detection framework. One-stage models represented by the YOLO series of models predicted the category and location of objects through a single forward propagation, achieving a detection speed close to real-time. However, the above models have certain limitations in object detection tasks: they usually require a large amount of labeled data for training, and due to the limited receptive field of convolution, it is difficult to capture global context information.

[0003] With the development of deep learning technology, the introduction of the Transformer architecture has brought new development opportunities to the field of computer vision. The Transformer model initially achieved great success in the field of natural language processing. Its core mechanism, the self-attention mechanism, can capture global dependencies in images and provide more comprehensive context information for downstream tasks. Therefore, some vision Transformer models such as Vision Transformer (ViT) have been proposed and shown great application potential. In this development process, self-supervised pre-training methods have gradually become the mainstream paradigm for pre-training visual large models. Self-supervised pre-training can learn deeper internal structures and visual feature representations of data by using a large amount of unlabeled data for training. This pre-training method not only improves the generalization ability of the model but also reduces the dependence on a large amount of labeled data, enabling it to achieve better transfer learning effects on various downstream tasks. Common visual self-supervised pre-training methods include Masked Image Modeling (MIM), etc.

[0004] However, pure vision model architectures mainly focus on single-modal processing and do not fully utilize cross-modal information. The development of text-image multi-modal large models marks a shift from single-vision task processing to cross-modal information fusion, aiming to integrate text and vision processing capabilities to achieve deeper understanding and interaction. Such models align visual features with text features, fuse information from different modalities, and expand the understanding and adaptation capabilities of single modalities to complex scenarios.

[0005] Recently, some studies such as the EVA (Exploring the Limits of Masked Visual Representation Learning at Scale) model can enable visual models to learn high-level semantic features in cross-modal contrastive learning by using visual features in cross-modal models as prediction targets in masked image modeling. At the same time, the structural and geometric information of images can be captured in masked image modeling, thus obtaining good performance. Although these studies and experiments perform well in general visual tasks, their pre-training processes mainly rely on publicly accessible datasets and are not specifically optimized for power scenarios. Meanwhile, the EVA model uses all tokens in the encoder stage to predict the image features output by the teacher model, and the model training is still time-consuming. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for power image object detection based on pre-training of cross-modal models in view of the deficiencies in the above-mentioned prior art. By constructing a multi-modal joint pre-training framework, the present invention innovatively combines dynamic masked image modeling with a multi-scale slicing strategy to solve the technical problem that the pre-trained visual model relied on for object detection in the power field fails to fully integrate semantic feature information.

[0007] The present invention adopts the following technical solutions: A method for power image object detection based on pre-training of cross-modal models includes the following steps: Carry out pre-training of a power visual model based on a cross-modal model; Fine-tune the pre-trained power visual model in the power object detection scenario to obtain the power visual model with the optimal performance; Input the power scene image into the obtained power visual model to realize defect and component recognition.

[0008] Preferably, the pre-training of the power visual model based on the cross-modal model is specifically as follows: Divide the input image into multiple image patches in a predefined manner; For the student model, randomly mask the image based on a preset masking ratio, input the remaining visible image patches into the encoder to obtain the representation of the visible image patch tokens; then input the tokens of the visible image patches and the tokens of the masked image patches into the decoder to obtain the output of the decoder; For the teacher model, input all the image patches into the encoder of the image model in the cross-modal model to output the encoded image features.

[0009] Preferably, the learning objective of the student model includes predicting the corresponding position results of the masked image patches after normalizing the output image features of the teacher model.

[0010] Preferably, during pre-training, general images and power images are mixed in a set ratio.

[0011] Preferably, when the resolution of the image data exceeds the set range, first set the resolution of the slices; when one side length of the image is less than the resolution of the set slices, enlarge the short side of the picture to the set resolution, use the center of the picture as the center point of slicing, and slice with the side length being the resolution of the slices; Then enlarge the image by several times, use the center of the picture as the center point of slicing, and slice with the side length being the resolution of the slices; Execute the above steps several times, and randomly select several from the obtained slicing results as the image data for training.

[0012] Preferably, when one side length of the enlarged image still is less than the resolution of the set slices, re-enlarge the short side of the picture to the set resolution, use the center of the picture as the center point of slicing, and slice with the side length being the resolution of the slices.

[0013] Preferably, the number of slices is .

[0014] In a second aspect, an embodiment of the present invention provides a power image target detection system based on pre-training of a text-image model, including: A construction module for pre-training a power vision model based on the text-image model; A fine-tuning module for fine-tuning the pre-trained power vision model in a power target detection scenario to obtain a power vision model with optimal performance; A detection module for inputting a power scene image into the obtained power vision model to implement defect and component recognition.

[0015] Preferably, in the construction module, pre-training the power vision model based on the text-image model is specifically as follows: Divide the input image into multiple image blocks in a predefined manner; For the student model, randomly mask the image based on a preset masking ratio, input the remaining visible image blocks into the encoder to obtain the representation of the visible image block tokens; then input the tokens of the visible image blocks and the tokens of the masked image blocks into the decoder together to obtain the output of the decoder; For the teacher model, input all the image blocks into the encoder of the image model in the text-image model to output the encoded image features.

[0016] Preferably, the learning objective of the student model includes predicting the corresponding position results of the masked image blocks after normalizing the output image features of the teacher model.

[0017] Preferably, when the resolution of the image data exceeds the set range, first set the resolution of the slice; when one side length of the image is less than the resolution of the set slice, enlarge the short side of the picture to the set resolution, use the center of the picture as the center point of the slice, and slice with the side length being the resolution of the slice; Then enlarge the image by several times, use the center of the picture as the center point of the slice, and slice with the side length being the resolution of the slice; Execute the above steps several times, and randomly select several from the obtained slice results as the image data for training.

[0018] Preferably, when one side length of the enlarged image still is less than the resolution of the set slice, re-enlarge the short side of the picture to the set resolution, use the center of the picture as the center point of the slice, and slice with the side length being the resolution of the slice, and the number of slices is .

[0019] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned power image target detection method based on pre-training of the graphic model are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned power image target detection method based on pre-training of the graphic model are implemented.

[0021] In a fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned power image target detection method based on pre-training of the graphic model are implemented.

[0022] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, and when the computer program is executed by the electronic device, the steps of the above-mentioned power image target detection method based on pre-training of the graphic model are implemented.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: A power image object detection method based on pre-training of a text-image model uses the text-image model as a teacher model to guide the visual model for representation learning. By mixing a general dataset and a power dataset, the model pre-training is completed, making full use of the advantage of the text-image model guiding the pre-training of the visual model. At the same time, considering the characteristics of power data, corresponding training strategies are formulated, enabling the model to learn visual features and patterns specific to power scenarios. The model encoder after pre-training serves as the backbone network of the object detection model and is further fine-tuned in various power scenarios such as power transmission, power distribution, and power transformation. Finally, the trained model is used to perform object detection on new power image data to achieve automatic identification and analysis of power equipment and defects.

[0024] Furthermore, the model adopts a "encoder + decoder" structure. In the encoder stage, only visible tokens are calculated, and a lighter structure is selected for the decoder. The output of the decoder is used to predict the image features of the teacher model, and the model converges faster, thus accelerating model training.

[0025] Furthermore, by processing smaller image slices, the video memory occupation and computational amount are effectively reduced, and the training speed is improved. After the image is magnified at multiple levels and then sliced, it is beneficial to capture objects at different scales. Starting from the center of the image and slicing in four non-overlapping directions can ensure that the main objects are completely included in the slices, and non-overlapping slices can reduce redundant calculations.

[0026] It can be understood that the beneficial effects of the above second to sixth aspects can refer to the relevant descriptions in the above first aspect and will not be elaborated here.

[0027] In summary, the method of the present invention can improve the model generalization ability, reduce the annotation cost, enhance the small object detection ability, improve the model robustness, accurately identify the subtle defects of the equipment, and improve the safety and reliability of the power system.

[0028] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly introduced below. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0030] Figure 1 is the flow chart of the present invention; Figure 2 is the schematic diagram of a computer device provided by an embodiment of the present invention; Figure 3 Block diagram of an electronic device provided according to an embodiment of the present invention.

[0031] Wherein, 60. computer device; 61. processor; 62. memory; 63. computer program; 600. electronic device; 610. processing unit; 620. storage unit; 6201. random access storage unit; 6202. cache storage unit; 6203. read-only storage unit; 6204. program / utilities; 6205. program module; 630. bus; 640. display unit; 650. input / output interface; 660. network adapter; 700. external device. Detailed implementation manners

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] In the description of the present invention, it should be understood that the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0034] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0035] It should be further understood that the term " / and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the contextually related objects.

[0036] It should be understood that although terms such as first, second, and third may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0037] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (stated condition or event)" or "in response to detecting (stated condition or event)".

[0038] Schematic diagrams of various structures according to the disclosed embodiments of the present invention are shown in the drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures and their relative sizes and positional relationships are only exemplary, and may actually deviate due to manufacturing tolerances or technical limitations. And those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0039] The present invention provides a method for power image target detection based on pre-training of a text-image model. Using the text-image model as a teacher model to guide the representation learning of a visual model with an "encoder + decoder" structure, by mixing a general dataset and a power dataset, and adopting a slicing strategy for high-resolution images, the model pre-training is completed. The encoder of the pre-trained model is used as the backbone network of the target detection model and is further fine-tuned in various power scenarios such as power transmission, power distribution, and power transformation. Finally, the trained model is used to perform target detection on new power image data to achieve automatic recognition and analysis of power equipment and defects.

[0040] Embodiment 1 Please refer to Figure 1 , a method for power image target detection based on pre-training of a text-image model according to the present invention, includes the following steps: S1. Carry out pre-training of a power visual model based on the text-image model; Use the structure of "encoder + decoder" as the structure of the visual pre-training model.

[0041] For example, the structure of the power vision model can follow the structure described in the literature "Masked autoencoders are scalable vision learners", or other structures of encoders and decoders can be constructed according to actual needs. The teacher model is a trained text-image model, such as the publicly available CILP model in the industry, etc.

[0042] The pre-training adopts the method of masked image modeling, and the specific steps are as follows: First, divide the input image into multiple image patches according to a predefined method; Specifically as follows: First, determine the side length of the image patch, and then divide the input image into non-overlapping image patches. The side length of the input image is generally an integer multiple of the side length of the image patch.

[0043] For the student model, randomly mask the image based on a preset masking ratio, input the remaining visible image patches into the encoder to obtain the representation of the visible image patch tokens; the tokens of the visible image patches and the tokens of the masked image patches are input into a more lightweight decoder together to obtain the output of the decoder.

[0044] For the teacher model, input all the image patches into the encoder of the image model in the text-image model to output the encoded image features.

[0045] The learning objective of the student model must include the prediction of the corresponding position results of the masked image patches after normalizing the output image features of the teacher model. Optionally, it can also include the reconstruction of the original image pixels, etc.

[0046] When performing model pre-training, mix the general images and power images in a certain proportion.

[0047] According to the experimental results of the mixing ratio of the power vision samples and the general samples during pre-training, the power dataset is optimal at 15% - 20%.

[0048] At the same time, aiming at the disadvantages that high-resolution images occupy a lot of computing resources, slow down the training speed, and a single size is not conducive to capturing target features of different scales, an image slicing strategy is designed, specifically as follows: For images with a resolution exceeding a certain range, first set the resolution of the slices. If there is a side length of the image less than the set resolution of the slices, enlarge the short side of the picture to the set resolution, and use the center of the picture as the center point of the slices, and slice with the side length of the slice resolution; Then enlarge the image by several times. If there is still a side length of the enlarged image less than the set resolution of the slices, similar to the above, enlarge the short side of the picture to the set resolution.

[0049] Then, taking the center of the picture as the center point of slicing, slice with the side length being the slicing resolution, and the number of slices is ; Execute the above steps several times, and randomly select several from the obtained sliced results as the image data for training..

[0050] S2. Fine-tune based on the pre-trained model in the power object detection scenario; First, collect image data in the power scenario, such as data in the power transmission scenario, power distribution scenario, and substation scenario, etc.; Annotate the collected images to determine the positions and categories of the components or defects to be recognized in the images; Use the encoder of the pre-trained model in step S1 as the backbone network of the object detection algorithm, and then connect a object detection head, such as yolo-head, etc., as the overall object detection network; Divide the annotated image dataset into a training set and a validation set according to a ratio; generally, set the ratio of the training set to the validation set to 8:2 or 9:1. Among them, when the amount of fine-tuning samples is small, 9:1 is selected.

[0051] Set the hyperparameters in the fine-tuning process, carry out model training, regularly evaluate the performance of the model using the validation set, and save the model weights with the best performance.

[0052] Use the fine-tuning method based on the pre-trained model to design the power object detection task, aiming to efficiently construct a high-precision and high-robustness power object detection model with limited labeled data through transfer learning and domain adaptation.

[0053] S3. Use the power object detection algorithm for inference application.

[0054] Deploy the model with the evaluation performance meeting the requirements to the actual power object detection system. The power object detection system receives power scenario images and uses the object detection model in step S2 to identify defects and components.

[0055] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0056] Embodiment 2 The present invention provides a power image target detection system based on pre-training of a graphic-text model, which can be used to implement the above-mentioned power image target detection method based on pre-training of a graphic-text model. Specifically, the power image target detection system based on pre-training of a graphic-text model includes a construction module, a fine-tuning module, and a detection module.

[0057] Among them, the construction module conducts pre-training of a power vision model based on a graphic-text model; The fine-tuning module fine-tunes the pre-trained power vision model in a power target detection scenario to obtain the power vision model with the optimal performance; The detection module inputs a power scenario image into the obtained power vision model to achieve defect and component recognition.

[0058] Embodiment 3 The present invention provides a terminal device, which includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operations of the power image target detection method based on pre-training of a graphic-text model, including: Conducting pre-training of a power vision model based on a graphic-text model; fine-tuning the pre-trained power vision model in a power target detection scenario to obtain the power vision model with the optimal performance; inputting a power scenario image into the obtained power vision model to achieve defect and component recognition.

[0059] Please refer to Figure 2, the terminal device is a computer device. The computer device 60 in this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, it implements the power image target detection method based on text-image model pre-training in the embodiment. To avoid repetition, it will not be elaborated here one by one. Alternatively, when the computer program 63 is executed by the processor 61, it implements the functions of each model / unit in the power image target detection system based on text-image model pre-training in the embodiment. To avoid repetition, it will not be elaborated here one by one.

[0060] The computer device 60 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art can understand that Figure 2 merely examples of the computer device 60, which do not constitute a limitation on the computer device 60, may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0061] The so-called processor 61 may be a central processing unit (CPU), or may also be other general-purpose processors, a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0062] The memory 62 may be an internal storage unit of the computer device 60, such as the hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk equipped on the computer device 60, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0063] Further, the memory 62 may also include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0064] Please refer to Figure 3 , the terminal device is an electronic device 600, and the electronic device 600 is presented in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0065] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the method part of this specification. For example, the processing unit 610 may execute steps as shown in Figure 1 .

[0066] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0067] The storage unit 620 may also include a program / utilities 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0068] The bus 630 may represent one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0069] The electronic device 600 can also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device (such as a router, a modem) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication can be carried out through the input / output interface 650. Moreover, the electronic device 600 can also communicate with one or more networks (such as a local area network, a wide area network, and / or a public network, such as the Internet) through the network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0070] Embodiment 4 The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. It can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. And, in this storage space, one or more instructions suitable for being loaded and executed by a processor are also stored, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples of the computer-readable storage medium here include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0071] The computer-readable storage medium also includes a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, radio frequency, etc., or any suitable combination of the above.

[0072] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network or a wide area network, or can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0073] One or more instructions stored in the computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the power image target detection method based on text-image model pre-training in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows: Carry out pre-training of the power vision model based on the text-image model; fine-tune the pre-trained power vision model in the power target detection scenario to obtain the power vision model with the optimal performance; input the power scene image into the obtained power vision model to achieve defect and component recognition.

[0074] The databases involved in the embodiments provided by the present invention can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on blockchain, etc., and is not limited thereto. The processors involved in the embodiments provided by the present invention can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and are not limited thereto.

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0076] Embodiment 5 1.1. Pre-training stage of the model The mae-vit-base-p16 in the method described in the literature "Masked autoencoders are scalable vision learners" is used as the structure of the visual pre-training model: The encoder is stacked by 12 Transformer modules with 768 dimensions and 12 heads.

[0077] The decoder is stacked by 8 Transformer modules with 512 dimensions and 16 heads.

[0078] The teacher model is the visual encoder of EVA-CLIP-8B.

[0079] The general dataset used for pre-training is the dataset obtained from public networks and web crawlers. The power dataset is sourced from the power inspection image sample library.

[0080] 1.1.1. Randomly select image samples, scale and crop them to a size of , and input them into the model; The size of the image patches is set to . The image is divided into 14×14 image patches according to the image patch size. Based on each image patch, a linear mapping is performed to generate the corresponding feature vector, and positional encoding is added.

[0081] 1.1.2. For the student model, denote the masking ratio γ = 75%, that is, randomly select 75% of the image patches as masked image patches, and the remaining 25% as visible image patches; Input the corresponding visible image patch part in the output of 1.1.1 into the encoder of the pre-training model to obtain a 49×768-dimensional vector output by the encoder. Concatenate the output of the encoder with the masked token, add positional encoding, and then input it into the decoder to obtain a 14×14×512-dimensional vector output by the decoder.

[0082] 1.1.3. For the teacher model, the output of 1.1.1 is input into the visual encoder of EVA-CLIP-8B, and then through InstanceNorm and LayerNorm, a 14×14×1280-dimensional vector is obtained as the output; 1.1.4. The learning objective of the student model is to predict the output results at the corresponding positions of the masked image patches in 1.1.3; Perform a linear mapping on the vector output in 1.1.2 to obtain a vector of size 14×14×1280. Use the mean squared error (MSE) as the loss function to calculate the masked feature prediction loss.

[0083] 1.1.5. Before the start of pre-training, use the image slicing strategy for images with a short-side resolution exceeding 300.

[0084] First, set the resolution of the slices to 360×360. If one side length of the image is less than 360, enlarge the short side of the image to 360, and use the center of the image as the center point of the slice to perform slicing with a side length of 360; then enlarge the image by 2.5 times, and also use the center of the image as the center point of the slice to perform slicing with a side length of 360. The number of slices is ⌊image width / 360⌋×⌊image height / 360⌋. Randomly select 5 slices from the obtained slice results as the image data for training.

[0085] In this embodiment, a total of 60 million slices of the general dataset and 10 million slices of the power dataset are mixed as samples for masked image modeling self-supervised pre-training. The parameters of the self-supervised pre-training refer to the literature "Masked autoencoders are scalable vision learners", the batch size is 4096, and the self-supervised pre-training is iterated 300,000 times.

[0086] 1.2. Model fine-tuning stage Collect a total of 20,000 transmission scene images. Annotate the collected images to determine that the categories to be recognized in the images are "small-sized fittings - bolts missing pins", "small-sized fittings - improper installation", "small-sized fittings - non-standard installation", "large-sized fittings - suspension clamp offset", "large-sized fittings - damper skewed", "large-sized fittings - damper slipped", "large-sized fittings - damper fallen off".

[0087] Use the encoder of the pre-trained model in the model pre-training stage as the backbone network of the object detection algorithm, followed by an object detection head Simple FPN. The specific structure is consistent with the literature "Exploring Plain Vision Transformer Backbones for Object Detection", that is, on the feature map output by the backbone network, convolutions or transposed convolutions with strides of {2, 1, , } are used to generate feature maps with scales of { , , , } (the scale of the feature map output by the backbone network is ). Feature maps of different scales are input into the convolutional modules of to obtain object detection results of different scales, including object coordinate information (xc, yc, w, h), confidence information (prob), and class information (classname).

[0088] During fine-tuning, the labeled image dataset is divided into a training set and a validation set in a ratio of 9:1. AdamW is selected as the optimizer, and a total of 30 epochs are iterated. During the fine-tuning process, after each epoch of training, the performance of the model is evaluated using the validation set, and the model weights with the highest average mAP on the validation set are saved.

[0089] 1.3. Inference and Application Stage Select the model with the highest average mAP on the validation set as the final model and deploy it to the actual power object detection system. The power object detection system receives transmission scene images and outputs defect categories including "Small-sized fitting class - bolt missing pin", "Small-sized fitting class - improper installation", "Small-sized fitting class - non-standard installation", "Large-sized fitting class - suspension clamp offset", "Large-sized fitting class - damper skewed", "Large-sized fitting class - damper slipped", and "Large-sized fitting class - damper fallen off" in the images.

[0090] Example 6 This example is the same as Example 1 except for the following content: (1) The visual encoder of EVA-CLIP-18B is selected as the teacher model For the teacher model, the output of 1.1.1 is input into the visual encoder of EVA-CLIP-18B, and then through InstanceNorm and LayerNorm, a 14×14×1536-dimensional vector is obtained as the output.

[0091] (2)The goal of the student model learning is to predict the output results at the corresponding positions of the masked image patches in 1.1.3 and reconstruct the original image pixels. Perform a linear mapping on the vectors output in 1.1.2 to obtain vectors of size 14×14×1536 and 14×14×768. Use the mean squared error MSE as the loss function to calculate the masked feature prediction loss dec_loss and the mean squared error img_loss of the masked image pixel reconstruction respectively.

[0092] The final loss is 0.9×dec_loss + 0.1×img_loss.

[0093] (3)Model fine-tuning stage Collect a total of 10,000 distribution scene images. Annotate the collected images to determine that the categories to be recognized in the images are "insulator damage", "arrester tripping", "fuse missing insulation cover", and "tower top damage".

[0094] (4)Inference and application stage The power object detection system receives the distribution scene image and outputs the defect categories of "insulator damage", "arrester tripping", "fuse missing insulation cover", and "tower top damage" contained in the image.

[0095] In summary, a power image object detection method and system based on text-image model pre-training according to the present invention fully utilizes the advantages of the text-image model to guide the pre-training of the visual model, and at the same time considers the characteristics of power data to formulate corresponding training strategies, enabling the model to learn the specific visual features and patterns of the power scene. In addition, the model adopts the "encoder + decoder" structure. In the encoder stage, only the visible tokens are calculated, and a lighter structure is selected for the decoder. The output of the decoder is used to predict the image features of the teacher model, and the model converges faster, thus accelerating the model training.

[0096] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0097] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in the form of hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0099] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the device or unit can be in electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, in each embodiment of the present invention, each functional unit may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0102] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the above-mentioned method embodiments of the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0103] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for realizing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the process in Figure 1One process or multiple processes and / or boxes Figure 1 The functions specified in one box or multiple boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process or multiple processes and / or boxes Figure 1 One process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one box or multiple boxes.

[0106] The above content is only to illustrate the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention fall within the protection scope of the claims of the present invention.

Claims

1. A method for detecting power image targets based on pre-training of image-text models, characterized in that: The following steps are involved: Carry out pre-training of electric power vision model based on graphic model; Fine-tune the pre-trained power vision model in the power target detection scenario to obtain the power vision model with the best performance; The power scene image is input into the obtained power vision model to realize defect and component identification.

2. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 1, characterized in that: The specific details of pre-training the power vision model based on the graphic model are as follows: Divide the input image into a plurality of image blocks according to a predefined method; For the student model, the image is randomly masked based on a pre-set mask ratio, and the remaining visible image blocks are input into the encoder to obtain the representation of the visible image block token; then the token of the visible image block and the token of the masked image block are input into the decoder together to obtain the output of the decoder; For the teacher model, all image blocks are input into the encoder of the image model in the graph-text model, and the encoded image features are output.

3. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 2 is characterized in that: The learning objective of the student model includes the prediction of the mask image patch for the corresponding position result after normalization of the teacher model output image features.

4. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 2, characterized in that: During pre-training, general images and power images are mixed in a set ratio.

5. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 2, characterized in that: When the resolution of the image data exceeds the set range, the resolution of the slice is set first; when the length of one side of the image is less than the set slice resolution, the short side of the image is enlarged to the set resolution, and the image center is used as the slice center point, and the slice is performed with the side length as the slice resolution; Then the image is enlarged several times, and sliced ​​with the center of the image as the center point of the slice and the side length as the slice resolution; The above steps are performed several times, and several images are randomly selected from the obtained slice results as image data for training.

6. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 5, characterized in that: When one of the sides of the enlarged image is still shorter than the set slice resolution, the short side of the image is re-enlarged to the set resolution, and the image center is used as the slice center point and the side length is the slice resolution.

7. The method for detecting electric power image targets based on pre-training of graphic and text models according to claim 5, characterized in that: The number of slices is .

8. A power image target detection system based on image-text model pre-training, characterized in that: include: Building modules to pre-train electric power vision models based on graphic models; The fine-tuning module fine-tunes the pre-trained power vision model in the power target detection scenario to obtain the power vision model with the best performance; The detection module inputs the power scene image into the obtained power vision model to realize defect and component identification.

9. The electric power image target detection system based on image-text model pre-training according to claim 8, characterized in that: In the construction module, the pre-training of the electric power vision model based on the graphic model is as follows: Divide the input image into a plurality of image blocks according to a predefined method; For the student model, the image is randomly masked based on a pre-set mask ratio, and the remaining visible image blocks are input into the encoder to obtain the representation of the visible image block token; then the token of the visible image block and the token of the masked image block are input into the decoder together to obtain the output of the decoder; For the teacher model, all image blocks are input into the encoder of the image model in the graph-text model, and the encoded image features are output.

10. The electric power image target detection system based on image-text model pre-training according to claim 9, characterized in that: The learning objective of the student model includes the prediction of the mask image patch for the corresponding position result after normalization of the teacher model output image features.

11. The electric power image target detection system based on image-text model pre-training according to claim 9, characterized in that: When the resolution of the image data exceeds the set range, the resolution of the slice is set first; when the length of one side of the image is less than the set slice resolution, the short side of the image is enlarged to the set resolution, and the image center is used as the slice center point, and the slice is performed with the side length as the slice resolution; Then the image is enlarged several times, and sliced ​​with the center of the image as the center point of the slice and the side length as the slice resolution; The above steps are performed several times, and several images are randomly selected from the obtained slice results as image data for training.

12. The electric power image target detection system based on image-text model pre-training according to claim 11, characterized in that: When one of the sides of the enlarged image is still smaller than the set slice resolution, the short side of the image is enlarged to the set resolution again, and the image center is used as the slice center point, and the side length is the slice resolution. The number of slices is .

13. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 7.

14. A computing device, characterized in that include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 7.

Images