Insulator state marking method and device, storage medium and electronic equipment

Abstract

The invention provides an insulator state marking method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a visible light reference image of a power transmission line of each monitoring segment when the power transmission line triggers an insulator fault, and obtaining an infrared image which is in space-time alignment with the visible light reference image; inputting the aligned multi-modal data into a dynamic mutual identification decision engine to obtain a first-order label, and obtaining environment audio data which is in space-time alignment with the visible light reference image; and correcting the first-order annotation according to the environment audio data to obtain a target annotation. According to the method, the multi-modal data is processed through the dynamic mutual identification decision engine, the insulator state annotation is quickly generated, and the first-order annotation is corrected according to the environment audio data, so that the target annotation with higher accuracy is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of safety monitoring, in particular to an insulator state labeling method and device, a storage medium and an electronic equipment. BACKGROUND

[0002] The power industry is the most important basic energy industry in economic development, and is a basic industry related to people's livelihood. As an advanced productive force and basic industry, the power industry is not only a strategic problem related to economic security, but also closely related to people's daily life and social stability.

[0003] The insulator can play an important role in supporting the conductor and preventing the current from returning to the ground in the power transmission line. Only when these two functions are guaranteed, can the use and operation safety of the entire power transmission line be ensured.

[0004] The length of the power transmission line is very long, and the number of insulators deployed in the power transmission line is also very large, which brings great difficulty to the inspection work of the insulator and becomes a problem that technicians in the field are concerned about. SUMMARY

[0005] The purpose of the present application is to provide an insulator state labeling method, device, storage medium and electronic equipment to improve the above problems.

[0006] In order to achieve the above purpose, the technical scheme adopted by the embodiments of the present application is as follows: In a first aspect, the embodiments of the present application provide an insulator state labeling method, which comprises: When the insulator fault of the power transmission line is triggered, visible light reference images of each monitoring section power transmission line are obtained, wherein the monitoring section power transmission line is a power transmission line with an insulator deployed; An infrared image that is spatiotemporally aligned with the visible light reference image is obtained; The aligned multi-modal data is input into a dynamic mutual certification decision engine to obtain a primary labeling, wherein the aligned multi-modal data includes the visible light reference image and the infrared image that is spatiotemporally aligned therewith, and the primary labeling includes state recognition results of each insulator in the monitoring section power transmission line and corresponding confidence levels, and the state recognition results include overheat anomaly recognition results and suspected crack texture recognition results; Environment audio data that is spatiotemporally aligned with the visible light reference image is obtained; The primary labeling is corrected according to the environment audio data to obtain a target labeling.

[0007] In a second aspect, the embodiments of the present application provide an insulator state labeling device, which comprises: The first processing unit is configured to acquire a visible light reference image of each monitoring section of the power transmission line when the power transmission line triggers an insulator fault, wherein the monitoring section of the power transmission line is the power transmission line on which an insulator is arranged; The first processing unit is further configured to acquire an infrared image that is spatiotemporally aligned with the visible light reference image; The first processing unit is further configured to input the aligned multi-modal data into a dynamic mutual certification decision engine to obtain a primary annotation, wherein the aligned multi-modal data includes the visible light reference image and the infrared image that is spatiotemporally aligned with the visible light reference image, and the primary annotation includes a state recognition result of each insulator in the monitoring section of the power transmission line and a corresponding confidence level, and the state recognition result includes an overheating anomaly recognition result and a suspected crack texture recognition result. The first processing unit is further configured to acquire environmental audio data that is spatiotemporally aligned with the visible light reference image; The second processing unit is configured to correct the primary annotation according to the environmental audio data to obtain a target annotation.

[0008] In a third aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method described above.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, and the memory is configured to store one or more programs; when the one or more programs are executed by the processor, the method described above is implemented.

[0010] Compared with the prior art, the insulator state annotation method, device, storage medium and electronic device provided by the embodiment of the present application acquire a visible light reference image of each monitoring section of the power transmission line when the power transmission line triggers an insulator fault, acquire an infrared image that is spatiotemporally aligned with the visible light reference image, input the aligned multi-modal data into a dynamic mutual certification decision engine to obtain a primary annotation, acquire environmental audio data that is spatiotemporally aligned with the visible light reference image, and correct the primary annotation according to the environmental audio data to obtain a target annotation. The multi-modal data is processed by the dynamic mutual certification decision engine to quickly generate the insulator state annotation, and the primary annotation is corrected according to the environmental audio data to obtain the target annotation with higher accuracy.

[0011] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are referred to. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0013] Figure 1 The structural schematic diagram of the electronic device provided by the embodiments of the present application.

[0014] Figure 2 The flowchart of the insulator state labeling method provided by the embodiments of the present application.

[0015] Figure 3 The unit schematic diagram of the insulator state labeling device provided by the embodiments of the present application.

[0016] In the figure: 10-processor; 11-memory; 12-bus; 13-communication interface; 701-first processing unit; 702-second processing unit. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art on the basis of the embodiments in the present application without creative labor are within the scope of the present application.

[0019] It should be noted that: similar labels and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second" and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0020] It should be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0021] In the description of the present application, it should be pointed out that the terms "upper", "lower", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0022] In the description of the present application, it should be further pointed out that unless otherwise explicitly specified and limited, the terms "arrangement", "connection" should be understood broadly, for example, it can be fixedly connected, or detachably connected, or integrally connected; it can be mechanically connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0023] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following examples and features in the examples can be combined with each other without conflict.

[0024] The electronic device provided by the embodiments of the present application can be a mobile phone device, a computer device, a server device, etc. Please refer to Figure 1 , a structural schematic diagram of the electronic device. The electronic device includes a processor 10, a memory 11 and a bus 12. The processor 10 and the memory 11 are connected through the bus 12, and the processor 10 is used to execute the executable modules stored in the memory 11, such as computer programs.

[0025] The processor 10 can be an integrated circuit chip with signal processing capability. In implementation, the steps of the insulator state labeling method can be completed by integrated logic circuits of hardware in the processor 10 or instructions in the form of software. The processor 10 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0026] The memory 11 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, for example, at least one disk memory.

[0027] The bus 12 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. Figure 1 Only one bidirectional arrow is used to represent the bus 12, but it does not mean that there is only one bus 12 or only one type of bus 12.

[0028] The memory 11 is used to store programs, for example, programs corresponding to the insulator state labeling device. The insulator state labeling device includes at least one software function module which can be stored in the memory 11 in the form of software or firmware or solidified in the operating system (OS) of the electronic device. After receiving an execution instruction, the processor 10 executes the program to implement the insulator state labeling method.

[0029] Possibly, the electronic device provided by the embodiment of the present application further includes a communication interface 13. The communication interface 13 is connected with the processor 10 through the bus.

[0030] It should be understood that, Figure 1 The structure shown is only a structural schematic diagram of part of the electronic device, and the electronic device can further include more components than those shown in the figure.Figure 1 more or less components than those shown, or with components arranged in different configurations and / or orders. Figure 1 Figure 1 The components shown in the various embodiments can be implemented in hardware, software, or a combination thereof.

[0031] The insulator state labeling method provided by the embodiment of the present application can be applied to, but is not limited to, the electronic device shown in the embodiment, and the specific process is shown in the following Figure 1 Figure 2 The insulator state labeling method comprises the following steps S10 to S50, which are specifically described as follows.

[0032] S10, acquiring a visible light reference image of each monitoring section power line when an insulator fault of the power line is triggered.

[0033] The monitoring section power line is a power line on which an insulator is deployed.

[0034] Optionally, S10, acquiring a visible light reference image of each monitoring section power line when an insulator fault of the power line is triggered, comprises the following step S101, which is specifically described as follows.

[0035] S101, taking a frame with the largest difference in visible light images in an observation time period before a time point when the insulator fault is triggered as the visible light reference image.

[0036] The largest difference here refers to the difference between the visible light images in the normal state, or refers to the difference between other visible light images in the observation time period.

[0037] S20, acquiring an infrared image that is spatiotemporally aligned with the visible light reference image.

[0038] Optionally, S20, acquiring an infrared image that is spatiotemporally aligned with the visible light reference image, comprises the following steps S201 and S202, which are specifically described as follows.

[0039] S201, acquiring a first infrared image and a second infrared image, the first infrared image being an infrared image generated earlier than the visible light reference image and closest in time to the generation time of the visible light reference image, and the second infrared image being an infrared image generated later than the visible light reference image and closest in time to the generation time of the visible light reference image.

[0040] S202, performing interpolation processing on the first infrared image and the second infrared image according to a first time interval and a second time interval to obtain the infrared image that is spatiotemporally aligned with the visible light reference image. The first time interval is an interval between the generation time of the first infrared image and the generation time of the visible light reference image, and the second time interval is an interval between the generation time of the second infrared image and the generation time of the visible light reference image.

[0041] ​​The interpolation processing can be, but is not limited to, linear interpolation processing.

[0042] Optionally, S20, acquiring the infrared image that is spatio-temporally aligned with the visible light reference image, comprises: S203, which is specifically described as follows.

[0043] S203, taking the infrared image whose generation time is closest to the generation time of the visible light reference image as the infrared image that is spatio-temporally aligned with the visible light reference image.

[0044] S30, inputting the aligned multi-modal data into a dynamic mutual verification decision engine to obtain the primary annotation.

[0045] The aligned multi-modal data comprises the visible light reference image and the infrared image that is spatio-temporally aligned therewith, and the primary annotation comprises state recognition results of each insulator in the monitoring section and corresponding confidence levels, wherein the state recognition results comprise overheat anomaly recognition results and suspected crack texture recognition results.

[0046] Optionally, S30, inputting the aligned multi-modal data into a dynamic mutual verification decision engine to obtain the primary annotation, comprises: S301, S302, S303, and S304, which are specifically described as follows.

[0047] S301, determining the position coordinates of each insulator in the visible light reference image.

[0048] S302, performing crack recognition on the visible light reference image based on the position coordinates of the insulators to obtain the suspected crack texture recognition results of the insulators and corresponding confidence levels.

[0049] S303, determining the infrared hot spots corresponding to the position coordinates of the insulators based on the coordinate mapping relationship between the visible light reference image and the infrared image.

[0050] S304, obtaining the overheat anomaly recognition results of the insulators and corresponding confidence levels based on the infrared hot spots.

[0051] S40, acquiring environmental audio data that is spatio-temporally aligned with the visible light reference image.

[0052] Optionally, S40, acquiring environmental audio data that is spatio-temporally aligned with the visible light reference image, comprises: S401 and S402, which are specifically described as follows.

[0053] S401, acquiring first environmental audio data and second environmental audio data, wherein the first environmental audio data is environmental audio data that is generated earlier than the visible light reference image and has a generation time closest to that of the visible light reference image, and the second environmental audio data is environmental audio data that is generated later than the visible light reference image and has a generation time closest to that of the visible light reference image.

[0054] S402, interpolating the first ambient audio data and the second ambient audio data according to the third time interval and the fourth time interval to obtain the ambient audio data spatio-temporally aligned with the visible light reference image.

[0055] The third time interval is an interval between a generation time of the first ambient audio data and a generation time of the visible light reference image, and the fourth time interval is an interval between a generation time of the second ambient audio data and the generation time of the visible light reference image.

[0056] S50, correcting the primary annotation according to the ambient audio data to obtain the target annotation.

[0057] Optionally, S50, the correcting the primary annotation according to the ambient audio data to obtain the target annotation comprises: S501, S502 and S503, which are specifically described as follows.

[0058] S501, when the primary annotation indicates that the insulator has an overheating anomaly, determining a wind vibration superposition amplitude corresponding to the ambient audio data.

[0059] S502, determining a crack superposition amplitude corresponding to the suspected crack texture recognition result and the confidence thereof.

[0060] S503, correcting the overheating anomaly recognition result according to the wind vibration superposition amplitude and the crack superposition amplitude to obtain the target annotation.

[0061] Here, the suspected crack texture recognition result can not be adjusted.

[0062] In the insulator state annotation provided by the embodiment of the present application, the multi-modal data is processed by the dynamic mutual verification decision engine to quickly generate the insulator state annotation (the primary annotation), and the primary annotation is corrected according to the ambient audio data to obtain the target annotation with higher accuracy.

[0063] Optionally, after the primary annotation is corrected, the correction record is fed back to the dynamic mutual verification decision engine to update the driving weight matrix used to generate the primary annotation. The correction record comprises the primary annotation, the target annotation, the visible light reference image, the infrared image spatio-temporally aligned therewith, and the ambient audio data.

[0064] The correction record is abstracted as an event vector and stored in an event graph for subsequent analysis and reference.

[0065] Please refer to Figure 3 , Figure 3 The insulator state annotation device provided by the embodiment of the present application can be applied to the electronic device described above.

[0066] The insulator state labeling device comprises a first processing unit 701 and a second processing unit 702.

[0067] The first processing unit 701 is configured to acquire a visible light reference image of each monitoring section power line when the insulator fault of the power line is triggered, wherein the monitoring section power line is a power line on which an insulator is arranged. The first processing unit 701 is further configured to acquire an infrared image that is spatiotemporally aligned with the visible light reference image. The first processing unit 701 is further configured to input the aligned multi-modal data into a dynamic mutual certification decision engine to obtain a primary labeling, wherein the aligned multi-modal data comprises the visible light reference image and the infrared image that is spatiotemporally aligned with the visible light reference image, and the primary labeling comprises a state recognition result of each insulator in the monitoring section power line and a corresponding confidence level, and the state recognition result comprises an overheating abnormality recognition result and a suspected crack texture recognition result. The first processing unit 701 is further configured to acquire environmental audio data that is spatiotemporally aligned with the visible light reference image. The second processing unit 702 is configured to correct the primary labeling according to the environmental audio data to obtain a target labeling.

[0068] It should be noted that the insulator state labeling device provided in the embodiment can execute the method processes shown in the method process embodiments to achieve the corresponding technical effects. For brevity, the part of the embodiment not mentioned can be referred to the corresponding content in the above embodiments.

[0069] The embodiment of the present application further provides a storage medium that stores computer instructions and programs, which, when read and executed, execute the insulator state labeling method of the above embodiments. The storage medium can include memory, flash memory, register, or a combination thereof.

[0070] The following provides an electronic device, which can be a mobile phone device, a computer device, a server device, etc. The electronic device can implement the above insulator state labeling method, as shown in the following figure. Figure 1 The electronic device comprises a processor 10, a memory 11, and a bus 12. The processor 10 can be a CPU. The memory 11 is configured to store one or more programs, which, when executed by the processor 10, execute the insulator state labeling method of the above embodiments.

[0071] In summary, the insulator state labeling method and device, storage medium and electronic equipment provided by the embodiments of the present application obtain visible light reference images of each monitoring section of the power transmission line when the power transmission line triggers an insulator fault, and obtain infrared images that are spatiotemporally aligned with the visible light reference images; input the aligned multi-modal data into a dynamic mutual verification decision engine to obtain primary labeling, obtain environmental audio data that is spatiotemporally aligned with the visible light reference images; and correct the primary labeling according to the environmental audio data to obtain target labeling. The multi-modal data is processed by the dynamic mutual verification decision engine to quickly generate insulator state labeling, and the primary labeling is corrected according to the environmental audio data to obtain target labeling with higher accuracy.

[0072] The above merely provides preferred embodiments of the present application but should not be used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall fall within the scope of the present application.

[0073] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Thus, the embodiments should be considered in all aspects as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be considered as limiting the involved claims.

Claims

1. An insulator state labeling method, characterized by, The method comprises: When an insulator fault of a power transmission line is triggered, visible light reference images of each monitoring section of the power transmission line are acquired, wherein the monitoring section of the power transmission line is a power transmission line on which an insulator is arranged; Infrared images that are spatiotemporally aligned with the visible light reference images are acquired; The aligned multi-modal data are input into a dynamic mutual verification decision engine to obtain a primary annotation, wherein the aligned multi-modal data comprise the visible light reference images and the infrared images that are spatiotemporally aligned therewith, and the primary annotation comprises state recognition results of each insulator in the monitoring section of the power transmission line and corresponding confidence levels, wherein the state recognition results comprise overheat anomaly recognition results and suspected crack texture recognition results; Environment audio data that are spatiotemporally aligned with the visible light reference images are acquired; The primary annotation is corrected according to the environment audio data to obtain a target annotation.

2. The insulator condition marking method of claim 1, wherein When an insulator fault of a power transmission line is triggered, visible light reference images of each monitoring section of the power transmission line are acquired, comprising: A frame of visible light image in an observation time period before a time point at which the insulator fault is triggered is taken as the visible light reference image, wherein the frame of visible light image has the largest difference.

3. The insulator condition marking method of claim 1, wherein The infrared images that are spatiotemporally aligned with the visible light reference images are acquired, comprising: A first infrared image and a second infrared image are acquired, wherein the first infrared image is an infrared image that is generated earlier than the visible light reference image and has a generation time closest to that of the visible light reference image, and the second infrared image is an infrared image that is generated later than the visible light reference image and has a generation time closest to that of the visible light reference image; The first infrared image and the second infrared image are subjected to interpolation processing according to a first time interval and a second time interval to obtain the infrared images that are spatiotemporally aligned with the visible light reference images; The first time interval is an interval between the generation time of the first infrared image and the generation time of the visible light reference image, and the second time interval is an interval between the generation time of the second infrared image and the generation time of the visible light reference image.

4. The insulator condition marking method of claim 1, wherein The infrared images that are spatiotemporally aligned with the visible light reference images are acquired, comprising: An infrared image that has a generation time closest to that of the visible light reference image is taken as the infrared image that is spatiotemporally aligned with the visible light reference image.

5. The insulator condition marking method of claim 1, wherein The aligned multi-modal data are input into a dynamic mutual verification decision engine to obtain a primary annotation, comprising: Coordinates of each insulator position in the visible light reference image are determined; Crack recognition is performed on the visible light reference image based on the coordinates of the insulator positions to acquire suspected crack texture recognition results of the insulators and corresponding confidence levels; Based on a coordinate mapping relationship between the visible light reference image and the infrared image, infrared hot spots corresponding to the coordinates of the insulator positions are determined; Overheat anomaly recognition results of the insulators and corresponding confidence levels are acquired based on the infrared hot spots.

6. The insulator condition marking method of claim 1, wherein The environment audio data that are spatiotemporally aligned with the visible light reference images are acquired, comprising: obtaining first ambient audio data and second ambient audio data, the first ambient audio data being ambient audio data generated closest in time before the visible light reference image, and the second ambient audio data being ambient audio data generated closest in time after the visible light reference image; interpolating the first ambient audio data and the second ambient audio data according to a third time interval and a fourth time interval to obtain ambient audio data spatio-temporally aligned with the visible light reference image; the third time interval being an interval between a generation time of the first ambient audio data and a generation time of the visible light reference image, and the fourth time interval being an interval between a generation time of the second ambient audio data and the generation time of the visible light reference image.

7. The insulator condition marking method of claim 1, wherein The correcting the preliminary annotation according to the ambient audio data to obtain a target annotation comprises: when the preliminary annotation indicates that the insulator has an overheating anomaly, determining a wind vibration superposition amplitude corresponding to the ambient audio data; determining a crack superposition amplitude corresponding to the crack texture identification result and the confidence thereof; correcting the overheating anomaly identification result according to the wind vibration superposition amplitude and the crack superposition amplitude to obtain the target annotation.

8. An insulator condition labeling apparatus characterized by comprising: The apparatus comprises: a first processing unit configured to, when an insulator fault is triggered on a power transmission line, obtain a visible light reference image of each monitoring section of the power transmission line, wherein the monitoring section of the power transmission line is a power transmission line on which an insulator is arranged; the first processing unit is further configured to obtain an infrared image spatio-temporally aligned with the visible light reference image; the first processing unit is further configured to input the aligned multi-modal data into a dynamic mutual verification decision engine to obtain a preliminary annotation, wherein the aligned multi-modal data comprises the visible light reference image and the infrared image spatio-temporally aligned therewith, and the preliminary annotation comprises a state identification result of each insulator in the monitoring section of the power transmission line and a corresponding confidence, the state identification result comprising an overheating anomaly identification result and a suspected crack texture identification result; the first processing unit is further configured to obtain ambient audio data spatio-temporally aligned with the visible light reference image; a second processing unit configured to correct the preliminary annotation according to the ambient audio data to obtain a target annotation.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-7.

10. An electronic device, comprising: comprises: a processor and a memory for storing one or more programs; when the one or more programs are executed by the processor, the method of any one of claims 1-7 is implemented.

Images