Semantic and visual information fusion-based high-risk operation scene identification and early warning method

Abstract

The invention relates to the technical field of operation scene recognition and early warning, in particular to a high-risk operation scene recognition and early warning method based on semantic and visual information fusion. The method comprises the following steps: relying on an edge computing device, combining a high-risk operation scene research and judgment rule, a confidence index and an identification result of a high-risk operation scene feature identification model, and uploading the identification result; the invention aims to provide a high-risk operation scene identification and early warning method based on semantic and visual information fusion so as to solve the problem that the existing artificial intelligence technology is low in high-risk operation scene identification precision.

Description

Technical Field

[0001] The present invention relates to the technical field of operation scene recognition and early warning, and in particular to a high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information. Background Art

[0002] The rapid development and widespread application of new-generation information technologies such as artificial intelligence and the Internet of Things across various industries have become the core driving force behind the digital and intelligent transformation and upgrading of power production operation safety management and control. Among them, the on-site intelligent safety management and control method based on target detection algorithms, which can identify and warn of operational violations, has become the mainstream model.

[0003] However, while AI technologies like target detection offer high accuracy for single-scene, single-target recognition, the accuracy of identifying operator violations decreases significantly when faced with the complex and ever-changing power operation environments, where factors such as personnel and machinery flow and occlusions are intertwined. This leads to a large number of false alarms, increasing the workload for safety inspectors in data investigation and violation confirmation. This is particularly true for high-risk operations such as those involving high-altitude work and large machinery. Patents for intelligent site safety management and control based on machine learning, deep learning, and cloud-edge collaboration have emerged. Most of these patents focus on identifying on-site violations or improving their accuracy by using different algorithms, improving algorithms, or integrating various sensing terminals. The intention is to achieve "large and comprehensive" violation identification in all professional operation scenarios such as power transmission, transformation, and distribution through artificial intelligence algorithms. The final result is that the identification effect is good only in some scenarios. The precision of on-site safety management is still based on "human management of safety" and supplemented by artificial intelligence technology. However, when there is a need to further improve the recognition accuracy of artificial intelligence technology for high-risk operation scenarios, reduce the number of inspectors who repeatedly and unfocusedly inspect hundreds of operation plans every day, and let limited inspection resources focus on high-risk operation scenarios and high-risk operation moments, empower the grassroots to reduce burdens and increase efficiency, and ensure on-site operation safety, it is a research matter of great significance.

[0004] Therefore, how to design a high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information, focusing on the recognition of high-risk operation scenes at power operation sites, through in-depth analysis of high-risk operation scene characteristics, refinement of operation scene analysis and judgment rules, use of target detection algorithm to analyze on-site video images, combined with the semantic analysis results of work ticket operation content, introduce confidence indicators to evaluate the operation scene recognition results, when the result is greater than the predetermined threshold, the recognition result will be pushed to the cloud platform to remind inspectors to focus on timely remote inspections, avoid inspectors indiscriminately conducting remote video inspections of each operation site, and improve inspection efficiency. Summary of the Invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information, so as to solve the problem that the existing artificial intelligence technology has low accuracy in identifying high-risk operation scenes.

[0006] To achieve the above object, the present invention adopts the following technical solution: comprising the following steps:

[0007] Step S1: Obtain a high-risk operation work ticket and pre-process it to remove spaces, punctuation marks, and words without entity meaning to obtain the corresponding text data;

[0008] Step S2: Manually extract characteristic keywords with operation scenario category information from the corresponding text data and construct a high-risk operation vocabulary;

[0009] Step S3: Acquire high-risk operation site pictures and pre-process them to obtain corresponding picture sample data;

[0010] Step S4: constructing a picture sample database according to the picture sample data representation result;

[0011] Step S5: Formulate high-risk operation scenario analysis and judgment rules by combining the high-risk operation vocabulary and the image sample database;

[0012] Step S6: A confidence index calculation mechanism for high-risk operation scene recognition results is obtained by training through semantic analysis of high-risk operation work tickets and analysis of operation video images at high-risk operation sites;

[0013] Step S7: Based on the labeled image sample data representation results, the YOLOx-s algorithm is used to train a feature recognition model for power high-risk operation scenarios;

[0014] Step S8: Relying on the edge computing device, combined with the high-risk operation scenario analysis rules, confidence indicators and the recognition results of the high-risk operation scenario feature recognition model, the recognition results are uploaded.

[0015] Furthermore, the step S3: obtaining high-risk operation site pictures and performing pre-processing, the pre-processing is specifically as follows:

[0016] For on-site image data, the LabelImg image annotation tool is used to label the basic feature targets of the power operation site and the feature targets of high-risk operation scenes in the images.

[0017] Furthermore, the step S5: combining the high-risk operation vocabulary and the image sample database to formulate high-risk operation scenario analysis and judgment rules, specifically as follows:

[0018] Based on the high-risk operation vocabulary, parse the corresponding text data to determine whether there are high-risk operation scenario keywords;

[0019] An image target detection algorithm is used to identify and return data information in the corresponding image data, and secondary reasoning based on business rules is used to determine whether there are characteristic targets in high-risk operation scenarios.

[0020] Furthermore, in step S6: "calculating the confidence index of the high-risk operation scene identification result", the confidence calculation formula is as follows:

[0021] cfi=wk*W wk +img*W img

[0022] Among them, W is the weight, which is a value between 0 and 1. wk With W img The sum is 1;

[0023] The calculation formula of wk is as follows:

[0024] wk=t i

[0025]

[0026] Among them, i represents the number of high-risk operation feature words, and the weight value corresponding to i can be adjusted according to actual business and application results;

[0027] The calculation formula of img is as follows:

[0028] img=b*W b +s*W s

[0029] Among them, W is the weight, which is a value between 0 and 1. b With W s The sum is 1;

[0030] Among them, b represents the number of characteristic targets of the power operation site appearing in the image;

[0031] Among them, s represents the number of characteristic targets of high-risk power operation scenes appearing in the image.

[0032] Furthermore, the number of characteristic targets of the power operation site appearing in the image is calculated as follows:

[0033]

[0034] Where i represents the number of basic characteristic targets of the power operation site appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i basic feature targets.

[0035] Furthermore, the number of characteristic targets of high-risk power operation scenarios appearing in the image is calculated as follows:

[0036]

[0037]

[0038] Where i represents the number of characteristic targets of high-risk power operation sites appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i scene feature targets.

[0039] The present invention has the following beneficial effects:

[0040] 1- This invention innovatively proposes a high-risk operation scene identification and early warning method based on the fusion of semantic and visual information. By analyzing the semantic analysis of power site operation work tickets and the recognition of operation image targets from two dimensions, it is judged whether there is a high-risk operation scene at the current operation site, and a confidence evaluation mechanism is proposed to fuse the judgment results of the two dimensions. A cloud-edge-end collaborative architecture with real-time edge analysis and cloud-end result display is used to assist safety inspectors in quickly, accurately and efficiently screening and locating work plans with high-risk operation scenes, changing the current safety supervision situation that relies on manual patrols, solving the current problems of low management efficiency, small coverage, and inability to supervise the entire process, as well as the server resource pressure caused by cloud-based intelligent analysis, improving the efficiency of safety monitoring at the operation site, effectively investigating and correcting on-site violations, and greatly promoting the improvement of the quality and efficiency of safety supervision.

[0041] 2- Compared with existing methods for on-site safety management of power operations based on machine learning, deep learning, and cloud-edge collaboration, this invention, for the first time, adopts a high-risk operation scene recognition method that combines text semantic analysis and image target analysis, formulates detailed secondary inference rules for high-risk operation scene recognition based on image target detection results, and proposes for the first time a confidence-based operation scene recognition result evaluation mechanism, which greatly improves the accuracy of recognition results and significantly reduces the false alarm rate. The high-risk operation vocabulary, high-risk operation scene recognition secondary inference rules, and confidence evaluation mechanism constructed by the present invention can be flexibly and dynamically added and adjusted according to business needs and usage scenarios, meeting the professional use needs of power transmission, substation, distribution, and infrastructure, and has a wide range of applications.

[0042] This invention utilizes cloud-edge-end collaboration for high-risk operation scenario identification and application. Compared to cloud-based high-risk operation scenario identification, this improves the timeliness and effectiveness of identification and reduces the pressure on cloud-based server resources such as CPUs and GPUs. (Electric power operation site videos are stored externally and need to be uploaded to the power grid intranet via a dedicated 4G / 5G network. Each province has tens of thousands of video terminals per day. If all of these terminals were to upload video data in real time, network channels would be strained. Intelligent analysis and judgment of each video feed would require a large amount of GPUs and memory in the cloud to deploy AI models.) BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of the method of the present invention;

[0044] Figure 2 Schematic diagram of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0046] refer to Figure 1 , in this embodiment,

[0047] Step S1: Obtain a high-risk operation work ticket and pre-process it to remove spaces, punctuation marks, and words without entity meaning to obtain the corresponding text data;

[0048] Step S2: Manually extract characteristic keywords with operation scenario category information from the corresponding text data and construct a high-risk operation vocabulary;

[0049] Step S3: Acquire high-risk operation site pictures and pre-process them to obtain corresponding picture sample data;

[0050] Step S4: constructing a picture sample database according to the picture sample data representation result;

[0051] Step S5: Formulate high-risk operation scenario analysis and judgment rules by combining the high-risk operation vocabulary and the image sample database;

[0052] Step S6: A confidence index calculation mechanism for high-risk operation scene recognition results is obtained by training through semantic analysis of high-risk operation work tickets and analysis of operation video images at high-risk operation sites;

[0053] Step S7: Based on the labeled image sample data representation results, the YOLOx-s algorithm is used to train a feature recognition model for power high-risk operation scenarios;

[0054] Step S8: Relying on the edge computing device, combined with the high-risk operation scenario analysis rules, confidence indicators and the recognition results of the high-risk operation scenario feature recognition model, the recognition results are uploaded to the safety risk management and supervision platform.

[0055] Furthermore, high-risk operation work tickets: obtain work tickets for high-risk operation scenarios from professional business systems such as power PMS, E-infrastructure, transmission and distribution, and infrastructure, and safety risk management and supervision platforms, and classify them according to types such as high-altitude operations, large-scale machinery operations, hot work operations, and confined space operations, and try to collect sub-category work tickets covering each major category.

[0056] Furthermore, for high-risk work site images: First, collect high-risk work site images from the existing violation images and operation videos on the power safety risk management and supervision platform. Second, assign dedicated personnel to visit specialized work sites such as transmission, substation, distribution, and infrastructure to collect high-risk work site images. Due to the complex and ever-changing weather conditions at work sites and the frequent movement of workers and machinery, the accuracy of image object detection algorithms based on machine learning and deep learning is easily affected by factors such as image clarity, angle, and distance. Therefore, manual image collection must be carried out in accordance with collection specifications.

[0057] Furthermore, image collection specifications stipulate that for the same type of operation scene, no more than 70 images should be collected at the same operation site, and no more than 15 images should be collected for the same operation scene at the same operation site. For example, no more than 70 images should be collected of a crane operating at substation A, and no more than 15 images should be collected for scene ① at substation A. The images of the crane in scene ① should be composed of images taken at different time periods, distances, angles, and coverage.

[0058] Furthermore, the step S3: obtaining high-risk operation site pictures and performing pre-processing, the pre-processing is specifically as follows:

[0059] For on-site image data, the LabelImg image annotation tool is used to label the basic feature targets of the power operation site and the feature targets of high-risk operation scenes in the images.

[0060] For the collected image data of high-risk work sites, such as high-altitude work, large machinery work, hot work, and confined space work, we use image annotation tools such as LabelImg to annotate basic features of power work sites, such as hard hats, work clothes, and safety fences, as well as high-risk work scene features, such as workers at height, towers, poles, cranes, boom trucks, sparks, oxygen, and foundation pit openings. The following table lists the vocabulary examples and annotated target labels:

[0061] Partial List of High-Risk Assignment Vocabulary

[0062]

[0063] List of target labels for image annotation of high-risk work scenarios

[0064]

[0065]

[0066] Furthermore, the step S5: combining the high-risk operation vocabulary and the image sample database to formulate high-risk operation scenario analysis and judgment rules, specifically as follows:

[0067] Based on the high-risk operation vocabulary, parse the corresponding text data to determine whether there are high-risk operation scenario keywords;

[0068] Using an image object detection algorithm, we identify and return data from the corresponding image data. We then perform secondary reasoning based on business rules to determine whether there are characteristic objects in high-risk operational scenarios. The secondary reasoning logic is shown in the table below.

[0069] Logic checklist for evaluating high-risk operation scenarios based on high-risk operation characteristic objectives

[0070]

[0071]

[0072] If at least one Category 1 high-risk operation scenario keyword appears in the text data of the work ticket name, work content, and safety measures, it is determined that the work ticket involves relevant high-risk operations.

[0073] If the keywords in the vocabulary are connected using the "&" symbol, the relevant keywords must appear in the work ticket at the same time to be determined as involving corresponding high-risk operations.

[0074] A work permit may involve two or more high-risk operations. For example, the installation of a power transmission line involves two high-risk operations: crane operation and height operation.

[0075] Furthermore, the characteristic targets of the work site identified by the image target detection algorithm will have a sim (similarity) value, which is used to judge the degree to which the identified target is close to the real target. The higher the sim value, the higher the accuracy of the target identified by the algorithm. However, the recognition of high-risk work scenes is the result of comprehensive analysis and judgment of the recognition results of the target detection algorithm. The sim of the identified target cannot be simply multiplied or added to calculate the accuracy judgment basis of the high-risk work scene recognition results. Therefore, the patent of this invention proposes a confidence (Confidence, abbreviated as "C") evaluation mechanism for the recognition of high-risk work scenes, and sets three evaluation factors: work ticket attributes (abbreviated as "t"), basic feature targets (abbreviated as "b"), and scene feature targets (abbreviated as "s") from the two dimensions of work ticket semantic analysis (abbreviated as "wk") and work video image analysis ("img").

[0076] In step S6, "confidence index calculation mechanism for high-risk operation scene identification results", the confidence calculation formula is as follows:

[0077] cfi=wk*W wk +img*W img

[0078] Among them, W is the weight, which is a value between 0 and 1. wk With W img The sum is 1;

[0079] The value of wk is related to the number of keywords obtained by parsing the work ticket that appear in the high-risk operation vocabulary. The greater the number, the greater the high-risk operation attribute value of the work ticket. The calculation formula for wk is as follows:

[0080] wk=t i

[0081]

[0082] Among them, i represents the number of high-risk operation feature words, and the weight value corresponding to i can be adjusted according to actual business and application results;

[0083] The calculation formula of img is as follows:

[0084] img=b*W b +s*W s

[0085] Among them, W is the weight, which is a value between 0 and 1. b With W s The sum is 1;

[0086] Here, b represents the number of power operation site characteristic targets in the image. The more characteristic targets appear, the more similar the scene is to a power operation site, which can reduce false positives caused by high-risk operation scene features such as cranes appearing on non-operation surfaces. For example, when conducting power distribution line maintenance work on the roadside, there may be large vehicles such as cranes passing by on the road.

[0087] Among them, s represents the number of characteristic targets of high-risk power operation scenes appearing in the image.

[0088] Furthermore, the authenticity of the work targets identified by the image target detection algorithm will be judged by the SIM indicator. The higher the SIM, the more accurate the target identified by the algorithm. The number of characteristic targets of the power work site appearing in the image is calculated as follows:

[0089]

[0090] Where i represents the number of basic characteristic targets of the power operation site appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i basic feature targets.

[0091] Furthermore, the more characteristic targets appear, the more likely the scene is to resemble a high-risk power operation site. At the same time, the authenticity of the operation targets identified by the image target detection algorithm will be judged using the SIM indicator. The higher the SIM, the more accurate the target identification by the algorithm. The number of characteristic targets of high-risk power operation scenes appearing in the image is calculated as follows:

[0092]

[0093]

[0094] Where i represents the number of characteristic targets of high-risk power operation sites appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i scene feature targets.

[0095] In summary, the confidence calculation formula for high-risk operation scene identification can be expressed as follows:

[0096]

[0097] Among them, x represents the number of high-risk operation words appearing in the work ticket, y represents the number of basic feature targets of the power operation site appearing in the picture, and z represents the number of feature targets of the power high-risk operation scene appearing in the picture. The confidence value is a value between 0 and 1, with 2 decimal places.

[0098] Furthermore, based on the labeled high-risk operation scene image sample data, the YOLOx-s algorithm is used to train the power high-risk operation scene feature recognition model. The operation target information identified by the model (including target labels, coordinate information, similarity, etc.) is combined with the sorted high-risk operation scene analysis logic and confidence indicators to perform secondary reasoning and interface encapsulation to obtain a high-risk operation scene recognition model that inputs work ticket information, operation images and outputs high-risk operation scene recognition results (including high-risk operation scene categories and confidence levels).

[0099] Furthermore, the present invention deploys a high-risk work scenario recognition model on an edge computing device with AI computing power. Through the Wi-Fi hotspot on the device, the video surveillance equipment (e.g., a mobile surveillance camera) at the work site is integrated to collect on-site video images for intelligent analysis. The recognition results and other information are then uploaded to the cloud-based security risk management and supervision platform according to predetermined rules and frequencies through the encryption module and 4G / 5G communication module on the device. The security risk management and supervision platform sorts the work plan for the day according to the order of high-risk work scenarios, making it easier for safety inspectors to quickly find high-risk work plans for key inspections.

[0100] The integration, upload, and cloud display of high-risk operation scenario identification results are as follows:

[0101] 1. Confidence threshold setting: Results are integrated and uploaded to the cloud only when CFI ≥ 0.3. The CFI threshold can be adjusted based on different training samples, deep learning algorithms, and application scenarios.

[0102] 2. Reporting frequency: Reports are sent every 20 minutes, and the reporting time can be flexibly set based on actual business needs. However, if different high-risk operation types are identified during the 20-minute interval, the identification results must be uploaded immediately and the timer will be reset.

[0103] 3. Recognition result integration: The recognition results are integrated in a list format. Each recognition result contains at least the scene type and scene confidence.

[0104] 4. Display of identification results: In the high-risk operation scenario screening module of the safety risk management and supervision platform, the work plan for the day is sorted in reverse order by the number of high-risk operation scenarios and the start time of the operation, and the high-risk operation scenario type is displayed after the work plan in the list.

[0105] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information, characterized by: The following steps are involved: Step S1: Obtain a high-risk operation work ticket and pre-process it to remove spaces, punctuation marks, and words without entity meaning to obtain the corresponding text data; Step S2: Manually extract characteristic keywords with operation scenario category information from the corresponding text data and construct a high-risk operation vocabulary; Step S3: Acquire high-risk operation site pictures and pre-process them to obtain corresponding picture sample data; Step S4: constructing a picture sample database according to the picture sample data representation result; Step S5: Formulate high-risk operation scenario analysis and judgment rules by combining the high-risk operation vocabulary and the image sample database; Step S6: A confidence index calculation mechanism for high-risk operation scene recognition results is obtained by training through semantic analysis of high-risk operation work tickets and analysis of operation video images at high-risk operation sites; Step S7: Based on the labeled image sample data representation results, the YOLOx-s algorithm is used to train a feature recognition model for power high-risk operation scenarios; Step S8: Relying on the edge computing device, combined with the high-risk operation scenario analysis rules, confidence indicators and the recognition results of the high-risk operation scenario feature recognition model, the recognition results are uploaded.

2. The high-risk operation scene recognition and early warning method based on semantic and visual information fusion according to claim 1 is characterized in that: Step S3: Obtain high-risk operation site pictures and perform pre-processing, the pre-processing is as follows: For on-site image data, the LabelImg image annotation tool is used to annotate the basic feature targets of the power operation site and the feature targets of high-risk operation scenes in the images.

3. The high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information according to claim 1 is characterized in that: Step S5: Formulate high-risk operation scenario analysis and judgment rules by combining the high-risk operation vocabulary and the image sample database, as follows: Based on the high-risk operation vocabulary, parse the corresponding text data to determine whether there are high-risk operation scenario keywords; An image target detection algorithm is used to identify and return data information in the corresponding image data, and secondary reasoning based on business rules is used to determine whether there are characteristic targets in high-risk operation scenarios.

4. The high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information according to claim 1 is characterized in that: In step S6: "calculation mechanism of confidence index of high-risk operation scene identification result", the confidence calculation formula is as follows: cfi=wk*W wk +img*W img Among them, W is the weight, which is a value between 0 and 1. wk With W img The sum is 1; The calculation formula of wk is as follows: wk=t i Among them, i represents the number of high-risk operation feature words, and the weight value corresponding to i can be adjusted according to actual business and application results; The calculation formula of img is as follows: img=b*W b +s*W s Among them, W is the weight, which is a value between 0 and 1. b With W s The sum is 1; Among them, b represents the number of characteristic targets of the power operation site appearing in the image; Among them, s represents the number of characteristic targets of high-risk power operation scenes appearing in the image.

5. The high-risk operation scene recognition and early warning method based on the fusion of semantic and visual information according to claim 4 is characterized in that: The number of characteristic targets of the power operation site appearing in the image is calculated using the following formula: Where i represents the number of basic characteristic targets of the power operation site appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i basic feature targets.

6. The high-risk operation scene recognition and early warning method based on semantic and visual information fusion according to claim 4 is characterized in that: The number of characteristic targets of high-risk power operation scenarios in the image is calculated as follows: Where i represents the number of characteristic targets of high-risk power operation sites appearing in the image. The weight value corresponding to i is adjusted based on actual business and application results. Represents the average similarity value of i scene feature targets.

Images