Safety supervision risk point identification and risk behavior alarm method and system based on multi-modal technology, and electronic equipment

Abstract

The invention discloses a safety supervision risk point identification and risk behavior alarm method and system based on a multi-modal technology, and electronic equipment, and relates to the technical field of electric power safety. The method comprises the following steps: extracting dangerous points from an image by using a risk point identification model and an object segmentation model, segmenting an object, fusing into a risk prompt graph, and sending the risk prompt graph to a monitor for real-time risk prompt. When a monitor is interested in other objects, a query text from the monitor is received, and after intention analysis and intention matching are performed on the query text, whether a visual language large model is used for answering or intention image extraction and re-recognition are further performed from an original image according to the intention of the monitor is determined. According to the invention, a flexible interaction mechanism can be provided in a live working risk monitoring scene, and key risk points are prevented from being missed in an AI fixed output mode.

Description

Technical Field

[0001] This invention relates to the field of power safety technology, specifically to a method, system, and electronic equipment for identifying safety risk points and alarming risk behaviors based on multimodal technology, used for intelligent monitoring in dangerous scenarios such as live-line work. Background Technology

[0002] Live-line work is a critical aspect of power system operation and maintenance, directly impacting the safety of workers and the stable operation of the power grid. Statistics show that approximately 35% of power industry safety accidents are related to live-line work, with over 60% of these accidents caused by inaccurate hazard identification. Therefore, establishing an efficient and accurate risk identification mechanism for live-line work is of great practical significance for preventing safety accidents and ensuring operational safety.

[0003] To address the high cost and difficulty of monitoring live-line work sites, existing technologies typically employ AI models to monitor hazardous points. However, these models only monitor hazards and push the results remotely. Live-line work sites are complex, and in many cases, the hazard identification results from the AI ​​model may not reflect the actual hazards the monitor is concerned with. Furthermore, the fixed output model of the AI ​​model prevents interaction with the monitor, potentially hindering their access to crucial information and impacting safety decisions.

[0004] Existing technology, as disclosed in a Chinese patent (publication number CN117370872A), provides a method and apparatus for optimizing control of power equipment, relating to the field of equipment optimization technology. The method includes: acquiring the real-time operating status of multiple power devices in the power grid, obtaining fault characteristics of each device, determining hazard points based on the fault characteristics, obtaining corresponding hazard indices, identifying the types of hazard points to obtain exposed contact information, performing optimization based on the hazard indices, and finally implementing maintenance control based on the optimal solution. This application primarily addresses the technical problems of not being able to monitor hazard points in real time, not being able to guarantee the accuracy of hazard location prediction, and not being able to provide timely warnings for hazard points. By acquiring the fault characteristics of each power device, analyzing the hazard characteristics to obtain hazard coefficients, and finally performing optimization based on the hazard coefficients, and finally implementing maintenance control based on the optimal strategy, the accuracy and real-time performance of hazard prediction can be effectively improved, reducing the threat of potential hazards to power equipment and personnel. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by disclosing a method, system and electronic device for identifying safety risk points and alarming risk behaviors based on multimodal technology, which solves the problem that existing AI monitoring systems may miss key risk points due to fixed output methods and lack of human-computer interaction.

[0006] To achieve the above objectives, the present invention provides a method, system, and electronic device for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology, comprising the following steps:

[0007] S1, acquire monitoring video of the live-line work site;

[0008] S2, extract the original image from the monitoring video and input the original image into a pre-built risk identification model to obtain the identification result; at the same time, input the original image into a pre-built segmentation model to obtain multiple objects and segmentation masks for the multiple objects;

[0009] The identification results include the boundary box of the dangerous area, the risk level of the dangerous area, and the risk type;

[0010] S3, map the recognition result and the segmentation mask of the multiple objects onto the original image to obtain a risk warning image;

[0011] S4, push the risk warning image to the remote monitoring terminal, and when receiving the query text from the remote monitoring terminal, perform intent parsing on the query text to obtain intent information;

[0012] The query text is generated based on the risk warning image;

[0013] S5, match the intent information with multiple object information in the risk warning image: when the intent information does not match any object information in the risk warning image, forward the query text and the risk warning image to the visual language big model;

[0014] When the intent information matches any target object information in the risk warning image, the intent image is extracted from the original image based on the segmentation mask of the target object;

[0015] S6, the intent image is sent to the risk recognition model to obtain the intent recognition result.

[0016] Furthermore:

[0017] In step S2, the method for constructing the risk identification model includes:

[0018] S2.1, Obtain historical image samples of live-line working scenarios;

[0019] S2.2, The historical image samples are labeled to obtain training data with label information, wherein the label information includes the bounding box of the risk point, the hazard type of the risk point, and the risk level of the risk point;

[0020] S2.3, Input the training data into the CNN convolutional neural network to obtain the prediction result;

[0021] S2.4, calculate the loss between the prediction result and the label information based on the pre-built risk identification model loss function, and adjust the parameters of the CNN convolutional neural network based on the loss and the minimum gradient method;

[0022] S2.5. Repeat steps S2.3-S2.4 until training is complete and the risk identification model is obtained.

[0023] Furthermore:

[0024] The loss function of the risk identification model is used to optimize the positioning accuracy of the boundary box of the dangerous area, the accuracy of risk type classification, and the rationality of risk level determination.

[0025] The mathematical expression for the loss function of the risk identification model is:

[0026] Formula 1: ;

[0027] Formula 2: ;

[0028] Formula 3: ;

[0029] Formula 4:

[0030] ;

[0031] In formulas 1, 2, 3, and 4, For the total loss, For bounding box loss, As the first weight, Losses classified as hazardous. As the second weight, Loss at the risk level For bounding box indexing, For the number of bounding boxes, Indicates the first The intersection-union ratio of the ground truth bounding boxes and the predicted bounding boxes. It is the Euclidean distance function. For the first One predicted bounding box coordinate, For the first A true bounding box coordinate, The length of the diagonal of the minimum closure region. For hazard type index, For the number of hazard types, A label vector of the danger type. For the predicted probability vector, For risk level index, For risk level numbers, For indicator functions, Indicates the true risk level. To output a vector of logical values, This is the activation function.

[0032] Furthermore:

[0033] The method for constructing the segmentation model in step S2 includes:

[0034] S2.6, Obtain historical image samples of live-line working scenarios;

[0035] S2.7, The historical image samples are labeled to obtain training data with label information, wherein the label information includes classification labels of multiple objects and segmentation masks of multiple objects;

[0036] S2.8, Input the training data into the CNN convolutional neural network to obtain the prediction mask;

[0037] S2.9, Calculate the loss between the prediction mask and the segmentation mask based on the training loss function of the pre-built segmentation model, and adjust the parameters of the CNN convolutional neural network based on the loss and the minimum gradient method;

[0038] S2.10, Repeat steps S2.3-S2.4 until training is complete and the segmentation model is obtained.

[0039] Furthermore:

[0040] The training loss function of the segmentation model is used to optimize the contour accuracy and class discrimination of the pixel-level segmentation mask of the object;

[0041] The mathematical expression for the training loss function of the segmentation model is:

[0042] Formula 5: ;

[0043] Formula 6: ;

[0044] Formula 7: ;

[0045] In formulas 6, 7, and 8, The weights represent the cross-entropy loss. For cross-entropy loss, for Loss weights, for loss, Image height, Image width, Represents pixel coordinates. Indexed by object type, For the number of object types, For type weights, Representation Object In the real mask The pixel value of the location, Representation Object In the prediction mask The pixel value of the location, To prevent the coefficient from being zero.

[0046] Furthermore:

[0047] In step S3, the recognition result and the segmentation mask of the multiple objects are mapped onto the original image to obtain a risk warning image, including:

[0048] S3.1, Based on the input-output image relationship of the model, the recognition result and the segmentation result are scaled, translated and mapped to obtain the recognition result and segmentation result in the original image;

[0049] S3.2, Filter out target risk areas with a risk level higher than or equal to a preset threshold (the preset threshold is "medium risk" and above) from the recognition results, and determine the risk object and the risk type of the risk object based on the target risk area in the original image and the segmentation result;

[0050] Among them, the risk object is an object that has an overlapping area with the target risk area exceeding a preset overlap threshold;

[0051] S3.3, The risk object and the risk type of the risk object are labeled on the original image to obtain a risk warning image.

[0052] Furthermore:

[0053] In step S4, the query text is parsed to obtain intent information, including:

[0054] S4.1, The query text is segmented and filtered to obtain multiple entities;

[0055] S4.2, convert the multiple entities into multiple word vectors, match the multiple word vectors with a pre-built intent entity lexicon, and when any word vector matches an intent word vector template in the intent entity lexicon, use the word vector as intent information;

[0056] The intent entity lexicon contains core entity word vectors for object categories, risk types, and equipment names in live-line working scenarios.

[0057] Furthermore:

[0058] In step S5, the intent information is matched with multiple object information in the risk warning image, including:

[0059] S5.1, The object information is vectorized to obtain the object recognition vector;

[0060] S5.2, Calculate the similarity between the intent information and the object recognition vector. The similarity calculation uses a cosine similarity algorithm. The similarity is determined when it is greater than or equal to a preset similarity threshold.

[0061] When the preset similarity threshold is ≥0.8, the object corresponding to the object recognition vector is determined to be the target object;

[0062] When the similarity between the intent information and any object recognition vector is less than a preset similarity threshold, it is determined that the intent information does not match any object information in the risk warning image.

[0063] In step S5, "extracting the intent image from the original image based on the segmentation mask of the target object" includes:

[0064] S5.3, fill with the segmentation mask to obtain an intention region mask that expands outward by n pixels based on the segmentation mask;

[0065] Wherein, n is 10-50 pixels, and is adaptively adjusted according to the object size;

[0066] S5.4 Multiply the intent region mask with the original image to obtain the intent image.

[0067] A safety supervision risk point identification and risk behavior alarm system based on multimodal technology includes:

[0068] The acquisition module is used to acquire monitoring videos of live-line work sites to provide raw data support for subsequent risk identification;

[0069] The risk monitoring module is used to extract raw images from monitoring videos, input them into a pre-built risk identification model and segmentation model respectively, and output identification results including dangerous area bounding boxes, risk levels, risk types, as well as multiple objects and their segmentation masks.

[0070] The risk alert module is used to map the recognition results and the segmentation mask of multiple objects to the original image, filter high-risk target risk areas and identify associated risk objects, and generate risk alert images after labeling risk objects and risk types.

[0071] The intent feedback module is used to push risk warning images to the remote monitoring terminal, receive query text generated by the monitoring terminal based on the image, perform word segmentation, entity filtering and word vector matching on the query text, and parse out the intent information.

[0072] The matching module is used to perform similarity matching between intent information and object information in risk warning images. If there is no match, the query text and risk warning image are forwarded to the visual language big model. When matching, the intent region mask is generated by expanding the segmentation mask of the target object, and then the intent image is extracted from the original image.

[0073] The intent region recognition module is used to input the extracted intent image into the risk recognition model, perform targeted risk recognition, and output the intent recognition results to the monitoring terminal.

[0074] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0075] The beneficial effects of this invention are as follows:

[0076] This invention integrates risk identification models, object segmentation models, and a large visual language model to construct an intelligent safety monitoring system with flexible interactive capabilities. This effectively solves the problem that traditional AI monitoring may miss key risk points due to its fixed output method. The system can not only automatically identify and associate risk areas with on-site objects and generate intuitive risk warning images, but also allows monitoring personnel to proactively focus on targets through natural language queries. It can adaptively perform risk re-identification or semantic responses based on the query intent, thereby significantly improving the accuracy, proactiveness, and decision support capabilities of risk perception in complex operational scenarios. It is particularly suitable for high-risk real-time monitoring scenarios such as live-line electrical work, helping to reduce accident rates and ensure operational safety. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a system architecture diagram of a method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology, as shown in one embodiment of this application.

[0079] Figure 2 This is a structural diagram of a safety supervision risk point identification and risk behavior alarm system based on multimodal technology, shown in one embodiment of this application;

[0080] Figure 3 This is a flowchart illustrating a method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology, as shown in one embodiment of this application.

[0081] Figure 4 This is a block diagram illustrating the risk warning image generation logic of a method for identifying safety supervision risk points and issuing risk behavior warnings based on multimodal technology, as shown in one embodiment of this application.

[0082] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Explanation of reference numerals in the attached figures

[0083] 401: Central Processing Unit; 402: Read-Only Memory (ROM); 403: Random Access Memory (RAM); 404: Bus; 405: Input / Output (I / O) Interface; 406: Input Section; 407: Output Section; 408: Storage Section; 409: Communication Section; 410: Driver; 411: Removable Media. Detailed Implementation

[0084] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0085] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0086] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0087] Example 1

[0088] like Figure 1 The diagram shown is a system architecture diagram of a safety supervision risk point identification and risk behavior alarm method based on multimodal technology, as illustrated in one embodiment of this application. Figure 1 As shown, the system of the present invention mainly includes three modules: a perception module, a recognition module, and an interaction module.

[0089] The sensing module is deployed at the live-line working site and includes cameras at multiple monitoring points to collect on-site video streams and send the video streams to the recognition module located in the cloud via network devices.

[0090] The identification module comprises a risk identification model, an object segmentation model, and a fusion unit. The risk identification model identifies hazardous areas in the image and outputs the hazardous area bounding box, risk level, and risk type. The object segmentation model segments work-related objects in the image and outputs the object segmentation mask and category. The outputs of the two models are fused in the fusion unit to generate a risk warning image.

[0091] The interaction module receives risk warning images and pushes them to a remote monitor. The monitor can view the risk warning images and input natural language query text for areas of interest. The intent parsing unit in the interaction module parses the query text and then determines the next step based on the parsing results: if the intent matches an object in the risk warning image, the region containing that object is extracted from the original image (intent image) for further recognition; otherwise, the query text and the risk warning image are input together into the Visual Language Model (VLM) for a response.

[0092] Example 2

[0093] Figure 2 This is a structural diagram of a safety supervision risk point identification and risk behavior alarm system based on multimodal technology, as shown in one embodiment of this application. Figure 2As shown, the system is specifically composed of an image acquisition module, a risk identification module, an object segmentation module, an image fusion module, a communication and interaction module, an intent processing module, and a decision-making and execution module. Specifically, the image acquisition module acquires monitoring videos and raw images of the work site; the risk identification module uses a risk identification model to analyze the raw images and outputs identification results including hazardous area bounding boxes, risk levels, and risk types; the object segmentation module uses an object segmentation model to analyze the raw images and outputs segmentation results containing multiple objects and their segmentation masks; the image fusion module fuses the identification and segmentation results to generate a risk warning image; the communication and interaction module pushes the risk warning image to a remote monitoring terminal and receives query text from the remote terminal based on the risk warning image; the intent processing module parses the query text to obtain intent information and matches the intent information with object information in the risk warning image; the decision-making and execution module performs branch operations based on the matching results: if the match is successful, it extracts the intent image from the raw image based on the segmentation mask of the matched target object and performs re-identification; if the match is unsuccessful, it uses a large visual language model to generate a response. The functions and collaborative relationships of the various modules of this system are consistent with the content defined in claim 8, and together they realize the flexible and accurate interactive safety monitoring risk alarm function of this invention.

[0094] Example 3

[0095] like Figure 3 As shown in the figure, a flowchart of a method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology is illustrated in one embodiment of this application; it mainly includes two stages: model construction and overall workflow.

[0096] Phase 1: Construction and Training of the Core Model

[0097] S2.1 Construction and Training of Risk Identification Model

[0098] This model is used to detect dangerous areas from images and predict their risk type and level. It is built based on a convolutional neural network (CNN), and the specific training steps are as follows:

[0099] The risk identification model is used to detect hazardous areas from images and predict their risk type and level. This model is built upon a convolutional neural network (CNN), and the specific training steps are as follows:

[0100] S2.11: Collect a large number of historical images of live-line working scenarios as training samples.

[0101] S2.12: Label the image. The labeling information includes the bounding box (i.e., the coordinates of the rectangle) of the risk point, the type of hazard (such as electric arc, insulation damage, insufficient safety distance, etc.), and the risk level (such as high, medium, low).

[0102] S2.13: Input the labeled dataset into the CNN network for training. In this embodiment, Faster R-CNN is used as the basic network architecture.

[0103] S2.14: During training, a multi-task loss function is used to optimize the network parameters. The loss function consists of three parts: bounding box regression loss, hazard classification loss, and risk level loss. The specific expression is as follows:

[0104] The mathematical expression for the loss function of the risk identification model is:

[0105] Formula 1: ;

[0106] in, For the bounding box regression loss, DIoULoss is used, and the calculation formula is as follows:

[0107] Formula 2: ;

[0108] in, For hazard classification loss, FocalLoss is used to address the imbalance between positive and negative samples. The calculation formula is as follows:

[0109] Formula 3: ;

[0110] in, For the risk level loss, an ordered regression loss is used, treating the risk level as an ordered variable for modeling. The calculation formula is as follows:

[0111] Formula 4:

[0112] ;

[0113] In formulas 1, 2, 3, and 4, For the total loss, For bounding box loss, As the first weight, Losses classified as hazardous. As the second weight, Loss at the risk level For bounding box indexing, For the number of bounding boxes, Indicates the first The intersection-union ratio of the ground truth bounding boxes and the predicted bounding boxes. It is the Euclidean distance function. For the first One predicted bounding box coordinate, For the first A true bounding box coordinate, The length of the diagonal of the minimum closure region. For hazard type index, For the number of hazard types, A label vector of the danger type. For the predicted probability vector, For risk level index, For risk level numbers, For indicator functions, Indicates the true risk level. To output a vector of logical values, This is the activation function.

[0114] During training, gradient descent was used to minimize the total loss, and the network parameters were continuously adjusted. The hyperparameters were set as follows: learning rate of 1e-3, Adam optimizer, batch size of 16, and training epochs of 100.

[0115] S2.15: Repeat the training process until the loss function converges to obtain a well-trained risk identification model.

[0116] The specific training and implementation of the object segmentation model, which is used to perform pixel-level segmentation of operation-related objects (such as operators, insulating rods, circuit breakers, etc.) in images.

[0117] S2.2 Parallel Risk Identification and Object Segmentation

[0118] The preprocessed original image is fed in parallel into two pre-trained deep learning models: a risk identification model and an object segmentation model.

[0119] S2.21: Collect historical images of live-line working scenarios as training samples.

[0120] S2.22: Perform pixel-level annotation on the image, generate a segmentation mask for each object, and label its category.

[0121] S2.23: Input the labeled data into the segmentation network for training. In this embodiment, the U-Net network architecture is used.

[0122] S2.24: Training is performed using a combined loss function, which is a weighted sum of cross-entropy loss and Dice loss.

[0123] The mathematical expression for the training loss function of the segmentation model is:

[0124] Formula 5: ;

[0125] Among them, cross-entropy loss The calculation formula is:

[0126] Formula 6: ;

[0127] Among them, Dice lost The calculation formula is:

[0128] Formula 7: ;

[0129] In formulas 6, 7, and 8, The weights represent the cross-entropy loss. For cross-entropy loss, for Loss weights, for loss, Image height, Image width, Represents pixel coordinates. Indexed by object type, For the number of object types, For type weights, Representation Object In the real mask The pixel value of the location, Representation Object In the prediction mask The pixel value of the location, To prevent the coefficient from being zero.

[0130] During training, the total loss is minimized using gradient descent to adjust the network parameters.

[0131] Parameter settings: =0.6$, =0.4, epsilon=1e-7, learning rate 1e-4, using SGD optimizer, trained for 80 epochs.

[0132] S2.25: Repeat training until the model converges to obtain a trained segmentation model.

[0133] Phase Two: Workflow Based on the Trained Model

[0134] S2.3 Data Acquisition and Preprocessing

[0135] First, multiple camera terminals deployed at the operation site are used to collect the monitoring video stream of the live working site in real time. Key frames are extracted from the video stream as the original images and are preprocessed by normalization (such as scaling to a fixed resolution) to meet the input requirements of the subsequent model.

[0136] S2.4 Parallel risk identification and object segmentation.

[0137] The preprocessed original images are input into the above two pre-trained deep learning models in parallel: the risk identification model and the object segmentation model.

[0138] S2.5 Risk prompt image generation

[0139] The outputs of the two models are fused to generate a risk prompt image. The fusion process includes coordinate mapping, risk-object association, and visual annotation.

[0140] S2.6 Interactive risk query.

[0141] The risk prompt image is pushed to the remote monitoring center. The monitoring personnel can input natural language query text based on this image. (For example: "What is the risk in the flashing area in the upper left corner?" or "Is the tool status normal next to the operator in blue clothes?"). The system analyzes the intent of the query text and extracts key entities (such as "upper left corner", "flashing area", "operator in blue clothes", "tool").

[0142] The system analyzes the intent of the query text and executes one of two paths according to the analysis result: 1) If the intent matches a specific object in the image, the object area is extracted for re-identification; 2) Otherwise, a vision-language large model is called to answer.

[0143] Specifically, natural language processing (NLP) techniques are used to split the continuous text string into independent words or subwords (Tokens). For example, the query "The status of the tool next to the operator in blue clothes" will be tokenized as: ["wear", "blue", "clothes", "of", "operator", "next", "to", "of", "tool", "status"].

[0144] Then, common function words and prepositions without specific reference meanings such as "of", "next to", "what" are removed, and the core entity words are retained. After screening in the above example, it may get: ["blue", "clothes", "operator", "tool", "status"].

[0145] Embodiment 5

[0146] Such as Figure 4As shown, this embodiment describes in detail the specific implementation process of intent matching, decision execution, and intent image extraction, which is executed after receiving the query text from the monitor and parsing it to obtain intent information.

[0147] Intent matching and decision-making. The specific steps are as follows:

[0148] S5.1: Object information vectorization; convert the category name of each object in the risk warning image into a word vector to obtain the object recognition vector.

[0149] S5.2: Similarity Calculation and Matching; Calculate the cosine similarity between the intent information word vector and the recognition vector of each object. If the highest similarity exceeds a preset threshold (e.g., 0.8), the match is considered successful, and the object is the target object; otherwise, the match fails.

[0150] S5.3: Decision Execution; Decision-making based on matching results: - Successful match: Proceed to the intent image extraction and re-recognition process. - Failed match: Input the query text and risk warning image together into the visual language large model (e.g., GPT-4V), and utilize its powerful multimodal understanding capabilities to generate a natural language response.

[0151] Intent Image Extraction: When a target object is matched, the region containing the object needs to be extracted from the original image for further recognition. To ensure that the extracted region contains the complete object and its surrounding context, the following steps are taken:

[0152] S5.4: Mask Expansion; Performs morphological dilation on the segmentation mask of the target object, using a circular structuring element with a radius of n pixels (e.g., n=5) to expand the mask outward by n pixels.

[0153] S5.5: Image cropping; Multiply the expanded mask by the original image to obtain an intentional image containing only the target object and its surrounding area. This focuses on the target object while preserving necessary contextual information.

[0154] Example 6

[0155] like Figure 5 As shown, it illustrates a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of this application. Figure 5 As shown, the computer system specifically includes the following components:

[0156] The computer system includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 402 or loaded from storage portion 408 into Random Access Memory (RAM) 403. RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0157] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0158] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs various functions defined in the system of this application.

[0159] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0162] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0163] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0164] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology. Its features are, Including the following steps: S1, acquire monitoring video of the live-line work site; S2, extract the original image from the monitoring video and input the original image into a pre-built risk identification model to obtain the identification result; at the same time, input the original image into a pre-built segmentation model to obtain multiple objects and segmentation masks for the multiple objects; The identification results include the boundary box of the dangerous area, the risk level of the dangerous area, and the risk type; S3, map the recognition result and the segmentation mask of the multiple objects onto the original image to obtain a risk warning image; S4, push the risk warning image to the remote monitoring terminal, and when receiving the query text from the remote monitoring terminal, perform intent parsing on the query text to obtain intent information; The query text is generated based on the risk warning image; S5, match the intent information with multiple object information in the risk warning image: when the intent information does not match any object information in the risk warning image, forward the query text and the risk warning image to the visual language big model; When the intent information matches any target object information in the risk warning image, the intent image is extracted from the original image based on the segmentation mask of the target object; S6, the intent image is sent to the risk recognition model to obtain the intent recognition result.

2. The method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology according to claim 1, Its features are, In step S2, the method for constructing the risk identification model includes: S2.1, Obtain historical image samples of live-line working scenarios; S2.2, The historical image samples are labeled to obtain training data with label information, wherein the label information includes the bounding box of the risk point, the hazard type of the risk point, and the risk level of the risk point; S2.3, Input the training data into the CNN convolutional neural network to obtain the prediction result; S2.4, calculate the loss between the prediction result and the label information based on the pre-built risk identification model loss function, and adjust the parameters of the CNN convolutional neural network based on the loss and the minimum gradient method; S2.

5. Repeat steps S2.3-S2.4 until training is complete and the risk identification model is obtained.

3. The method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology according to claim 2, Its features are, The loss function of the risk identification model is used to optimize the positioning accuracy of the boundary box of the dangerous area, the accuracy of risk type classification, and the rationality of risk level determination. The mathematical expression for the loss function of the risk identification model is: Official 1: ; Official 2: ; Official 3: ; Official 4: ; In formulas 1, 2, 3, and 4, For the total loss, For bounding box loss, As the first weight, Losses classified as hazardous. As the second weight, Loss at the risk level For bounding box indexing, For the number of bounding boxes, Indicates the first The intersection-union ratio of the ground truth bounding boxes and the predicted bounding boxes. It is the Euclidean distance function. For the first One predicted bounding box coordinate, For the first The coordinates of the actual bounding box. The length of the diagonal of the minimum closure region. For hazard type index, For the number of hazard types, A label vector of the danger type. For the predicted probability vector, For risk level index, For risk level numbers, For indicator functions, Indicates the true risk level. To output a vector of logical values, This is the activation function.

4. The method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology according to claim 1, Its features are, The method for constructing the segmentation model in step S2 includes: S2.6, Obtain historical image samples of live-line working scenarios; S2.7, The historical image samples are labeled to obtain training data with label information, wherein the label information includes classification labels of multiple objects and segmentation masks of multiple objects; S2.8, Input the training data into the CNN convolutional neural network to obtain the prediction mask; S2.9, Calculate the loss between the prediction mask and the segmentation mask based on the training loss function of the pre-built segmentation model, and adjust the parameters of the CNN convolutional neural network based on the loss and the minimum gradient method; S2.10, Repeat steps S2.8-S2.9 until training is complete and the segmentation model is obtained.

5. The method for identifying safety supervision risk points and alarming risk behaviors based on multimodal technology according to claim 4, Its features are, The training loss function of the segmentation model is used to optimize the contour accuracy and class discrimination of the pixel-level segmentation mask of the object; The mathematical expression for the training loss function of the segmentation model is: Official 5: ; Official 6: ; Official 7: ; In formulas 6, 7, and 8, The weights represent the cross-entropy loss. For cross-entropy loss, for Loss weights, for loss, Image height, Image width, Represents pixel coordinates. Indexed by object type, For the number of object types, For type weights, Representation Object In the real mask The pixel value of the location, Representation Object In the prediction mask The pixel value of the location, To prevent the coefficient from being zero.

6. The method for identifying safety supervision risk points and alarming risk behaviors based on multimodal technology according to claim 1, Its features are, In step S3, the recognition result and the segmentation mask of the multiple objects are mapped onto the original image to obtain a risk warning image, including: S3.1, Based on the input-output image relationship of the model, the recognition result and the segmentation result are scaled, translated and mapped to obtain the recognition result and segmentation result in the original image; S3.2, Filter out target risk areas with risk levels higher than or equal to a preset threshold from the recognition results, and determine the risk object and the risk type of the risk object based on the target risk areas and segmentation results in the original image; Among them, the risk object is an object that has an overlapping area with the target risk area exceeding a preset overlap threshold; S3.3, The risk object and the risk type of the risk object are labeled on the original image to obtain a risk warning image.

7. The method for identifying safety supervision risk points and alarming risk behaviors based on multimodal technology according to claim 1, Its features are, In step S4, the query text is parsed to obtain intent information, including: S4.1, The query text is segmented and filtered to obtain multiple entities; S4.2, convert the multiple entities into multiple word vectors, match the multiple word vectors with a pre-built intent entity lexicon, and when any word vector matches an intent word vector template in the intent entity lexicon, use the word vector as intent information; The intent entity lexicon contains core entity word vectors for object categories, risk types, and equipment names in live-line working scenarios.

8. The method for identifying safety supervision risk points and alarming risk behaviors based on multimodal technology according to claim 1, Its features are, In step S5, the intent information is matched with multiple object information in the risk warning image, including: S5.1, The object information is vectorized to obtain the object recognition vector; S5.2, Calculate the similarity between the intent information and the object recognition vector. The similarity calculation uses a cosine similarity algorithm. The similarity is determined when it is greater than or equal to a preset similarity threshold. When the preset similarity threshold is ≥0.8, the object corresponding to the object recognition vector is determined to be the target object; When the similarity between the intent information and any object recognition vector is less than a preset similarity threshold, it is determined that the intent information does not match any object information in the risk warning image. In step S5, "extracting the intent image from the original image based on the segmentation mask of the target object" includes: S5.3, fill with the segmentation mask to obtain an intention region mask that expands outward by n pixels based on the segmentation mask; Wherein, n is 10-50 pixels, and is adaptively adjusted according to the object size; S5.4 Multiply the intent region mask with the original image to obtain the intent image.

9. A safety supervision risk point identification and risk behavior alarm system based on multimodal technology. The method for identifying safety supervision risk points and issuing risk behavior alarms based on multimodal technology, as described in any one of claims 1 to 8, is adopted. Its features are, include: The acquisition module is used to acquire monitoring videos of live-line work sites to provide raw data support for subsequent risk identification; The risk monitoring module is used to extract raw images from monitoring videos, input them into a pre-built risk identification model and segmentation model respectively, and output identification results including dangerous area bounding boxes, risk levels, risk types, as well as multiple objects and their segmentation masks. The risk alert module is used to map the recognition results and the segmentation mask of multiple objects to the original image, filter high-risk target risk areas and identify associated risk objects, and generate risk alert images after labeling risk objects and risk types. The intent feedback module is used to push risk warning images to the remote monitoring terminal, receive query text generated by the monitoring terminal based on the image, perform word segmentation, entity filtering and word vector matching on the query text, and parse out the intent information. The matching module is used to perform similarity matching between intent information and object information in risk warning images. If there is no match, the query text and risk warning image are forwarded to the visual language big model. When matching, the intent region mask is generated by expanding the segmentation mask of the target object, and then the intent image is extracted from the original image. The intent region recognition module is used to input the extracted intent image into the risk recognition model, perform targeted risk recognition, and output the intent recognition results to the monitoring terminal.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. Its features are, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

Images