Power generation enterprise operation risk identification method and system based on multi-modal large model

Abstract

The invention discloses a power generation enterprise operation risk identification method and system based on a multi-modal large model. The method comprises the following steps: S1, inputting and constructing a multi-modal power generation enterprise operation safety data set and carrying out data preprocessing; s2, inputting the data set into a vertical multi-mode large model constructed by using an ALBEF large model frame, and performing pre-training; s3, identifying a field operation safety risk by using the trained model and giving an alarm; and S4, based on the intelligent agent and the multi-modal large model, realizing operation safety knowledge question answering of the power generation enterprise staff, inputting an operation site image and a natural language question instruction to the multi-modal large model, and generating an answer. According to the invention, operation safety risk identification can be carried out by fusing operation image, text and video multi-mode data, and the method is suitable for various application scenarios such as safety helmet wearing identification, smoking identification, open fire detection, overflow detection, post-sleeping identification, calling identification, personnel gathering identification and the like, so that the accuracy of operation safety risk identification is improved; and the safety production level of power generation enterprises is improved.

Description

Technical Field

[0001] The present invention relates to the field of production safety risk prevention and control of power generation enterprises, and in particular to a method and system for identifying operational risks of power generation enterprises based on a multi-modal large model. Background Art

[0002] In recent years, artificial intelligence technology has achieved remarkable application results in many fields such as natural language processing (NLP) and computer vision, driving the digital and intelligent transformation of various industries. At present, artificial intelligence technology usually focuses on a single modality such as text, image, and audio, and cannot realize the processing and recognition of multimodal data, resulting in high R&D and application costs of artificial intelligence and difficulty in large-scale application.

[0003] Especially in view of the problems of complex operation sites, large number of construction workers, various types of operation safety risks, and insufficient supervision in power generation enterprises, power generation enterprises often use video surveillance terminals to monitor on-site operations. Then, how to conduct real-time analysis, risk identification and alarm of video surveillance terminal data, and at the same time, provide a multimodal operation safety risk retrieval question and answer system for power generation enterprise staff, is the key focus of power generation enterprise safety production. The patent with invention number 202410403204.3 discloses "a method for identifying risks in power operations based on deep learning". This method effectively locates the risk position in the power scene picture through a deep learning algorithm, and improves the accuracy of the risk position in the image. However, this method only realizes the risk identification of the image, and cannot realize the identification of multimodal data, and has certain limitations; the patent with invention number 202410920456.3 discloses "a method and system for identifying the behavior of safe production workers based on multimodal information fusion". This method records the working process of the workers through video, and applies a deep learning model to identify the working behavior and mark the coordinates. It can automate the entire process of identifying and assessing the behavior of safe production workers. However, this method does not explain the universal applicability of the deep learning model to various behavioral risk types of workers. Summary of the invention

[0004] In order to solve the existing problems, the present invention provides a method and system for identifying operational risks of power generation enterprises based on a multi-modal large model. The specific scheme is as follows:

[0005] The method for identifying operational risks of power generation enterprises based on a multimodal large model includes the following steps:

[0006] S1, construct a multimodal power generation enterprise operation safety data set and perform data preprocessing;

[0007] S2, using the ALBEF large model architecture to establish a multimodal large model, inputting the data set into the above model, and pre-training the multimodal large model;

[0008] S3, using the trained multi-modal large model to identify safety risks in on-site operations and issue warnings;

[0009] S4, based on intelligent agents and multimodal big models, realizes the question and answer of work safety knowledge of power generation company staff. The work site images and natural language question instructions are input into the multimodal big model to generate answers.

[0010] Preferably, the data set in step S1 includes a production safety text data set, a work site image data set, a work site video data set, and a list of common violations at the work site; the production safety text data set includes but is not limited to production safety standards, safety management standards, standardized operation specifications for maintenance sites, work supervision logs, work plan forms, and a safety risk knowledge question and answer question bank; the work site image data set is key frames extracted from the work site video.

[0011] Preferably, the data preprocessing includes data cleaning, noise reduction, and image enhancement.

[0012] Preferably, step S2 is specifically as follows: using the ALBEF large model architecture to establish a multimodal large model, inputting the multimodal data set into the model, using the image encoder and the text encoder to independently encode the text data and the image data, using the multimodal encoder to fuse the image features and the text features through the cross-modal attention mechanism, determining the pre-trained objective function and pre-training the multimodal large model.

[0013] Preferably, the objective function is The corresponding expression is:

[0014]

[0015] In the formula, represents the image-text contrast learning loss function, represents the masked language modeling loss function, Represents the image-text matching loss function;

[0016] The image-text contrast learning loss function calculation formula is:

[0017] In the formula, H represents the cross entropy, y i2t (I), y t2i (T) represents the corresponding ground truth label;

[0018] The masked language modeling loss function calculation formula is:

[0019] In the formula, T represents the text after masking, P msk(I, T) represents the model's predicted probability of the token after the mask, y msk represents a one-hot vocabulary distribution, where the probability of the ground truth tag is 1;

[0020] The image text matching loss function calculation formula is:

[0021] In the formula, y itm represents a two-dimensional one-hot label, P itm (I,T) represents the probability of two classes.

[0022] Preferably, S4 comprises the following steps:

[0023] S41: The user terminal inputs question information in the form of pictures or texts;

[0024] S42: The agent obtains the question information from the user and pre-processes the input data;

[0025] S43: The agent calls the multimodal large model to generate response information;

[0026] S44: The intelligent agent combines information of the question and the answer of the user, and stores the question-answer pair in the memory bank;

[0027] S45: The intelligent agent cleans the information in the memory bank and stores it in the multimodal database, providing incremental training data for the multimodal large model and optimizing the large model.

[0028] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. After the computer program is run, any of the above methods is executed.

[0029] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0030] Preferably, a system based on any of the above methods comprises:

[0031] Model building module, using ALBEF model architecture to pre-train multimodal large models;

[0032] The data processing module records images and extracts image information of each frame based on the real-time uploaded surveillance video, and performs data preprocessing;

[0033] The risk identification and warning module uses the trained multimodal large model to identify the risk type and risk location of power operations. The images processed by the data processing module are transmitted to the model in real time. The model will output the operation risk category and location, and display the alarm.

[0034] In the intelligent question-and-answer module, users input questions and answers of the image or text type in the search box, and the intelligent agent calls the multimodal large model to recognize the message and reply to the user. At the same time, the intelligent agent performs expansion tasks, which include extracting standard question-and-answer pairs based on the dialogue data and importing the standard question-and-answer pairs into the database.

[0035] The beneficial effects of the present invention are:

[0036] The present invention uses a contrastive loss to align image and text data before cross-modal interaction, applies a multimodal encoder, and fuses image features with text features through cross-modal attention, solving the image and text alignment and interaction problems in the multimodal field. The present invention also proposes a knowledge question-answering system based on the interaction between a multimodal large model and an intelligent agent. The intelligent agent memory function can extract standard question-answer pairs from the dialogue data generated by user questions and answers, and store them in a database, which can enhance the intelligent level of multimodal large model question-answering task processing and improve the work efficiency of power generation enterprise operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 The figure is a flow chart of the method for identifying operational safety risks of power generation enterprises based on a multimodal large model.

[0039] Figure 2 This is the ALBEF model architecture diagram of the present invention.

[0040] Figure 3 This is a risk identification and warning flow chart of the present invention.

[0041] Figure 4 This is a flow chart of the intelligent question and answer method of the present invention.

[0042] Figure 5 This is a module diagram of the power generation enterprise operation safety risk identification system based on a multi-modal large model. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 The present invention provides a method and system for identifying operation risks of power generation enterprises based on a multi-modal large model. The specific scheme is as follows:

[0045] The method for identifying operational risks of power generation enterprises based on a multimodal large model includes the following steps:

[0046] S1, construct a multimodal power generation enterprise operation safety dataset and perform data preprocessing.

[0047] The data set in step S1 includes a production safety text data set, a work site image data set, a work site video data set, and a list of common violations at the work site; the production safety text data set includes but is not limited to production safety standards, safety management standards, standardized operation specifications for maintenance sites, work supervision logs, work plan forms, and a safety risk knowledge question and answer question bank; the work site image data set is the key frames extracted from the work site video.

[0048] The data preprocessing includes data cleaning, noise reduction, and image enhancement.

[0049] S2, using the ALBEF large model architecture to establish a multimodal large model, input the data set into the above model, and pre-train the multimodal large model.

[0050] Specifically, the ALBEF large model architecture is used to build a multimodal large model, such as Figure 2 This is the ALBEF model architecture diagram. The multimodal dataset is input into the model. The image encoder and text encoder (i.e., text editor) are used to independently encode the text data and image data. The multimodal encoder is used to fuse image features and text features through a cross-modal attention mechanism. The pre-training objective function is determined and the multimodal large model is pre-trained.

[0051] The objective function is The corresponding expression is:

[0052]

[0053] In the formula, represents the image-text contrast learning loss function, represents the masked language modeling loss function, represents the image-text matching loss function;

[0054] The image-text contrast learning loss function calculation formula is:

[0055] In the formula, H represents the cross entropy, y i2t (I), y t2i (T) represents the corresponding ground truth label;

[0056] The masked language modeling loss function calculation formula is:

[0057] In the formula, T represents the text after masking, P msk (I, T) represents the model's predicted probability of the token after the mask, y msk represents a one-hot vocabulary distribution, where the probability of the ground truth tag is 1;

[0058] The image text matching loss function calculation formula is:

[0059] In the formula, y itm represents a two-dimensional one-hot label, P itm (I,T) represents the probability of two classes.

[0060] S3, using the trained multimodal large model to identify on-site operation safety risks and issue alarms.

[0061] The process of identifying and warning of operational safety risks is as follows: Figure 3 As shown:

[0062] S31, monitoring terminal collects the video of the operation site;

[0063] S32, extraction of video key frames;

[0064] S33, preprocessing the data;

[0065] S34, using multimodal large models to identify operational safety risks;

[0066] S35, output the operation risk type.

[0067] S4, based on intelligent agents and multimodal big models, realizes the question and answer of work safety knowledge of power generation company staff. The work site images and natural language question instructions are input into the multimodal big model to generate answers.

[0068] like Figure 4 , said S4 comprises the following steps:

[0069] S41: The user terminal inputs question information in the form of pictures or texts;

[0070] S42: The agent obtains the question information from the user and pre-processes the input data;

[0071] S43: The agent calls the multimodal large model to generate response information;

[0072] S44: The intelligent agent combines information of the question and the answer of the user, and stores the question-answer pair in the memory bank;

[0073] S45: The intelligent agent cleans the information in the memory bank and stores it in the multimodal database, providing incremental training data for the multimodal large model and optimizing the large model.

[0074] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. After the computer program is run, any of the above methods is executed.

[0075] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0076] like Figure 5 , a system based on any of the above methods, comprising:

[0077] Model building module, using ALBEF model architecture to pre-train multimodal large models;

[0078] The data processing module records images and extracts image information of each frame based on the real-time uploaded surveillance video, and performs data preprocessing;

[0079] The risk identification and warning module uses the trained multimodal large model to identify the risk type and risk location of power operations. The images processed by the data processing module are transmitted to the model in real time. The model will output the operation risk category and location, and display the alarm.

[0080] In the intelligent question-and-answer module, users input questions and answers of the image or text type in the search box, and the intelligent agent calls the multimodal large model to recognize the message and reply to the user. At the same time, the intelligent agent performs expansion tasks, which include extracting standard question-and-answer pairs based on the dialogue data and importing the standard question-and-answer pairs into the database.

[0081] The present invention uses a contrastive loss to align image and text data before cross-modal interaction, applies a multimodal encoder, and fuses image features with text features through cross-modal attention, solving the image and text alignment and interaction problems in the multimodal field. The present invention also proposes a knowledge question-answering system based on the interaction between a multimodal large model and an intelligent agent. The intelligent agent memory function can extract standard question-answer pairs from the dialogue data generated by user questions and answers, and store them in a database, which can enhance the intelligent level of multimodal large model question-answering task processing and improve the work efficiency of power generation enterprise operators.

[0082] The present invention can integrate multimodal data of operation images, texts, and videos to identify operation safety risks, and is suitable for various application scenarios such as helmet wearing recognition, smoking recognition, open flame detection, water overflow detection, sleeping on the job recognition, phone call recognition, and personnel gathering recognition, thereby improving the accuracy of operation safety risk identification and enhancing the safety production level of power generation enterprises.

[0083] The invention also discloses a computer-readable storage medium and a computer system respectively.

[0084] Specifically, a computer-readable storage medium stores a computer program on the medium, and after the computer program is run, the above method is executed. A computer system includes a processor and a storage medium, and the storage medium stores a computer program on the storage medium. The processor reads and runs the computer program from the storage medium to execute the above method.

[0085] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0086] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0087] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0088] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0089] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

[0090] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying operational risks of power generation enterprises based on a multimodal large model, characterized in that: The following steps are involved: S1, construct a multimodal power generation enterprise operation safety data set and perform data preprocessing; S2, using the ALBEF large model architecture to establish a multimodal large model, inputting the data set into the above model, and pre-training the multimodal large model; S3, using the trained multi-modal large model to identify safety risks in on-site operations and issue warnings; S4, based on intelligent agents and multimodal big models, realizes the question and answer of work safety knowledge of power generation company staff. The work site images and natural language question instructions are input into the multimodal big model to generate answers.

2. The method according to claim 1, characterized in that: The data set in step S1 includes a production safety text data set, a work site image data set, a work site video data set, and a list of common violations at the work site; the production safety text data set includes but is not limited to production safety standards, safety management standards, standardized operation specifications for maintenance sites, work supervision logs, work plan forms, and a safety risk knowledge question and answer question bank; the work site image data set is the key frames extracted from the work site video.

3. The method according to claim 1, characterized in that: The data preprocessing includes data cleaning, noise reduction, and image enhancement.

4. The method according to claim 1, characterized in that: Step S2 is specifically as follows: using the ALBEF large model architecture to establish a multimodal large model, inputting the multimodal data set into the model, using the image encoder and the text encoder to independently encode the text data and the image data, using the multimodal encoder to fuse the image features and the text features through the cross-modal attention mechanism, determining the pre-training objective function and pre-training the multimodal large model.

5. The method according to claim 4, characterized in that: The objective function is The corresponding expression is: In the formula, represents the image-text contrast learning loss function, represents the masked language modeling loss function, Represents the image-text matching loss function; The image-text contrast learning loss function calculation formula is: In the formula, H represents the cross entropy, y i2t (I), y t2i (T) represents the corresponding ground truth label; The masked language modeling loss function calculation formula is: In the formula, T represents the text after masking, P msk (I, T) represents the model's predicted probability of the token after the mask, y msk represents a one-hot vocabulary distribution, where the probability of the ground truth tag is 1; The image text matching loss function calculation formula is: In the formula, y itm represents a two-dimensional one-hot label, P itm (I,T) represents the probability of two classes.

6. The method according to claim 1, characterized in that The S4 comprises the following steps: S41: The user terminal inputs question information in the form of pictures or texts; S42: The agent obtains the question information from the user and pre-processes the input data; S43: The agent calls the multimodal large model to generate response information; S44: The intelligent agent combines information of the question and the answer of the user, and stores the question-answer pair in the memory bank; S45: The intelligent agent cleans the information in the memory bank and stores it in the multimodal database, providing incremental training data for the multimodal large model and optimizing the large model.

7. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and after the computer program is run, the method according to any one of claims 1 to 6 is executed.

8. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein a computer program is stored in the storage medium, and the processor reads and runs the computer program from the storage medium to execute the method as claimed in any one of claims 1 to 6.

9. A system based on the method according to any one of claims 1 to 6, characterized in that: Includes a model building module that uses the ALBEF model architecture to pre-train large multimodal models; The data processing module records images and extracts image information of each frame based on the real-time uploaded surveillance video, and performs data preprocessing; The risk identification and warning module uses the trained multi-modal large model to identify the risk type and risk location of power operations; The images processed by the data processing module are transmitted to the model in real time. The model will output the operation risk category and location and display an alarm. In the intelligent question-and-answer module, users input questions and answers of the image or text type in the search box, and the intelligent agent calls the multimodal large model to recognize the message and reply to the user. At the same time, the intelligent agent performs expansion tasks, which include extracting standard question-and-answer pairs based on the dialogue data and importing the standard question-and-answer pairs into the database.

Images