Intelligent safety production monitoring method and device

Abstract

The application discloses a kind of safety production intelligent monitoring method and device, the method comprises: using safety production big model, in combination with the video of safety production operation process, job type definition, identify the job type of safety production;Obtain the safety rule corresponding to the job type identified, the safety rule is obtained using safety production big model;Using safety production big model, in combination with the job type and corresponding safety rule, environmental parameter, obtain environmental data;Using multimodal big model, in combination with the job type and corresponding safety rule, environmental data, the video of safety production operation process, obtain risk behavior analysis result;Using safety production big model, in combination with the risk behavior analysis result, obtain risk behavior suggestion measure.The application can efficiently and accurately identify risk behavior, and give suggestion measure.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of safety production data processing and monitoring, and in particular to a safety production intelligent monitoring method and device. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior art is prior art nor does it constitute an admission of any description in this section as prior art to an application described herein and / or in another application also owned by the applicant of the present application.

[0003] Special operations are indispensable in the production and operation process of chemical enterprises, and carrying out special operations, especially carrying out hot work and work in confined spaces, is one of the main reasons for frequent accidents. The main reason for frequent accidents in special operation links is that the enterprise special operation management system is not fully implemented, the risk identification before operation is not clear, the risk control during operation is not in place, and the emergency disposal ability of the guardian is insufficient.

[0004] In order to strengthen the safety risk control of special operation links and curb the occurrence of major production safety accidents during special operations, especially during hot work and work in confined spaces, there are currently laws and regulations that stipulate the safety requirements for special operations such as hot work, work in confined spaces, blind plate plugging work, high-altitude work, hoisting work, temporary power work, earthwork, and circuit breaking work in dangerous chemical enterprises. However, in the current safety production, whether it meets the laws and regulations mainly relies on human judgment, which is prone to judgment errors and low efficiency. SUMMARY

[0005] The embodiments of the present application provide a safety production intelligent monitoring method, which can efficiently and accurately identify risk behaviors by analyzing the video in the safety production process, and give suggestions, including:

[0006] Using a safety production large model, in combination with the video of the safety production operation process and the definition of the operation type, the operation type of safety production is identified, and the safety production large model is obtained by training a machine learning model based on historical video, historical safety rule files, historical environmental parameters and historical suggestions of the safety production operation process;

[0007] Obtain the safety rules corresponding to the identified operation type, which are obtained using the safety production large model;

[0008] Using a safety production large model, in combination with the operation type and the corresponding safety rules and environmental parameters, environmental data is obtained;

[0009] The multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environment data and historical videos of safety production operation processes.

[0010] The risk behavior suggestion measure is obtained by using the safety production large model in combination with the risk behavior analysis result.

[0011] The embodiment of the application provides a safety production intelligent monitoring device, which can efficiently and accurately identify risk behaviors by analyzing videos in a safety production process and give suggestion measures.

[0012] The operation type identification module is configured to identify the operation type of safety production by using a safety production large model in combination with a video of a safety production operation process and operation type definition, wherein the safety production large model is obtained by training a machine learning model based on historical videos of safety production operation processes, historical safety rule files, historical environment parameters and historical suggestion measures.

[0013] The safety rule obtaining module is configured to obtain safety rules corresponding to the identified operation type, wherein the safety rules are obtained by using the safety production large model.

[0014] The environment data obtaining module is configured to obtain environment data by using the safety production large model in combination with the operation type and corresponding safety rules and environment parameters.

[0015] The risk behavior analysis result obtaining module is configured to obtain a risk behavior analysis result by using a multi-modal large model in combination with the operation type and corresponding safety rules, environment data and a video of a safety production operation process, wherein the multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environment data and historical videos of safety production operation processes.

[0016] The suggestion measure obtaining module is configured to obtain a risk behavior suggestion measure by using the safety production large model in combination with the risk behavior analysis result.

[0017] The embodiment of the application further provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the safety production intelligent monitoring method when executing the computer program.

[0018] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the safety production intelligent monitoring method.

[0019] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the safety production intelligent monitoring method.

[0020] In the embodiment of the present application, a safety production large model is used to identify the work type of safety production in combination with the video of the safety production work process and the definition of the work type, the safety production large model is obtained by training a machine learning model based on historical video of the safety production work process, historical safety rule files, historical environment parameters and historical recommended measures, the safety rules corresponding to the identified work type are obtained, the safety rules are obtained by using the safety production large model, the environment data are obtained by using the safety production large model in combination with the work type and the corresponding safety rules and environment parameters, the risk behavior analysis result is obtained by using a multi-modal large model in combination with the work type and the corresponding safety rules, environment data and video of the safety production work process, the multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environment data and historical video of the safety production work process, and the risk behavior recommended measures are obtained by using the safety production large model in combination with the risk behavior analysis result. Compared with the step of monitoring the risk in the safety production process by artificial judgment in the prior art, the method proposed in the embodiment of the present application can automatically and efficiently identify the work type and safety rules, and then accurately analyze the risk behavior analysis result and give the risk behavior recommended measures, the whole process does not need artificial judgment, the efficiency is higher, and the user experience is good. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0022] Figure 1 The flowchart of the safety production intelligent monitoring method in the embodiment of the present application;

[0023] Figure 2 The flowchart of identifying the work type of safety production in the embodiment of the present application;

[0024] Figure 3 The flowchart of constructing the safety rules in the embodiment of the present application;

[0025] Figure 4 The safety rule representation example in the embodiment of the present application;

[0026] Figure 5A flowchart for generating a safety rule in an embodiment of the present application;

[0027] Figure 6 A flowchart for obtaining environmental data in an embodiment of the present application;

[0028] Figure 7 A flowchart for obtaining a risk behavior analysis result in an embodiment of the present application;

[0029] Figure 8 A flowchart for obtaining a risk behavior recommendation measure in an embodiment of the present application;

[0030] Figure 9 An example of a preset recommendation measure output structure in an embodiment of the present application;

[0031] Figure 10 A flowchart for generating a safety production report in an embodiment of the present application;

[0032] Figure 11 A schematic diagram of a safety production intelligent monitoring device in an embodiment of the present application;

[0033] Figure 12 A schematic diagram of a safety production intelligent monitoring system in an embodiment of the present application;

[0034] Figure 13 A schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0035] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, further detailed description of the embodiments of the present application will be given below with reference to the accompanying drawings. Herein, the schematic embodiments of the present application and their descriptions are used to explain the present application, but not as a limitation of the present application.

[0036] Figure 1 A flowchart of a safety production intelligent monitoring method in an embodiment of the present application, the method is applied to a first node in an asynchronous distributed system, the first node is any node in the asynchronous distributed system, and the method comprises:

[0037] Step 101, using a safety production large model, combining a video of a safety production operation process and an operation type definition, identifying an operation type of safety production, the safety production large model is obtained by training a machine learning model based on historical videos of safety production operation processes, historical safety rule files, historical environmental parameters and historical recommendation measures;

[0038] Step 102, obtaining a safety rule corresponding to the identified operation type, the safety rule is obtained using the safety production large model;

[0039] Step 103, using a safety production large model, combining the job type and corresponding safety rules, environmental parameters, obtaining environmental data;

[0040] Step 104, using a multi-modal large model, combining the job type and corresponding safety rules, environmental data, and video of the safety production job process, obtaining risk behavior analysis results, the multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environmental data, and historical video of the safety production job process;

[0041] Step 105, using a safety production large model, combining the risk behavior analysis results, obtaining risk behavior recommendation measures.

[0042] Each step will be described in detail below.

[0043] First of all, it needs to be pointed out that the embodiments of the present application take chemical special operations as an example, but it can be applied to any industry involving safety production, such as mining industry, construction industry, transportation industry, etc.

[0044] In step 101, using a safety production large model, combining the video of the safety production job process and the job type definition, identifying the job type of safety production, the safety production large model is obtained by training a machine learning model based on historical video of the safety production job process, historical safety rule files, historical environmental parameters, and historical recommendation measures;

[0045] In the embodiments of the present application, the safety production large model is obtained by training (fine-tuning) a machine learning model, which is an intelligent model for the field of industrial safety production, realizing job safety prediction and early warning, risk analysis, and multi-modal intelligent question and answer through large language model + industry knowledge + environmental information fusion.

[0046] Figure 2 The flowchart for identifying the job type of safety production in the embodiments of the present application, in an embodiment, using a safety production large model, combining the video of the safety production job process and the job type definition, identifying the job type of safety production, including:

[0047] Step 201, analyzing the video of the safety production job process, if a key event is detected, continuously intercepting multiple frames of images before, during, and after the key event occurs;

[0048] When acquiring images from the monitoring camera, a key event detection algorithm is introduced. By analyzing the motion features, device state changes and other information in the video stream, key events such as device startup, personnel entering the work area, tool use, etc. are identified. When these key events are detected, the frame extraction operation is automatically triggered, and multiple frames of images before, during and after the key event occur are continuously intercepted, ensuring complete recording of the key operation process. For example, in a fire operation, when the ignition action is detected, not only the image at the moment of ignition is intercepted, but also the images of the device preparation state before ignition and the flame situation after ignition are intercepted, providing more comprehensive information for job type identification.

[0049] Subsequently, according to the light conditions, picture complexity and other factors of the work site, the resolution and compression ratio of the image are dynamically adjusted. An image quality evaluation algorithm is used to analyze the video picture in real time. When the picture light is dark or there are complex textures, the image resolution is increased to retain more details; when the picture content is relatively simple, the resolution is appropriately reduced and the compression ratio is optimized to reduce data storage. At the same time, combined with edge computing technology, part of the image processing work is completed on the edge device close to the monitoring camera, reducing data transmission pressure and improving image acquisition efficiency.

[0050] Step 202, according to the risk level and complexity of the safety production operation scene, determine the role of the safety production large model and the corresponding operation type identification instruction.

[0051] A dynamic generation model of the role and operation type identification instruction can be established, which automatically adjusts the role of the safety production large model and the operation type identification instruction according to the risk level of the operation scene. For example, when the safety production operation scene is a restricted space operation and the risk level is high, the role of the safety production large model is set to correspond to the prompt description "You are a chemical high-risk operation type identification specialist", and the operation type identification instruction is "Please check the key safety elements such as personnel protective equipment, ventilation equipment, gas detection instruments in the image according to the provided materials and operation type definition, and make a comprehensive judgment on the operation type"; when the safety production operation scene is a soil moving operation scene and the risk level is low, the role corresponds to the prompt description "You are a basic operation type identification assistant", and the operation type identification instruction is simplified to "Determine the operation type according to the operation tools and operation mode in the image". Through this dynamic adjustment, the safety production large model is more suitable for the identification needs of different operation scenes. The above dynamic generation model can be obtained by training.

[0052] Step 203, determine the text information extracted from the image as the operation type features that the safety production large model needs to pay attention to;

[0053] By utilizing semantic analysis technology in natural language processing, the large-scale safety production model is guided to focus on the operation type features in the image that are related to the text information extracted from the image, such as "there are large amounts of combustible materials piled up on site", thereby enhancing the semantic richness of the prompt words, helping the large-scale model to understand the operation scene more accurately and improve recognition accuracy.

[0054] In step 204, the role played by the large model of safe production and the operation type identification instructions, operation type characteristics, operation type definition and preset output operation type structure are input into the large model of safe production as prompt words to obtain the current operation type of safe production.

[0055] Preset job types include live work, motion work, confined space work, and work at height, with no restrictions here. The prompt word for the large safety production model can be marked as prompt word A. The large safety production model is trained based on a large number of safety production job roles, job type identification instructions, job type characteristics, and defined job types. The preset output job type structure can be in JSON format, with no restrictions here.

[0056] Embodiments of the present invention employ an interactive prompt mechanism to automatically generate supplementary prompts when the reliability of the recognition results output by the safety production model is low. For example, if the safety production model is uncertain about a particular job type, the supplementary prompt could be, "Please further examine the worker's identification card or the nameplate information on the equipment in the image to assist in determining the job type." The supplementary prompt and the original image are then re-entered into the safety production model for secondary recognition. This interactive approach gradually guides the safety production model to acquire more effective information, improving recognition accuracy.

[0057] In an embodiment of the present invention, multiple large-scale safety production models with different architectures and training data are introduced to construct a collaborative recognition network. Each large-scale safety production model has different advantages. For example, some large-scale safety production models are good at identifying the equipment form in complex scenarios, and some large-scale safety production models have a high accuracy rate in identifying personnel actions. The operation process images and prompt words are input into multiple large-scale safety production models at the same time. Each large-scale safety production model independently identifies the operation type, and then through the model fusion algorithm, such as confidence-weighted voting, Bayesian fusion and other methods, the output results of multiple large-scale safety production models are integrated to obtain the final operation type identification conclusion. This heterogeneous model collaboration mode can give full play to the strengths of different large-scale safety production models, reduce the limitations of a single large-scale safety production model, and improve the overall recognition performance.

[0058] In the embodiment of the present application, a dynamic evolution system of the safety production large model is established, and the safety production large model is automatically optimized according to the recognition result feedback in the actual work scene. The difference between the actual work type and the model output result of each recognition is recorded, and when it is found that the recognition error rate of a certain type of work exceeds the threshold, the image, prompt word and correct result of the related work are automatically collected to form a new training data set, and the corresponding safety production large model is incrementally trained. At the same time, the performance of all models is evaluated regularly, the models with poor performance are eliminated, and new safety production large models are introduced to supplement, so as to maintain the advancement and adaptability of the models.

[0059] In addition to the recognized work type, the embodiment of the present application also supports user-specified work type for subsequent safety rules corresponding to the work type.

[0060] Step 102, obtaining the safety rules corresponding to the recognized work type, wherein the safety rules are obtained by using the safety production large model;

[0061] Specifically, the safety rules are obtained by extracting all safety rule files, and different safety rules correspond to different work types. After identifying the work type, the corresponding safety rules can be directly obtained;

[0062] Figure 3 A flowchart for constructing safety rules in the embodiment of the present application is provided. In an embodiment, the method further comprises:

[0063] Step 301, obtaining all safety rule files;

[0064] Step 302, extracting the text content in all safety rule files as safety rule text of the safety production large model;

[0065] In specific implementation, safety production rule files in related industries can be automatically collected, whether in electronic text form or in other forms such as image text. For files in non-electronic text form, advanced OCR technology and deep learning models are integrated to realize high-precision text recognition, accurately converting the text in the image into editable text content. At the same time, parsing of multiple file formats is supported, such as processing Word and PDF files by using regular expression matching and semantic analysis technology, parsing Excel tables by using data pattern recognition algorithm, and extracting key information in PPT documents by using visual element analysis.

[0066] Step 303, determining the role of the safety production large model and the corresponding rule extraction instruction;

[0067] Step 304, the role and the corresponding rule extraction instruction, safety rule text, safety rule example, preset output rule template are taken as the prompt word of the safety production large model, input to the safety production large model, and the safety rule is obtained;

[0068] The preset output rule template can adopt the structure as shown in Figure 4 , Figure 4 The safety rule representation example in the embodiment of the application includes rule abbreviation, complete rule description, rule category (such as personnel safety / equipment safety / environment safety, etc.), severity (low / medium / high / urgent), applicable scene, only: general / personal protection / fire operation / high operation, necessary safety equipment or condition, prohibited items or behavior, environmental requirements, alarm content when risk is found, content that needs to be notified, measures that need to be taken, etc.

[0069] In specific embodiments, the extracted safety rules can be manually intervened, including adding rules, deleting, modifying rules, matching operation types, etc.; the safety rules after manual intervention form a reliable safety production rule library, and the environmental parameters are, for example, wind speed ≤ 5 m / s, relative humidity ≤ 80%, etc.

[0070] Figure 5 The flowchart of the safety rule update generated in the embodiment of the application, in an embodiment, after obtaining the safety rule, further includes:

[0071] Step 501, analyze the context environment in the text content, extract key information, and construct a context environment feature text;

[0072] When analyzing the context environment in the text content, the text content can be cleaned first to remove special symbols, invalid white space characters and other noise data. The text content is processed by a word segmentation tool, and the part of speech is labeled, which prepares for subsequent analysis. The text content after word segmentation is encoded to obtain the semantic vector representation of the text content. The context environment in which the text content is located is analyzed by the semantic vector representation, key information such as related scene keywords (fire operation, high operation, etc.), and degree words (serious, urgent, etc.) describing safety regulations are extracted, and a context environment feature text is constructed.

[0073] Step 502, analyze the obtained safety rule to obtain missing fields and / or ambiguous descriptions;

[0074] It is judged Figure 4 whether the severity field is completely filled in, whether each requirement in the condition field is clear, etc.

[0075] Step 503, based on the context environment feature text and the obtained safety rule, generate a supplementary prompt word;

[0076] For example, if Figure 4 If the severity field is missing, a supplementary prompt word is generated according to the risk consequences described in the text content based on the obtained safety rules.

[0077] In step 504, the supplementary prompt word and the missing field and / or ambiguous description are input into the safety production large model to obtain updated safety rules.

[0078] In step 103, the safety production large model is used to obtain environmental data in combination with the job type, the corresponding safety rules and the environmental parameters.

[0079] Figure 6 For the flowchart of obtaining environmental data in the embodiments of the present application, in an embodiment, the method further comprises:

[0080] In step 601, a safety rule knowledge graph is constructed based on all safety rules, and the safety rule knowledge graph is used to store the relationship between the job type and the corresponding environmental parameters.

[0081] When constructing the safety rule knowledge graph, named entity recognition (NER) technology is needed to label key entities such as job type and environmental parameters. For example, it is identified that the environmental parameters required for fire operation include wind speed ≤ 5 m / s, relative humidity ≤ 80%, etc.

[0082] Using the safety production large model in combination with the job type, the corresponding safety rules and the environmental parameters, the environmental data is obtained, including:

[0083] In step 602, the role of the safety production large model is determined.

[0084] For example, the role of the safety production large model is an environmental perception assistant, and the corresponding prompt description can be "You are an environmental perception assistant".

[0085] In step 603, based on the job type, the safety rule knowledge graph is queried to obtain environmental parameters and serve as environmental parameters.

[0086] In specific embodiments, based on the parsed job type, the related environmental parameters are retrieved in the safety rule knowledge graph. For example, when outdoor construction involves high-altitude operation, environmental parameters such as wind speed, air temperature and rainfall probability need to be obtained.

[0087] In step 604, the time characteristics and spatial characteristics of the job type are analyzed, and environmental data obtaining instructions containing time and space dimensions are obtained according to the safety rules corresponding to the job type.

[0088] For example, for a pipeline welding operation lasting 24 hours, the environmental data obtaining instruction can be described as: "Please obtain the environmental data of the operation area every hour in the next 24 hours according to the provided materials by calling an external environmental perception tool through the MCP protocol".

[0089] In step 605, the role of the safety production large model, the environmental data obtaining instruction, the environmental parameter, the environmental data example, and the preset output environmental data template are used as the prompt words of the safety production large model to call an external environmental perception tool through the MCP protocol to obtain the environmental data.

[0090] The environmental perception tool can be an existing tool, service, or model. The safety production large model calls the environmental perception tool through the MCP protocol to obtain the desired environmental data.

[0091] In step 104, a multi-modal large model is used to obtain a risk behavior analysis result in combination with the job type, the corresponding safety rules, the environmental data, and the video of the safety production operation process. The multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environmental data, and historical videos of safety production operation processes.

[0092] Figure 7 For the flowchart of obtaining the risk behavior analysis result in the embodiment of the present application, in an embodiment, a multi-modal large model is used to obtain a risk behavior analysis result in combination with the job type, the corresponding safety rules, the environmental data, and the video of the safety production operation process, including:

[0093] In step 701, the role of the safety production large model is determined.

[0094] For example, the role of the safety production large model can be described as: "You are a professional safety consultant specializing in safety issues in industrial scenarios."

[0095] In step 702, the constraint conditions in the safety rules are extracted.

[0096] In specific implementation, the safety rule text can be semantically encoded, and then the core intent of the semantic encoding is identified by an intent classification model to extract the constraint conditions in the safety rules, such as wearing a safety belt and the device grounding resistance ≤ 4Ω.

[0097] In step 703, the emotional intensity and the corresponding expression method are determined based on the risk level of the safety production operation scene and the constraint conditions.

[0098] In specific implementation, an emotional mapping function can be designed to map the risk level to the emotional intensity, such as high risk corresponding to serious / urgent tone, and a tone library can be constructed to include expression methods of different emotional intensities, such as must strictly comply with and suggest for reference.

[0099] Step 704, based on the emotional intensity and corresponding expression, determine the non-safety risk analysis instruction;

[0100] For example, the non-safety risk analysis instruction can be "please combine the operation type, safety rules, environmental data and non-safety risk examples to determine whether the operation behavior in the image conforms to the specification, and output according to the preset output structure".

[0101] Step 705, according to the risk point distribution map corresponding to the operation type, determine the frame extraction frequency of each risk point of the video of the safety production operation process, and extract frames from the video of the safety production operation process according to the frame extraction frequency of each risk point, to obtain images.

[0102] The video of the safety production operation process can be divided into time sequence segments in chronological order. At the same time, the standard operation steps of this operation type are sorted out, the possible risk points in each step are determined, and a risk point distribution map is established, which corresponds one-to-one between the risk points and the operation process time nodes. For example, in the fire operation process, the risk points corresponding to the steps of ignition and welding are fire and explosion.

[0103] Based on time series analysis, combine the operation process and risk point distribution to set the initial frame extraction frequency. Before the operation starts, according to the operation type and risk level, different frame extraction frequencies are preset for different stages. For example, in the fire operation, the frame extraction frequency can be set to 1-2 pictures per minute in the non-critical operation stage; while in the key operation stage such as ignition and welding, the frame extraction frequency is increased to 1-2 pictures per second, to ensure that the key operation details are fully recorded.

[0104] In specific implementation, the target detection algorithm can be used to analyze the monitoring video screen in real time. The target detection algorithm continuously scans the objects, behaviors and other elements in the screen, extracts frames from the video of the safety production operation process according to the frame extraction frequency of each risk point, and obtains images.

[0105] Step 706, input the role of the safety production large model, the non-safety risk analysis instruction, the environmental data, the non-safety risk example and the preset output structure as the prompt words of the multi-modal large model into the safety production large model, to obtain the risk behavior analysis result;

[0106] The safety production large model is obtained after training the machine learning model.

[0107] The risk behavior analysis result includes information such as whether a key behavior conforms to a safety specification, whether a device state is normal, and whether a scene has a potential risk. The analysis result is presented in a visual form, such as a risk heat map or a key behavior detection report, to provide a safety management personnel with an intuitive and accurate decision basis, so that corresponding measures can be taken in a timely manner to ensure work safety.

[0108] In step 105, a risk behavior recommendation measure is obtained by using the safety production large model in combination with the risk behavior analysis result.

[0109] During training (fine-tuning), the safety production large model can be trained based on historical recommendation measures at the same time, so that the safety production large model can intelligently analyze risk behavior recommendation measures. Specifically, the risk behavior analysis result can be fused with multi-source data such as historical accident case data and industry best practice data to jointly constitute the content of a prompt word. By analyzing the processing mode of similar risk behaviors in historical accident cases and the safety measures in industry best practices, the connotation of the prompt word is enriched, the safety production large model is guided to learn from past experience, and more feasible and effective recommendation measures are output, thereby realizing safety production analysis and recommendation generation based on multi-dimensional data.

[0110] Figure 8 For the flowchart of obtaining the risk behavior recommendation measure in the embodiment of the present application, in an embodiment, a risk behavior recommendation measure is obtained by using the safety production large model in combination with the risk behavior analysis result, including:

[0111] Step 801, determining a role played by the safety production large model and a recommendation measure instruction;

[0112] For example, the role played by the safety production large model can be "you are a professional production safety inspection expert", and the recommendation measure instruction can be "please analyze the safety rules, risk description, severity, and recommendation measures violated by the risk behavior analysis result".

[0113] Step 802, forming a prompt word of the safety production large model by using the historical safety accident case data, the preset recommendation measure output structure, the role played by the safety production large model, and the recommendation measure instruction, and inputting the prompt word into the safety production large model to obtain a recommendation measure.

[0114] Figure 9 For an example of the preset recommendation measure output structure in the embodiment of the present application, it includes a safety rule identifier, a risk description, a severity, and a recommendation measure.

[0115] Figure 10 For the flowchart of generating the safety production report in the embodiment of the present application, in an embodiment, the method further includes:

[0116] Step 1001, determine the role of the safety production large model and the safety report generation instruction;

[0117] For example, the role of the safety production large model can be "you are a safety production report making expert", and the safety report generation instruction can be "please combine the risk behavior analysis result and the recommended measures to form a detection report".

[0118] Step 1002, form the prompt words of the safety production large model by combining the risk behavior analysis result and the recommended measures, the preset safety report output structure, the role of the safety production large model and the safety report generation instruction, and input them into the safety production large model to obtain the safety report.

[0119] During training (fine-tuning) of the safety production large model, the safety production large model can be trained based on historical safety reports at the same time, so that the safety production large model can intelligently generate a safety report.

[0120] In specific implementation, the preset safety report output structure can adopt a markdown format, and the final safety report can include an overall situation overview, main problem analysis, risk level, risk behavior analysis result and recommended measures, follow-up tracking plan, etc.

[0121] The embodiment of the application also proposes a safety production intelligent monitoring device, which has a principle similar to the safety production intelligent monitoring method, and will not be described here.

[0122] Figure 11 The safety production intelligent monitoring device is shown in the schematic diagram of the embodiment of the application, which includes:

[0123] The work type identification module 1101 is configured to use the safety production large model to identify the work type of safety production by combining the video of the safety production work process and the work type definition, wherein the safety production large model is obtained by training a machine learning model based on historical video of the safety production work process, historical safety rule files, historical environmental parameters and historical recommended measures.

[0124] The safety rule obtaining module 1102 is configured to obtain the safety rule corresponding to the identified work type, wherein the safety rule is obtained by using the safety production large model.

[0125] The environmental data obtaining module 1103 is configured to use the safety production large model to obtain environmental data by combining the work type and the corresponding safety rule and environmental parameters.

[0126] The risk behavior analysis result obtaining module 1104 is configured to obtain a risk behavior analysis result by using a multi-modal large model in combination with the job type, the corresponding safety rule, the environmental data, and the video of the safety production job process, wherein the multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environmental data, and historical videos of safety production job processes;

[0127] The suggestion measure obtaining module 1105 is configured to obtain a risk behavior suggestion measure by using a safety production large model in combination with the risk behavior analysis result.

[0128] In an embodiment, the job type identification module 1101 is configured to:

[0129] analyze the video of the safety production job process, and if a key event is detected, continuously capture multiple frames of images before, during, and after the key event;

[0130] determine a role of the safety production large model and a corresponding job type identification instruction according to the risk level and the complexity of the safety production job scene;

[0131] determine the textual information extracted from the images as a job type feature that needs to be focused on by the safety production large model;

[0132] input the role of the safety production large model and the job type identification instruction, the job type feature, the job type definition, and the preset output job type structure as prompt words of the safety production large model to the safety production large model, and obtain the job type of the current safety production.

[0133] In an embodiment, the apparatus further includes a safety rule construction module configured to:

[0134] obtain all safety rule files;

[0135] extract the text content in all safety rule files as safety rule text of the safety production large model;

[0136] determine a role of the safety production large model and a corresponding rule extraction instruction;

[0137] input the role and the corresponding rule extraction instruction, the safety rule text, the safety rule example, and the preset output rule template as prompt words of the safety production large model to the safety production large model, and obtain the safety rule.

[0138] In an embodiment, the safety rule construction module is further configured to:

[0139] after obtaining the safety rule, analyze the context environment in the text content by using a pre-trained model, extract key information, and construct a context environment feature text;

[0140] analyzing the obtained safety rules to obtain missing fields and / or ambiguous descriptions;

[0141] generating a supplementary prompt word based on the context environment feature text and the obtained safety rules;

[0142] inputting the supplementary prompt word, and the missing fields and / or ambiguous descriptions into the safety production large model to obtain updated safety rules.

[0143] In an embodiment, the safety rule construction module is further configured to:

[0144] constructing a safety rule knowledge graph based on all safety rules, the safety rule knowledge graph being configured to store relationships between job types and corresponding environment parameters;

[0145] The environment data obtaining module 1103 is configured to:

[0146] determining a role of the safety production large model;

[0147] querying the safety rule knowledge graph based on the job type to obtain an environment parameter and serving as an environment parameter;

[0148] analyzing time characteristics and space characteristics of the job type, and obtaining an environment data obtaining instruction containing a time dimension and a space dimension according to a safety rule corresponding to the job type;

[0149] inputting the role of the safety production large model, the environment data obtaining instruction, the environment parameter, an environment data example, and a preset output environment data template as a prompt word of the safety production large model into the safety production large model to obtain environment data.

[0150] In an embodiment, the risk behavior analysis result obtaining module 1104 is configured to:

[0151] determining a role of the safety production large model;

[0152] extracting a constraint condition in the safety rule;

[0153] determining an emotional intensity and a corresponding expression manner based on a risk level of a safety production job scene and the constraint condition;

[0154] determining a non-safety risk analysis instruction based on the emotional intensity and the corresponding expression manner;

[0155] determining a frame extraction frequency of each risk point of a video of a safety production job process according to a risk point distribution graph corresponding to the job type, and extracting frames of the video of the safety production job process according to the frame extraction frequency of each risk point to obtain an image;

[0156] The playing role of the safety production large model, the non-safety risk analysis instruction, the environmental data, the non-safety risk example and the preset output structure are input into the safety production large model as prompt words of the multi-modal large model, and a risk behavior analysis result is obtained.

[0157] In an embodiment, the recommended measure obtaining module 1105 is configured to:

[0158] determine the playing role of the safety production large model and the recommended measure instruction;

[0159] form the playing role of the safety production large model and the recommended measure instruction into prompt words of the safety production large model, and input the prompt words into the safety production large model to obtain the recommended measure.

[0160] In an embodiment, the device further comprises a safety report generating module configured to:

[0161] determine the playing role of the safety production large model and the safety report generation instruction;

[0162] form the risk behavior analysis result and the recommended measure, the preset safety report output structure, the playing role of the safety production large model and the safety report generation instruction into prompt words of the safety production large model, and input the prompt words into the safety production large model to obtain the safety report.

[0163] Figure 12 FIG. 1 is a schematic diagram of a safety production intelligent monitoring system according to an embodiment of the present application. The system is used to implement a safety production intelligent monitoring device. The safety production intelligent monitoring system comprises an infrastructure layer, a data layer, a model layer and an application layer.

[0164] The infrastructure layer provides the system with access to a monitoring camera, a video file and a safety production rule file.

[0165] The safety rule database and the risk behavior analysis result database of the data layer are respectively used to save safety rules extracted from the safety production rule file and artificially set safety rules, job record related data and risk behavior analysis results.

[0166] The model layer provides a safety production large model, a multi-modal large model and other models mentioned in the embodiments of the present application.

[0167] The application layer comprises a user interface, an environment perception service, a job type identification service, a rule extraction service, a safety detection service and a video detection service.

[0168] The user interface is an entry for user operation in the entire system.

[0169] The rule extraction service extracts safety rules from the safety production rule file through various models and saves them to the database. The constructed safety rules can support manual intervention.

[0170] The job management service is responsible for the management of jobs and related data.

[0171] The video processing service is responsible for pulling streams from monitoring cameras, saving videos at intervals, and obtaining job frames from videos and delivering them to the job detection service for detection.

[0172] The job detection service detects job frames through a multi-modal large model, and the detection results are saved to the database.

[0173] In summary, in the method and device proposed in the embodiment of the application, a safety production large model is used to identify the job type of safety production in combination with the video of the safety production job process and the job type definition. The safety production large model is obtained by training a machine learning model based on historical video of the safety production job process, historical safety rule files, historical environment parameters, and historical recommended measures. The safety rules corresponding to the identified job type are obtained, and the safety rules are obtained using the safety production large model. The environment data is obtained using the safety production large model in combination with the job type and the corresponding safety rules and environment parameters. The risk behavior analysis result is obtained using a multi-modal large model in combination with the job type and the corresponding safety rules, environment data, and video of the safety production job process. The multi-modal large model is obtained by training a machine learning model based on historical safety rules, historical environment data, and historical video of the safety production job process. The risk behavior recommended measures are obtained using the safety production large model in combination with the risk behavior analysis result. Compared with the step of monitoring the risk in the safety production process by manual judgment in the prior art, the method proposed in the embodiment of the application can automatically and efficiently identify the job type and safety rules, and then accurately analyze the risk behavior analysis result and give the risk behavior recommended measures. The entire process does not require manual judgment, is more efficient, and has good user experience. In addition, all the models proposed in the embodiment of the application can be continuously updated and iterated, so that the models understand the safety rules more deeply and adapt to multi-modal information fusion: fusion of text, image, video, audio, and environment data, to realize multi-dimensional and stereoscopic perception and accurate judgment of complex job scenes, which is significantly better than traditional image recognition models. Relying on the powerful general understanding and reasoning ability of the large model, it can adapt to different enterprises, different job types, and complex scenes, and is easy to migrate and expand. It can realize a closed-loop job process from risk detection, alarm triggering, rule comparison to report generation, greatly improving management efficiency and response speed.

[0174] The embodiment of the application also provides a computer device, Figure 13Fig. 13 is a schematic diagram of a computer device according to an embodiment of the present application. The computer device 1300 comprises a memory 1310, a processor 1320, and a computer program 1330 stored in the memory 1310 and executable on the processor 1320, wherein the processor 1320 implements the data visualization method based on the data exchange model as described above when executing the computer program 1330.

[0175] The embodiment of the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the data visualization method based on the data exchange model.

[0176] The embodiment of the present application also provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to implement the data visualization method based on the data exchange model.

[0177] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s) and / or combinations thereof.

[0179] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s) and / or combinations thereof.

[0180] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0181] The above described specific embodiments, the purpose, technical solutions and beneficial effects of the present application are further described in detail, it should be understood that the above described is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for intelligent monitoring of production safety, characterized in that: include: A large production safety model is used to identify the type of production safety operation in combination with videos of the production safety operation process and job type definitions. The large production safety model is obtained by training a machine learning model based on historical videos of the production safety operation process, historical safety rule files, historical environmental parameters, and historical recommended measures. A dynamic generation model for role-playing and job type identification instructions is established, and the role-playing and job type identification instructions of the large production safety model are automatically adjusted according to the risk level of the operation scenario. When the reliability of the recognition result output by the large production safety model is low, supplementary prompt words are automatically generated. The supplementary prompt words and the original image are re-entered into the large production safety model for secondary recognition. The large production safety model is automatically optimized based on the recognition result feedback in the actual operation scenario. Obtaining safety rules corresponding to the identified job type, wherein the safety rules are obtained using a large production safety model; Use the large safety production model to obtain environmental data by combining the operation type and corresponding safety rules and environmental parameters; Using a multimodal large model, combining the operation type with corresponding safety rules, environmental data, and videos of safe production operations, to obtain risk behavior analysis results. The multimodal large model is obtained by training a machine learning model based on historical safety rules, historical environmental data, and historical videos of safe production operations. Use the large production safety model and the risk behavior analysis results to obtain recommended risk behavior measures; Also includes: Obtain all safety rule files; extract the text content in all safety rule files and use it as the safety rule text of the safety production big model; determine the role played by the safety production big model and the corresponding rule extraction instructions; use the role played and the corresponding rule extraction instructions, safety rule text, safety rule examples, and preset output rule templates as prompt words of the safety production big model, input them into the safety production big model to obtain safety rules; after obtaining the safety rules, analyze the context in the text content, extract key information, and construct context feature text; analyze the obtained safety rules to obtain missing fields and / or fuzzy descriptions; generate supplementary prompt words based on the context feature text and the obtained safety rules; input the supplementary prompt words, as well as the missing fields and / or fuzzy descriptions into the safety production big model to obtain updated safety rules.

2. The method according to claim 1, wherein Use the safety production model, combined with videos of safety production operation processes and operation type definitions, to identify safety production operation types, including: Analyze videos of production safety operations and, if a critical event is detected, continuously capture multiple frames of images before, during, and after the critical event occurs. Determine the role of the safety production model and the corresponding operation type identification instructions based on the risk level and complexity of the safety production operation scenario; The text information extracted from the image is determined as the operation type characteristics that the large-scale safe production model needs to pay attention to; The role played by the large model of safe production and the operation type identification instructions, operation type characteristics, operation type definition and preset output operation type structure are input into the large model of safe production as prompt words to obtain the current operation type of safe production.

3. The method according to claim 1, wherein Also includes: Based on all security rules, a security rule knowledge graph is constructed, which is used to store the relationship between job types and corresponding environmental parameters; Using the large safety production model, combined with the operation type and corresponding safety rules and environmental parameters, environmental data is obtained, including: Determine the role of the large-scale safety production model; Based on the job type, query the safety rule knowledge graph to obtain environmental parameters and use them as environmental parameters; Analyze the temporal and spatial characteristics of the operation type, and obtain instructions for obtaining environmental data including temporal and spatial dimensions according to the safety rules corresponding to the operation type; The role played by the large-scale safe production model, instructions for obtaining environmental data, environmental parameters, environmental data examples and preset output environmental data templates are used as prompt words of the large-scale safe production model to call external environmental perception tools through the MCP protocol to obtain environmental data.

4. The method according to claim 1, wherein Using a multimodal large model, combined with the operation type and corresponding safety rules, environmental data, and videos of safe production operation processes, we can obtain risk behavior analysis results, including: Determine the role of the large-scale safety production model; Extracting constraints from the security rules; Determine the emotional intensity and corresponding expression based on the risk level of the safe production operation scenario and the constraints; Determining a non-safety risk analysis instruction based on the emotion intensity and the corresponding expression; Determine, based on a risk point distribution map corresponding to the operation type, a frame extraction frequency for each risk point in a video of the safe production operation process, and extract frames from the video of the safe production operation process according to the frame extraction frequency for each risk point to obtain an image; The roles played by the large-scale safe production model, non-safety risk analysis instructions, environmental data, non-safety risk examples and preset output structure are used as prompt words of the multimodal large-scale model and input into the large-scale safe production model to obtain the risk behavior analysis results.

5. The method according to claim 1, wherein Using the large production safety model and combining the risk behavior analysis results, we can obtain recommended measures for risk behavior, including: Determine the roles and recommended measures of the safety production model; The historical safety accident case data, the preset recommended measures output structure, the role played by the large safety production model and the recommended measures instructions are formed into the prompt words of the large safety production model, and input into the large safety production model to obtain the recommended measures.

6. The method according to claim 1, wherein Also includes: Determine the roles played by the safety production model and the instructions for generating safety reports; The risk behavior analysis results and recommended measures, the preset safety report output structure, the role played by the safety production model and the safety report generation instructions are formed into prompt words of the safety production model and input into the safety production model to obtain a safety report.

7. An intelligent production safety monitoring device, characterized in that: include: The job type recognition module is used to identify the job type of safe production using a large safety production model, combined with videos of safe production operation processes and job type definitions. The large safety production model is obtained by training a machine learning model based on historical videos of safe production operation processes, historical safety rule files, historical environmental parameters, and historical recommended measures. Among them, a dynamic generation model for role playing and job type recognition instructions is established, and the role playing and job type recognition instructions of the large safety production model are automatically adjusted according to the risk level of the operation scenario. When the recognition result output by the large safety production model has low credibility, supplementary prompt words are automatically generated; the supplementary prompt words and the original image are re-entered into the large safety production model for secondary recognition; and the large safety production model is automatically optimized based on the recognition result feedback in the actual operation scenario. A safety rule acquisition module is used to obtain safety rules corresponding to the identified job type, wherein the safety rules are obtained using a large safety production model; An environmental data acquisition module is used to obtain environmental data using a large safety production model in combination with the operation type and corresponding safety rules and environmental parameters; A risk behavior analysis result acquisition module is used to obtain risk behavior analysis results using a multimodal large model, combining the operation type and corresponding safety rules, environmental data, and videos of safe production operation processes. The multimodal large model is obtained by training a machine learning model based on historical safety rules, historical environmental data, and historical videos of safe production operation processes. A suggested measures acquisition module is used to use the large production safety model and the risk behavior analysis results to obtain risk behavior suggested measures; The security rule construction module is used to: obtain all security rule files; extract the text content in all security rule files and use it as the security rule text of the safe production big model; determine the role played by the safe production big model and the corresponding rule extraction instructions; use the role played and the corresponding rule extraction instructions, security rule text, security rule examples, and preset output rule templates as prompt words of the safe production big model, input them into the safe production big model to obtain security rules; after obtaining the security rules, use the pre-trained model to analyze the context in the text content, extract key information, and construct context feature text; analyze the obtained security rules to obtain missing fields and / or fuzzy descriptions; generate supplementary prompt words based on the context feature text and the obtained security rules; input the supplementary prompt words, as well as the missing fields and / or fuzzy descriptions into the safe production big model to obtain updated security rules.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images