Safety production supervision system and method based on multi-modal large model

Abstract

The invention discloses a safety production supervision system and method based on a multi-modal large model, and aims to solve the problem of large inspection workload of manual checking of potential safety hazards in an existing enterprise production scene. By acquiring image information and position information of a hidden danger scene to be recognized, potential safety hazards possibly occurring in the enterprise production process are detected and marked by utilizing technologies such as a visual large model and a large language model, and rectification suggestions are given according to corresponding safety regulations. The system can monitor the potential safety hazards in the production process of an enterprise in an all-weather manner, remarkably improves the potential safety hazard monitoring efficiency, reduces the manual workload, reduces the labor cost, and provides a basic guarantee for the safety production of the enterprise.

Description

Technical Field

[0001] The present invention belongs to the technical field of safe production, and in particular relates to a safe production supervision system and method based on a multi-modal large model. Background Art

[0002] There are potential safety hazards in the daily production process of enterprises. In order to reduce and effectively avoid potential safety hazards, potential safety hazard detection and investigation work is essential. At present, the work of enterprise potential safety hazard investigation mainly relies on safety management personnel to go to the site for manual observation and investigation in accordance with the safety standards and specifications of each enterprise. However, the ability of various safety management personnel in production enterprises to discover professional potential safety hazards varies. At the same time, the huge inspection workload can easily cause fatigue of safety management personnel, resulting in inadequate discovery of safety issues and easy omission of abnormal phenomena. Therefore, standardization and automation of identifying production safety hazards and reducing the workload of manual inspections have become important requirements in enterprise safety production. Summary of the invention

[0003] To solve the above problems, the present invention discloses a production safety supervision system and method based on a multimodal large model, which can monitor the safety hazards in the production process of an enterprise around the clock, significantly improve the efficiency of hazard monitoring, reduce manual workload, reduce labor costs, and provide basic guarantees for the safe production of the enterprise.

[0004] To achieve the above object, the technical solution of the present invention is as follows: A safety production supervision system based on a multimodal large model, comprising: Multimedia information collection terminals, used to collect image information and location information in enterprise production scenarios, including but not limited to monitoring equipment, smart phones, and patrol glasses with shooting and positioning functions; The visual model is used to detect, identify and describe image content information and generate image descriptions; A database storing the correspondence between location information and hidden danger inspection scope, the correspondence between hidden danger description words and hidden danger standard prompt words, and the hidden danger information finally detected; A text similarity comparison module, used for receiving the image description and the hidden danger description and outputting the text similarity between them; A threshold judgment module is used to judge whether the similarity of a given text is higher than a given threshold; Standard prompt word matching module: used to match the hidden danger standard prompt words corresponding to the hidden danger description in the database. The hidden danger standard prompt words are suitable for subsequent image segmentation and annotation and knowledge base retrieval tasks; An image segmentation and annotation module, including an image segmentation and annotation model, is used to segment and annotate hidden danger areas on hidden danger images based on hidden danger standard prompt words; The knowledge base retrieval module includes the RAG knowledge base and the large language model, which is used to retrieve the sources of hidden dangers in the safety standard documents in the RAG knowledge base, and summarize and output the hidden danger rectification measures using the large language model; The result output module is used to output and record the hidden danger inspection results, including whether there are hidden dangers, the time and location of hidden dangers, the name of hidden danger type, hidden danger segmentation and annotation images, suggestions for hidden danger rectification measures, and early warning prompts.

[0005] Furthermore, the knowledge base retrieval module includes the RAG knowledge base and the large language model: For the RAG knowledge base, the text in the safety standard document is divided into different text segments, which can be used to complete the semantic matching retrieval between each text segment and the hidden danger standard prompt words, and the text content with the highest matching degree is used as the source information of the hidden danger in the safety standard document; The large language model, including the large language model deployed locally or in the cloud, analyzes and summarizes the source information of the hidden dangers in the safety standard documents retrieved from the RAG knowledge base and the preset guide words to obtain suggestions for hidden danger rectification measures.

[0006] Furthermore, the text segmentation of the safety standard document in the RAG knowledge base is as follows: when faced with a text with multi-level title marking, firstly, based on the first-level title, the content under each first-level title is divided into a corresponding number of text segments; then based on the second-level title, the content under the second-level title is segmented again, and so on, the text content is segmented hierarchically according to different levels of titles, so that the segmented text segments contain both long text fields containing the overall context and contextual connections of the text, and short text fields that are only related to specific sub-topics, focus on local key information, and can accurately adapt to specific retrieval directions; when faced with a text that has not been calibrated with multi-level titles, the text is sent to a large language model, and the large language model performs multi-level title calibration. After the calibration is completed, the text segmentation steps with multi-level title calibration are followed.

[0007] The safety production supervision method implemented based on the above system can collect image information and location information in the enterprise production scene, use visual analysis, natural language processing and other technologies to detect whether there are safety hazards, segment and mark the detected hazards, provide rectification measures and early warnings. The steps are as follows: S1. The multimedia information collection terminal collects image information and location information in the enterprise production scene; S2, passing the image information into the visual model to generate image description; S3, searching the database for a hidden danger description within the hidden danger inspection range that matches the corresponding location information; S4, the image description and the hidden danger description are passed into the text similarity comparison module to obtain the text similarity between the two. S5: The text similarity is passed to the threshold judgment module, and the relationship between the text similarity and the threshold is used to determine whether there is a hidden danger. S6. According to the hidden danger description words of the existing hidden dangers, the corresponding hidden danger standard prompt words are matched in the database; S7, transferring the image information and the standard hidden danger prompt words into the image segmentation and annotation module to perform segmentation and annotation of the hidden danger area; S8. The hidden danger standard prompt words are transferred to the knowledge base retrieval module, the source of the hidden danger in the safety standard document is searched in the RAG knowledge base, and the hidden danger rectification measures are summarized and outputted using the large language model; S9. Output and record the hidden danger inspection results, including whether there are hidden dangers, the time and location of hidden dangers, the name of hidden danger type, hidden danger segmentation and annotation images, suggestions for hidden danger rectification measures, and provide early warnings to safety management personnel.

[0008] Furthermore, step S3 searches the database for the hidden danger range that should be checked for the corresponding location information, retrieves the input location information in the database, and outputs descriptions of multiple different hidden dangers within the hidden danger checking project range that matches the location information.

[0009] Furthermore, in step S8, the safety standard document is pre-divided into text segments of different lengths in the RAG knowledge base according to the hierarchical segmentation rules, and a parameter N is set. The parameter N is the number of sources of hidden dangers in the safety standard document retrieved in the RAG knowledge base. When the hidden danger standard prompt words are passed into the knowledge base retrieval module, the semantic similarity between the segmented text segments and the hidden danger standard prompt words is calculated, and the first N text segments with the highest semantic similarity to the hidden danger standard prompt words in the text segment are passed into the large language model.

[0010] The beneficial effects of the present invention are: The present invention detects and troubleshoots potential safety hazards in the production process of enterprises through the application of artificial intelligence methods such as large visual models and large language models. It can monitor potential safety hazards in the production scenes of enterprises around the clock. It has high timeliness and unified output format standards, significantly improves the frequency of potential safety hazard detection, reduces manual workload, and provides a basic guarantee for the safe production of enterprises. By obtaining the location information corresponding to the image information and querying the corresponding hidden danger inspection range in the database, the scope of hidden danger inspection is narrowed, the time for hidden danger detection is shortened, the speed of hidden danger detection is improved, and hidden dangers can be warned and handled more timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of the system structure of the present invention.

[0012] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0014] The present invention describes a production safety supervision system and method based on a multimodal large model. In order to address the current problem that safety hazard detection in enterprise production relies on manual labor and has a large inspection workload, it uses artificial intelligence-based methods such as visual large models and large language models to detect and detect safety hazards in the enterprise production process, thereby reducing manual workload and performing safety hazard monitoring and management around the clock.

[0015] like Figure 1 As shown, the system structure of the present invention comprises: The multimedia information collection terminal is responsible for transferring the image information into the visual model and the location information into the database; The visual model is responsible for generating image descriptions and passing them to the text similarity comparison module; The database is responsible for transferring the description of the hidden dangers to be checked to the text similarity comparison module; The text similarity comparison module is responsible for calculating the text similarity between the image description and the description of the hidden danger to be checked and passing it to the threshold judgment module; The threshold judgment module is responsible for comparing whether the similarity of the input text is higher than a given threshold and judging whether there are hidden dangers; The standard prompt word matching module is responsible for transferring the hidden danger standard prompt words corresponding to the hidden danger description words to the image segmentation and annotation module and the knowledge base retrieval module; The image segmentation and annotation module is responsible for segmenting and annotating the potential danger areas in the image; The knowledge base retrieval module is responsible for searching the sources of hidden dangers in safety standard documents and outputting suggestions for hidden danger rectification measures; The result output module is responsible for displaying the results of hidden danger inspection.

[0016] like Figure 2 As shown, the specific steps of implementing the present invention include: S1. Multimedia information collection terminal collects image information and location information in enterprise production scenarios: Utilize terminal devices that can obtain image information and location information, including but not limited to monitoring equipment, smart phones, and patrol glasses with shooting and positioning functions, to collect image information and corresponding location information in the enterprise's production scenarios. For example, a certain image information is an image taken by a mobile phone in the enterprise's warehouse No. 2, and the location information is obtained by the positioning function of the mobile phone: "Warehouse No. 2".

[0017] S2, pass the image information into the visual model and generate image description: Based on the image information collected by the multimedia information collection terminal, a visual large model such as CLIP generates an image description corresponding to the image information, such as "there is a fire extinguisher on the left side of the image, and there are rust marks on the fire extinguisher."

[0018] S3. Search the database for the hidden danger description within the hidden danger inspection range that matches the corresponding location information: According to the location information collected by the multimedia information collection terminal, the hidden danger description words within the hidden danger inspection range that matches the corresponding location information are searched in the database. For example, if the location information is Warehouse No. 2, the corresponding hidden danger description words within the hidden danger inspection range include "fire-fighting equipment is damaged, rusted, and blocked" and "safety exit signs are tilted and detached."

[0019] S4. The image description and the hidden danger description are passed into the text similarity comparison module to obtain the text similarity between them: The image description words, such as "there is a fire extinguisher in the image, and there are rust marks on the fire extinguisher", and the hidden danger description words, such as "the fire extinguisher is stained, rusted, and blocked", "the safety exit sign is tilted and falling off", are compared for text similarity, and the semantic similarity between the texts is calculated. The text similarity comparison algorithm includes the Word2Vec algorithm, etc., to obtain the text similarity between them.

[0020] S5. The text similarity is passed to the threshold judgment module, and the relationship between the text similarity and the threshold is used to determine whether there is a hidden danger: The similarity between the image description and the hidden danger description is compared with the preset text similarity threshold. If the similarity between the image description and a hidden danger description is greater than or equal to the threshold, it is determined that the hidden danger corresponding to the hidden danger description exists in the image. If the similarity between the image description and a hidden danger description is less than the threshold, it is determined that the hidden danger corresponding to the hidden danger description does not exist in the image. If there is no hidden danger, jump to step S9 to directly output that there is no hidden danger. If there is a hidden danger, perform the following steps.

[0021] S6. According to the description of the hidden danger, the corresponding standard prompt words of the hidden danger are matched in the database: According to the hidden danger description, the database is queried to match the hidden danger description with the corresponding hidden danger standard prompt word. For example, if the hidden danger description is "the fire extinguisher is rusted", the matching corresponding hidden danger standard prompt word is "the fire extinguisher box is rusted". The hidden danger standard prompt word can be better suitable for subsequent image segmentation and annotation and knowledge base retrieval tasks.

[0022] S7, the image information and the standard hidden danger prompt words are transmitted to the image segmentation and annotation module to perform hidden danger area segmentation and annotation: Use an image segmentation and annotation model, such as Grounded-SAM, to detect the pixel coordinate positions of hidden dangers in the image based on the input image information and standard hidden danger prompt words, add a dark semi-transparent mask to the hidden danger area, and annotate the standard hidden danger prompt words with text next to the hidden danger area.

[0023] S8. The hidden danger standard prompt words are passed into the knowledge base retrieval module, and the source of the hidden danger in the safety standard document is searched in the RAG knowledge base. The large language model is used to summarize and output the hidden danger rectification measures: First, the safety standard document text is pre-segmented into segments according to a specific segmentation method. The segmentation method is multi-level title loop nested text segmentation, including: If the safety standard document has been calibrated with multi-level titles (i.e., it contains first-level, second-level, third-level and above titles), the hierarchical segmentation mechanism is activated; first, based on the first-level title, the content under it is divided into independent text segments one by one, and the number of segmented text segments is consistent with the number of first-level titles; then, based on the first-level title, the content under each second-level title is divided into independent text segments one by one, and the number of segmented text segments is consistent with the number of second-level titles; and so on, the subsequent titles at each level are segmented in turn until the last level; if the safety standard document has not been calibrated with multi-level titles (i.e., it does not contain first-level, second-level, third-level and above titles), the safety standard document is now input into the large language model, and with the help of the understanding and analysis capabilities of the large language model, the text is calibrated with multi-level titles. After the title calibration is completed, the above-mentioned established segmentation process is followed.

[0024] The RAG knowledge base will match the first N segments (N can be set freely) with the highest semantic similarity in the segmented text segments according to the input standard hidden danger prompt words, pass these N segments into the large language model, and send guide words at the same time, for example, "The safety hidden danger detected by the user is: <Standard hidden danger prompt words>, please give suggestions for hidden danger rectification measures based on the knowledge base search content."

[0025] S9. Output and record the hidden danger inspection results, including whether there are hidden dangers, the time and location of hidden dangers, the name of hidden danger type, hidden danger segmentation and annotation images, suggestions for hidden danger rectification measures, and early warning prompts: On the front end, including the web and client application interfaces, it is displayed whether there are hidden dangers. If there are no hidden dangers, it is output that no hidden dangers are detected; if there are hidden dangers, it is output that hidden dangers are detected, and the time and place of occurrence of the hidden dangers, the name of the hidden danger type, the hidden danger segmentation and annotation image, and the suggestions for hidden danger rectification measures are output. The above information is stored in the database for subsequent viewing; at the same time, early warning prompts are issued to notify the person in charge of production safety to deal with the safety hazards in a timely manner.

Claims

1. A safety production supervision system based on a multimodal large model, characterized in that: include: Multimedia information collection terminal, used to collect image information and location information in the enterprise production scene, the multimedia information collection terminal includes monitoring equipment, smart phones, and patrol glasses with shooting and positioning functions; The visual model is used to receive image information and generate image descriptions; A database storing the correspondence between location information and hidden danger inspection scope, the correspondence between hidden danger description words and hidden danger standard prompt words, and the hidden danger information finally detected; A text similarity comparison module, used for receiving the image description and the hidden danger description and outputting the text similarity between them; A threshold judgment module is used to judge whether the similarity of a given text is higher than a given threshold; Standard prompt word matching module: used to match the hidden danger standard prompt words corresponding to the hidden danger description in the database. The hidden danger standard prompt words are suitable for subsequent image segmentation and annotation and knowledge base retrieval tasks; An image segmentation and annotation module, including an image segmentation and annotation model, is used to segment and annotate hidden danger areas on hidden danger images based on hidden danger standard prompt words; The knowledge base retrieval module includes the RAG knowledge base and the large language model, which is used to retrieve the sources of hidden dangers in the safety standard documents in the RAG knowledge base, and summarize and output the hidden danger rectification measures using the large language model; The result output module is used to output and record the hidden danger inspection results, including whether there are hidden dangers, the time and location of hidden dangers, the name of hidden danger type, hidden danger segmentation and annotation images, suggestions for hidden danger rectification measures, and early warning prompts.

2. The multi-modal large model-based safety production supervision system according to claim 1 is characterized in that: For the knowledge base retrieval module, it includes RAG knowledge base and large language model: For the RAG knowledge base, the text in the safety standard document is segmented into different text segments, which is used to complete the semantic matching retrieval between each text segment and the hidden danger standard prompt words, and the text content with the highest matching degree is used as the source information of the hidden danger in the safety standard document; The large language model, including the large language model deployed locally or in the cloud, analyzes and summarizes the source information of the hidden dangers in the safety standard documents retrieved from the RAG knowledge base and the preset guide words to obtain suggestions for hidden danger rectification measures.

3. The multi-modal large model-based safety production supervision system according to claim 2 is characterized in that: Safety standards document text segmentation in the RAG knowledge base: When faced with a text with multi-level title marking, firstly, based on the first-level title, the content under each first-level title is divided into a corresponding number of text segments; then based on the second-level title, the content under the second-level title is divided again, and so on, the text content is hierarchically segmented according to different levels of titles, so that the segmented text segments contain both long text segments containing the overall context and contextual connections of the text, and short text fields that are only related to specific sub-topics, focus on local key information, and can accurately adapt to specific search directions; when faced with a text that has not been calibrated with multi-level titles, the text is sent to the large language model, and the large language model performs multi-level title calibration. After the calibration is completed, the above-mentioned text segmentation steps with multi-level title calibration are followed.

4. A supervision method for a production safety supervision system based on a multimodal large model as claimed in any one of claims 1 to 3, characterized in that: The following steps are involved: S1. The multimedia information collection terminal collects image information and location information in the enterprise production scene; S2, passing the image information into the visual model to generate image description; S3, searching the database for the hidden danger description within the hidden danger inspection range that matches the corresponding location information; S4, passing the image description and the hidden danger description into a text similarity comparison module to obtain the text similarity between the two; S5, passing the text similarity into the threshold judgment module, and judging whether there is a hidden danger according to the relationship between the text similarity and the threshold; S6. According to the hidden danger description words of the existing hidden dangers, the corresponding hidden danger standard prompt words are matched in the database; S7, transferring the image information and the standard hidden danger prompt words into the image segmentation and annotation module to perform segmentation and annotation of the hidden danger area; S8. The hidden danger standard prompt words are transferred to the knowledge base retrieval module, the source of the hidden danger in the safety standard document is searched in the RAG knowledge base, and the hidden danger rectification measures are summarized and outputted using the large language model; S9. Output and record the hidden danger inspection results, including whether there are hidden dangers, the time and location of hidden dangers, the name of hidden danger type, hidden danger segmentation and annotation images, suggestions for hidden danger rectification measures, and early warning prompts.

5. The supervision method according to claim 4, characterized in that: Step S3 searches the database for the hidden danger range that should be checked for the corresponding location information, retrieves the input location information in the database, and outputs descriptions of multiple different hidden dangers within the hidden danger checking project range that matches the location information.

6. The supervision method according to claim 4, characterized in that: In step S8, the safety standard document is pre-divided into text segments of different lengths in the RAG knowledge base according to the hierarchical segmentation rules, and a parameter N is set. The parameter N is the number of sources of hidden dangers in the safety standard document retrieved in the RAG knowledge base. When the hidden danger standard prompt words are passed into the knowledge base retrieval module, the semantic similarity between the segmented text segments and the hidden danger standard prompt words is calculated, and the first N text segments with the highest semantic similarity to the hidden danger standard prompt words in the text segments are passed into the large language model.

Images