Multi-modal intelligent safety production question-answering method and system based on local RAG

Abstract

The application discloses a multi-modal safety production intelligent question and answer method and system based on a local RAG, and relates to the technical field of intelligent safety production management and artificial intelligence interaction. The method comprises the following steps: deploying a multi-modal large model in an enterprise intranet in a localized manner to construct a local safety production knowledge base; collecting multi-modal data for semantic analysis of the multi-modal data; matching a user question intention based on HanLP word segmentation and a pre-defined intention keyword dictionary, and determining a scene semantic description according to a scene risk level; querying based on a user original question, applying three-way asynchronous parallel retrieval in the local safety production knowledge base, generating a candidate set and performing sorting; generating a structured prompt word according to the sorted retrieval result, the scene semantic description and the user original question, generating a final structured answer by calling the multi-modal large model, and outputting a structured answer. The application can realize localized safety production intelligent question and answer in the enterprise intranet.

Description

Technical Field

[0001] This invention relates to the fields of intelligent management of safe production and artificial intelligence interaction technology, and in particular to a multimodal intelligent question-and-answer method and system for safe production based on local RAG. Background Technology

[0002] Safety in production is the cornerstone of enterprise operation. Traditional safety Q&A mainly relies on manually consulting safety manuals, consulting experts in the field, or comparing operating procedures item by item, which has problems such as low response efficiency, poor timeliness, and high labor costs.

[0003] In recent years, artificial intelligence technologies, represented by large-scale models, have provided new solutions for safety production question answering. For example, industrial production operation safety knowledge question answering methods based on large-scale models achieve automatic retrieval and reasoning answers of safety knowledge by constructing a safety standard knowledge base and using binary similarity measures; intelligent construction safety management systems based on large-scale multimodal language models integrate visual perception and RAG database retrieval technology to achieve safety hazard identification and early warning at construction sites; and a unified multimodal large-scale model architecture replaces multi-scenario dedicated small models, combined with the Milvus vector database to achieve automatic association between violation judgment and knowledge provisions.

[0004] However, the aforementioned existing technologies still have the following shortcomings: First, there are data security risks. Most existing systems rely on cloud deployment, uploading sensitive enterprise safety data (such as equipment parameters, operation videos, and internal regulations) to the cloud for processing, posing a risk of data leakage. For high-risk industries such as chemical and power, local data storage is essential.

[0005] Second, modal support is limited. Some systems only support plain text question-and-answer, and cannot handle the multimodal inputs such as images and videos commonly encountered in on-site inspections. Even if some systems have introduced multimodal models, their deployment is still mainly cloud-based.

[0006] Third, the retrieval accuracy is insufficient. The existing RAG system uses a general text segmentation strategy and has not been optimized for the structured characteristics of security specification clauses (such as chapter hierarchy, risk level, and applicable device types), resulting in limited matching between the retrieved results and the user's actual scenario. Furthermore, the single vector database has bottlenecks in dealing with the diverse needs of security knowledge retrieval.

[0007] Fourth, system architecture limitations. Existing technical solutions mostly use Python and dedicated AI frameworks, which are difficult to integrate with the enterprise's existing Java technology stack (such as Spring Boot microservice architecture), resulting in high system transformation costs, complex deployment, and difficult maintenance.

[0008] Fifth, the problem of model illusion. When lacking precise knowledge support, general-purpose large models may generate false answers that contradict safety regulations, potentially causing serious consequences in safe production scenarios.

[0009] Based on this, the present invention proposes a multimodal intelligent question-answering method and system for safe production based on local RAG. Summary of the Invention

[0010] This invention provides a multimodal intelligent question-answering method for safe production based on local RAG, including: The multimodal large model is deployed locally on the enterprise intranet, and communication between the Java backend and the model service is realized through Spring Boot's RestTemplate + HttpClient connection pool; A local safety production knowledge base is built, which adopts a storage and retrieval architecture that uses Milvus + ES + MySQL in collaboration. Collect multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model; Based on the semantic parsing results, the user's question intent is matched using HanLP word segmentation and a predefined intent keyword dictionary, and the semantic description of the scenario is determined according to the scenario risk level. Based on the user's original question query, a three-way asynchronous parallel retrieval is applied in the local safety production knowledge base to generate a candidate set. The retrieval results in the candidate set are scored for safety relevance and sorted according to the score. Based on the sorted search results, the semantic description of the scene, and the user's original question, structured prompts are generated. The final structured answer is then generated by calling a multimodal big model and output.

[0011] The above-described multimodal intelligent question-answering method for safe production based on local RAG comprises: Milvus for storing 1024-dimensional vectors of text blocks and scene semantic descriptions, using IVF_FLAT indexes and inner product distance metrics, and responsible for semantic similarity retrieval; Elasticsearch for storing full-text data, using the IK Chinese word segmenter, responsible for precise keyword matching and multi-field weighted retrieval; and MySQL for storing text block metadata, associated with Milvus and ES through a unified ID, supporting complex business queries and transaction management. The three components work collaboratively and are uniformly managed through Java's Spring Data JPA + Spring Data Elasticsearch + milvus-sdk-java.

[0012] The above-described multimodal intelligent question-answering method for safe production based on local RAG includes a dedicated preprocessing method tailored to the structured characteristics of safety regulations, comprising the following sub-steps: The security specification documents are parsed and segmented into hierarchical levels. Each segmented safety specification text block is automatically labeled with three-dimensional structured tags according to equipment type, operation type, and risk type; A multi-engine index is constructed based on the segmented security specification text blocks and the annotated 3D structured tags.

[0013] The above-described multimodal intelligent question-answering method for safe production based on local RAG includes constructing a multi-engine index, comprising: The vector index uses an embedding model to convert the segmented text blocks into 1024-dimensional vectors, and then uses Milvus's Java SDK to batch insert vector data to create IVF_FLAT indexes. Full-text indexing synchronously writes text block content and tag fields to Elasticsearch 8.x, creates an IK Chinese word segmentation index, sets the text type field to use the ik_max_word segmenter, and the keyword type field to use precise tag filtering.

[0014] The above-described multimodal intelligent question-answering method for safe production based on local RAG (Radar Image Processing) uses multimodal data, which consists of user-input interactive information, including at least one modality of data such as text questions, on-site captured images, and monitoring video streams. Specifically, it receives user-uploaded image / video files and text parameters through Spring Boot's MultipartFile interface, with the text encoded in UTF-8. If images or videos are included, the visual understanding API of the local multimodal large model service is called through RestTemplate to generate a structured scene semantic description JSON containing the identified object, feature description, and risk level.

[0015] The above describes a multimodal intelligent question-answering method for safe production based on local RAG, in which the local multimodal large-model service interface is called through a Spring Boot HTTP client: If the input contains image data, the image is normalized for resolution and format. If the input includes video data, keyframe sequences are extracted at a predetermined sampling rate. A visual understanding request is submitted to the multimodal large model service via a RESTful API. The request body includes image Base64 data and prompt words. The multimodal large model returns a structured scene semantic description JSON text, including the name of the identified object, description of the identified features, and preliminary risk level assessment. The system deserializes the returned JSON into a Java object using the Jackson library and caches it in Redis to avoid repeated calls. For plain text input, text parameters are received directly through Spring's @RequestParam, without going through the visual understanding channel.

[0016] The above-described multimodal intelligent question-answering method for safe production based on local RAG, wherein a hybrid approach combining rule-based and model-based methods is used for user intent matching, including: The predefined intent keyword dictionary is loaded through Spring's ResourceLoader. The dictionary is categorized as follows: device status query, operation specification consultation, hidden danger investigation report, accident emergency response, and safety training knowledge. After the user's question is segmented by HanLP and matched with the keyword dictionary, if the highest matching score of the category is lower than the threshold, the local multimodal large model is called to supplement the intent classification.

[0017] The above-mentioned multimodal intelligent question-answering method for safe production based on local RAG includes the following: the scene risk level determination includes: determining whether the current scene is low-risk, medium-risk, or high-risk based on three dimensions: equipment type weight, visual feature weight, and user question urgency, using a weighted scoring method.

[0018] The above-described multimodal intelligent question-answering method for safe production based on local RAG includes three-way asynchronous parallel retrieval, specifically comprising: Text vector similarity retrieval is performed using MilvusServiceClient; Visual semantic text vector retrieval is performed using MilvusServiceClient; Use ElasticsearchRestTemplate to perform keyword inverted index retrieval; The results from all three paths are deduplicated by ID using the distinct() method of Java Stream and then merged to form a candidate set.

[0019] This invention also provides a multimodal intelligent question-and-answer system for safe production based on local RAG, comprising: The multimodal large model deployment module is used to deploy multimodal large models locally on the enterprise intranet and realize the communication between the Java backend and the model service through Spring Boot's RestTemplate + HttpClient connection pool. The local safety production knowledge base construction module is used to build a local safety production knowledge base. The local safety production knowledge base adopts a storage and retrieval architecture that uses Milvus + ES + MySQL in collaboration. The multimodal data acquisition and parsing module is used to acquire multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model. The user intent and scenario semantic generation module is used to match user question intent based on HanLP word segmentation and a predefined intent keyword dictionary, and to determine the scenario semantic description based on the scenario risk level, based on the semantic parsing results. The local RAG retrieval module is used to generate a candidate set by applying three-way asynchronous parallel retrieval in the local safety production knowledge base based on the user's original question query, and to score the safety relevance of the retrieval results in the candidate set and sort them according to the score. The structured answer output module is used to generate structured prompts based on the sorted search results, scene semantic descriptions, and the user's original question. It then generates the final structured answer by calling a multimodal big model and outputs the structured answer.

[0020] The beneficial effects achieved by this invention are as follows: 1. Adopting a full-chain local deployment architecture, the multimodal large language model (Qwen3.5-35B-A3B), vector database (Milvus), full-text search engine (Elasticsearch), business database (MySQL), and backend service (SpringBoot) are all deployed on local servers within the enterprise intranet. User questions, on-site images / videos, specification documents, intermediate search results, and generated answers are never transmitted to the cloud, fundamentally eliminating the risk of data leakage and meeting the compliance requirements of high-risk industries that "data does not leave the intranet".

[0021] 2. Supports multimodal input combining text-based questions, on-site images, and monitoring video. This invention provides visual understanding capabilities through a locally deployed multimodal large language model (Qwen3.5-VL series), supporting inspection personnel to interact using a combination of photos / videos and text / voice questions. The system automatically identifies on-site equipment, its status, and potential hazards, eliminating the need for users to manually input complex technical descriptions, greatly lowering the barrier to entry and enabling on-site personnel (not just safety experts) to easily obtain professional-level safety guidance.

[0022] 3. This invention employs a dual-search engine architecture combining Milvus and Elasticsearch, along with a MySQL business database, to achieve complementary advantages between vector retrieval and keyword retrieval, thereby improving the accuracy and flexibility of clause retrieval. Addressing the hierarchical structure of safety regulations, this invention designs a chapter-based hierarchical segmentation strategy and a multi-dimensional tagging system (equipment type, job type, risk level). Simultaneously, the dual-engine architecture of Milvus (vector retrieval) + Elasticsearch (full-text retrieval) balances the generalization capability of semantic similarity retrieval with the accuracy of precise keyword matching. Furthermore, a risk level-aware re-ranking mechanism is introduced, significantly improving the matching degree between search results and the current scenario.

[0023] 4. Implemented using the Java technology stack, seamlessly integrating with existing enterprise systems. This invention is implemented using the Java technology stack of JDK 21 + SpringBoot 3, utilizing mature Java ecosystem components such as milvus-sdk-java, Spring Data Elasticsearch, and Spring Data JPA. It is naturally compatible with the existing Spring Cloud microservice architecture, Maven / Gradle build system, and unified operation and monitoring platform of large and medium-sized enterprises. The system can be directly integrated into the enterprise's existing backend project as a Maven module, eliminating the need to maintain a separate Python runtime environment, significantly reducing introduction costs and implementation resistance.

[0024] 5. This invention suppresses model illusions through enhanced retrieval and strong constraints. It employs a dual mechanism of strong constraint prompts ("Answer only based on the provided information; speculation or supplementary information not mentioned in the materials is strictly prohibited") and retrieval content injection to minimize the degree of freedom in the generated model. All regulatory bases in the answers are accompanied by specific article sources (regulation name + article number), allowing users to jump to the original text for verification with one click. This achieves traceable and auditable intelligent Q&A for safe production, fundamentally avoiding safety hazards caused by illusionary content. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0026] Figure 1 This is a flowchart of a multimodal intelligent question-and-answer method for safe production based on local RAG, provided in Embodiment 1 of this application; Figure 2This is a schematic diagram of a multimodal intelligent question-and-answer system for safe production based on local RAG, provided in Embodiment 2 of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1 like Figure 1 As shown, Embodiment 1 of this application provides a multimodal intelligent question-answering method for safe production based on local RAG, including: Step S1: Deploy the multimodal large model locally on the enterprise intranet, and realize the communication between the Java backend and the model service through Spring Boot's RestTemplate + HttpClient connection pool; RestTemplate is an HTTP client tool provided by Spring, used to call other people's HTTP interfaces (send GET / POST requests) from the Java backend; HttpClient is an HTTP component of Apache (a web server), implementing socket network communication at the underlying level; RestTemplate + HttpClient connection pooling reuses TCP connections, avoiding the creation and destruction of network connections for each call, thus improving performance. This embodiment, through Spring Boot's RestTemplate + HttpClient connection pooling, can reduce the three-way handshake during each call, reduce the overhead of frequently creating and destroying sockets, and improve the efficient communication between the Java backend and the model service.

[0029] This invention employs end-to-end local deployment, storing all security-sensitive production data (including user queries, on-site images, specification documents, and intermediate search results) on local servers within the enterprise's intranet, without transmitting them to the cloud, thus meeting the data compliance requirements of high-risk industries. Implemented using a Java technology stack of JDK 21 + Spring Boot 3, it is naturally compatible with the enterprise's existing Java microservice architecture and can be directly integrated via Maven / Gradle, significantly reducing system introduction and modification costs, without requiring an additional Python runtime environment.

[0030] This invention differs from existing Python-based AI solutions by innovatively building the entire system using JDK 21 + Spring Boot 3 as the core technology stack. Backend service layer: Spring Boot 3 + Spring Data JPA + Spring Data Elasticsearch + Spring Data Redis; Asynchronous processing: Spring @Async + ThreadPoolTaskExecutor (10 cores / 50 maximum / 200 queues); Vector database client: milvus-sdk-java 2.3.x; HTTP communication: RestTemplate + HttpClient connection pool (maxTotal=20); Streaming output: Server-Sent Events (SSE) enables word-by-word push; Build tool: Maven 3.9.x.

[0031] The system can be directly imported into an enterprise's existing Spring Boot project as a Maven module, and seamlessly integrated with the enterprise service registry, configuration center, and monitoring platform.

[0032] Step S2: Construct a local safety production knowledge base. The local safety production knowledge base adopts a storage and retrieval architecture that uses Milvus + ES + MySQL in collaboration. Building and maintaining a safety production standard knowledge base on a local server within the enterprise intranet provides a data foundation for subsequent local RAG retrieval. Local RAG (Local Retrieval-Augmented Generation) refers to a technical architecture that generates safety production standard documents by structurally segmenting them, building vector indexes on a local server, and enhancing retrieval capabilities. All data in the entire chain does not leave the enterprise intranet.

[0033] The architecture uses a three-database collaboration approach: Milvus + Elasticsearch (ES) + MySQL. Milvus: Stores 1024-dimensional vectors of text blocks and scene semantic descriptions, using IVF_FLAT indexing and inner product distance metric, and is responsible for semantic similarity retrieval. Milvus is an open-source vector database specifically designed to provide millisecond-level similarity retrieval for billions of vector datasets. It features data persistence, metadata filtering, and a cloud-native compute separation architecture, supports multiple index types such as HNSW and IVF, and is the core middleware in the RAG system connecting unstructured data and large language models.

[0034] Elasticsearch stores full-text data and uses the IK Chinese word segmenter (ik_max_word / ik_smart) to handle precise keyword matching and multi-field weighted retrieval (chunk_text weight 2.0, tags device type weight 1.5). Elasticsearch (ES) is a distributed full-text search engine based on Apache Lucene, possessing core capabilities such as real-time data indexing, multi-field full-text search, aggregation analysis, and distributed high availability. It provides services via a RESTful API and supports deployments from single machines to large-scale clusters. In this invention, Elasticsearch handles the keyword inverted index retrieval of safety production specification clauses (in conjunction with IK Chinese word segmentation), forming a dual-engine synergy with Milvus vector retrieval—ES handles precise term matching, while Milvus handles semantic generalization and recall, with both complementing each other to improve overall search accuracy.

[0035] MySQL: Stores text block metadata (id, hierarchical path, tag JSON, source document, timestamp), and associates it with Milvus and Elasticsearch through a unified ID, supporting complex business queries and transaction management.

[0036] The three components work together, and through unified management using Java's Spring Data JPA + Spring Data Elasticsearch + milvus-sdk-java, they solve the technical challenge that a single search engine cannot simultaneously handle "semantic generalization" and "exact matching".

[0037] Alternatively, you can use the Qdrant + Elasticsearch + PostgreSQL approach. Qdrant is a vector database written in Rust and also supports a Java client (qdrant-java). Its filtering mechanism is richer than Milvus and supports nested conditional queries. PostgreSQL replaces MySQL, and with the pgvector extension, it can store vectors and metadata in the same database at the same time, reducing cross-database query overhead. It is suitable for scenarios where you want to reduce the complexity of database operation and maintenance.

[0038] Alternatively, we can use the Weaviate + MySQL + RedisSearch approach. Weaviate natively integrates vector search and keyword search (BM25), supporting both vector similarity search and keyword matching simultaneously, without the need for additional Elasticsearch deployment. RedisSearch provides lightweight full-text search capabilities, suitable for scenarios where extremely high search accuracy is not required. This solution reduces the number of databases from three to two (Weaviate + MySQL), lowering the complexity of system deployment.

[0039] This invention addresses the structured nature of safety specification clauses by designing a specialized preprocessing method distinct from general text segmentation. It employs Apache POI / PDFBox to parse the safety specification document, segmenting the text into hierarchical blocks and labeling them according to device type, job type, and risk level. An embedding model is used to convert the text blocks into 1024-dimensional vectors, which are then batch-written into Milvus via milvus-sdk-java to create an IVF_FLAT index. Spring Data Elasticsearch is used to synchronize the text content and tags to Elasticsearch 8.x to create an IK Chinese word segmentation index. Finally, Spring Data JPA stores the metadata in a MySQL 8.0 knowledge_chunk table, with all three linked by a unified ID.

[0040] Specifically, the preprocessing method includes the following sub-steps: Step S21: Parse the security specification document and perform hierarchical segmentation of the security specification. Use Apache POI or PDFBox to parse security specification documents in Word / PDF format; The code is segmented according to the hierarchical structure of "General Provisions → Specific Provisions → Chapters → Clauses → Detailed Rules", and regular expressions are used to match the chapter title patterns (such as "Chapter X" and "Article X"). Each text block records its hierarchical path (e.g., "Safety Production Law → Chapter 3 → Article 45") and its parent association.

[0041] Step S22: Automatically label each segmented safety specification text block with three-dimensional structured tags according to equipment type, operation type, and risk type; For each text block, labels are added according to three dimensions: equipment type (e.g., pressure vessel, pipeline, electrical equipment), operation type (e.g., hot work, work at height, confined space work), and risk level (low / medium / high). The label data is stored in JSON format with the following field structure: {"deviceTypes": [], "operationTypes": [], "riskLevel":""}.

[0042] By segmenting security specifications hierarchically and using three-dimensional structured tags, we ensure that each retrieval unit is a complete and self-consistent specification (rather than fragmented text fragments), while providing structured tag support for subsequent risk level perception re-ranking.

[0043] Step S23: Construct a multi-engine index based on the segmented security specification text blocks and the annotated 3D structured tags; Vector Index (Milvus): The segmented text blocks are converted into 1024-dimensional vectors using an embedding model (such as text2vec-large-chinese or bge-large-zh). Vector data is then inserted in batches using the Milvus Java SDK (milvus-sdk-java) to create IVF_FLAT indexes. The nlist parameter is set to 1024, and the distance metric is inner product (IP). Full-text index (Elasticsearch): Synchronously write text block content and tag fields to Elasticsearch 8.x, create IK Chinese word segmentation index, set text type fields to use ik_max_word segmenter, and keyword type fields to use precise tag filtering; It adopts a dual-search engine architecture of Milvus (vector search) and Elasticsearch (full-text search), which takes into account the generalization ability of semantic similarity search and the accuracy of keyword exact matching. Compared with a single search engine, the F1 score is improved by about 15% in the scenario of searching for security regulations.

[0044] The metadata of the text chunks (ID, hierarchical path, tag JSON, source document name, creation time, and update time) is stored in the knowledge_chunk table of the MySQL 8.0 database. The table structure includes: id (BIGINT primary key), chunk_text (LONGTEXT), hierarchy_path (VARCHAR 500), tags (JSON), doc_source (VARCHAR 200), created_at (DATETIME), and updated_at (DATETIME). The id stored in MySQL is consistent with the vector id in Milvus and the document id in Elasticsearch, and the three databases are joined by id.

[0045] This step completes the structured preprocessing of the security specification document, multi-dimensional tag annotation, and database sharding index construction. The three databases, MySQL, Milvus, and Elasticsearch, are linked through a unified and unique ID. The entire process is executed asynchronously on the intranet server using Spring Boot's @Async asynchronous task execution to avoid blocking the main process.

[0046] Step S3: Collect multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model; This invention acquires user-input interactive information, i.e., multimodal data. Multimodal data refers to a collection of data originating from different information carriers and with different forms of expression. Its core includes text, images, audio, video, and structured data (tables, database records), etc. These modal data exhibit heterogeneity and complementarity, requiring fusion technology for collaborative interpretation. This invention supports a combined input method of "on-site photo / video recording + text question," eliminating the need for inspection personnel to manually input complex descriptions, lowering the barrier to entry, and improving on-site operational efficiency.

[0047] The interactive information includes receiving user-uploaded image / video files and text parameters via Spring Boot's MultipartFile interface, with the text encoded in UTF-8. If images or videos are included, the visual understanding API of the local multimodal large model service (Qwen3.5-35B-A3B) is called via RestTemplate to generate a structured scene semantic description JSON containing the identified object, feature description, and risk level.

[0048] This invention utilizes the MultipartFile file upload interface provided by the Spring Boot 3 framework. Spring Boot 3 is a major upgrade to the Spring framework, built on Spring Framework 6 and requiring JDK 17 or later. Its core features include: full adoption of the Jakarta EE specification (replacing the old javax namespace), support for AOT compilation and GraalVM native images (significantly improving startup speed and greatly reducing memory usage), enhanced observability (integrated Micrometer Tracing), and improved auto-configuration mechanisms. In enterprise-level development, Spring Boot 3 significantly simplifies the construction, development, and deployment process of Java backend applications through the "convention over configuration" principle, seamlessly integrating with ecosystem components such as Spring Cloud. This invention selects Spring Boot 3 as the basic framework, fully leveraging its AOT compilation to optimize startup speed and its natural integration capabilities with components such as Spring Data JPA and Spring Data Elasticsearch to achieve efficient and secure intelligent question-and-answer system development.

[0049] The Spring Boot 3 framework supports the following data collection methods: Text input: Submit security-related questions to users via forms on terminal devices (web or mobile), with text encoded in UTF-8; Image acquisition: Take photos of the equipment status and working environment on site using cameras on smartphones and inspection robots. JPEG and PNG formats are supported, and the size of a single image shall not exceed 10MB. Video Acquisition: Real-time monitoring video streams of the production site are acquired through fixed surveillance cameras or inspection robots, using H.264 encoding, and the system extracts key frames at a sampling rate of 1 frame per second.

[0050] After standardizing the format and Base64 encoding or URL path conversion of the collected multimodal data, it is transmitted to the local multimodal large model through the Spring Boot Service layer for semantic parsing of the multimodal data, generating a structured scene semantic description.

[0051] Specifically, the system uses a Spring Boot HTTP client to call the locally deployed multimodal large model service interface (such as the OpenAI-compatible API of Qwen3-VL-32B-Instruct). If the input includes image data, the image is first normalized in resolution (uniformly scaled to 448×448 pixels) and format standardized. If it includes video data, keyframe sequences (up to 5 frames) are extracted at a sampling rate of 1fps. A visual understanding request is submitted to the multimodal large model service via a RESTful API. The request body includes the image Base64 data and the prompt "Please identify the equipment type, status characteristics, and potential hazards in this safety production scenario." The multimodal large model returns a structured JSON text describing the scene's semantics, including the name of the identified object, description of the identified features, and a preliminary risk level assessment. The system deserializes the returned JSON into a Java object using the Jackson library and caches it in Redis (with an expiration time of 5 minutes) to avoid repeated calls. For plain text input, text parameters are received directly through Spring's @RequestParam, bypassing the visual understanding channel.

[0052] Step S4: Based on the semantic parsing results, match the user's question intent using HanLP word segmentation and a predefined intent keyword dictionary, and determine the semantic description of the scenario based on the scenario risk level; The system performs joint analysis of the user-input text question and the scene semantic description generated by the multimodal semantic understanding module to identify the user's question intent and determine the risk level of the current scene.

[0053] Specifically, a hybrid approach combining rules and models is used for user intent matching, including: The predefined intent keyword dictionary (YAML format) is loaded through Spring's ResourceLoader. The dictionary is categorized as follows: Equipment status query (keywords: status, running, parameters, temperature, pressure), operation procedure consultation (how, can, is allowed, steps), hazard investigation report (abnormal, rust, leak, loose, crack), accident emergency response (urgent, leak, fire, explosion, immediately), and safety training knowledge (definition, principle, why, what).

[0054] After the user's question is segmented by HanLP, it is matched with the keyword dictionary. If the highest matching score of the category is lower than the threshold of 0.5, the local large model is called to supplement the intent classification. Scenario risk level determination: Scenario risk scores are calculated based on the following dimensions: Based on three dimensions—device type weight, visual feature weight, and user problem urgency—a weighted scoring method is used to determine whether the current scenario is low-risk, medium-risk, or high-risk.

[0055] Equipment type weight (W1): Based on the pre-configured equipment hazard level dictionary table in MySQL (e.g., pressure vessel = 0.9, pipeline = 0.6, office equipment = 0.1). Visual feature weight (W2): The severity of the abnormal features identified by the multimodal model (severe = 0.9, moderate = 0.6, mild = 0.3). User issue urgency (W3): Detects urgent keywords in the issue through regular expression matching (0.9 for matching "immediate", "urgent", "leak", "fire", otherwise 0.2).

[0056] The overall risk score is Score = 0.5×W1 + 0.3×W2 + 0.2×W3, which divides the scenario into three levels: "low risk" (Score<0.4), "medium risk" (0.4≤Score≤0.7), and "high risk" (Score>0.7).

[0057] The above logic is encapsulated as a Spring @Service component, and the dependency is injected via @Autowired.

[0058] Step S5: Based on the user's original question query, apply three-way asynchronous parallel retrieval in the local safety production knowledge base to generate a candidate set, score the safety relevance of the retrieval results in the candidate set, and sort them according to the scores; Based on user queries, a three-way parallel search is performed in the local security knowledge base to retrieve the most relevant security provisions for the current scenario. The three-way search is executed asynchronously in parallel using CompletableFuture, which reduces the response time by approximately 60% compared to serial search; combined with Redis caching and @Async asynchronous tasks, the system throughput is significantly improved.

[0059] Three-way asynchronous parallel retrieval, specifically including: ① Perform text vector similarity retrieval using MilvusServiceClient (topK=100); The user's text question is encoded into a 1024-dimensional vector using an embedding model. Vector similarity retrieval is then performed using the search method of MilvusServiceClient with the following parameters: topK=100, nprobe=16, and an optional expr filter (e.g., pre-filtering by risk level). The result is a list of vector similarity scores and corresponding IDs.

[0060] ② Perform visual semantic text vector retrieval using MilvusServiceClient (topK=100); The scene semantic description text generated by the multimodal semantic understanding module is also vectorized for retrieval; the retrieval parameters are the same as those for text vector recall, topK=100; the results are merged with the text vector recall results by ID.

[0061] ③ Perform keyword inverted index retrieval using ElasticsearchRestTemplate (multi_match query, topK=100).

[0062] The Elasticsearch multi_match query is used to perform precise matching and retrieval of security domain keywords in user questions and scenario semantic descriptions. The query fields include: chunk_text (weight 2.0), tags.deviceTypes (weight 1.5), and tags.operationTypes (weight 1.5). The IK tokenizer's ik_smart mode is used to return the Top-100 results.

[0063] The results from all three paths are deduplicated by ID using the distinct() method of Java Stream and then merged to form a candidate set.

[0064] The candidate search results are re-ranked in multiple dimensions, placing the most relevant entries at the top. This re-ranking is performed in Java memory using a weighted summation algorithm. The comprehensive score for each candidate entry is calculated as follows: text FinalScore = α × SimScore + β × RiskMatch + γ × DeviceMatch+ δ × TimeScore Among them, SimScore is the semantic relevance score, which is the cosine similarity value returned by Milvus (range 0~1); RiskMatch is the risk level matching degree, which is 1.0 if the risk level in the candidate text label is completely consistent with the determined scenario risk level, 0.5 if they are adjacent levels, and 0 otherwise; DeviceMatch is the device type matching degree, which is 1.0 if the device type in the candidate text label matches the visual recognition result, 0.5 if they partially match (belonging to the same category), and 0 if they do not match; TimeScore is the timeliness weight, which is extracted from the publication date of the doc_source field stored in MySQL, with 1.0 for publication within the last 6 months, 0.8 for publication within the last 2 years, and 0.5 for publication more than 2 years ago; weight coefficients: α=0.5, β=0.25, γ=0.15, δ=0.1.

[0065] Sort the candidate set in descending order of FinalScore, and take the top-5 to 8 entries as the search results. The sorting logic is encapsulated in a Comparator implementation class and executed via Collections.sort().

[0066] Step S6: Based on the sorted search results, scene semantic description and user's original question, generate structured prompt words, generate the final structured answer by calling the multimodal big model, and output the structured answer; The search results' chunk text, scene semantic description, and the user's original question are concatenated into structured prompts. These prompts are then used to call the ChatCompletion API (temperature=0.3) of the multimodal large model Qwen3.5-35B-A3B via Spring Boot's RestTemplate. This generates a structured answer containing problem diagnosis, regulatory basis, handling suggestions, and reference cases, which is then pushed to the front end via SSE streaming. This invention ensures that the generated answer is verifiable by constraining the prompts to "answer only based on the provided information" and combining this with a search enhancement mechanism. The generated answer includes the source of the regulatory clause (specific regulatory name and chapter number), meeting the requirements of traceable and auditable information sources in safety production scenarios.

[0067] The prompt word construction template is as follows: text [System Command] You are a professional assistant in the field of workplace safety. Please answer based solely on the information provided below; speculation or additions of safety regulations not mentioned in the materials are strictly prohibited.

[0068] [Scene Information] Scene description: {sceneDescription} Risk Level: {riskLevel} [Reference Articles] {retrievedChunks} [User Issues] {userQuestion}

Answer Requirements

[0069] After generating structured answers, the system displays the question-and-answer results to the user in a structured format, and supports multi-turn dialogues and incremental updates to the knowledge base.

[0070] The system uses Spring Boot 3's WebSocket or SSE to implement real-time communication with the front end. When outputting answers, the structured answers (containing four fields: diagnosis, basis, measures, and case) are serialized into JSON and returned to the front end via Jackson.

[0071] In terms of multi-turn dialogue, the system maintains the dialogue history of the current session through Redis caching (key is sessionId, value is the list of dialogue history, expiration time is 30 minutes). When asking questions in the next round, the historical context is concatenated into the prompt words to achieve continuous question and answer.

[0072] Regarding knowledge updates, the system provides the following RESTful API endpoints to support the addition, modification, and deletion of local security specification files, with operations synchronously updating the MySQL, Milvus, and Elasticsearch databases: POST / api / knowledge / chunk: Add a security specification text block and write it synchronously to MySQL, Milvus, and Elasticsearch; PUT / api / knowledge / chunk / {id}: Updates the text block, modifying it synchronously across the three databases; DELETE / api / knowledge / chunk / {id}: Deletes a text block, simultaneously deleting it across all three databases. POST / api / knowledge / rebuild: Fully rebuild vector indexes and ES indexes.

[0073] The above operations ensure MySQL transaction consistency through Spring's @Transactional annotation, and ensure eventual consistency of Milvus and Elasticsearch operations through manual coding.

[0074] Example 2 like Figure 2 As shown, Embodiment 2 of the present invention provides a multimodal intelligent question-and-answer system for safe production based on local RAG, comprising: a user layer, an access layer, an application layer, a model layer, and a storage layer; The user layer includes web, mobile, and surveillance camera interfaces; the access layer connects the user layer to the application layer; the model layer includes a multimodal large model deployment module for deploying multimodal large models locally on the enterprise intranet, and uses Spring Boot's RestTemplate + HttpClient connection pool to enable communication between the Java backend and the model service; the storage layer includes a local security production knowledge base construction module for building a local security production knowledge base, which adopts a Milvus + ES + MySQL three-database collaborative storage and retrieval architecture. The application layer specifically includes: The multimodal data acquisition and parsing module is used to acquire multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model. The user intent and scenario semantic generation module is used to match user question intent based on HanLP word segmentation and a predefined intent keyword dictionary, and to determine the scenario semantic description based on the scenario risk level, based on the semantic parsing results. The local RAG retrieval module is used to generate a candidate set by applying three-way asynchronous parallel retrieval in the local safety production knowledge base based on the user's original question query, and to score the safety relevance of the retrieval results in the candidate set and sort them according to the score. The secure answer generation module generates structured prompts based on the sorted search results, contextual semantic descriptions, and the user's original question. It then generates the final structured answer by calling a multimodal big data model and outputs the structured response.

[0075] The application layer also includes a security-related sorting module, which sorts the search results and then inputs them into a security answer generation module to generate a structured answer; the human-computer interaction and knowledge base update module connects to the storage layer and updates the three databases in the storage layer.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal intelligent question-and-answer method for safe production based on local RAG, characterized in that, include: The multimodal large model is deployed locally on the enterprise intranet, and the communication between the Java backend and the model service is realized through Spring Boot's RestTemplate + HttpClient connection pool; A local safety production knowledge base is built, which adopts a storage and retrieval architecture that uses Milvus + ES + MySQL in collaboration. Collect multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model; Based on the semantic parsing results, the user's question intent is matched using HanLP word segmentation and a predefined intent keyword dictionary, and the semantic description of the scenario is determined according to the scenario risk level. Based on the user's original question query, a three-way asynchronous parallel retrieval is applied in the local safety production knowledge base to generate a candidate set. The retrieval results in the candidate set are scored for safety relevance and sorted according to the score. Based on the sorted search results, the semantic description of the scene, and the user's original question, structured prompts are generated. The final structured answer is then generated by calling a multimodal big model and output.

2. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 1, characterized in that, Milvus stores 1024-dimensional vectors of text blocks and scene semantic descriptions, using IVF_FLAT indexes and inner product distance metrics, and is responsible for semantic similarity retrieval; Elasticsearch stores full-text data, using the IK Chinese word segmenter, and is responsible for precise keyword matching and multi-field weighted retrieval; MySQL stores text block metadata, is associated with Milvus and ES through a unified ID, and supports complex business queries and transaction management; the three work together and are managed uniformly through Java's Spring Data JPA + Spring Data Elasticsearch + milvus-sdk-java.

3. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 1, characterized in that, In view of the structured nature of safety regulations, a dedicated preprocessing method is set up, which includes the following sub-steps: The security specification documents are parsed and segmented into hierarchical levels. Each segmented safety specification text block is automatically labeled with three-dimensional structured tags according to equipment type, operation type, and risk type; A multi-engine index is constructed based on the segmented security specification text blocks and the annotated 3D structured tags.

4. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 3, characterized in that, Building multi-engine indexes includes: The vector index uses an embedding model to convert the segmented text blocks into 1024-dimensional vectors, and then uses Milvus's Java SDK to batch insert vector data to create the IVF_FLAT index. Full-text indexing synchronously writes text block content and tag fields to Elasticsearch 8.x, creates an IK Chinese word segmentation index, sets the text type field to use the ik_max_word segmenter, and the keyword type field to use precise tag filtering.

5. The multimodal intelligent question-answering method for safe production based on local RAG as described in claim 1, characterized in that, Multimodal data consists of interactive information input by the user, including at least one modality of data such as text questions, on-site captured images, and monitoring video streams. Specifically, it receives image / video files and text parameters uploaded by the user through the MultipartFile interface of Spring Boot, with the text encoded in UTF-8. If images or videos are included, the visual understanding API of the local multimodal large model service is called through RestTemplate to generate a structured scene semantic description JSON containing the identified objects, feature descriptions, and risk levels.

6. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 5, characterized in that, Call the locally deployed multimodal large model service interface through the Spring Boot HTTP client: If the input contains image data, the image is normalized for resolution and format. If the input includes video data, keyframe sequences are extracted at a predetermined sampling rate. A visual understanding request is submitted to the multimodal large model service via a RESTful API. The request body includes image Base64 data and prompt words. The multimodal large model returns a structured scene semantic description JSON text, including the name of the identified object, the description of the identified features, and preliminary risk level information. The system deserializes the returned JSON into a Java object using the Jackson library and caches it in Redis to avoid repeated calls. For plain text input, text parameters are received directly through Spring's @RequestParam, without going through the visual understanding channel.

7. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 1, characterized in that, A hybrid approach combining rule-based and model-based methods is employed for user intent matching, including: The predefined intent keyword dictionary is loaded through Spring's ResourceLoader. The dictionary is categorized as follows: device status query, operation specification consultation, hidden danger investigation report, accident emergency response, and safety training knowledge. After the user's question is segmented by HanLP and matched with the keyword dictionary, if the highest matching score of the category is lower than the threshold, the local multimodal large model is called to supplement the intent classification.

8. The multimodal intelligent question-answering method for safe production based on local RAG as described in claim 1, characterized in that, The risk level assessment of a scenario includes: determining whether the current scenario is low-risk, medium-risk, or high-risk based on three dimensions: device type weight, visual feature weight, and urgency of user issues, using a weighted scoring method.

9. The multimodal intelligent question-and-answer method for safe production based on local RAG as described in claim 1, characterized in that, Three-way asynchronous parallel retrieval, specifically including: Text vector similarity retrieval is performed using MilvusServiceClient; Visual semantic text vector retrieval is performed using MilvusServiceClient; Use ElasticsearchRestTemplate to perform keyword inverted index retrieval; The results from all three paths are deduplicated by ID using the distinct() method of Java Stream and then merged to form a candidate set.

10. A multimodal intelligent question-and-answer system for safe production based on local RAG, characterized in that, include: The multimodal large model deployment module is used to deploy multimodal large models locally on the enterprise intranet and realize the communication between the Java backend and the model service through Spring Boot's RestTemplate + HttpClient connection pool. The local safety production knowledge base construction module is used to build a local safety production knowledge base. The local safety production knowledge base adopts a storage and retrieval architecture that uses Milvus + ES + MySQL in collaboration. The multimodal data acquisition and parsing module is used to acquire multimodal data and perform semantic parsing of the multimodal data based on the local multimodal large model. The user intent and scenario semantic generation module is used to match user question intent based on HanLP word segmentation and a predefined intent keyword dictionary, and to determine the scenario semantic description based on the scenario risk level, based on the semantic parsing results. The local RAG retrieval module is used to generate a candidate set by applying three-way asynchronous parallel retrieval in the local safety production knowledge base based on the user's original question query, and to score the safety relevance of the retrieval results in the candidate set and sort them according to the score. The structured answer output module is used to generate structured prompts based on the sorted search results, scene semantic descriptions, and the user's original question. It then generates the final structured answer by calling a multimodal big model and outputs the structured answer.

Images