Method for constructing security hazard database based on multi-modal fusion and dynamic updating

Abstract

This invention discloses a method for constructing a safety hazard database based on multimodal fusion and dynamic updates. The method includes: collecting multimodal hazard data using a multi-source data stream acquisition interface to obtain a dynamic multimodal data stream; wherein the dynamic multimodal data stream completes the effective judgment, real-time integration, and continuous acquisition of multi-source multimodal hazard data through set operations, time-series alignment operators, and a data acquisition intensity quantification model; performing real-time preprocessing on the dynamic multimodal data stream; using multimodal fusion and adaptive learning methods to extract and associate features from the preprocessed data, and dynamically updating the hazard feature vector and database index through online clustering and incremental embedding learning; periodically reconstructing and optimizing the database based on the dynamically updated feature vectors using an adaptive optimization mechanism; and generating a hazard trend report and performance indicators based on the optimization results. This improves the initiative and effectiveness of safety management.

Description

Technical Field

[0001] This invention belongs to the field of security hazard database construction technology, and more specifically, relates to a method and system for constructing a security hazard database based on multimodal fusion and dynamic updating. Background Technology

[0002] In fields such as construction and industrial production, the identification and management of safety hazards are crucial for ensuring production safety. Traditional safety hazard management relies heavily on manual inspections and single-modal data recording, such as using paper ledgers to record textual information about hazards or manually reviewing surveillance images to determine hazards. This approach has significant shortcomings. First, the data source is limited, failing to comprehensively cover various types of hazard information, including text, images, and sensor data, leading to blind spots in hazard identification. Second, the data lacks effective integration and dynamic updates; historical hazard data is difficult to correlate with newly added hazard data, failing to form a complete hazard characteristic system. This results in hazard analysis remaining superficial, making it difficult to uncover potential hazard patterns, and leaving safety management decisions without strong data support.

[0003] With the development of IoT and big data technologies, multi-source, multi-modal data collection has become possible. Sensors at construction sites, historical hazard records in external databases, and user feedback can all provide hazard-related data. However, these multi-source, multi-modal data differ in format, dimension, and time, making effective integration into a unified data stream a major challenge. Furthermore, existing databases are mostly static, unable to dynamically adjust to new hazard characteristics and query needs, resulting in data storage redundancy, low retrieval efficiency, and an inability to provide timely support for real-time monitoring and early warning of safety hazards.

[0004] In practical safety management, enterprises urgently need a method that can comprehensively integrate multi-source, multi-modal hazard data, dynamically update hazard characteristics, and adaptively optimize the database structure to achieve accurate identification, dynamic analysis, and effective early warning of safety hazards. However, current technologies have significant shortcomings in multi-modal data fusion and dynamic database optimization, failing to meet this need. If these problems cannot be solved, safety hazard management will remain reactive, making it difficult to effectively prevent safety accidents and posing significant risks to enterprise production safety and the lives and property of personnel. Therefore, constructing a safety hazard database based on multi-modal fusion and dynamic updates has significant practical significance and application value. Summary of the Invention

[0005] This invention aims to address the problems of single data sources, difficulty in integrating multimodal data, and lack of dynamic adaptability in traditional safety hazard management. Through multi-source, multimodal data collection, fusion, correlation, dynamic updating, and adaptive optimization, it achieves comprehensive capture of hazard data, accurate feature mining, and efficient database operation. This provides reliable data support for safety hazard identification, early warning, and management decisions, enhancing the initiative and effectiveness of safety management.

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for constructing a security vulnerability database based on multimodal fusion and dynamic updates, comprising: S1. Continuously collect multimodal hazard data from construction sites, external databases, and user feedback using a multi-source data stream acquisition interface, including text, images, sensor readings, and spatiotemporal information, to obtain a dynamic multimodal data stream; among which, the dynamic multimodal data stream completes the effective judgment, real-time integration, and continuous acquisition of multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions; S2. Real-time preprocessing of dynamic multimodal data streams using streaming data cleaning; S3. Use multimodal fusion and adaptive learning methods to extract and associate features from the preprocessed data, and dynamically update the hidden danger feature vector and database index through online clustering and incremental embedding learning; S4. Based on the dynamically updated feature vectors, the database is periodically reconstructed and optimized using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. S5. Based on the optimization results, generate a hazard trend report and performance indicators using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction.

[0007] Furthermore, the method for determining validity in S1 is as follows: Let the set of access channels of the acquisition system be . , These correspond to the on-site sensing device channel, the external database channel, and the user interaction channel, respectively; At any moment The data transmission status is , ,in, This indicates normal transmission. Indicates an interruption; the set of modal types compatible with the channel is: The complete set of target acquisition modes is Define the channel validity determination function. :

[0008] in, The cardinality of a set is the number of elements; it only applies when... At that time, the channel Only the collected data enters the preprocessing process to ensure the effectiveness of the cleaned data.

[0009] Furthermore, the real-time integration method in S1 is as follows: Assume each channel is within the time window The set of multimodal data samples acquired internally is , , This represents the total number of data sets. It is a set The first in There are data elements, among which, This indicates the dataset number to which the element belongs. This indicates the index of the element in the corresponding dataset, and the sample. The timestamp is Modal identifier is , Indicates the first The set of modality identifiers corresponding to each dataset; Define timing alignment operator This maps samples from different channels and modalities to a unified time axis.

[0010] in, Indicates the current time reference point; These represent category labels, used to group and filter data by category. Define dynamic multimodal data flow for:

[0011] in, The logical AND operation is used to mark the overall validity of a data stream; Indicates the first The acquisition channels corresponding to each dataset.

[0012] Furthermore, the method for continuous data acquisition in S1 is as follows: Set time window Number of valid samples collected in each channel for:

[0013] in, Indicates the first The effective data volume of each acquisition channel; Indicates the first The first dataset One data element; It is a validity determination function used to determine the validity of data elements. Whether it is effective, among which It is the first One acquisition channel, It is a data element Timestamp; when When this happens, the data element is deemed valid; The system's preset minimum effective sample threshold is Define the intensity acquisition function. :

[0014] when At this time, the system triggers dynamic expansion of the acquisition channel to ensure the continuity of the data stream to meet subsequent processing requirements.

[0015] Furthermore, the streaming data cleaning in S2 specifically includes: Assume the data acquisition device transmission status verification result is: , This indicates that the transmission was stable and complete. This indicates an anomaly; the data timestamp is... The real-time scrolling time window is , Given the window duration corresponding to the data update frequency, the validity of the data is determined as follows:

[0016] when At that time, the data enters the subsequent process; Meanwhile, the dedicated effective domains for text, sensors, and images are defined as follows: , , The data subject is The selection criteria are as follows:

[0017] when At that time, data subject The data was determined to be valid. Furthermore, let the database structured field template be... The semantic similarity matching function for text data is: Sensor data unit conversion operator is The image pixel normalization operator is The aligned data is as follows:

[0018] in, They represent different types of features. This indicates a "mapping" relationship, that is, establishing a connection between the features on the left and the result of the function operation on the right; Then through Perform field-by-field validation to ensure a perfect format match. The cardinality of a set, i.e., the number of elements. Represents the set of aligned fields With template field collection The intersection of.

[0019] Furthermore, the feature extraction process in S3 is as follows: The feature extraction process takes cleaned multi-source, multi-modal data as input and is completed through modal feature deconstruction, cross-modal feature association, and effective feature selection, as follows: Modal feature deconstruction: Let text, sensor, and image data be respectively... , , Through semantic word segmentation operators Extract the set of text keywords and extract sensor data change features through time-series difference analysis. Through pixel gradient operator Extract image edge features to form a single-modal basic feature set. ; Cross-modal feature association: Defining the feature association degree function ,in, For different modal feature sets, The cardinality of a set, i.e., the number of elements; filtering. Strongly correlated feature pairs, through set union operation Integrating cross-modal features yields a set of strongly correlated cross-modal features. ; Effective feature selection: optimizing the required feature dimensions based on the database. To constrain the calculation of feature information content ,in, The probability of feature occurrence; according to Before selecting descending order These features form the final core feature set. It directly supports dynamic database updates.

[0020] Furthermore, the association process in S3 is as follows: Let the basic feature sets of each single mode after cleaning be... , , The database historical association feature feedback set is The association process is accomplished through three steps: unified feature space mapping, cross-modal association quantification, and strong association feature selection. The specific formulas are as follows: Unified feature space mapping: Applying modality-specific mapping operators to text, sensor, and image feature sets respectively. ,Right now:

[0021] Output uniform dimension feature vectors , , ; It is a modal type identifier, which clarifies... The range of values ​​represents three different modalities: text, sensor, and image. Cross-modal correlation metric: eigenvectors after any two unified spaces Cross-modal correlation strength is calculated using the cosine similarity formula. :

[0022] Strongly correlated feature selection: using historical correlated feature feedback set Based on this, the correlation threshold is calculated. :

[0023] in, The cardinality of a set, i.e., the number of elements. Represent two types of feature vectors after a unified spatial mapping in history; filter those that satisfy... Strongly correlated feature pairs are obtained through vector concatenation operations. Integrate to form a cross-modal correlation feature set The filtering results directly support the construction of database indexes.

[0024] Furthermore, the adaptive optimization mechanism in S4 is specifically as follows: Let the dynamically updated feature vector set be... The data timestamp set is , Represents the dynamically updated feature vector set Total data volume; historical query record set is The query error set is , For the first The difference between the historical query results and the actual values, ; Represents the historical query record set and query error set The total number of records; the adaptive optimization mechanism achieves dynamic database adjustment through the following three steps: Data Freshness Measurement: Defining a Freshness Judgment Function ,in, , For the current time, The difference between the current time and the average of historical timestamps. For this is the first The timestamp of each data item; when At that time, the data is determined to be fresh data; Integrating query frequency and error feedback: Defining a query validity function ,in, It is the first 100 historical query records For query Number of occurrences This represents the total number of historical query records. It is the first Error of each query Represents the query error set The maximum error value in the query; this function filters the features corresponding to queries that are "high-frequency and low-error". Storage structure and retrieval parameter adjustment: for dynamically updated feature vector sets Simultaneously, features that satisfy data freshness and query validity are selected, and these features are integrated to form a new storage structure. :

[0025] in, The criteria for determining "data freshness" are determined by... Definition: Data timestamps must meet freshness requirements; The criteria for determining whether a query is valid are determined by... The definition is that a query is a high-frequency, low-error effective query; By calculating the cardinality difference between the cardinality of the feature set in the new storage structure and the cardinality difference between the intersection of the dynamic feature set and the historical query set, the retrieval parameters that minimize this difference are found. :

[0026] in, Indicates the first 100 historical query records; It is the mathematical symbol for the number of elements in a set.

[0027] As a second aspect of the present invention, a security vulnerability database construction system based on multimodal fusion and dynamic updating is also provided, comprising: The multi-source multimodal data acquisition unit is used to continuously collect multimodal hazard data from the construction site, external databases, and user feedback using a multi-source data stream acquisition interface. This data includes text, images, sensor readings, and spatiotemporal information, resulting in a dynamic multimodal data stream. The dynamic multimodal data stream effectively determines, integrates in real time, and continuously acquires multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions. The streaming data real-time preprocessing unit is used to perform real-time preprocessing on dynamic multimodal data streams using streaming data cleaning. The feature extraction and association update unit is used to extract and associate features from the preprocessed data using multimodal fusion and adaptive learning methods. Through online clustering and incremental embedding learning, it dynamically updates the hidden danger feature vector and database index. The database adaptive optimization and reconstruction unit is used to periodically reconstruct and optimize the database based on dynamically updated feature vectors using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. The hazard reporting and indicator generation unit is used to generate hazard trend reports and performance indicators based on the optimization results using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction.

[0028] As a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being executed by a processor of any one of the methods for constructing a security vulnerability database based on multimodal fusion and dynamic updating.

[0029] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The safety hazard database construction method based on multimodal fusion and dynamic updating of the present invention continuously collects multimodal hazard data from construction sites, external databases, and user feedback through a multi-source data stream acquisition interface. Utilizing set operations, time-series alignment operators, and a data acquisition intensity quantification model, it effectively determines, integrates, and continuously collects multi-source multimodal hazard data, forming a dynamic multimodal data stream. This technical feature enables multi-dimensional and multi-channel data capture of safety hazards at construction sites, breaking the limitations of single data sources, ensuring data comprehensiveness and real-time performance, and providing rich and timely raw data support for subsequent hazard analysis. This allows safety hazard identification to cover various scenarios such as text descriptions, image presentations, and sensor monitoring, avoiding missed hazard identification due to data loss or lag.

[0030] 2. The safety hazard database construction method based on multimodal fusion and dynamic updating of the present invention performs real-time preprocessing of dynamic multimodal data streams using streaming data cleaning, and then uses multimodal fusion and adaptive learning methods to extract and associate features from the preprocessed data. Online clustering and incremental embedding learning are used to dynamically update the hazard feature vectors and database indexes. This technical feature solves the problem of difficulty in fusing multimodal data due to differences in format and dimension, realizes effective association and feature mining of different types of hazard data, enables the database to dynamically adapt to newly emerging hazard features, improves the accuracy of hazard identification, and shifts the feature description of safety hazards from the one-sidedness of a single modality to the comprehensiveness of multimodal fusion, laying a technical foundation for in-depth hazard analysis and early warning.

[0031] 3. The safety hazard database construction method based on multimodal fusion and dynamic updates of this invention utilizes an adaptive optimization mechanism to periodically reconstruct and optimize the database based on dynamically updated feature vectors. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. Then, based on the optimization results, it uses real-time data mining and visualization engines to generate hazard trend reports and performance indicators. This technical feature enables the database to have self-optimization capabilities, dynamically adjusting according to actual usage, ensuring efficient data storage and accurate retrieval. Simultaneously, it transforms hazard data into intuitive trend reports and performance indicators, providing data support for safety management decisions. This shifts safety hazard management from passive response to proactive early warning, helping enterprises to promptly identify hazard patterns and formulate targeted safety prevention and control strategies. Attached Figure Description

[0032] Figure 1 This is a flowchart of a security vulnerability database construction method based on multimodal fusion and dynamic updating according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal data processing flow according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the visual application results of an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the text editing and annotation of safety hazard data according to an embodiment of the present invention; Figure 5 This is a system unit diagram of an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Example 1 Please refer to Figure 1 This embodiment 1 provides a method for constructing a security vulnerability database based on multimodal fusion and dynamic updating, including: S1. Continuously collect multimodal hazard data from construction sites, external databases, and user feedback using a multi-source data stream acquisition interface, including text, images, sensor readings, and spatiotemporal information, to obtain a dynamic multimodal data stream; among which, the dynamic multimodal data stream completes the effective judgment, real-time integration, and continuous acquisition of multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions; S2. Real-time preprocessing of dynamic multimodal data streams using streaming data cleaning; S3. Use multimodal fusion and adaptive learning methods to extract and associate features from the preprocessed data, and dynamically update the hidden danger feature vector and database index through online clustering and incremental embedding learning; S4. Based on the dynamically updated feature vectors, the database is periodically reconstructed and optimized using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. S5. Based on the optimization results, generate a hazard trend report and performance indicators using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction.

[0035] This embodiment 1 further elaborates on the above steps.

[0036] (1) Multi-source multimodal data acquisition In fields such as construction, traditional safety hazard management relies on manual inspections and single-modal data recording. This results in problems such as limited data sources, lack of effective integration and dynamic updates, leading to blind spots in hazard identification, superficial analysis, and a lack of data support for safety management decisions. While technological advancements have made multi-source, multi-modal data acquisition possible, the differences in format, dimensions, and time between these data sources make integration difficult. Furthermore, existing databases are mostly static, unable to be dynamically adjusted, resulting in redundant data storage, low retrieval efficiency, and an inability to provide timely support for safety hazard monitoring and early warning.

[0037] Based on this, the construction of the safety hazard database first utilizes a multi-source data stream acquisition interface to continuously collect multimodal hazard data, including text, images, sensor readings, and spatiotemporal information, from sensing devices at the construction site, external databases, and user interaction channels. In the process of building the safety hazard database, determining the effectiveness of the acquisition channels, real-time integration of multi-source data, and ensuring continuous acquisition are key measures to address the shortcomings of traditional safety management data.

[0038] The validity of the data acquisition channels is determined by classifying them into three categories: on-site sensing devices, external databases, and user interactions. Data is filtered based on the data transmission status and modal type compatibility of each channel. This is because only by ensuring the validity of the data source can a reliable foundation be provided for subsequent processing, avoiding deviations in hazard identification due to poor data quality. This solves the problem of single data source and difficulty in guaranteeing data quality in traditional management.

[0039] Real-time integration of multi-source, multi-modal data is achieved by using time-series alignment operators. Samples from different channels and modalities are mapped to a unified time axis and form a dynamic multi-modal data stream. This is done because multi-modal data differ in format and dimension, making direct fusion difficult. A unified time axis and data stream format can break this fusion dilemma and provide comprehensive and coherent information support for in-depth analysis and early warning of potential risks.

[0040] Statistical analysis of the number of continuously collected valid samples and dynamic channel expansion triggered by the collection intensity function are essential to ensure the continuity and sufficiency of the data stream. Only with a sufficient volume of valid data can the database possess self-optimization capabilities, dynamically adjusting its storage structure and retrieval parameters based on actual usage. This ensures efficient data storage and accurate retrieval, helping enterprises promptly identify potential risks and develop targeted security control strategies.

[0041] Based on the above three objectives, the following different solutions are proposed in this embodiment 1: The specific method for determining validity is as follows: Let the set of access channels of the acquisition system be . , These correspond to the on-site sensing device channel, the external database channel, and the user interaction channel, respectively; At any moment The data transmission status is , ,in, This indicates normal transmission. Indicates an interruption; the set of modal types compatible with the channel is: The complete set of target acquisition modes is Define the channel validity determination function. :

[0042] in, The cardinality of a set is the number of elements; it only applies when... At that time, the channel Only the collected data enters the preprocessing process to ensure the effectiveness of the cleaned data.

[0043] The specific method for real-time integration is as follows: Assume each channel is within the time window The set of multimodal data samples acquired internally is , , This represents the total number of data sets. It is a set The first in There are data elements, among which, This indicates the dataset number to which the element belongs. This indicates the index of the element in the corresponding dataset, and the sample. The timestamp is Modal identifier is , Indicates the first The set of modality identifiers corresponding to each dataset; Define timing alignment operator This maps samples from different channels and modalities to a unified time axis.

[0044] in, Indicates the current time reference point; These represent category labels, used to group and filter data by category. Define dynamic multimodal data flow for:

[0045] in, The logical AND operation is used to mark the overall validity of a data stream; Indicates the first The acquisition channels corresponding to each dataset.

[0046] For continuous data collection, the specific method is as follows: Set time window Number of valid samples collected in each channel for:

[0047] in, Indicates the first The effective data volume of each acquisition channel; Indicates the first The first dataset One data element; It is a validity determination function used to determine the validity of data elements. Whether it is effective, among which It is the first One acquisition channel, It is a data element Timestamp; when When this happens, the data element is deemed valid; The system's preset minimum effective sample threshold is Define the intensity acquisition function. :

[0048] when At this time, the system triggers dynamic expansion of the acquisition channel to ensure the continuity of the data stream to meet subsequent processing requirements.

[0049] (2) Real-time preprocessing of streaming data Please refer to Figure 2 After acquiring dynamic multimodal data streams, real-time preprocessing is required through streaming data cleaning. The core purpose is to remove invalid and non-standard data, ensuring that the data entering the subsequent fusion stage has timeliness, validity, and uniform format, thus laying a solid foundation for multimodal feature association and database construction.

[0050] The core of real-time preprocessing for streaming data cleaning of dynamic multimodal data streams is a progressive process of "triple verification + standardized alignment" to ensure data quality and achieve format uniformity. First, valid data is filtered using a dual verification of "transmission status + time window," eliminating incomplete and expired data caused by equipment failures to ensure data real-time performance and integrity. Next, based on the specific valid fields of text, sensors, and images, redundant data from non-target modalities is filtered out, locking in core valid information. Finally, by comparing with the database's structured field templates and establishing field mappings through standardized processing specific to each modality, format uniformity is achieved through full field matching verification, providing a high-quality, highly adaptable data foundation for subsequent multimodal fusion and database construction.

[0051] Specifically, streaming data cleaning involves: Assume the data acquisition device transmission status verification result is: , This indicates that the transmission was stable and complete. This indicates an anomaly; the data timestamp is... The real-time scrolling time window is , Given the window duration corresponding to the data update frequency, the validity of the data is determined as follows:

[0052] when At that time, the data enters the subsequent process; Meanwhile, the dedicated effective domains for text, sensors, and images are defined as follows: , , The data subject is The selection criteria are as follows:

[0053] when At that time, data subject The data was determined to be valid. Furthermore, let the database structured field template be... The semantic similarity matching function for text data is: Sensor data unit conversion operator is The image pixel normalization operator is The aligned data is as follows:

[0054] in, They represent different types of features. This indicates a "mapping" relationship, that is, establishing a connection between the features on the left and the result of the function operation on the right; Then through Perform field-by-field validation to ensure a perfect format match. The cardinality of a set, i.e., the number of elements. Represents the set of aligned fields With template field collection The intersection of.

[0055] (3) Feature extraction and association update In the construction of the safety hazard database, the feature extraction and association process using multimodal fusion and adaptive learning specifically addresses many pain points in traditional management. The feature extraction process is as follows: The feature extraction process takes cleaned multi-source, multi-modal data as input and is completed through modal feature deconstruction, cross-modal feature association, and effective feature selection, as follows: Modal feature deconstruction: Let text, sensor, and image data be respectively... , , Through semantic word segmentation operators Extract the set of text keywords and extract sensor data change features through time-series difference analysis. Through pixel gradient operator Extract image edge features to form a single-modal basic feature set. ; Cross-modal feature association: Defining the feature association degree function ,in, For different modal feature sets, The cardinality of a set, i.e., the number of elements; filtering. Strongly correlated feature pairs, through set union operation Integrating cross-modal features yields a set of strongly correlated cross-modal features. ; Effective feature selection: optimizing the required feature dimensions based on the database. To constrain the calculation of feature information content ,in, The probability of feature occurrence; according to Before selecting descending order These features form the final core feature set. It directly supports dynamic database updates.

[0056] The process of its association is as follows: Let the basic feature sets of each single mode after cleaning be... , , The database historical association feature feedback set is The association process is accomplished through three steps: unified feature space mapping, cross-modal association quantification, and strong association feature selection. The specific formulas are as follows: Unified feature space mapping: Applying modality-specific mapping operators to text, sensor, and image feature sets respectively. ,Right now:

[0057] Output uniform dimension feature vectors , , ; It is a modal type identifier, which clarifies... The range of values ​​represents three different modalities: text, sensor, and image. Cross-modal correlation metric: eigenvectors after any two unified spaces Cross-modal correlation strength is calculated using the cosine similarity formula. :

[0058] Strongly correlated feature selection: using historical correlated feature feedback set Based on this, the correlation threshold is calculated. :

[0059] in, The cardinality of a set, i.e., the number of elements. Represent two types of feature vectors after a unified spatial mapping in history; filter those that satisfy... Strongly correlated feature pairs are obtained through vector concatenation operations. Integrate to form a cross-modal correlation feature set The filtering results directly support the construction of database indexes.

[0060] From a practical perspective, this process, through dynamically updating hazard feature vectors and database indexes, enables safety management departments to grasp the latest characteristics and patterns of hazards in real time, such as newly emerging equipment failure characteristics and behavioral patterns of violations. This allows for the precise formulation of prevention and control strategies, significantly improving the timeliness and accuracy of safety governance, effectively reducing the probability of safety accidents, and protecting the lives and property of personnel.

[0061] In terms of the specific problems it addresses, it solves the difficulty of effectively integrating multimodal data in traditional safety management. Previously, text, image, and sensor data were isolated and unable to work collaboratively. This process, through unified feature space mapping and cross-modal correlation quantification, breaks down modal barriers, allowing different types of data to corroborate and complement each other. For example, a textual description of a hazard location can be combined with images and sensor environmental data to form a complete hazard scenario. Simultaneously, it solves the problem of lagging hazard feature updates. Traditional databases struggle to quickly adapt to new hazard forms, but through online clustering and incremental embedding learning, it can absorb feature changes from new data in real time, ensuring the database always reflects the latest hazard situation and that safety warnings and decisions are always based on the latest and most comprehensive information.

[0062] In short, this process, through efficient fusion and dynamic learning of multimodal data, enables real-time capture of safety hazard characteristics and continuous optimization of the database, providing core technical support for the transformation of safety management from passive response to proactive prevention, and effectively improving the intelligence and refinement of safety management.

[0063] Furthermore, this embodiment also dynamically updates the safety hazard database. Online clustering and incremental embedding learning are the core technical means to achieve continuous optimization of hazard feature vectors and database indexes. The two work together to enable the database to adapt to new hazard feature patterns in real time, providing dynamic and accurate information support for safety management.

[0064] Online clustering is an algorithm that does not require a pre-determined number of clusters and can adjust the cluster structure in real time as new data is added. During the update of hazard features, whenever new pre-processed multimodal data is input, online clustering automatically groups hazard data with similar characteristics into one category. For example, a newly emerging electrical equipment overheating hazard, with its multimodal features such as the textual description of the fault phenomenon, changes in temperature data collected by sensors, and abnormal equipment appearance shown in images, will be identified by online clustering as similar to historical hazard cases and thus grouped into the same cluster. In this way, the system can discover new hazard types or new manifestations of the same hazard type in real time, continuously enriching the hazard feature category system.

[0065] Incremental embedding learning updates the feature embeddings of new input data without retraining the entire model. It compares and merges the feature vectors of the new data with existing feature vectors in the database, dynamically adjusting the embedding representation of each feature vector. For example, when a previously rare fire hazard becomes more frequent due to aging equipment, incremental embedding learning strengthens the weight and representation accuracy of the feature vectors for that type of hazard in the database, making it easier to match in subsequent hazard identification and retrieval.

[0066] When dynamically updating the feature vectors of potential hazards, online clustering first categorizes new data to determine its hazard category or a new category. Incremental embedding learning then adjusts the feature vectors corresponding to that category based on the clustering results, making them more accurately reflect the latest characteristics of that type of hazard. Simultaneously, the database index is also dynamically updated. The index is a structure in the database used for fast data retrieval. When the feature vectors of potential hazards change, the index is adjusted according to the new feature vector relationships, ensuring that subsequent retrieval requests for that type of hazard can quickly and accurately locate the relevant data.

[0067] For example, a new potential collapse hazard caused by improper stacking of construction materials emerges at a construction site. Online clustering identifies its similarity to historical collapse hazards and categorizes it into the same class. Incremental embedding learning then integrates the new features of this type of hazard (such as textual descriptions of material type and stacking method, pressure changes monitored by sensors, and stacking layout in on-site images) into the original feature vector. The database index is also updated synchronously, so that subsequent searches for "construction material stacking collapse hazard" can immediately be linked to the latest feature information, providing accurate reference for safety inspectors.

[0068] This dynamic update mechanism transforms the safety hazard database from a static information repository into a living "safety brain" that continuously learns new hazard knowledge and updates its own "cognition." This enables real-time identification, accurate early warning, and efficient handling of hazards in safety management, significantly improving the foresight and effectiveness of safety management.

[0069] (4) Database adaptive optimization and reconstruction After the dynamic update of the hazard feature vector is completed, the database needs to be periodically reconstructed and optimized based on the adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters around three dimensions: data freshness, query frequency, and error feedback, to ensure that the database always maintains high-efficiency storage and retrieval performance, providing strong support for the rapid identification and handling of safety hazards.

[0070] Let the dynamically updated feature vector set be... The data timestamp set is , Represents the dynamically updated feature vector set Total data volume; historical query record set is The query error set is , For the first The difference between the historical query results and the actual values, ; Represents the historical query record set and query error set The total number of records; the adaptive optimization mechanism achieves dynamic database adjustment through the following three steps: The first step is data freshness measurement. The system analyzes the timestamps of the dynamically updated feature vector set, comparing the current time with the timestamps of each feature vector to calculate the average difference between the current time and historical timestamps. This average difference is used as a threshold to determine whether the feature vectors are fresh data. Only data whose timestamps meet the freshness requirements are included in subsequent optimization considerations. This step filters out old data that has lost its real-time reference value due to time lag, ensuring that the data participating in optimization is timely. Specifically: Data Freshness Measurement: Defining a Freshness Judgment Function ,in, , For the current time, The difference between the current time and the average of historical timestamps. For this is the first The timestamp of each data item; when At that time, the data is determined to be fresh data; Next is the query frequency and error feedback integration stage. The system counts the occurrences of each query in historical query records and, combined with its corresponding query error, filters out those queries that occur frequently and have low error rates—the "high-frequency, low-error" queries. These queries reflect users' core needs for potential data, and their corresponding characteristics are more practically valuable. This filtering process clarifies the key areas for database optimization. Specifically: Integrating query frequency and error feedback: Defining a query validity function ,in, It is the first 100 historical query records For query Number of occurrences This represents the total number of historical query records. It is the first Error of each query Represents the query error set The maximum error value in the query; this function filters the features corresponding to queries that are "high-frequency and low-error". Finally, there's the storage structure and retrieval parameter adjustment stage. The system integrates feature vectors that satisfy data freshness and query validity to construct a new storage structure. During construction, the difference in the number of elements at the intersection of the new storage structure's feature set, the dynamic feature set, and the historical query set is calculated. By adjusting the retrieval parameters, this difference is minimized. This adjusted storage structure better matches actual query needs, and the retrieval parameters make the database more efficient when processing high-frequency, valid queries, significantly improving the speed and accuracy of data retrieval. This allows security management departments to quickly obtain the necessary vulnerability information and promptly carry out prevention and control work. Details are as follows: Storage structure and retrieval parameter adjustment: for dynamically updated feature vector sets Simultaneously, features that satisfy data freshness and query validity are selected, and these features are integrated to form a new storage structure. :

[0071] in, The criteria for determining "data freshness" are determined by... Definition: Data timestamps must meet freshness requirements; The criteria for determining whether a query is valid are determined by... The definition is that a query is a high-frequency, low-error effective query; By calculating the cardinality difference between the cardinality of the feature set in the new storage structure and the cardinality difference between the intersection of the dynamic feature set and the historical query set, the retrieval parameters that minimize this difference are found. :

[0072] in, Indicates the first 100 historical query records; It is the mathematical symbol for the number of elements in a set.

[0073] The entire adaptive optimization process is a cyclical process. As new data is continuously input and new queries are continuously generated, the database will continuously adjust itself to maintain the best storage and retrieval status, providing solid database support for the dynamic management of security risks.

[0074] (5) Hazard reporting and indicator generation After completing the adaptive optimization of the database, the system will generate hazard trend reports and performance indicators based on the optimization results, using real-time data mining and visualization engines, providing intuitive and quantitative reference for safety management decisions.

[0075] Real-time data mining performs in-depth analysis of optimized hazard data in the database, uncovering potential patterns in hazards across time, space, and type. For example, it analyzes the frequency changes of various hazards over different time periods to identify peak hazard times; it explores the distribution of hazards in different regions to pinpoint areas with concentrated hazards; and it identifies the correlations between different types of hazards, such as whether a certain type of equipment failure is often accompanied by other types of hazards.

[0076] The visualization engine presents these patterns obtained from data mining in intuitive forms such as charts and graphs, forming a hazard trend report. For example, a line chart can show the time trend of hazard occurrence frequency, a heat map can show the spatial distribution of hazards, and a correlation diagram can show the relationship between different hazard types. At the same time, the visualization engine also calculates and displays the database's performance indicators, such as data retrieval response time, query result accuracy, and database storage utilization. These indicators reflect the database's operating status and service capabilities.

[0077] By generating hazard trend reports, safety management departments can clearly grasp the development trend of hazards and formulate targeted prevention and control measures in advance, such as strengthening inspections during periods of high hazard incidence and increasing monitoring equipment in areas with concentrated hazards. The display of performance indicators helps technical personnel to promptly identify problems in database operation. For example, if the search response time is too long, the database structure or search algorithm can be further optimized to ensure that the database can always efficiently support safety management work. The entire process is continuous; as the database is constantly updated and optimized, the hazard trend reports and performance indicators are also updated in real time, providing continuous and dynamic decision support for safety management.

[0078] Based on the above methods, this embodiment 1 also provides an example of a result, such as... Figure 3 and Figure 4 As shown: Figure 3 This interface presents the visual application results of the safety hazard database construction system. It serves as the output carrier after the entire methodology is implemented: the "Total Data" in the system overview module corresponds to the effective data continuously accumulated after multi-source data stream collection and streaming cleansing; "Safety Hazard Rate" and "Unprocessed Hazards" are the real-time dynamic statistical results of the hazard data after adaptive optimization, intuitively reflecting the database's operational efficiency. The hazard type distribution pie chart in the data visualization and reporting module is a visual representation of hazard data categorized by type after "cross-modal feature association and effective filtering" in the methodology; the hazard rate trend chart for the past 7 days is the analysis result of the time-dimensional patterns of hazards in the "real-time data mining" stage of the methodology; and the generated hazard report list is a direct result of the "visualization engine generating trend reports" in the methodology, providing a practical reference document for safety management.

[0079] Figure 4The interface showcases the text editing and annotation interface for safety hazard data. This function is a key step in the "multimodal data preprocessing and feature deconstruction" part of the methodology: the interface allows for editing of hazard text, standardized annotation of hazard types (equipment, personnel, etc.) and severity, which corresponds to the "semantic segmentation and effective domain selection of text data" process in the methodology. By clarifying the hazard type, severity, and other attributes, it provides a structured text feature foundation for the subsequent "unified feature space mapping." At the same time, the annotation operation record allows for traceability of the annotator and the content, ensuring data reliability and making the results of subsequent cross-modal feature association more accurate.

[0080] The results of these two interfaces fully cover the entire process of this method, from "multi-source data acquisition - structured annotation - feature fusion - dynamic optimization - visualization output", demonstrating the practicality and applicability of the method in the management of security hazard data.

[0081] Example 2 Please refer to Figure 5 This embodiment 2 also provides a security vulnerability database construction system based on multimodal fusion and dynamic updating, including: The multi-source multimodal data acquisition unit is used to continuously collect multimodal hazard data from the construction site, external databases, and user feedback using a multi-source data stream acquisition interface. This data includes text, images, sensor readings, and spatiotemporal information, resulting in a dynamic multimodal data stream. The dynamic multimodal data stream effectively determines, integrates in real time, and continuously acquires multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions. The streaming data real-time preprocessing unit is used to perform real-time preprocessing on dynamic multimodal data streams using streaming data cleaning. The feature extraction and association update unit is used to extract and associate features from the preprocessed data using multimodal fusion and adaptive learning methods. Through online clustering and incremental embedding learning, it dynamically updates the hidden danger feature vector and database index. The database adaptive optimization and reconstruction unit is used to periodically reconstruct and optimize the database based on dynamically updated feature vectors using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. The hazard reporting and indicator generation unit is used to generate hazard trend reports and performance indicators based on the optimization results using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction.

[0082] Example 3 This embodiment 3 also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement any step of a method for constructing a security vulnerability database based on multimodal fusion and dynamic updates.

[0083] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0085] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a security hazard database based on multi-modal fusion and dynamic update, characterized in that, include: S1. Continuously collect multimodal hazard data from construction sites, external databases, and user feedback using a multi-source data stream acquisition interface, including text, images, sensor readings, and spatiotemporal information, to obtain a dynamic multimodal data stream; among which, the dynamic multimodal data stream completes the effective judgment, real-time integration, and continuous acquisition of multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions; S2. Real-time preprocessing of dynamic multimodal data streams using streaming data cleaning; S3. Use multimodal fusion and adaptive learning methods to extract and associate features from the preprocessed data, and dynamically update the hidden danger feature vector and database index through online clustering and incremental embedding learning; S4. Based on the dynamically updated feature vectors, the database is periodically reconstructed and optimized using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. S5. Based on the optimization results, generate a hazard trend report and performance indicators using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction. 2.The method according to claim 1, characterized in that, The method for determining validity in S1 is as follows: Let the access channel set of the acquisition system be , corresponding to the field sensing device channel, the external database channel, and the user interaction channel, respectively; the channel The data transmission state of the channel at time is , , wherein represents normal transmission, and represents interruption; the modal type set compatible with the channel is , and the target acquisition modal set is ; define a channel validity determination function : in, The cardinality of a set is the number of elements; it only applies when... At that time, the channel Only the collected data enters the preprocessing process to ensure the effectiveness of the cleaned data.

3. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 2, characterized in that, The real-time integration method in S1 is as follows: Assume each channel is within the time window The set of multimodal data samples acquired internally is , , This represents the total number of data sets. It is a set The first in There are data elements, among which, This indicates the dataset number to which the element belongs. This indicates the index of the element in the corresponding dataset, and the sample. The timestamp is Modal identifier is , Indicates the first The set of modality identifiers corresponding to each dataset; Define timing alignment operator This maps samples from different channels and modalities to a unified time axis. in, Indicates the current time reference point; These represent category labels, used to group and filter data by category. Define dynamic multimodal data flow for: in, The logical AND operation is used to mark the overall validity of a data stream; Indicates the first The acquisition channels corresponding to each dataset.

4. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 3, characterized in that, The method for continuous data acquisition in S1 is as follows: Set time window Number of valid samples collected in each channel for: in, Indicates the first The effective data volume of each acquisition channel; Indicates the first The first dataset One data element; It is a validity determination function used to determine the validity of data elements. Whether it is effective, among which It is the first One acquisition channel, It is a data element Timestamp; when When this happens, the data element is deemed valid; The system's preset minimum effective sample threshold is Define the intensity acquisition function. : when At this time, the system triggers dynamic expansion of the acquisition channel to ensure the continuity of the data stream to meet subsequent processing requirements.

5. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 1, characterized in that, The streaming data cleaning in S2 specifically involves: Assume the data acquisition device transmission status verification result is: , This indicates that the transmission was stable and complete. Indicates an anomaly; Data timestamp is The real-time scrolling time window is , Given the window duration corresponding to the data update frequency, the validity of the data is determined as follows: when At that time, the data enters the subsequent process; Meanwhile, the dedicated effective domains for text, sensors, and images are defined as follows: , , The data subject is The selection criteria are as follows: when At that time, data subject The data was determined to be valid. Furthermore, let the database structured field template be... The semantic similarity matching function for text data is: Sensor data unit conversion operator is The image pixel normalization operator is The aligned data is as follows: in, They represent different types of features. This indicates a "mapping" relationship, that is, establishing a connection between the features on the left and the result of the function operation on the right; Then through Perform field-by-field validation to ensure a perfect format match. The cardinality of a set, i.e., the number of elements. Represents the set of aligned fields With template field collection The intersection of.

6. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 1, characterized in that, The feature extraction process in S3 is as follows: The feature extraction process takes cleaned multi-source, multi-modal data as input and is completed through modal feature deconstruction, cross-modal feature association, and effective feature selection, as follows: Modal feature deconstruction: Let text, sensor, and image data be respectively... , , Through semantic word segmentation operators Extract the set of text keywords and extract sensor data change features through time-series difference analysis. Through pixel gradient operator Extract image edge features to form a single-modal basic feature set. ; Cross-modal feature association: Define feature correlation function ,in, For different modal feature sets, The cardinality of a set, i.e., the number of elements; filtering. Strongly correlated feature pairs, through set union operation Integrating cross-modal features yields a set of strongly correlated cross-modal features. ; Effective feature selection: optimizing the required feature dimensions based on the database. To constrain the calculation of feature information content ,in, The probability of feature occurrence; according to Before selecting descending order These features form the final core feature set. It directly supports dynamic database updates.

7. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 1, characterized in that, The association process in S3 is as follows: Let the basic feature sets of each single mode after cleaning be... , , The database historical association feature feedback set is The association process is accomplished through three steps: unified feature space mapping, cross-modal association quantification, and strong association feature selection. The specific formulas are as follows: Unified feature space mapping: Applying modality-specific mapping operators to text, sensor, and image feature sets respectively. ,Right now: Output uniform dimension feature vectors , , ; It is a modal type identifier, which clarifies... The range of values ​​represents three different modalities: text, sensor, and image. Cross-modal correlation metric: eigenvectors after any two unified spaces Cross-modal correlation strength is calculated using the cosine similarity formula. : Strongly correlated feature selection: using historical correlated feature feedback set Based on this, the correlation threshold is calculated. : in, The cardinality of a set, i.e., the number of elements. Represent two types of feature vectors after a unified spatial mapping in history; filter those that satisfy... Strongly correlated feature pairs are obtained through vector concatenation operations. Integrate to form a cross-modal correlation feature set The filtering results directly support the construction of database indexes.

8. The method for constructing a security vulnerability database based on multimodal fusion and dynamic updating according to claim 1, characterized in that, The adaptive optimization mechanism in S4 is specifically as follows: Let the dynamically updated feature vector set be... The data timestamp set is , Represents the dynamically updated feature vector set Total data volume; historical query record set is The query error set is , For the first The difference between the historical query results and the actual values, ; Represents the historical query record set and query error set The total number of records; the adaptive optimization mechanism achieves dynamic database adjustment through the following three steps: Data Freshness Measurement: Defining a Freshness Judgment Function ,in, , For the current time, The difference between the current time and the average of historical timestamps. For this is the first The timestamp of each data item; when At that time, the data is determined to be fresh data; Integrating query frequency and error feedback: Defining a query validity function ,in, It is the first 100 historical query records For query Number of occurrences This represents the total number of historical query records. It is the first Error of each query Represents the query error set The maximum error value in the query; this function filters the features corresponding to queries that are "high-frequency and low-error". Storage structure and retrieval parameter adjustment: for dynamically updated feature vector sets Simultaneously, features that satisfy data freshness and query validity are selected, and these features are integrated to form a new storage structure. : in, The criteria for determining "data freshness" are determined by... Definition: Data timestamps must meet freshness requirements; The criteria for determining whether a query is valid are determined by... The definition is that a query is a high-frequency, low-error effective query; By calculating the cardinality difference between the cardinality of the feature set in the new storage structure and the cardinality difference between the intersection of the dynamic feature set and the historical query set, the retrieval parameters that minimize this difference are found. : in, Indicates the first 100 historical query records; It is the mathematical symbol for the number of elements in a set.

9. A security vulnerability database construction system based on multimodal fusion and dynamic updating, characterized in that, include: The multi-source multimodal data acquisition unit is used to continuously collect multimodal hazard data from the construction site, external databases, and user feedback using a multi-source data stream acquisition interface. This data includes text, images, sensor readings, and spatiotemporal information, resulting in a dynamic multimodal data stream. The dynamic multimodal data stream effectively determines, integrates in real time, and continuously acquires multi-source multimodal hazard data through set operations, time-series alignment operators, and acquisition intensity functions. The streaming data real-time preprocessing unit is used to perform real-time preprocessing on dynamic multimodal data streams using streaming data cleaning. The feature extraction and association update unit is used to extract and associate features from the preprocessed data using multimodal fusion and adaptive learning methods. Through online clustering and incremental embedding learning, it dynamically updates the hidden danger feature vector and database index. The database adaptive optimization and reconstruction unit is used to periodically reconstruct and optimize the database based on dynamically updated feature vectors using an adaptive optimization mechanism. This mechanism automatically adjusts the storage structure and retrieval parameters according to data freshness, query frequency, and error feedback. The hazard reporting and indicator generation unit is used to generate hazard trend reports and performance indicators based on the optimization results using real-time data mining and visualization engines; Feature extraction and association involve deconstructing single-modal features and associating cross-modal features from cleaned multi-source multimodal data, then filtering the information content to obtain a core feature set, mapping single-modal features to a unified space, quantifying the association strength, and then selecting strongly associated feature pairs to form a feature set that supports dynamic database updates and index construction.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8: a method for constructing a security vulnerability database based on multimodal fusion and dynamic updates.

Images