Multi-modal computing power knowledge base construction method and device, equipment, medium and product

Abstract

The invention discloses a multi-modal computing power knowledge base construction method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting multi-source data in a computing power safety scene, and carrying out the multi-modal association labeling of the multi-source data, and obtaining multi-modal computing power safety data; performing differential preprocessing on the multi-modal computing power security data to obtain preprocessed multi-modal computing power security data; performing multi-modal analysis on the preprocessed multi-modal computing power security data to obtain multi-modal text content and time sequence feature vectors corresponding to the multi-modal computing power security data; performing text block segmentation processing on the text content, and performing semantic vector calculation on the text content after the text block segmentation processing to obtain a new semantic vector containing a context; and generating a multi-modal RAG knowledge base based on the time sequence feature vector and the semantic vector containing the context. According to the method, the comprehensiveness and the accuracy of security event response can be improved by constructing the multi-modal computing power knowledge base.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium and product for constructing a multimodal computing power knowledge base. Background Technology

[0002] With the large-scale deployment of cloud computing, edge computing, and AI computing infrastructure, computing power security has become a core guarantee for the development of the digital economy, directly related to the stable operation of computing infrastructure and the secure conduct of digital businesses. The current computing environment faces complex security threats, including ransomware attacks, hardware supply chain vulnerabilities, and the abuse of computing resources. The efficiency of response to such security incidents highly depends on the knowledge base's ability to integrate and analyze multi-source security data.

[0003] Current mainstream knowledge bases are primarily unimodal text-based, supporting only structured storage and retrieval of text data. This fails to meet the demands for cross-modal information utilization in current computing power security scenarios, thus limiting the comprehensiveness and accuracy of security incident response. Therefore, there is an urgent need to provide a multimodal knowledge base capable of handling cross-modal information to overcome the limitations of unimodal technology, improve the comprehensiveness and accuracy of security incident response in computing power security scenarios, and provide stronger data support for the stable operation of computing infrastructure. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in related technologies, the purpose of this application is to provide a method, apparatus, equipment, medium and product for constructing a multimodal computing power knowledge base. By constructing a multimodal computing power knowledge base, the comprehensiveness and accuracy of security incident response can be improved, and data support can be provided for the stable operation of computing power infrastructure.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for constructing a multimodal computing power knowledge base, including: Collect multi-source data under computing power security scenarios, and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data; Differential preprocessing is performed on the multimodal computing power security data to obtain preprocessed multimodal computing power security data; Multimodal parsing is performed on the preprocessed multimodal computing power security data to obtain the multimodal text content and temporal feature vector corresponding to the multimodal computing power security data; The multimodal text content is segmented into text blocks, and semantic vectors are calculated on the segmented text content to obtain new semantic vectors containing context. A multimodal RAG knowledge base is generated based on the temporal feature vector and the semantic vector containing the context.

[0006] Optionally, the step of performing multimodal association annotation on the multi-source data to obtain multimodal computing power security data includes: annotating the multi-source data based on a weakly supervised learning model and manual verification to obtain the multimodal computing power security data.

[0007] Optionally, the multimodal computing power security data includes labeled text data, labeled image or video data, labeled audio data, and labeled structured data; the labeled text data includes computing power security logs, security policy documents, threat intelligence text, and security alarm information; the labeled image or video data includes monitoring screens of the physical environment of computing power nodes, network traffic visualization charts, and screenshots of security device interfaces; the labeled audio data includes hardware fault alarm sounds and security system voice prompts; the labeled structured data includes security device indicators and network traffic data packet characteristics; the differential preprocessing of the multimodal computing power security data to obtain preprocessed multimodal computing power security data includes: performing differential preprocessing on the labeled text data, the labeled image or video data, the labeled audio data, and the labeled structured data respectively to obtain preprocessed text data, preprocessed image / video data, preprocessed audio data, and preprocessed structured data.

[0008] Optionally, the step of performing multimodal parsing on the preprocessed multimodal computing power security data to obtain multimodal text content and temporal feature vectors corresponding to the multimodal computing power security data includes: parsing the preprocessed text data using the Apache Tika tool to obtain document text content in a preset format; separating the audio data and frame image data in the preprocessed image / video data using the FFmpeg tool; parsing the image data and separated frame image data of the preprocessed image / video data using the Tesseract OCR engine or a multimodal large model to obtain image text content; parsing the preprocessed audio data into audio text content using the Whisper ASR model; and parsing the preprocessed structured data using an LSTM time series model to obtain the temporal feature vectors.

[0009] Optionally, the step of performing text block segmentation on the multimodal text content and calculating semantic vectors on the segmented text content to obtain new semantic vectors containing context includes: segmenting the multimodal text content into M text blocks based on paragraph separators and line breaks according to the maximum number of characters in a preset text block, where M is a positive integer and M≥10; calculating the initial semantic vector of each text block in a preset dimension using an Embedding model, and clustering the M initial semantic vectors into N clusters using a K-Means clustering algorithm, where N is a positive integer and 1<N<M; traversing the N clusters and merging all text blocks within each cluster to generate merged text; generating a token vector of a preset dimension for each token in the merged text using an Embedding model; and aggregating the token vectors corresponding to each text block within a cluster using an attention weighted summation method to generate the new semantic vector containing context for each text block, wherein there are M new semantic vectors.

[0010] Optionally, generating a multimodal RAG knowledge base based on the temporal feature vector and the semantic vector containing context includes: importing M new semantic vectors and the temporal feature vector into a Milvus vector database, establishing a vector index, wherein the vector index adopts the IVF_FLAT index type; storing the multimodal original data association identifiers corresponding to the new semantic vector and the temporal feature vector, generating a reverse mapping between the new semantic vector and the temporal feature vector and the corresponding original multimodal data; and obtaining the multimodal RAG knowledge base based on the vector index, the reverse mapping between the new semantic vector and the temporal feature vector and the corresponding original multimodal data, and the new semantic vector and the temporal feature vector.

[0011] Secondly, this application provides a multimodal computing power knowledge base construction device, comprising: The data collection and annotation module is used to collect multi-source data in computing power security scenarios and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data. The preprocessing module is used to perform differentiated preprocessing on the multimodal computing power security data to obtain preprocessed multimodal computing power security data. The data parsing module is used to perform multimodal parsing on the preprocessed multimodal computing power security data to obtain multimodal text content and temporal feature vectors corresponding to the multimodal computing power security data; The text segmentation module is used to perform text block segmentation processing on the multimodal text content, and to calculate the semantic vector of the text content after text block segmentation processing to obtain a new semantic vector containing the context. The knowledge base generation module is used to generate a multimodal RAG knowledge base based on the temporal feature vector and the semantic vector containing the context.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multimodal computing power knowledge base construction method described in any one of the above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal computing power knowledge base construction method described in any one of the above.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal computing power knowledge base construction method described above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, medium, and product for constructing a multimodal computing power knowledge base. It obtains multimodal computing power security data by collecting multi-source data in computing power security scenarios and performing multimodal association annotation on the multi-source data. Preprocessing of the multimodal computing power security data yields preprocessed multimodal computing power security data. Multimodal parsing of the preprocessed multimodal computing power security data yields corresponding multimodal text content and temporal feature vectors. Text content is segmented into text blocks, and semantic vectors are calculated from the segmented text content to obtain new semantic vectors containing context. A multimodal RAG knowledge base is generated using the temporal feature vectors and the context-containing semantic vectors. This multimodal RAG knowledge base addresses the technical limitations of existing single-modal knowledge bases, which can only process text data and cannot integrate key non-text data such as hardware fault images and alarm audio. It achieves systematic integration of multimodal data in computing power security scenarios, providing multi-dimensional data support for security event analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is an application environment diagram of a multimodal computing power knowledge base construction method according to an embodiment of this application; Figure 2 A flowchart illustrating a method for constructing a multimodal computing power knowledge base according to an embodiment of this application; Figure 3 A schematic diagram of the functional modules of a multimodal computing power knowledge base construction device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The multimodal computing power knowledge base construction method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Server 104 collects multi-source data from terminal device 102 in the computing power security scenario, performs multimodal association annotation on the multi-source data to obtain multimodal computing power security data; performs differentiated preprocessing on the multimodal computing power security data to obtain preprocessed multimodal computing power security data; performs multimodal parsing on the preprocessed multimodal computing power security data to obtain multimodal text content and temporal feature vectors corresponding to the multimodal computing power security data; performs text block segmentation on the multimodal text content, and calculates semantic vectors on the segmented text content to obtain new semantic vectors containing context; based on the temporal feature vectors and the semantic vectors containing context, a multimodal RAG knowledge base is generated.

[0021] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0022] It should be noted that RAG (Retrieval-Augmented Generation) is a technical framework that combines external knowledge base retrieval with large language model (LLM) generation. This application can solve the technical defects of existing single-modal knowledge bases in processing only text data and failing to integrate key non-text data such as hardware fault images and alarm audio by constructing a multimodal RAG knowledge base. It can realize the systematic integration of multimodal data in computing power security scenarios and provide multi-dimensional data support for security event analysis.

[0023] In one exemplary embodiment, such as Figure 2 As shown, a method for constructing a multimodal computing power knowledge base is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S205. Wherein: Step S201: Collect multi-source data under computing power security scenarios, and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data.

[0024] In the example implementation, the multi-source data in the computing power security scenario includes text, images, videos, audio, structured data, etc., covering all types of data in the computing power security scenario. Among them, text data is the most direct carrier of semantic information in the computing power security scenario, mainly covering four categories: security logs, policy documents, threat intelligence, and alarm information. Among them, computing power security logs include server access logs (such as user login records and data transmission logs) and vulnerability scanning reports (such as vulnerability locations and risk level assessment results); security policy documents include computing power security management specifications, device protection configuration instructions, and access control rule documents; threat intelligence texts include CVE vulnerability descriptions (such as vulnerability number, scope of impact, and exploitation method) and ATT&CK framework (standardized threat model texts such as network attack lifecycle and attack technology classification); security alarm information includes real-time triggered security event notification texts (such as "SQL injection attempt detected on a certain server" and "Hardware temperature exceeds the limit alarm").

[0025] Image data primarily targets non-textual anomalies in the physical environment, visualizations, and device interfaces within computing power security scenarios. This image data mainly includes monitoring footage of the physical environment of computing power nodes, network traffic visualizations, and screenshots of security device interfaces. Figure 3 Major categories. Monitoring screens of the physical environment of computing nodes include abnormal equipment lights in server racks (such as red fault lights and flashing alarm lights), hardware failure images (such as damaged interfaces and stopped fans), and abnormal images of the data center environment (such as water accumulation and foreign object intrusion); network traffic visualization charts include real-time traffic monitoring line charts, abnormal traffic distribution charts, and port occupancy rate bar charts; security device interface screenshots include firewall configuration interface screenshots, intrusion detection system (IDS) alarm interface screenshots, and vulnerability scanning tool result display screenshots.

[0026] Video data is a temporal extension of image data, with dynamic physical monitoring and equipment operation processes at its core. It mainly includes three categories: dynamic monitoring videos of the physical environment of computing nodes, videos of security equipment operation, and dynamic visualization videos of network traffic. Specifically, dynamic monitoring videos of the physical environment of computing nodes include real-time operation videos of server racks (used to capture changes in equipment lighting and abnormal hardware actions) and dynamic monitoring videos of data center entrances and exits (to prevent unauthorized personnel from accessing computing equipment); videos of security equipment operation include demonstration videos of vulnerability scanning tools and videos of equipment status changes when security alarms are triggered (such as the dynamic process of alarm lights flashing); and dynamic visualization videos of network traffic include real-time fluctuation videos of network bandwidth and dynamic chart videos of abnormal traffic attacks.

[0027] The audio data corresponds to sound alarms and prompts in computing power security scenarios, mainly including two categories: hardware fault alarm sounds and security system voice prompts. Hardware fault alarm sounds include abnormal server fan noise (such as stuttering or high-frequency screeching), power supply failure beeps, and hard drive read / write abnormal sounds; security system voice prompts include security alarm voice notifications (such as "Ransomware attack detected, please handle immediately") and device status voice broadcasts (such as "CPU utilization of computing node 1 exceeds 90%"). The audio formats used in this application embodiment are mainly mainstream audio formats such as mp3, wav, and ogg.

[0028] Structured data comprises quantifiable metrics of computing power equipment and network characteristics, used to capture numerical anomalies. It mainly includes three categories: computing power equipment performance metrics, network operation metrics, and network traffic characteristic data. Computing power equipment performance metrics include CPU utilization, memory usage, GPU temperature, GPU computing load, hard disk storage utilization, and server power supply voltage. Network operation metrics include network bandwidth usage, data packet transmission rate, port throughput, and TCP / UDP connection count. Network traffic characteristic data includes standardized numerical features such as abnormal data packet size distribution, access frequency of specific IP addresses, and malicious port access records.

[0029] In some embodiments, step S201 above, which involves performing multimodal association annotation on multi-source data to obtain multimodal computing power security data, includes: annotating multi-source data based on a weakly supervised learning model and manual verification to obtain multimodal computing power security data.

[0030] Understandably, weakly supervised learning models can be used to label multimodal computing security data, while key data such as high-risk vulnerability descriptions and emergency alarms can be manually verified.

[0031] It should be noted that during manual verification, verifiers perform verification based on a pre-built security domain labeling system and label rule definitions, consisting of "first-level label - second-level label - label attribute". The security domain labeling system is shown in Table 1 below: Table 1

[0032] Tagging rules are defined as the judgment rules for each secondary tag, and can include multimodal feature descriptions. For example, the tag rule for "ransomware attack" is: text data containing keywords such as "file encryption," "ransomware message," and "LockBit / Ryuk," or structured data displaying "sudden increase in CPU utilization + abnormal disk I / O," or monitoring images containing "ransomware pop-up interface," or audio containing "system abnormal alarm voice." Meeting any one of these modal features is sufficient for a preliminary match to the tag. Verification personnel manually annotate the tags according to the tagging rules.

[0033] When using a weakly supervised learning model for annotation, the Label Propagation algorithm is selected as the core weakly supervised model. Furthermore, based on a pre-built security domain labeling system, similarity thresholds (e.g., text data similarity ≥ 0.8, image feature distance ≤ 0.2) are set to automatically annotate high-confidence data (≥ 90%), directly retaining the annotation results. Low-confidence data (< 90%) is automatically annotated but not included in subsequent processes, awaiting manual verification. Multimodal association annotation is also employed, such as establishing semantic associations across modal data (matching text alerts with corresponding monitoring image timestamps).

[0034] By using a labeling mechanism that combines automatic labeling with manual verification through weakly supervised learning models, we can improve data labeling efficiency while ensuring labeling accuracy, thus balancing data processing speed and labeling quality.

[0035] Step S202: Perform differentiated preprocessing on the multimodal computing power security data to obtain preprocessed multimodal computing power security data.

[0036] In conjunction with the above embodiments, multimodal computing power security data includes labeled text data, image or video data, audio data, and structured data; labeled text data includes computing power security logs, security policy documents, threat intelligence texts, and security alarm information; labeled image or video data includes monitoring screens of the physical environment of computing power nodes, network traffic visualization charts, and screenshots of security device interfaces; labeled audio data includes hardware fault alarm sounds and security system voice prompts; labeled structured data includes security device indicators and network traffic data packet characteristics; therefore, differentiated preprocessing refers to processing labeled text data, image or video data, audio data, and structured data using different processing methods. That is, differentiated preprocessing is performed on labeled text data, image or video data, audio data, and structured data respectively to obtain preprocessed text data, preprocessed image / video data, preprocessed audio data, and preprocessed structured data.

[0037] Specifically, when preprocessing the annotated text data, redundant information such as duplicate log entries and invalid characters is removed. The terms "vulnerability," "attack," and "vulnerability" are unified into a pre-defined standardized expression. The jieba word segmentation tool is used to perform word segmentation operations, combined with part-of-speech tagging. The TF-IDF algorithm is used to extract security keywords such as "SQL injection," "DDoS attack," and "hardware vulnerability."

[0038] When preprocessing the labeled image / video data, Gaussian filtering for noise reduction, size normalization, and key region extraction are performed sequentially on the labeled image data. Specifically, the labeled image data is normalized to 512×512 pixels, and the YOLOv8 model is used to extract key regions such as server panel indicator lights and hardware interfaces. The labeled video data is subjected to frame extraction at a frequency of 25 frames per second, retaining key frames containing abnormal features, and then processed according to the preprocessing flow of the labeled image data. The key frames with abnormal features can be limited according to a pre-built security domain labeling system, and no specific restrictions are specified here.

[0039] When preprocessing the labeled audio data, the labeled audio data format is converted into mp3, wav, and ogg audio formats. When preprocessing the labeled structured data, the Min-Max normalization algorithm is used to convert computing power indicators such as CPU utilization, memory usage, GPU temperature, and network bandwidth to the [0,1] interval, forming a standardized computing power indicator dataset.

[0040] Step S203: Perform multimodal parsing on the preprocessed multimodal computing power security data to obtain the multimodal text content and temporal feature vector corresponding to the multimodal computing power security data.

[0041] Optionally, the Apache Tika tool is used to parse the preprocessed text data to obtain document text content in PDF, DOC, DOCX, TXT, Excel, and CSV formats; the FFmpeg tool is used to separate the audio data and frame image data in the preprocessed image / video data; the Tesseract OCR engine or a multimodal large model is used to parse the image data and separated frame image data of the preprocessed image / video data to obtain image text content; the Whisper ASR model is used to parse the preprocessed audio data into audio text content, and metadata such as audio duration and sampling rate are retained, so that the audio text and metadata are stored together; the LSTM time series model is used to parse the preprocessed structured data to obtain time series feature vectors.

[0042] Step S204: Perform text block segmentation on the multimodal text content, and calculate the semantic vector of the text content after text block segmentation to obtain a new semantic vector containing the context.

[0043] In the example embodiment, text block segmentation processing refers to dividing multimodal text content into text blocks of the same length according to a preset length. In this embodiment, the preset length is the maximum number of characters in the preset text block. The specific length value is based on actual application and is not specifically limited in this application.

[0044] Optionally, step S204 may include steps S2041 to S2045, specifically: Step S2041: Based on the maximum number of characters in a preset text block, the multimodal text content is divided into M text blocks according to paragraph separators and line breaks, where M is a positive integer and M≥10; Step S2042: The Embedding model is used to calculate the initial semantic vector of each text block in a preset dimension, and the K-Means clustering algorithm is used to cluster the M initial semantic vectors into N clusters, where N is a positive integer and 1 < N < M. Step S2043: Traverse N clusters and merge all text blocks within each cluster to generate merged text; Step S2044: Use the Embedding model to generate a token vector of a preset dimension for each token in the merged text; Step S2045: Aggregate the token vector corresponding to each text block within the cluster using an attention weighted summation method to generate a new semantic vector containing contextual association for each text block. There are M new semantic vectors.

[0045] It should be noted that in other embodiments, text semantic vectors (768-dimensional), image description vectors (768-dimensional), audio text vectors (768-dimensional), and structured temporal vectors (mapped to 768-dimensional) can be placed in the same feature space. By training a loss function that "brings the distance between positive sample pairs (such as text + image with the same fault) and widens the distance between negative sample pairs (such as text + image with different faults)," the vectors of different modalities can be made directly semantically comparable, replacing the single-modal intra-clustering in the above scheme and solving the problem of cross-modal vectors having different dimensions and semantics that cannot be directly matched.

[0046] Step S205: Generate a multimodal RAG knowledge base based on temporal feature vectors and semantic vectors containing context.

[0047] Optionally, step S205 may include steps S2501 to S2053, namely: Step S2501: Import the M new semantic vectors and temporal feature vectors into the Milvus vector database and create a vector index. The vector index adopts the IVF_FLAT index type. Step S2502: Store the multimodal original data association identifiers corresponding to the new semantic vector and temporal feature vector, and generate the reverse mapping between the new semantic vector and temporal feature vector and the corresponding original multimodal data; Step S2503: Based on the inverse mapping of vector index, new semantic vector and temporal feature vector to the corresponding original multimodal data, new semantic vector and temporal feature vector, the multimodal RAG knowledge base is obtained.

[0048] It should be noted that, in other embodiments, in order to improve the correlation between the labeled structured data and the labeled text data, the temporal feature vector generated by LSTM and the corresponding text embedding vector can be fused through an "attention mechanism" to generate a temporal-text fusion vector, which replaces the problem of storing the temporal feature vector and the text vector separately in the above scheme. By implementing steps S201 to S205 above, multi-source data in computing power security scenarios are collected, and multi-modal association annotation is performed on the multi-source data to obtain multi-modal computing power security data. Preprocessing of the multi-modal computing power security data is performed to obtain preprocessed multi-modal computing power security data. Multi-modal parsing of the preprocessed multi-modal computing power security data yields multi-modal text content and temporal feature vectors corresponding to the multi-modal computing power security data. Text content is segmented into text blocks, and semantic vectors are calculated from the segmented text content to obtain new semantic vectors containing context. A multi-modal RAG knowledge base is generated using the temporal feature vectors and the semantic vectors containing context. This multi-modal RAG knowledge base addresses the technical shortcomings of existing single-modal knowledge bases, which can only process text data and cannot integrate key non-text data such as hardware fault images and alarm audio. It achieves systematic integration of multi-modal data in computing power security scenarios, providing multi-dimensional data support for security event analysis.

[0049] Furthermore, this application can comprehensively cover multi-source data in computing power security scenarios from text data, image or video data, audio data, and structured data. By combining differentiated preprocessing and multimodal parsing, multimodal computing power security data can be obtained. This solves the technical defects of existing single-modal text knowledge bases, which can only process text data and cannot integrate key non-text data such as hardware failure images and alarm audio. It realizes the systematic integration of multimodal data in computing power security scenarios and provides full-dimensional data support for security event analysis. Establishing precise cross-modal data associations through multimodal association annotation can improve the comprehensiveness and accuracy of security incident response; generating new semantic vectors containing context through K-Means clustering + Embedding model can ensure the semantic relevance of text block vectors; and transforming structured data into time-series feature vectors through LSTM time series model can achieve vector dimension adaptation between structured data and other modal data, ensuring the efficiency of security incident retrieval and response.

[0050] By constructing a standardized process of data acquisition, data annotation, data preprocessing, data parsing, generation of new semantic vectors and temporal feature vectors, and storage, a multimodal computing power knowledge base can be built relatively quickly. The multimodal RAG knowledge base can directly support the intelligent retrieval and analysis of computing power security events. For example, when troubleshooting hardware faults, the system can quickly match multimodal data (fault images, repair text, alarm audio) of similar historical faults, providing operation and maintenance personnel with a visual and traceable basis for decision-making. This makes up for the shortcomings of existing single-modal knowledge bases in intelligent retrieval and scenario-based knowledge reuse, and provides more intelligent and reliable support for the stable operation of computing power infrastructure.

[0051] In another exemplary embodiment of this application, in order to protect user privacy and core computing power parameters, the embodiments of this application need to desensitize the multi-source data after collecting it.

[0052] In some embodiments, the desensitization process uses conventional mask replacement.

[0053] Furthermore, in other embodiments, to prevent the leakage of enterprise computing power parameters in computing power security scenarios, differential privacy technology can be introduced in the stages of generating new semantic vectors with context using the K-Means clustering + Embedding model and generating time-series feature vectors using the LSTM time series model to transform structured data. Specifically, noise conforming to ε-differential privacy is added to the time-series vectors of sensitive computing power indicators (such as GPU computing power values ​​and the number of cluster nodes), for example, ε is set to 0.1, to ensure privacy protection without affecting anomaly identification; noise is added to text blocks containing user privacy (such as operation logs of maintenance personnel), which can reduce the risk that sensitive information may still be inferred from the vectors after desensitization in existing solutions, meeting the requirement of data usability without visibility in computing power security scenarios.

[0054] During the multi-source data acquisition phase, sensitive data (such as user privacy and core computing power parameters) are anonymized, and dynamic computing power data is obtained by connecting to real-time monitoring systems such as Prometheus / Zabbix. This not only meets the stringent requirements for data compliance in the field of computing power security, but also ensures that the multimodal RAG knowledge base can access the latest computing power status data in a timely manner, avoiding the omission of security threats due to data lag.

[0055] Based on the same inventive concept, this application also provides a multimodal computing power knowledge base construction device for implementing the multimodal computing power knowledge base construction method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the multimodal computing power knowledge base construction device provided below can be found in the limitations of the multimodal computing power knowledge base construction method described above, and will not be repeated here.

[0056] In one exemplary embodiment, such as Figure 3 As shown, a multimodal computing power knowledge base construction device is provided. The multimodal computing power knowledge base construction device 300 includes a data acquisition and annotation module 301, a preprocessing module 302, a data parsing module 303, a text segmentation module 304, and a knowledge base generation module 305, wherein: The data collection and annotation module 301 is used to collect multi-source data in computing power security scenarios and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data. The preprocessing module 302 is used to perform differentiated preprocessing on the multimodal computing power security data to obtain preprocessed multimodal computing power security data; The data parsing module 303 is used to perform multimodal parsing on the preprocessed multimodal computing power security data to obtain the multimodal text content and time-series feature vector corresponding to the multimodal computing power security data; The text segmentation module 304 is used to perform text block segmentation processing on the multimodal text content, and to calculate the semantic vector of the text content after text block segmentation processing to obtain a new semantic vector containing the context. The knowledge base generation module 305 is used to generate a multimodal RAG knowledge base based on temporal feature vectors and semantic vectors containing context.

[0057] As an optional implementation, the above-mentioned data collection and annotation module 301 is specifically used to annotate multi-source data based on a weakly supervised learning model and manual verification to obtain multimodal computing power security data.

[0058] As an optional implementation, the aforementioned multimodal computing power security data includes labeled text data, labeled image or video data, labeled audio data, and labeled structured data; the labeled text data includes computing power security logs, security policy documents, threat intelligence texts, and security alarm information; the labeled image or video data includes monitoring screens of the physical environment of computing power nodes, network traffic visualization charts, and screenshots of security device interfaces; the labeled audio data includes hardware fault alarm sounds and security system voice prompts; the labeled structured data includes security device indicators and network traffic data packet characteristics; the aforementioned preprocessing module 302 is specifically used to perform differentiated preprocessing on the labeled text data, labeled image or video data, labeled audio data, and labeled structured data respectively to obtain preprocessed text data, preprocessed image / video data, preprocessed audio data, and preprocessed structured data.

[0059] As an optional implementation, the data parsing module 303 is specifically used to: use the Apache Tika tool to parse the preprocessed text data to obtain document text content in a preset format; use the FFmpeg tool to separate the audio data and frame image data in the preprocessed image / video data; use the Tesseract OCR engine or a multimodal large model to parse the image data and the separated frame image data of the preprocessed image / video data to obtain image text content; use the Whisper ASR model to parse the preprocessed audio data into audio text content; and use the LSTM time series model to parse the preprocessed structured data to obtain time-series feature vectors.

[0060] As an optional implementation, the text segmentation module 304 is specifically used to: segment the multimodal text content into M text blocks based on paragraph separators and line breaks according to the maximum number of characters in a preset text block, where M is a positive integer and M≥10; calculate the initial semantic vector of each text block in a preset dimension using an Embedding model, and cluster the M initial semantic vectors into N clusters using a K-Means clustering algorithm, where N is a positive integer and 1<N<M; traverse the N clusters and merge all text blocks within each cluster to generate merged text; generate a token vector of a preset dimension for each token in the merged text using an Embedding model; and aggregate the token vectors corresponding to each text block within a cluster using an attention weighted summation method to generate a new semantic vector containing contextual association for each text block, with M new semantic vectors.

[0061] As an optional implementation, the knowledge base generation module 305 is specifically used to: import M new semantic vectors and temporal feature vectors into the Milvus vector database; establish a vector index, which adopts the IVF_FLAT index type; store the multimodal original data association identifiers corresponding to the new semantic vectors and temporal feature vectors; generate a reverse mapping between the new semantic vectors and temporal feature vectors and the corresponding original multimodal data; and generate a multimodal RAG knowledge base based on the vector index, the reverse mapping between the new semantic vectors and temporal feature vectors and the corresponding original multimodal data, and the new semantic vectors and temporal feature vectors.

[0062] This implementation method involves collecting multi-source data from computing power security scenarios and performing multimodal association annotation on the multi-source data to obtain multimodal computing power security data. Differential preprocessing of the multimodal computing power security data yields preprocessed multimodal computing power security data. Multimodal parsing of the preprocessed multimodal computing power security data yields corresponding multimodal text content and temporal feature vectors. Text content is segmented into blocks, and semantic vectors are calculated from the segmented text content to obtain new semantic vectors containing context. A multimodal RAG knowledge base is generated using the temporal feature vectors and the context-containing semantic vectors. This multimodal RAG knowledge base addresses the technical limitations of existing single-modal knowledge bases, which can only process text data and cannot integrate key non-text data such as hardware fault images and alarm audio. It achieves systematic integration of multimodal data in computing power security scenarios, providing multi-dimensional data support for security event analysis.

[0063] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data for constructing a multimodal computing knowledge base. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a method for constructing a multimodal computing knowledge base.

[0064] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0066] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0067] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0068] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0069] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0070] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0071] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing a multimodal computing power knowledge base, characterized in that, The method for constructing the multimodal computing power knowledge base includes: Collect multi-source data under computing power security scenarios, and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data; Differential preprocessing is performed on the multimodal computing power security data to obtain preprocessed multimodal computing power security data; Multimodal parsing is performed on the preprocessed multimodal computing power security data to obtain the multimodal text content and temporal feature vector corresponding to the multimodal computing power security data; The multimodal text content is segmented into text blocks, and semantic vectors are calculated on the segmented text content to obtain new semantic vectors containing context. A multimodal RAG knowledge base is generated based on the temporal feature vector and the semantic vector containing the context.

2. The method for constructing a multimodal computing power knowledge base according to claim 1, characterized in that, The step of performing multimodal association annotation on the multi-source data to obtain multimodal computing power security data includes: The multi-source data is labeled based on a weakly supervised learning model and manual verification to obtain the multimodal computing power security data.

3. The method for constructing a multimodal computing power knowledge base according to claim 1, characterized in that, The multimodal computing power security data includes labeled text data, labeled image or video data, labeled audio data, and labeled structured data; the labeled text data includes computing power security logs, security policy documents, threat intelligence text, and security alarm information; the labeled image or video data includes monitoring screens of the physical environment of computing power nodes, network traffic visualization charts, and screenshots of security device interfaces; the labeled audio data includes hardware fault alarm sounds and security system voice prompts. The labeled structured data includes security device indicators and network traffic data packet characteristics; The differential preprocessing of the multimodal computing power security data to obtain preprocessed multimodal computing power security data includes: Differential preprocessing is performed on the labeled text data, labeled image or video data, labeled audio data, and labeled structured data to obtain preprocessed text data, preprocessed image / video data, preprocessed audio data, and preprocessed structured data.

4. The method for constructing a multimodal computing power knowledge base according to claim 3, characterized in that, The step of performing multimodal parsing on the preprocessed multimodal computing power security data to obtain the multimodal text content and temporal feature vector corresponding to the multimodal computing power security data includes: The preprocessed text data is parsed using the Apache Tika tool to obtain document text content in a preset format; The FFmpeg tool is used to separate the audio data and frame image data in the preprocessed image / video data. The Tesseract OCR engine or a multimodal large model is used to parse the image data and the separated frame image data of the preprocessed image / video data to obtain the image text content. The preprocessed audio data is parsed into audio text content using the Whisper ASR model; The preprocessed structured data is parsed using an LSTM time series model to obtain the time series feature vector.

5. The method for constructing a multimodal computing power knowledge base according to claim 1, characterized in that, The process of segmenting the multimodal text content into text blocks and calculating semantic vectors on the segmented text content to obtain new semantic vectors containing context includes: Based on the maximum number of characters in a preset text block, the multimodal text content is divided into M text blocks according to paragraph separators and line breaks, where M is a positive integer and M≥10; The Embedding model is used to calculate the initial semantic vector of each text block in a preset dimension, and the K-Means clustering algorithm is used to cluster the M initial semantic vectors into N clusters, where N is a positive integer and 1 < N < M; Traverse N clusters and merge all text blocks within each cluster to generate merged text; An embedding model is used to generate a token vector of a preset dimension for each token in the merged text; The token vectors corresponding to each text block within the cluster are aggregated using an attention weighted summation method to generate a new semantic vector containing contextual associations for each text block. There are M such new semantic vectors.

6. The method for constructing a multimodal computing power knowledge base according to claim 5, characterized in that, The generation of a multimodal RAG knowledge base based on the temporal feature vector and the semantic vector containing context includes: Import the M new semantic vectors and the temporal feature vectors into the Milvus vector database to create a vector index, which adopts the IVF_FLAT index type. Store the multimodal original data association identifiers corresponding to the new semantic vector and the temporal feature vector, and generate the reverse mapping between the new semantic vector and the temporal feature vector and the corresponding original multimodal data; The multimodal RAG knowledge base is obtained based on the vector index, the new semantic vector, the inverse mapping of the temporal feature vector to the corresponding original multimodal data, the new semantic vector, and the temporal feature vector.

7. A multimodal computing power knowledge base construction device, characterized in that, The multimodal computing power knowledge base construction device includes: The data collection and annotation module is used to collect multi-source data in computing power security scenarios and perform multi-modal association annotation on the multi-source data to obtain multi-modal computing power security data. The preprocessing module is used to perform differentiated preprocessing on the multimodal computing power security data to obtain preprocessed multimodal computing power security data. The data parsing module is used to perform multimodal parsing on the preprocessed multimodal computing power security data to obtain multimodal text content and temporal feature vectors corresponding to the multimodal computing power security data; The text segmentation module is used to perform text block segmentation processing on the multimodal text content, and to calculate the semantic vector of the text content after text block segmentation processing to obtain a new semantic vector containing the context. The knowledge base generation module is used to generate a multimodal RAG knowledge base based on the temporal feature vector and the semantic vector containing the context.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the multimodal computing power knowledge base construction method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal computing power knowledge base construction method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal computing power knowledge base construction method as described in any one of claims 1-6.

Images