A method and system for retrieving and processing port equipment inspection information based on big data

Abstract

This invention discloses a method and system for retrieving and processing port equipment inspection information based on big data. The method involves collecting standard multimodal data; inputting the standard multimodal data into a pre-trained multimodal feature fusion model to generate a joint deep feature vector; inputting the joint deep feature vector into a cross-modal deep hash coding network to generate a corresponding binary hash code; storing the binary hash code in a retrieval database; generating a corresponding retrieval hash code in response to a retrieval request; calculating the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database; retrieving historical inspection events based on the Hamming distance; constructing a heterogeneous inspection knowledge graph based on the historical inspection events; and using a GNN (Graph Neural Network) for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request. This reduces reliance on human experience, lowers the misjudgment rate, and improves the overall intelligence level of port equipment inspection.

Description

Technical Field

[0001] This invention relates to the field of port intelligent operation and maintenance technology, and in particular to a method and system for retrieving and processing port equipment inspection information based on big data. Background Technology

[0002] As global logistics hubs, ports rely heavily on the stable operation of core equipment such as cranes and conveyors, which directly determines their operational efficiency and safety. Inspection work is a crucial link in ensuring the reliable operation of this equipment. Currently, port equipment inspections largely depend on manual checks and single-modal data monitoring, which presents several challenges: multi-source inspection data exhibits heterogeneity, lacks standardized preprocessing procedures, suffers from high data noise and inconsistent formats, hindering multi-dimensional collaborative analysis; traditional feature extraction methods cannot effectively uncover deep correlations between different modalities, resulting in insufficient feature representation accuracy; inspection data storage is costly and retrieval efficiency is low, making it difficult to quickly match similar historical inspection events, leading to delays in anomaly detection; anomaly tracing relies heavily on manual experience, lacking systematic knowledge association and reasoning mechanisms, making it difficult to accurately pinpoint the root cause of faults, easily leading to misjudgments and processing delays. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing a method and system for retrieving and processing port equipment inspection information based on big data.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following steps in the above-mentioned port equipment inspection information retrieval and processing method based on big data:

[0005] Collect multimodal inspection data of port equipment, and preprocess the multimodal inspection data to obtain standard multimodal data;

[0006] The standard multimodal data is input into a pre-trained multimodal feature fusion model. Image features are extracted using an improved visual Transformer, audio features are extracted using Mel spectrograms and CNN convolutional neural networks, and feature fusion is performed based on cross-modal attention to generate a joint deep feature vector.

[0007] The joint deep feature vector is input into a cross-modal deep hash coding network to generate a corresponding binary hash code, and the binary hash code is stored in a retrieval database.

[0008] In response to a retrieval request, a corresponding retrieval hash code is generated, the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database is calculated, and historical inspection events are retrieved based on the Hamming distance.

[0009] Based on historical inspection events, a heterogeneous knowledge graph of inspection is constructed, and a GNN graph neural network is used for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

[0010] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of collecting multimodal inspection data of port equipment and preprocessing the multimodal inspection data to obtain standard multimodal data includes:

[0011] Collect multimodal inspection data of port equipment, including visual images, audio signals, vibration signals and equipment operating status parameters;

[0012] Gaussian filtering is used to remove noise from the visual image, histogram equalization is used to adjust the image contrast, and then the image is sized and normalized. The YOLOv8 target detection algorithm is used to crop the image of the device part to obtain standard image data.

[0013] The audio signal is pre-emphasized, and the audio signal is framed using STFT short-time Fourier transform. Background noise in the port environment is eliminated by Wiener filtering to obtain a standard audio signal.

[0014] Wavelet transform is used to denoise the vibration signal, remove high-frequency interference signals and baseline drift, and the 3σ criterion is used to identify and remove outliers in the vibration signal to obtain the standard vibration signal.

[0015] Outliers in the equipment's operating status parameters are identified using box plots. These outliers are then corrected by combining the equipment's historical normal parameter ranges. Linear regression is used to fill in the deleted data, resulting in standard operating parameter data.

[0016] Standard image data, standard audio signals, standard vibration signals, and standard operating parameter data are correlated and aligned according to acquisition time and device number to obtain standard multimodal data.

[0017] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of inputting the standard multimodal data into a pre-trained multimodal feature fusion model, extracting image features using an improved visual Transformer, extracting audio features through Mel spectrograms and CNN convolutional neural networks, and performing feature fusion based on cross-modal attention to generate a joint deep feature vector includes:

[0018] In the Transformer model, positional encoding optimization is added to the PatchEmbedding layer of ViT to reduce the number of layers in the Transformer encoder. Residual connections and LayerNorm normalization are added to the encoder, and image features are extracted through global average pooling.

[0019] The standard audio signal is converted into a Mel spectrogram, and the one-dimensional audio signal is converted into a two-dimensional spectral feature map. A CNN convolutional neural network is used as the audio feature extraction branch. The network structure includes 4 convolutional layers and 2 max pooling layers to extract texture features and frequency features from the spectrogram to obtain audio features.

[0020] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of inputting the standard multimodal data into a pre-trained multimodal feature fusion model, extracting image features using an improved visual Transformer, extracting audio features through Mel spectrograms and CNN convolutional neural networks, and performing feature fusion based on cross-modal attention to generate a joint deep feature vector includes:

[0021] A cross-modal attention matrix is ​​constructed, with image features as queries and audio features as keys and values. The attention weights of each modality feature are calculated using the attention formula, and the features are weighted and summed to generate a joint deep feature vector.

[0022] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of inputting the joint depth feature vector into a cross-modal depth hash coding network to generate a corresponding binary hash code, and storing the binary hash code in the retrieval database, includes:

[0023] A cross-modal deep hash coding network is established. The network structure includes three fully connected layers, the activation function is tanh, and the output value is finally converted into a binary hash code through the sign function.

[0024] The joint deep feature vectors are batch-input into the trained cross-modal deep hash coding network to generate corresponding binary hash codes with a hash code length of 64 bits.

[0025] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of responding to a retrieval request, generating a corresponding retrieval hash code, calculating the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database, and retrieving historical inspection events based on the Hamming distance includes:

[0026] In response to a retrieval request, the query request is parsed, and query information is extracted, including the query device number, device location, query modality type, and anomaly description. The query information is then standardized to obtain standard query data.

[0027] Standard query data is input into a pre-trained multimodal feature fusion model to generate a query joint deep feature vector. The query joint deep feature vector is then input into a trained cross-modal deep hash coding network to generate the corresponding query hash code.

[0028] Filter out all binary hash codes within the corresponding partition, and use the Hamming distance calculation formula to calculate the Hamming distance between the query hash code and the filtered historical binary hash codes.

[0029] Furthermore, in the aforementioned method for retrieving and processing port equipment inspection information based on big data, the step of constructing a heterogeneous inspection knowledge graph based on historical inspection events and using a GNN graph neural network for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request includes:

[0030] The node features of the heterogeneous knowledge graph of the inspection are input into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128. The activation function is LeakyReLU and the dropout probability is 0.2.

[0031] Furthermore, the port equipment inspection information retrieval and processing system based on big data includes the following modules:

[0032] The port data acquisition module is used to collect multimodal inspection data of port equipment and preprocess the multimodal inspection data to obtain standard multimodal data.

[0033] The feature vector extraction module is used to input the standard multimodal data into a pre-trained multimodal feature fusion model, extract image features using an improved visual Transformer, extract audio features using Mel spectrograms and CNN convolutional neural networks, perform feature fusion based on cross-modal attention, and generate a joint deep feature vector.

[0034] The hash encoding storage module is used to input the joint deep feature vector into the cross-modal deep hash encoding network, generate the corresponding binary hash code, and store the binary hash code into the retrieval database;

[0035] The retrieval calculation and matching module is used to respond to a retrieval request, generate a corresponding retrieval hash code, calculate the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database, and retrieve historical inspection events based on the Hamming distance;

[0036] The causal tracing module is used to construct a heterogeneous graph of inspection knowledge based on historical inspection events, and to use a GNN graph neural network for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

[0037] Furthermore, in the port equipment inspection information retrieval and processing system based on big data, the retrieval calculation and matching module includes the following sub-modules:

[0038] The parsing submodule is used to respond to retrieval requests, parse the query requests, extract query information, including the query device number, device location, query modality type and anomaly description, and standardize the query information to obtain standard query data;

[0039] The input submodule is used to input standard query data into a pre-trained multimodal feature fusion model to generate a query joint deep feature vector, and input the query joint deep feature vector into a trained cross-modal deep hash coding network to generate the corresponding query hash code.

[0040] The calculation submodule is used to filter out all binary hash codes within the corresponding partition. It uses the Hamming distance calculation formula to calculate the Hamming distance between the query hash code and the filtered historical binary hash codes.

[0041] Furthermore, in the port equipment inspection information retrieval and processing system based on big data, the retrieval causal tracing module includes the following sub-modules:

[0042] The source tracing submodule is used to input the node features of the heterogeneous knowledge graph of the inspection into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128, and the activation function is LeakyReLU with a dropout probability of 0.2.

[0043] Its beneficial effects are as follows: By targeting denoising, normalization, and correlation alignment, it eliminates data heterogeneity and noise interference, improves data quality, and lays a solid foundation for subsequent analysis; by adopting an improved multimodal feature fusion model, it accurately extracts the core features of each modality and achieves efficient fusion, and the generated joint deep feature vector can comprehensively represent the equipment status, improving the accuracy of feature representation; by using cross-modal deep hash coding, it converts high-dimensional features into compact binary hash codes, reducing storage costs, and by combining Hamming distance calculation, it enables rapid retrieval of similar inspection events, significantly improving inspection efficiency; based on the inspection knowledge heterogeneous graph and GNN inference, it achieves accurate causal tracing of anomalies, outputs clear tracing paths and processing suggestions, reduces reliance on human experience, and lowers the misjudgment rate; overall, it improves the intelligence level of port equipment inspection, reduces equipment downtime due to failure, lowers inspection costs, and ensures the continuity and safety of port operations. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0045] Figure 1This is a schematic diagram of the first embodiment of the port equipment inspection information retrieval and processing method based on big data in this invention.

[0046] Figure 2 This is a schematic diagram of the second embodiment of the port equipment inspection information retrieval and processing method based on big data in this invention.

[0047] Figure 3 This is a schematic diagram of the first embodiment of the port equipment inspection information retrieval and processing system based on big data in this invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms "one," "an," and "this" used herein may also include the plural forms. It should be further understood that the terminology used in this specification includes the presence of features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0050] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, the port equipment inspection information retrieval and processing method based on big data includes the following steps:

[0051] Step 101: Collect multimodal inspection data of port equipment, preprocess the multimodal inspection data to obtain standard multimodal data;

[0052] Specifically, this embodiment collects multimodal inspection data of port equipment, including visual images, audio signals, vibration signals, and equipment operating status parameters. Gaussian filtering is used to remove noise from the visual images, histogram equalization is used to adjust image contrast, and then the images are sized and normalized. The equipment parts are cropped using the YOLOv8 target detection algorithm to obtain standard image data. The audio signal is pre-emphasized, and the audio signal is framed using STFT short-time Fourier transform. Wiener filtering is used to eliminate background noise in the port environment to obtain a standard audio signal. Wavelet transform is used to denoise the vibration signal, removing high-frequency interference signals and baseline drift. The 3σ criterion is used to identify and remove outliers in the vibration signal to obtain a standard vibration signal. Box plotting is used to identify outliers in the equipment operating status parameters. Combined with the historical normal parameter range of the equipment, outliers are corrected, and linear regression is used to complete the deleted data to obtain standard operating parameter data. The standard image data, standard audio signal, standard vibration signal, and standard operating parameter data are associated and aligned according to the acquisition time and equipment number to obtain standard multimodal data.

[0053] High-definition industrial cameras are used and deployed at key and vulnerable parts of the equipment. Static images are captured at a frequency of 1 frame every 30 minutes, while dynamic videos of the equipment in operation are recorded at the same time. The video covers three scenarios: normal operation, minor abnormalities, and obvious malfunctions. During the acquisition process, environmental interference such as port dust, strong light, and rain is avoided to ensure image clarity.

[0054] High-sensitivity audio sensors are used and fixed to the drive, rotating and sealing parts of the equipment to collect audio signals in real time during equipment operation. The sensors focus on collecting signals such as equipment vibration noise, friction noise and abnormal noise, and simultaneously record the collection time, equipment location and ambient temperature.

[0055] Piezoelectric vibration sensors are used and attached to core stress-bearing parts such as bearing housings, motor housings, and drive shafts. Vibration acceleration, velocity, and displacement data are collected at a sampling frequency of 100Hz, with each sampling period lasting no less than 10 seconds to avoid data distortion caused by sensor loosening.

[0056] Through the interface of the port equipment PLC control system and SCADA monitoring system, the equipment operating parameters, including voltage, current, power, speed, oil temperature, pressure, etc., are collected in real time at a frequency of once per minute. The equipment number, operating conditions, no-load / load, operator and other basic information are synchronously associated.

[0057] First, Gaussian filtering with a 5×5 convolution kernel and a standard deviation of 0.5 is used to remove image noise. Then, histogram equalization is used to adjust the image contrast and solve the image blurring problem in strong / low light environments at the port. Next, the image is sized uniformly to 224×224 pixels and normalized, with pixel values ​​normalized to the [0,1] range. Bilinear interpolation is used to supplement the missing pixels after image scaling. Finally, the target detection algorithm YOLOv8 is used to crop the key parts of the equipment image and remove redundant background information to obtain standardized image data.

[0058] First, the original audio signal is pre-emphasized with a pre-emphasis coefficient of 0.97 to improve the clarity of high-frequency signals. Then, a short-time Fourier transform (STFT) is used to divide the audio signal into frames with a frame length of 25ms and a frame shift of 10ms to remove inter-frame redundancy. Background noise in the port environment, such as ocean waves and vehicle noise, is eliminated by Wiener filtering. Missing audio segments are filled in using linear interpolation. Finally, the audio signal is normalized to the [-1,1] interval to obtain a standardized audio signal.

[0059] Wavelet transform was used with db4 as the wavelet basis and 5 decomposition layers to denoise the vibration signal, removing high-frequency interference signals and baseline drift. The collected vibration acceleration and velocity data were subjected to integral / differential transformation to unify the data type. The 3σ criterion was used to identify and remove outliers in the vibration signal, and missing data was filled using the K-nearest neighbor interpolation method with K=5. Finally, the vibration signal was normalized to a unified data dimension of 1×1000 to obtain a standardized vibration signal.

[0060] Outliers in the parameters are identified using box plots. These outliers are then corrected or removed by combining them with the historical normal parameter ranges of the equipment. For missing parameter values, linear regression is used to fill in the missing values ​​based on historical data of the equipment under the same operating conditions. All operating parameters are uniformly converted into floating-point data and organized in the format of equipment number-acquisition time-parameter type-parameter value to obtain standardized operating parameter data.

[0061] The preprocessed visual images, audio signals, vibration signals, and operating status parameters are correlated and aligned according to the acquisition time and device number, and the output is standard multimodal data that can be directly input into the model.

[0062] Step 102: Input standard multimodal data into the pre-trained multimodal feature fusion model, extract image features using the improved visual Transformer, extract audio features using Mel spectrogram and CNN convolutional neural network, perform feature fusion based on cross-modal attention, and generate a joint deep feature vector.

[0063] Specifically, in this embodiment, positional encoding optimization is added to the PatchEmbedding layer of the ViT layer in the Transformer model to reduce the number of layers in the Transformer encoder. Residual connections and LayerNorm normalization are added to the encoder, and image features are extracted through global average pooling. The standard audio signal is converted into a Mel spectrogram, and the one-dimensional audio signal is converted into a two-dimensional spectral feature map. A CNN convolutional neural network is used as the audio feature extraction branch. The network structure includes four convolutional layers and two max pooling layers to extract texture and frequency features from the spectrogram to obtain audio features. A cross-modal attention matrix is ​​constructed, using image features as queries and audio features as keys and values. The attention weights of each modality feature are calculated using the attention formula, and the features are weighted and summed to generate a joint deep feature vector.

[0064] An improved visual Transformer, with ViT as the image feature extraction branch, is used to process standardized image data. Improvements include: adding positional encoding optimization to the PatchEmbedding layer of ViT to enhance the spatial correlation of features; reducing the number of layers in the Transformer encoder to lower computational complexity; and adding residual connections and LayerNorm normalization to the encoder to avoid gradient vanishing. Finally, global average pooling is used to extract image feature vectors with a dimension of 1×768.

[0065] First, the standardized audio signal is converted into a Mel spectrogram, transforming the one-dimensional audio signal into a two-dimensional spectral feature map with a size of 224×224. A CNN convolutional neural network is used as the audio feature extraction branch. The network structure includes four convolutional layers with 3×3 kernels and a stride of 1, one or two max pooling layers with 2×2 kernels and a stride of 2, and one fully connected layer. The convolutional layers use the ReLU activation function to extract texture and frequency features from the spectrogram. The fully connected layer outputs an audio feature vector with a dimension of 1×768.

[0066] Standardized vibration signals are processed using a 1D-CNN convolutional neural network to extract time-domain and frequency-domain features, outputting a vibration feature vector with a dimension of 1×256. Standardized operating parameters are then mapped through a fully connected layer, outputting an operating parameter feature vector with a dimension of 1×256. The vibration feature vector and the operating parameter feature vector are concatenated to obtain a fused feature vector with a dimension of 1×512, serving as the comprehensive feature of the non-visual / audio modality. The non-visual / audio fused feature vector is then mapped from 1×512 to 768 dimensions through a fully connected layer, maintaining consistency with the dimensions of the image and audio feature vectors, resulting in a single-modal feature vector with three dimensions of 1×768.

[0067] A cross-modal attention matrix is ​​constructed, with image features as the query (Q), audio features and non-visual / audio features as the keys (K and V). The attention weights of each modality feature are calculated using the attention formula. The weighted and fused feature vector is then input into a multilayer perceptron (MLP), a two-layer MLP with 1024 hidden neurons, for non-linear feature mapping. A Dropout layer with a dropout probability of 0.3 is added to avoid overfitting. Finally, L2 normalization is applied to output a joint deep feature vector with a dimension of 1×1024.

[0068] Step 103: Input the joint deep feature vector into the cross-modal deep hash coding network to generate the corresponding binary hash code, and store the binary hash code in the retrieval database;

[0069] Specifically, in this embodiment, a cross-modal deep hash coding network is established. The network structure includes three fully connected layers, the activation function is tanh, and the output value is converted into a binary hash code by the sign function. The joint deep feature vector is input in batches into the trained cross-modal deep hash coding network to generate the corresponding binary hash code. The hash code length is set to 64 bits.

[0070] A cross-modal deep hash coding network is constructed. This network takes a joint deep feature vector as input and outputs a fixed-length binary hash code, a 0 / 1 sequence. The network structure includes three fully connected layers, with tanh as the activation function. Finally, the output value is converted into a binary hash code through the sign function, where the output value is 1 if it is greater than 0, and 0 otherwise.

[0071] A contrastive loss function is used to train the hash coding network. During training, the joint feature vectors corresponding to multimodal data under the same device and the same working condition are used as positive sample pairs, and data from different devices or different working conditions are used as negative sample pairs. The network parameters are optimized by minimizing the hash code distance of positive sample pairs and maximizing the hash code distance of negative sample pairs. The training batch size is 64, the learning rate is 5e-6, and the number of iterations is 80 rounds until the network loss value converges. The convergence threshold is 1e-4.

[0072] All joint deep feature vectors are batch-input into the trained cross-modal deep hash coding network to generate corresponding binary hash codes with a length of 64 bits. An associated tag is added to each binary hash code, including information such as device number, data acquisition time, device location, inspection condition, and joint feature vector ID, forming a key-value pair of hash code and associated tag to facilitate information association during subsequent retrieval.

[0073] Redis was selected as the query database, and the database was partitioned according to the port equipment type. Each partition corresponds to the hash code data of a type of equipment, avoiding data confusion between different equipment and improving retrieval efficiency. A hash index was built for each equipment partition, with the index key being the equipment number + equipment location, and the index value being all binary hash codes and associated tags corresponding to that location, enabling fast retrieval by equipment and location. Hash code-associated tag key-value pairs were stored in batches into the corresponding partitions, using an asynchronous write method to avoid affecting data processing efficiency. At the same time, a data backup mechanism was established to synchronously back up the hash code data to the MySQL database to prevent data loss due to Redis database failure. A scheduled update task was set up to synchronously store newly generated binary hash codes into the database.

[0074] Step 104: In response to the retrieval request, generate the corresponding retrieval hash code, calculate the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database, and retrieve historical inspection events based on the Hamming distance;

[0075] Specifically, in this embodiment, in response to a retrieval request, the query request is parsed, and query information is extracted, including the query device number, device location, query modality type, and anomaly description. The query information is standardized to obtain standard query data. The standard query data is input into a pre-trained multimodal feature fusion model to generate a query joint deep feature vector. The query joint deep feature vector is input into a trained cross-modal deep hash coding network to generate the corresponding query hash code. All binary hash codes within the corresponding partition are selected, and the Hamming distance is calculated using the Hamming distance formula between the query hash code and the selected historical binary hash codes.

[0076] The system receives user query requests, which support multimodal input, including the following types: visual images of key parts of the equipment; abnormal audio during equipment operation; equipment vibration signal data; abnormal operating parameters of the equipment; and mixed-modal queries. The system parses the query requests, extracts core query information, including the equipment number, equipment location, query modality type, and anomaly description. It then performs synchronous preprocessing on the multimodal data input according to the preprocessing standards in step 1.2 to obtain standardized query data.

[0077] The preprocessed standardized query data is input into the pre-trained multimodal feature fusion model according to the feature extraction and fusion process in step 2 to generate a query joint deep feature vector. Then, the query joint deep feature vector is input into the trained cross-modal deep hash coding network to generate the corresponding query hash code, a 64-bit binary sequence. After generating the query hash code, its validity is verified. If it is an invalid hash code, the user is prompted to re-enter the query data, or the query data is re-preprocessed and re-encoded to ensure the validity of the query hash code.

[0078] From the retrieval database, based on the device number and device location parsed from the query request, all binary hash codes within the corresponding partition are filtered out; the Hamming distance calculation formula is used to calculate the Hamming distance between the query hash code and each filtered historical binary hash code in batches;

[0079] Hamming distance characterizes the similarity between two hash codes; the smaller the distance, the more similar the corresponding inspection data. Parallel computation is used during the calculation to improve efficiency and ensure a query response time of no more than one second. A Hamming distance threshold is set, and historical binary hash codes with a Hamming distance ≤ the threshold are selected. The corresponding historical inspection events are considered similar to the current query request. The selected historical inspection events are sorted in ascending order of Hamming distance, with the smallest distance indicating the most similar event. Simultaneously, the associated tag information for each historical inspection event is extracted, including collection time, anomaly description, handling plan, and inspection personnel, and integrated into a search result list. This list contains core information such as similarity, Hamming distance, inspection time, equipment status, anomaly description, and handling measures.

[0080] Step 105: Construct a heterogeneous knowledge graph of inspection based on historical inspection events, and use a GNN graph neural network for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

[0081] Specifically, in this embodiment, the node features of the inspection knowledge heterogeneous graph are input into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128, and the activation function is LeakyReLU with a dropout probability of 0.2.

[0082] Using retrieved historical inspection events as core data, and combining information such as the port equipment inspection knowledge base, equipment manuals, and maintenance records, a heterogeneous inspection knowledge graph is constructed. The nodes and edges of the heterogeneous graph are defined as follows, ensuring that nodes cover all elements and edges represent relationships:

[0083] Node Definitions: Equipment Node: Represents specific port equipment, with attributes including equipment number, equipment type, installation time, rated parameters, maintenance history, etc.; Anomaly Type Node: Represents the specific type of equipment anomaly, with attributes including anomaly name, anomaly level, anomaly manifestation, etc.; Inspection Event Node: Represents a single inspection record, with attributes including inspection time, inspection personnel, equipment location, inspection conditions, hash code associated ID, etc.; Cause Node: Represents the root cause of the equipment anomaly, with attributes including cause name, cause type, scope of impact, etc.; Handling Measures Node: Represents the handling method for the anomaly, with attributes including handling steps, required tools, handling cycle, handling effect, etc.

[0084] Edge definitions: Equipment-Anomaly Type edge, representing a type of anomaly that has occurred in a certain piece of equipment, with edge attributes being the number of occurrences and the time of first occurrence; Anomaly Type-Inspection Event edge, representing a type of anomaly discovered in an inspection event, with edge attribute being the anomaly confirmation method; Inspection Event-Cause edge, representing the root cause of an anomaly in an inspection event, with edge attribute being the basis for cause confirmation; Anomaly Type-Cause edge, representing common root causes of a certain type of anomaly, with edge attribute being the probability of cause occurrence; Cause-Handling Measures edge, representing the handling method corresponding to a certain type of anomaly cause, with edge attribute being the effectiveness rate of the handling measure; Equipment-Inspection Event edge, representing a specific inspection event corresponding to a certain piece of equipment, with edge attribute being the inspection method.

[0085] After the heterogeneous graph is constructed, a graph database, Neo4j, is used to store the heterogeneous graph data and establish a node index to facilitate the GNN model to quickly read the graph data. At the same time, the heterogeneous graph is updated regularly to add new nodes and edges, thereby improving the accuracy of inference.

[0086] A graph attention network (GAT) is selected as the GNN inference model. Based on the constructed heterogeneous inspection knowledge graph, anomaly causal tracing inference is performed. The node features of the heterogeneous inspection knowledge graph are input into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128. The activation function is LeakyReLU, and the dropout probability is 0.2 to avoid overfitting.

[0087] The GAT model's attention mechanism calculates the attention weights between each node and its neighboring nodes. Based on these weights, the features of neighboring nodes are aggregated, and the feature vector of the current node is updated. After feature aggregation through two attention layers, the final feature vector of each node is obtained, which represents the node's association and importance. Taking the anomaly type node corresponding to the current query request as the starting point for inference, and based on the updated node feature vector, the GAT model's inference layer mines all causal nodes corresponding to this anomaly type node, calculates the association probability of each causal node (i.e., the probability that the cause leads to the current anomaly), and simultaneously mines the corresponding handling measure nodes for each causal node, forming a preliminary causal chain of anomaly type-cause-handling measure.

[0088] By combining the retrieved historical inspection event nodes, the preliminary causal chain is verified, and causal relationships with high frequency and high probability (≥80%) are selected, while invalid causal chains with low probability are removed. Simultaneously, the association probability of the cause nodes is adjusted based on the current operating parameters of the equipment to ensure the accuracy of the inference results. The optimized causal relationships are then organized into a complete anomaly causal tracing path, with the path output following the logic of querying anomalies – potential causes – cause verification – handling suggestions.

[0089] Its beneficial effects are as follows: By targeting denoising, normalization, and correlation alignment, it eliminates data heterogeneity and noise interference, improves data quality, and lays a solid foundation for subsequent analysis; by adopting an improved multimodal feature fusion model, it accurately extracts the core features of each modality and achieves efficient fusion, and the generated joint deep feature vector can comprehensively represent the equipment status, improving the accuracy of feature representation; by using cross-modal deep hash coding, it converts high-dimensional features into compact binary hash codes, reducing storage costs, and by combining Hamming distance calculation, it enables rapid retrieval of similar inspection events, significantly improving inspection efficiency; based on the inspection knowledge heterogeneous graph and GNN inference, it achieves accurate causal tracing of anomalies, outputs clear tracing paths and processing suggestions, reduces reliance on human experience, and lowers the misjudgment rate; overall, it improves the intelligence level of port equipment inspection, reduces equipment downtime due to failure, lowers inspection costs, and ensures the continuity and safety of port operations.

[0090] Please see Figure 2 In the big data-based port equipment inspection information retrieval and processing method, standard multimodal data is input into a pre-trained multimodal feature fusion model. An improved visual Transformer is used to extract image features, and Mel-spectrum mapping and CNN convolutional neural networks are used to extract audio features. Feature fusion is then performed based on cross-modal attention to generate a joint deep feature vector. The process includes the following steps:

[0091] Step 201: In the Transformer model, add positional encoding optimization to the PatchEmbedding layer of ViT to reduce the number of layers in the Transformer encoder. Add residual connections and LayerNorm normalization to the encoder and extract image features through global average pooling.

[0092] Step 202: Convert the standard audio signal into a Mel spectrogram, and convert the one-dimensional audio signal into a two-dimensional spectral feature map; use a CNN convolutional neural network as the audio feature extraction branch. The network structure includes 4 convolutional layers and 2 max pooling layers to extract texture features and frequency features from the spectrogram to obtain audio features.

[0093] The above describes embodiments of the port equipment inspection information retrieval and processing method based on big data of the present invention. Please refer to [link / reference]. Figure 3 The port equipment inspection information retrieval and processing system based on big data includes the following modules:

[0094] The port data acquisition module is used to collect multimodal inspection data of port equipment and preprocess the multimodal inspection data to obtain standard multimodal data.

[0095] The feature vector extraction module is used to input standard multimodal data into a pre-trained multimodal feature fusion model, extract image features using an improved visual Transformer, extract audio features using Mel spectrograms and CNN convolutional neural networks, perform feature fusion based on cross-modal attention, and generate a joint deep feature vector.

[0096] The hash encoding storage module is used to input the joint deep feature vector into the cross-modal deep hash encoding network, generate the corresponding binary hash code, and store the binary hash code into the retrieval database;

[0097] The retrieval calculation and matching module is used to respond to retrieval requests, generate corresponding retrieval hash codes, calculate the Hamming distance between the retrieval hash codes and the binary hash codes in the retrieval database, and retrieve historical inspection events based on the Hamming distance.

[0098] The causal tracing module is used to construct a heterogeneous graph of inspection knowledge based on historical inspection events, and to use a GNN graph neural network for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

[0099] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended technical solutions and equivalents.

Claims

1. A method for retrieving and processing port equipment inspection information based on big data, characterized in that, The port equipment inspection information retrieval and processing method includes the following steps: Collect multimodal inspection data of port equipment, including visual images, audio signals, vibration signals and equipment operating status parameters, and preprocess the multimodal inspection data to obtain standard multimodal data; In the Transformer model, positional encoding optimization is added to the PatchEmbedding layer of ViT to reduce the number of layers in the Transformer encoder. Residual connections and LayerNorm normalization are added to the encoder, and image features are extracted through global average pooling. The standard audio signal is converted into a Mel spectrogram, and the one-dimensional audio signal is converted into a two-dimensional spectral feature map. A CNN convolutional neural network is used as the audio feature extraction branch. The network structure includes 4 convolutional layers and 2 max pooling layers to extract texture and frequency features from the spectrogram to obtain audio features. A cross-modal attention matrix is ​​constructed, with image features as queries and audio features as keys and values. The attention weights of each modality feature are calculated using the attention formula, and the features are weighted and summed to generate a joint deep feature vector. The joint deep feature vector is input into a cross-modal deep hash coding network to generate a corresponding binary hash code, and the binary hash code is stored in a retrieval database. In response to a retrieval request, a corresponding retrieval hash code is generated, the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database is calculated, and historical inspection events are retrieved based on the Hamming distance. Based on historical inspection events, a heterogeneous knowledge graph of inspection is constructed, and a GNN graph neural network is used for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

2. The port equipment inspection information retrieval and processing method based on big data as described in claim 1, characterized in that, The multimodal inspection data of the port equipment is collected, and the multimodal inspection data is preprocessed to obtain standard multimodal data, including: Gaussian filtering is used to remove noise from the visual image, histogram equalization is used to adjust the image contrast, and then the image is sized and normalized. The YOLOv8 target detection algorithm is used to crop the image of the device part to obtain standard image data. The audio signal is pre-emphasized, and the audio signal is framed using STFT short-time Fourier transform. Background noise in the port environment is eliminated by Wiener filtering to obtain a standard audio signal. Wavelet transform is used to denoise the vibration signal, remove high-frequency interference signals and baseline drift, and the 3σ criterion is used to identify and remove outliers in the vibration signal to obtain a standard vibration signal. Outliers in the equipment's operating status parameters are identified using box plots. These outliers are then corrected by combining the equipment's historical normal parameter ranges. Linear regression is used to fill in the deleted data, resulting in standard operating parameter data. Standard image data, standard audio signals, standard vibration signals, and standard operating parameter data are correlated and aligned according to acquisition time and device number to obtain standard multimodal data.

3. The port equipment inspection information retrieval and processing method based on big data as described in claim 1, characterized in that, The step of inputting the joint deep feature vector into a cross-modal deep hash coding network to generate a corresponding binary hash code, and storing the binary hash code in a retrieval database, includes: A cross-modal deep hash coding network is established. The network structure includes three fully connected layers, the activation function is tanh, and the output value is finally converted into a binary hash code through the sign function. The joint deep feature vectors are batch-input into the trained cross-modal deep hash coding network to generate corresponding binary hash codes with a hash code length of 64 bits.

4. The port equipment inspection information retrieval and processing method based on big data as described in claim 1, characterized in that, In response to a retrieval request, a corresponding retrieval hash code is generated, the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database is calculated, and historical inspection events are retrieved based on the Hamming distance, including: In response to a retrieval request, the query request is parsed, and query information is extracted, including the query device number, device location, query modality type, and anomaly description. The query information is then standardized to obtain standard query data. Standard query data is input into a pre-trained multimodal feature fusion model to generate a query joint deep feature vector. The query joint deep feature vector is then input into a trained cross-modal deep hash coding network to generate the corresponding query hash code. Filter out all binary hash codes within the corresponding partition, and use the Hamming distance calculation formula to calculate the Hamming distance between the query hash code and the filtered historical binary hash codes.

5. The port equipment inspection information retrieval and processing method based on big data as described in claim 1, characterized in that, The process of constructing a heterogeneous knowledge graph based on historical inspection events and using a GNN (Graph Neural Network) for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request includes: The node features of the heterogeneous knowledge graph of the inspection are input into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128. The activation function is LeakyReLU and the dropout probability is 0.

2.

6. A system for implementing the port equipment inspection information retrieval and processing method based on big data as described in claim 1, characterized in that, The system includes the following modules: The port data acquisition module is used to collect multimodal inspection data of port equipment and preprocess the multimodal inspection data to obtain standard multimodal data. The feature vector extraction module is used to input the standard multimodal data into a pre-trained multimodal feature fusion model, extract image features using an improved visual Transformer, extract audio features using Mel spectrograms and CNN convolutional neural networks, perform feature fusion based on cross-modal attention, and generate a joint deep feature vector. The hash encoding storage module is used to input the joint deep feature vector into the cross-modal deep hash encoding network, generate the corresponding binary hash code, and store the binary hash code into the retrieval database; The retrieval calculation and matching module is used to respond to a retrieval request, generate a corresponding retrieval hash code, calculate the Hamming distance between the retrieval hash code and the binary hash code in the retrieval database, and retrieve historical inspection events based on the Hamming distance; The causal tracing module is used to construct a heterogeneous graph of inspection knowledge based on historical inspection events, and to use a GNN graph neural network for reasoning to output the causal tracing path of the anomaly corresponding to the retrieval request.

7. The system as described in claim 6, characterized in that, The retrieval calculation and matching module includes the following sub-modules: The parsing submodule is used to respond to retrieval requests, parse the query requests, extract query information, including the query device number, device location, query modality type and anomaly description, and standardize the query information to obtain standard query data; The input submodule is used to input standard query data into a pre-trained multimodal feature fusion model to generate a query joint deep feature vector, and input the query joint deep feature vector into a trained cross-modal deep hash coding network to generate the corresponding query hash code. The calculation submodule is used to filter out all binary hash codes within the corresponding partition. It uses the Hamming distance calculation formula to calculate the Hamming distance between the query hash code and the filtered historical binary hash codes.

8. The system as described in claim 6, characterized in that, The causal tracing module includes the following sub-modules: The source tracing submodule is used to input the node features of the heterogeneous knowledge graph of the inspection into the GAT model. The node features are represented by one-hot encoding combined with node attribute features to generate the initial feature vector of the node. The model is set with two attention layers, each with a hidden layer dimension of 128, and the activation function is LeakyReLU with a dropout probability of 0.2.

Images