Maintenance case-oriented cross-modal retrieval and intelligent recommendation system

Abstract

This invention relates to the fields of computer information retrieval and artificial intelligence, specifically disclosing a cross-modal retrieval and intelligent recommendation system for maintenance cases. It includes: a multimodal data acquisition and preprocessing module for collecting text description data, fault image data, and sensor waveform data from a maintenance case database, and standardizing and normalizing various types of data; a cross-modal semantic alignment and feature extraction module, which includes a text semantic encoding submodule, an image visual feature encoding submodule, and a waveform temporal feature encoding submodule; a dynamic adaptive fusion module; a cross-modal retrieval and matching module; and an intelligent recommendation and decision support module. This system achieves accurate cross-modal semantic matching and efficient retrieval through multimodal feature extraction, dynamic adaptive fusion, and a dual-tower retrieval network, and generates a comprehensive recommendation report, significantly improving the accuracy and real-time performance of maintenance case retrieval.

Description

Technical Field

[0001] This invention belongs to the field of computer information retrieval and artificial intelligence technology, specifically relating to a cross-modal retrieval and intelligent recommendation system for maintenance cases. Background Technology

[0002] In the field of industrial equipment maintenance and fault diagnosis, the efficient and accurate retrieval of historical maintenance cases to guide on-site maintenance decisions is a key technology for improving equipment operation and maintenance efficiency and reliability. Maintenance cases typically contain multiple modal data such as text descriptions, fault images, and sensor waveforms, which together constitute the knowledge base for fault diagnosis.

[0003] A cross-modal retrieval and intelligent recommendation system for maintenance cases aims to provide maintenance personnel with accurate case matching and decision support through comprehensive analysis of multimodal data. Its core objective is to quickly locate historical cases most relevant to the current fault situation from massive amounts of heterogeneous maintenance data, thereby assisting in fault localization and maintenance plan formulation.

[0004] Existing technologies face significant challenges in processing multimodal data from maintenance cases. Due to the inherent differences in semantic granularity between different modalities such as text descriptions and images, traditional feature extraction and alignment methods struggle to achieve accurate mapping across modal semantic spaces, leading to feature alignment distortion.

[0005] This semantic gap prevents the system from accurately capturing the deep semantic relationships between text queries and image examples when performing cross-modal retrieval, thus limiting the relevance and accuracy of the search results. Furthermore, maintenance sites have extremely high requirements for retrieval efficiency and real-time performance; existing methods, while ensuring retrieval accuracy, often fail to meet the practical needs of rapid response. Therefore, how to effectively integrate multimodal maintenance data to achieve accurate and efficient cross-modal retrieval and recommendation has become a pressing technical challenge in this field. Summary of the Invention

[0006] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies, such as inaccurate cross-modal semantic space mapping, feature alignment distortion, and the difficulty in balancing retrieval efficiency and accuracy due to differences in semantic granularity between different modalities of data, including text and images. This invention provides a cross-modal retrieval and intelligent recommendation system for maintenance cases. The system achieves deep semantic understanding and efficient, accurate retrieval of multimodal maintenance case data by constructing a hierarchical semantic alignment and dynamic adaptive fusion mechanism.

[0007] The technical solution of this invention is as follows. The system includes a multimodal data acquisition and preprocessing module, a cross-modal semantic alignment and feature extraction module, a dynamic adaptive fusion module, a cross-modal retrieval and matching module, and an intelligent recommendation and decision support module.

[0008] The multimodal data acquisition and preprocessing module is responsible for collecting text description data, fault image data, and sensor waveform data from the maintenance case library, and performing standardization and normalization on various types of data. The cross-modal semantic alignment and feature extraction module further includes a text semantic encoding submodule, an image visual feature encoding submodule, and a waveform temporal feature encoding submodule.

[0009] The text semantic encoding submodule employs a deep semantic encoding network based on an attention mechanism to extract multi-level semantic feature vectors from maintenance text descriptions. The image visual feature encoding submodule uses a convolutional neural network and a spatial pyramid pooling structure to extract multi-scale visual feature vectors from fault images. The waveform temporal feature encoding submodule uses a long short-term memory network and a self-attention mechanism to extract temporal dynamic feature vectors from sensor waveform data.

[0010] The dynamic adaptive fusion module receives multimodal feature vectors from the cross-modal semantic alignment and feature extraction module, and calculates the weight distribution of each modality feature in the global semantic space based on the gated fusion unit and multi-head cross-attention mechanism, thereby generating a unified cross-modal joint semantic representation vector.

[0011] The cross-modal retrieval and matching module receives query requests and cross-modal joint semantic representation vectors. By constructing a dual-tower retrieval network, it performs deep semantic embedding on both the query end and the case database end. It then uses a maximum inner product search algorithm and an approximate nearest neighbor index to calculate the semantic similarity score between the query and each case. The intelligent recommendation and decision support module sorts the search results based on the semantic similarity score and generates a comprehensive recommendation report containing the most relevant repair cases and their repair solutions.

[0012] Furthermore, the specific implementation process of the text semantic encoding submodule is as follows. First, the input maintenance text is segmented and vectorized to obtain a word embedding sequence. Then, the word embedding sequence is input into a deep semantic encoding network based on a multi-head self-attention mechanism. This network contains at least three encoder layers, each of which contains a self-attention sublayer and a feedforward neural network sublayer.

[0013] The self-attention sublayer calculates the attention weights between each word and all other words to capture long-distance dependencies in the text. The feedforward neural network sublayer performs non-linear transformations and feature enhancements on the output of the self-attention sublayer. Finally, pooling operations aggregate the encoded word sequence features into a fixed-dimensional text semantic feature vector.

[0014] Furthermore, the specific implementation process of the image visual feature encoding submodule is as follows: A pre-trained convolutional neural network is used as the basic feature extractor, which contains at least 5 convolutional layers and 2 fully connected layers. The fault image is input into the convolutional neural network, and local and global features of the image are extracted layer by layer through convolution and pooling operations.

[0015] The feature map from the last convolutional layer of the convolutional neural network is then input into a spatial pyramid pooling structure. This structure uses three pooling windows of different scales to perform multi-scale pooling on the feature map, generating a fixed-length multi-scale visual feature vector. The multi-scale visual feature vector is then passed through a fully connected layer for dimensionality reduction and feature compression, outputting the image visual feature vector.

[0016] Furthermore, the specific implementation process of the waveform timing feature encoding submodule is as follows: The sensor waveform data is segmented and normalized to obtain timing data segments of equal length. The timing data segments are input into a Long Short-Term Memory (LSTM) network, which contains at least two hidden layers, each containing 128 memory units.

[0017] The Long Short-Term Memory (LSTM) network processes the input sequence step by step, capturing long-term dependencies and dynamic patterns in the waveform data. The final hidden state of the LSM network is then fed into a self-attention mechanism layer, which calculates the importance weights of features at different time steps and performs a weighted summation of features from all time steps to generate a waveform temporal feature vector.

[0018] Furthermore, the specific implementation process of the dynamic adaptive fusion module is as follows. First, the text semantic feature vector, image visual feature vector, and waveform temporal feature vector are projected onto a common semantic space of the same dimension. Then, a gated fusion unit is used to calculate the initial fusion weights of each modality feature. The gated fusion unit receives the concatenated vector of all modality features as input and outputs the weight coefficients of each modality through a fully connected layer and a sigmoid activation function.

[0019] A multi-head cross-attention mechanism is further employed to refine the initial fusion weights. This mechanism uses textual semantic features as the query vector and image visual features and waveform temporal features as the key and value vectors, respectively, to calculate cross-attention scores between text and image, and between text and waveform. Finally, the initial fusion weights and cross-attention scores are weighted and combined to obtain the final fusion weights for each modality. Based on these final fusion weights, the features of each modality are weighted and summed to generate a cross-modal joint semantic representation vector.

[0020] Furthermore, the specific implementation process of the cross-modal retrieval and matching module is as follows: A dual-tower retrieval network is constructed, where the query tower is responsible for deep semantic embedding of the user-input text or image query, and the case tower is responsible for deep semantic embedding of the cross-modal joint semantic representation vector in the case library. Both the query tower and the case tower adopt a multilayer perceptron structure, with each perceptron layer containing at least two fully connected layers and a ReLU activation function.

[0021] The semantic similarity score between the query and the case embedding vectors is obtained by performing an inner product operation. A maximum inner product search algorithm is then used to quickly find the K most similar cases to the query in the case database, where K is a preset threshold for the number of searches. To accelerate the retrieval process, the system constructs an approximate nearest neighbor index based on a hierarchical, navigable small world. This index achieves efficient similarity search by constructing a multi-level graph structure.

[0022] Furthermore, the specific implementation process of the intelligent recommendation and decision support module is as follows: Based on the semantic similarity score output by the cross-modal retrieval and matching module, all retrieved maintenance cases are sorted in descending order. The top N most relevant cases are selected, where N is a preset recommendation quantity threshold. For each recommended case, its original maintenance text description, fault image, sensor waveform data, and maintenance plan record are extracted.

[0023] The extracted original repair text descriptions, fault images, sensor waveform data, and repair plan records are integrated into a structured comprehensive recommendation report, which is presented to repair personnel through a visual interface. The comprehensive recommendation report also includes a confidence score based on historical repair success rates, calculated by statistically analyzing the repair success rates of similar cases.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. By constructing a hierarchical semantic alignment and dynamic adaptive fusion mechanism, the system can accurately capture the deep semantic relationships between text, image and waveform data, effectively bridging the semantic gap between different modal data and significantly improving the accuracy and semantic consistency of cross-modal retrieval.

[0026] 2. By adopting a dual-tower structure retrieval network and near nearest neighbor indexing technology, millisecond-level retrieval response speed is achieved while ensuring retrieval accuracy, meeting the stringent real-time requirements of maintenance sites.

[0027] 3. By integrating multimodal feature extraction, adaptive fusion and intelligent recommendation into a single system architecture, it provides an end-to-end solution from data input to decision output, which greatly improves the utilization efficiency of maintenance case knowledge and the scientific nature of maintenance decisions. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall technical solution architecture of the cross-modal retrieval and intelligent recommendation system for maintenance cases proposed in this invention;

[0029] Figure 2 This is a schematic diagram of the core principle framework of the dynamic adaptive fusion module in this invention;

[0030] Figure 3 This is a flowchart illustrating the logical flow of the cross-modal semantic alignment and feature extraction module in this invention.

[0031] Figure 4 This is a schematic diagram of the dual-tower structure interaction and data flow of the cross-modal retrieval and matching module in this invention;

[0032] Figure 5 This is a flowchart illustrating the recommendation generation and report integration process of the intelligent recommendation and decision support module in this invention. Detailed Implementation

[0033] Please refer to the attached document. Figure 1 To be continued Figure 5 This embodiment details the specific implementation of a cross-modal retrieval and intelligent recommendation system for maintenance cases. This system is a complex, integrated engineering system where software and hardware work together. It is deployed on a server cluster equipped with high-performance graphics processing units and a central processing unit, and stores massive amounts of maintenance case data through a distributed file system.

[0034] The core objective of the system is to process multimodal data from the maintenance site, including text descriptions, fault images, and sensor waveform data, and through a series of sophisticated calculation modules, ultimately output highly relevant maintenance case recommendation reports to assist maintenance personnel in making decisions.

[0035] The multimodal data acquisition and preprocessing module serves as the entry point for the system's data flow. This module establishes a persistent network connection with an external maintenance case database through a pre-defined data interface protocol. The data interface protocol employs the Transmission Control Protocol (TCP) as the underlying communication guarantee and defines a data exchange format based on JavaScript object representation.

[0036] This format explicitly stipulates that each repair case data package must contain three independent data fields, corresponding to text description data, fault image data, and sensor waveform data, respectively. The text description data field stores the fault phenomenon description, operation record, and final solution filled in by the repair personnel, and its character encoding is uniformly UTF-8 format to ensure global character set compatibility.

[0037] The fault image data field stores digital images of the faulty parts of the equipment captured by an industrial camera or high-resolution scanner. The image format is preferably lossless compressed PNG or high-compression JPEG, with an image resolution of no less than 1920×1080 pixels. The sensor waveform data field stores time-series voltage or current signals acquired from the equipment's sensors, with a sampling rate of no less than 10,000 points per second and a data precision of 16-bit signed integers.

[0038] After data acquisition is completed, the preprocessing process is initiated immediately. For text description data, preprocessing includes removing special characters, unifying capitalization, and performing Chinese word segmentation. The word segmentation tool uses a custom dictionary segmenter based on a Hidden Markov Model, which contains more than 50,000 professional terms in the maintenance field.

[0039] For fault image data, preprocessing includes size normalization to 256×256 pixels, pixel value normalization to a floating-point range of 0 to 1, and histogram equalization to enhance image contrast. For sensor waveform data, preprocessing includes DC component removal, low-pass filtering based on a moving average window to suppress high-frequency noise, and standardizing the data length to 1024 sampling points by truncation or zero padding. All preprocessed data are assigned a unique timestamp and case identifier and stored in the system's temporary buffer database, awaiting subsequent module calls.

[0040] Please refer to the attached document. Figure 3 The cross-modal semantic alignment and feature extraction module receives standardized data from the preprocessing module and initiates three parallel feature encoding sub-processes. The text semantic encoding sub-module is responsible for extracting deep semantic information from the text description. This sub-module first inputs the preprocessed word segmentation sequence into the word embedding layer, which is a trainable lookup table with a dimension of 300. This layer maps each word to a 300-dimensional dense real-valued vector, thereby transforming the discrete word sequence into a continuous word embedding sequence.

[0041] Subsequently, the word embedding sequence is fed into a deep semantic encoding network based on a multi-head self-attention mechanism. This network consists of three identical encoder layers stacked on top of each other. Each encoder layer contains two core sub-layers. The first sub-layer is a multi-head self-attention sub-layer, which internally runs eight independent attention heads in parallel. Each attention head linearly projects the input sequence into a 64-dimensional space, forming a query matrix, a key matrix, and a value matrix, respectively. The attention score is calculated following a scaled dot product attention mechanism.

[0042] Specifically, for each attention head, its output is obtained through the following calculation:

[0043] ;

[0044] Among them, the query matrix Key matrix Sum matrix All are obtained by linear transformation of the input sequence. It is the scaling factor, here It equals 64. This represents the transpose. The outputs of the eight attention heads are concatenated in the last dimension, and then subjected to a linear transformation to produce the final output of this self-attention sublayer.

[0045] The second sublayer is a feedforward neural network sublayer, which consists of two fully connected layers. The first fully connected layer increases the input dimension from 300 to 1024 and introduces non-linearity using the ReLU activation function; the second fully connected layer reduces the dimension from 1024 back to 300.

[0046] Residual connections and layer normalization operations are applied after each sublayer to stabilize the training process and accelerate convergence. After processing by three encoder layers, the network outputs a word sequence feature matrix rich in contextual information. Finally, average pooling is performed on this matrix along the sequence dimension to generate a fixed 300-dimensional text semantic feature vector.

[0047] The image visual feature encoding submodule processes fault images in parallel. This submodule uses a ResNet50 model pre-trained on the ImageNet dataset as the basic feature extractor. The ResNet50 model contains 49 convolutional layers and 1 fully connected layer, with the last convolutional layer outputting a 7×7×2048-dimensional feature map. This feature map is then fed into a spatial pyramid pooling structure.

[0048] This structure employs three different pooling window sizes: 1×1, 2×2, and 4×4. For the 1×1 window, global average pooling is performed directly on the global feature map, producing a single 2048-dimensional feature vector. For the 2×2 window, the feature map is divided into a 2×2 grid, and average pooling is performed within each grid, producing four 2048-dimensional feature vectors.

[0049] For a 4×4 window, the feature map is divided into a 4×4 grid. Average pooling is performed within each grid, producing 16 feature vectors of 2048 dimensions each. All pooling results are concatenated along their feature dimensions to obtain a multi-scale visual feature vector of 43008 dimensions (1 + 4 + 16 × 2048). This high-dimensional vector is then dimensionality-reduced through a fully connected layer with 1024 output units, using the ReLU activation function to finally output a 1024-dimensional image visual feature vector.

[0050] The waveform timing feature encoding submodule is specifically designed for processing sensor waveform data. The preprocessed 1024-point timing data segment is first fed into a two-layer Long Short-Term Memory (LSTM) network. Each layer of this network contains 128 memory units. The LTM network processes the input sequence sequentially, and its internal gate control mechanisms (input gate, forget gate, output gate) dynamically adjust the flow of information and the updating of memory, thereby effectively capturing long-term dependencies and dynamic patterns in the waveform.

[0051] After processing all 1024 time steps, the hidden state of the network at the last time step, a 128-dimensional vector, is extracted as a preliminary temporal representation. This hidden state is then fed into a self-attention mechanism layer. The self-attention layer first transforms the input through a linear transformation to generate a query vector, a key vector, and a value vector.

[0052] Then, attention weights are calculated by performing a dot product operation on the query vector and all key vectors, and applying the Softmax function to obtain the relative importance score of the feature at each time step. Finally, these weights are multiplied by the corresponding value vectors and summed to generate a 128-dimensional waveform time-series feature vector. This vector concentrates the most discriminative information in the waveform sequence.

[0053] Please refer to the attached document. Figure 2 The dynamic adaptive fusion module is responsible for integrating the feature vectors from the three modalities into a unified semantic representation. This module first maps the 300-dimensional text semantic feature vector, the 1024-dimensional image visual feature vector, and the 128-dimensional waveform temporal feature vector to a 512-dimensional common semantic space through three independent linear projection layers. The projection layers use a fully connected structure with weight matrices of dimensions 300×512, 1024×512, and 128×512, respectively. The projected feature vectors are denoted as text projection features, image projection features, and waveform projection features, respectively.

[0054] Next, the gated fusion unit begins its operation. This unit concatenates the text projection features, image projection features, and waveform projection features along their feature dimensions, forming a 1536-dimensional concatenated vector. This concatenated vector is fed into the gated network, which consists of a fully connected network with 512 hidden neurons and a fully connected layer with an output dimension of 3. Finally, a sigmoid activation function outputs three scalar values ​​between 0 and 1, representing the initial fusion weights for the text, image, and waveform, respectively. The sum of these three weights is normalized to 1 through a softmax layer.

[0055] Subsequently, the system initiates a multi-head cross-attention mechanism to fine-tune the initial weights. This mechanism uses text projection features as the query vector, and the concatenated vector of image projection features and waveform projection features as the key and value vectors. Specifically, the query vector is a 512-dimensional text projection feature, and the key and value vectors are 1024-dimensional vectors concatenated from image and waveform projection features. This cross-attention mechanism also employs eight heads. Each head linearly projects the query, key, and value into a 64-dimensional space and calculates the cross-attention score. The outputs of the eight heads are concatenated and linearly transformed to obtain a 512-dimensional context vector. This context vector is then transformed into two adjustment factors through a fully connected layer and a sigmoid activation function, corresponding to the image modality and waveform modality, respectively. The adjustment factor for the text modality is fixed at 1.

[0056] The initial fusion weights are multiplied element-wise with their corresponding adjustment factors to obtain refined weights for each modality. Finally, these refined weights are normalized again using a Softmax layer to obtain the final fusion weights for each modality. Based on these final fusion weights, the text projection features, image projection features, and waveform projection features are weighted and summed to generate a 512-dimensional cross-modal joint semantic representation vector. This vector integrates the core semantic information from the three modalities.

[0057] Please refer to the attached document. Figure 4 The cross-modal retrieval and matching module is responsible for receiving user queries and matching them with cases in the case library based on similarity. This module constructs a dual-tower retrieval network. The query tower is responsible for processing user input, which can be either text or image queries. For text queries, the processing flow is completely consistent with the text semantic encoding submodule, ultimately generating a 512-dimensional query embedding vector. For image queries, the processing flow is completely consistent with the image visual feature encoding submodule, ultimately generating a 512-dimensional query embedding vector.

[0058] The case tower is responsible for processing the cross-modal joint semantic representation vectors of all cases in the case library. This vector has already been generated by the dynamic adaptive fusion module. The case tower further transforms it using a multilayer perceptron, which contains two fully connected layers: the first layer has 256 neurons using the ReLU activation function, and the second layer has 128 neurons, outputting a final 128-dimensional case embedding vector. The query tower has a symmetrical structure to the case tower, also using a two-layer perceptron, mapping the 512-dimensional query features to a 128-dimensional query embedding vector. Similarity is calculated using the vector inner product. For a given query embedding vector... and case embedding vector Its semantic similarity score Calculated using the following formula:

[0059] ;

[0060] in, and These are the query embedding vector and the case embedding vector, respectively. Each component. To achieve millisecond-level retrieval in a library containing millions of cases, the system constructs an approximate nearest neighbor index based on a hierarchical, navigable small world. This index structure is built starting from an underlying graph of case embedding vectors, where each node represents a case embedding vector. A highly interconnected graph is formed by repeatedly inserting new nodes and connecting them to their nearest neighbors.

[0061] Building upon this foundation, a higher-level graph is constructed by randomly selecting a subset of nodes. This higher-level graph is a sparse representation of the lower-level graph. During retrieval, starting from the highest level, the node closest to the query vector is found. Then, the process is repeated layer by layer downwards, performing a greedy search at each level. Ultimately, the K nearest neighbor cases most similar to the query vector are found in the lower-level graph, where K is typically set to 100. This process significantly reduces the number of cases requiring precise similarity calculations.

[0062] Please refer to the attached document. Figure 5 The intelligent recommendation and decision support module is the final output of the system. This module receives the top K candidate cases and their semantic similarity scores from the retrieval module. First, the module sorts all candidate cases in descending order based on their semantic similarity scores.

[0063] Subsequently, the system's preset recommendation threshold N takes effect. N is typically set to 10, meaning the top 10 most relevant cases are selected from the sorted list as the final recommendation results. For each selected recommendation case, the module retrieves complete multimodal data from its original storage location, including the original maintenance text description, high-resolution fault images, complete sensor waveform data, and detailed maintenance plan records. This scattered data is integrated into a structured comprehensive recommendation report.

[0064] This report is generated using Hypertext Markup Language (HTML) to ensure compatibility across different terminal devices. The report begins with a summary listing the numbers of all recommended cases, their main fault symptoms, and core similarity scores. Clicking on any case number expands to reveal its detailed information. The detailed information view consists of four side-by-side panels: a text description panel displaying the complete repair text; an image panel showing fault images in a scalable format; a waveform panel dynamically drawing sensor waveforms using a JavaScript charting library; and a solution panel progressively listing the repair operations performed, the replaced parts, and their models.

[0065] In addition, the comprehensive recommendation report calculates and displays a confidence score for each recommended case. This score is not based on semantic similarity, but is calculated by querying a historical repair record database. The system statistically analyzes all historical cases that are highly similar to the current recommended case in terms of fault symptoms, and calculates the percentage of successful repairs among these historical cases; this percentage is used as the confidence score.

[0066] For example, if there are 50 similar historical cases, and 45 of them were successfully repaired, the confidence score is 90%. The report is ultimately presented to maintenance personnel through a web-based visual interface. The interface provides filtering and sorting functions, allowing maintenance personnel to further filter the recommended results based on confidence scores, fault types, or equipment models to assist in making final maintenance decisions. The entire system, from data input to report generation, forms a complete closed loop, greatly improving the accuracy of maintenance knowledge retrieval and decision-making efficiency.

[0067] Based on the above embodiments, this embodiment optimizes the connection method between the gated fusion unit and the multi-head cross-attention mechanism in the dynamic adaptive fusion module, and provides an alternative implementation scheme, aiming to further improve the discrimination ability of fusion weights and the robustness of the system.

[0068] In the above embodiments, the initial fusion weights output by the gated fusion unit and the adjustment factors output by the multi-head cross-attention mechanism are directly multiplied element-wise, and then Softmax normalization is performed to obtain the final fusion weights.

[0069] This embodiment makes a key modification to the process by introducing a weight refinement network. Specifically, after the gated fusion unit calculates the initial fusion weight vectors for text, image, and waveform, these initial weight vectors are no longer directly multiplied by the adjustment factor.

[0070] Meanwhile, the multi-head cross-attention mechanism still functions normally, but its output is no longer just adjustment factors for images and waveforms. Instead, it expands to an attention weight vector with three elements and the same dimension as the initial weight vector. This attention weight vector reflects the importance distribution of each modality feature from the perspective of text query.

[0071] Next, the initial fused weight vector and the attention weight vector are fed in parallel into a novel weight refinement network. This network is a small, fully connected neural network whose input layer receives the concatenation of two 3D vectors, resulting in a 6-dimensional input vector. The network contains a hidden layer with 16 neurons using the ReLU activation function, and an output layer with 3 neurons. The output layer is then followed by a Softmax activation function to ensure the output follows a probability distribution.

[0072] The role of this weight refinement network is to learn how to non-linearly and adaptively combine static weight information from the gated fusion unit with dynamic context weight information from the cross-attention mechanism. By training on labeled data, the network can learn when to trust the judgment of the gated fusion unit more and when to rely more on the context information of the cross-attention mechanism, thereby generating more discriminative final fusion weights.

[0073] The direct benefit of this modification is enhanced flexibility in the fusion strategy. For example, when the text description is very clear but the image is blurry, the gating unit may assign a higher weight to the text, and the cross-attention mechanism will also reinforce the importance of the text. The refinement network will then combine both to give a very high final weight to the text. Conversely, when the text description is ambiguous but the image features are highly discriminative, the refinement network can appropriately reduce the text weight from the gating unit while increasing the image weight from the cross-attention mechanism, thus obtaining a more reasonable fusion result.

[0074] Compared to simple element-wise multiplication, this non-linear combination method can capture more complex inter-modal dependencies, theoretically generating a superior cross-modal joint semantic representation vector, thereby potentially improving the accuracy of subsequent retrieval and recommendation. The remaining parts of the system, including the multimodal data acquisition and preprocessing module, the cross-modal semantic alignment and feature extraction module, the cross-modal retrieval and matching module, and the intelligent recommendation and decision support module, are implemented in the same way as the embodiments described above.

Claims

1. A system for maintenance case oriented cross-modal retrieval and intelligent recommendation, characterized in that, The method comprises the following steps: a multi-modal data acquisition and preprocessing module is used to acquire text description data, fault image data and sensor waveform data from a maintenance case library, and to standardize and normalize various types of data; a cross-modal semantic alignment and feature extraction module is used to receive standardized data from the multi-modal data acquisition and preprocessing module, and to extract multi-modal feature vectors; the cross-modal semantic alignment and feature extraction module comprises a text semantic encoding submodule, an image visual feature encoding submodule and a waveform time series feature encoding submodule; a dynamic adaptive fusion module is used to receive multi-modal feature vectors from the cross-modal semantic alignment and feature extraction module, and to calculate the weight distribution of each modal feature in the global semantic space based on a gating fusion unit and a multi-head cross-attention mechanism, and then to generate a unified cross-modal joint semantic representation vector; a cross-modal retrieval and matching module is used to receive a query request and a cross-modal joint semantic representation vector, to construct a double-tower structure retrieval network, to respectively perform deep semantic embedding on the query end and the case library end, and to calculate the semantic similarity score between the query and each case by using a maximum inner product search algorithm and an approximate nearest neighbor index; the specific implementation process of the cross-modal retrieval and matching module is as follows: a double-tower structure retrieval network is constructed, wherein the query tower is responsible for deep semantic embedding of the text query or image query input by the user, and the case tower is responsible for deep semantic embedding of the cross-modal joint semantic representation vector in the case library; the query tower and the case tower both adopt a multi-layer perceptron structure, and each layer of the perceptron contains at least two fully connected layers and a ReLU activation function; the query embedding vector and the case embedding vector are subjected to inner product operation to obtain the semantic similarity score between the query and the case; a maximum inner product search algorithm is used to quickly find the top K cases most similar to the query in the case library, wherein K is a preset retrieval quantity threshold; in order to speed up the retrieval process, the system constructs an approximate nearest neighbor index based on a hierarchical navigable small world, and the index realizes efficient similarity search by constructing a multi-layer graph structure; an intelligent recommendation and decision support module is used to sort the retrieval results according to the semantic similarity score, and to generate a comprehensive recommendation report containing the most relevant maintenance cases and their maintenance schemes. 2.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, the text semantic encoding submodule adopts a deep semantic encoding network based on an attention mechanism to extract multi-level semantic feature vectors from the maintenance text description; the specific implementation process of the text semantic encoding submodule is as follows: first, the input maintenance text is subjected to word segmentation and vectorization to obtain a word embedding sequence; then, the word embedding sequence is input into a deep semantic encoding network based on a multi-head self-attention mechanism, and the network contains at least three encoder layers, each of which contains a self-attention sublayer and a feedforward neural network sublayer; the self-attention sublayer calculates the attention weight between each word and all other words to capture long-distance dependencies in the text; the feedforward neural network sublayer performs nonlinear transformation and feature enhancement on the output of the self-attention sublayer; finally, the encoded word sequence features are aggregated into a fixed-dimensional text semantic feature vector through a pooling operation. 3.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The image visual feature encoding submodule uses a convolutional neural network and a spatial pyramid pooling structure to extract multi-scale visual feature vectors from fault images. The specific implementation process of the image visual feature encoding submodule is as follows: a pre-trained convolutional neural network is used as the basic feature extractor, which contains at least 5 convolutional layers and 2 fully connected layers; The fault image is input into a convolutional neural network, and local and global features of the image are extracted layer by layer through convolution and pooling operations; The feature map of the last convolutional layer of the convolutional neural network is then input into the spatial pyramid pooling structure, which uses three pooling windows of different scales to perform multi-scale pooling on the feature map, generating a fixed-length multi-scale visual feature vector. The multi-scale visual feature vectors are reduced in dimensionality and compressed in a fully connected layer, and the output is an image visual feature vector. 4.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The waveform timing feature encoding submodule uses a long short-term memory network and a self-attention mechanism to extract timing dynamic feature vectors from sensor waveform data; The specific implementation process of the waveform timing feature encoding submodule is as follows: the sensor waveform data is segmented and normalized to obtain timing data segments of equal length; The time-series data segment is input into a long short-term memory network, which contains at least two hidden layers, each containing 128 memory units; Long Short-Term Memory (LSTM) networks process input sequences step by step to capture long-term dependencies and dynamic patterns in waveform data. The final hidden state of the Long Short-Term Memory network is then input into the self-attention mechanism layer. The self-attention mechanism layer calculates the importance weights of features at different time steps and performs a weighted summation of features at all time steps to generate a waveform temporal feature vector. 5.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The specific implementation process of the dynamic adaptive fusion module is as follows: First, the text semantic feature vector, image visual feature vector, and waveform temporal feature vector are projected onto a common semantic space of the same dimension; then, a gated fusion unit is used to calculate the initial fusion weights of each modality feature. The gated fusion unit receives the concatenated vector of features from all modalities as input, and outputs the weight coefficients of each modality through a fully connected layer and a sigmoid activation function. The initial fusion weights are further refined by adopting a multi-head cross-attention mechanism. The multi-head cross-attention mechanism uses text semantic features as query vectors and image visual features and waveform temporal features as key vectors and value vectors to calculate the cross-attention scores between text and image, and between text and waveform. Finally, the initial fusion weights and cross-attention scores are weighted and combined to obtain the final fusion weights for each modality feature. The features of each modality are weighted and summed according to the final fusion weights to generate a cross-modal joint semantic representation vector. 6.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The specific implementation process of the intelligent recommendation and decision support module is as follows: based on the semantic similarity score output by the cross-modal retrieval and matching module, all retrieved repair cases are sorted in descending order; Select the top N most relevant cases, where N is a preset threshold for the number of recommendations; For each recommended case, extract its original repair text description, fault image, sensor waveform data, and repair plan record; The extracted original maintenance text descriptions, fault images, sensor waveform data, and maintenance plan records are integrated into a structured comprehensive recommendation report, which is then presented to maintenance personnel through a visual interface. The comprehensive recommendation report also includes a confidence score based on historical repair success rates, which is calculated by statistically analyzing the repair success rates of similar cases. 7.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The multimodal data acquisition and preprocessing module preprocesses the text description data by removing special characters, unifying capitalization, and performing Chinese word segmentation. Preprocessing of fault image data includes size normalization, pixel value normalization, and histogram equalization to enhance image contrast; The preprocessing of the sensor waveform data includes DC component removal, low-pass filtering based on a moving average window to suppress high-frequency noise, and unifying the data length to 1024 sampling points by truncation or zero padding. 8.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 5, characterized in that, The connection method between the gated fusion unit and the multi-head cross-attention mechanism in the dynamic adaptive fusion module is optimized using a weighted refining network. The weight refinement network receives the initial fusion weight vector from the gated fusion unit and the attention weight vector from the multi-head cross-attention mechanism, and performs nonlinear combination through a fully connected neural network to generate a more discriminative final fusion weight. 9.The maintenance case oriented cross-modal retrieval and intelligent recommendation system according to claim 1, characterized in that, The system is deployed on a server cluster equipped with a high-performance graphics processing unit and a central processing unit, and stores massive amounts of repair case data through a distributed file system. The system establishes a persistent network connection with an external maintenance case database through a preset data interface protocol. The data interface protocol uses the Transmission Control Protocol as the underlying communication guarantee and defines a data exchange format based on JavaScript object representation.

Images