Architecture graph retrieval method and device, electronic equipment, storage medium and product

Abstract

The invention discloses an architecture graph retrieval method and device, electronic equipment, a storage medium and a product. The architecture graph retrieval method comprises the following steps: acquiring a retrieval request, and determining a query vector corresponding to the retrieval request; the similarity between the query vector and an architecture graph vector of each architecture graph in an architecture graph library is calculated, and the architecture graph vector of each architecture graph is determined according to fusion features of text features, image features and relation features of the architecture graphs; and according to the similarity between the query vector and the architecture graph vector of each architecture graph, determining a retrieval result of the architecture graph. According to the technical scheme, the framework graph vector is determined by utilizing the fusion features of the text features, the image features and the relation features of the framework graph, the comprehensive information of the framework graph can be accurately captured, the framework graph vector similar to the query vector is retrieved according to the comprehensive information, and the correlation and the accuracy of the retrieval result are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data retrieval technology, and in particular, to an architecture diagram retrieval method, device, electronic device, storage medium and product. Background Art

[0002] In modern software engineering, system architecture diagrams are an important tool for describing the design of complex software systems. They help understand the composition structure of the system, the interactions between components, and the dependencies. However, when faced with a large number of complex architecture diagrams, how to efficiently and accurately retrieve architecture diagrams related to specific requirements has become an urgent problem to be solved.

[0003] Existing architecture diagram retrieval technologies mainly rely on keyword matching or simple metadata search, which often fail to fully understand and utilize the rich information in architecture diagrams. Specifically, traditional text retrieval methods can only search based on the descriptive text of the diagram, while ignoring the visual and structured information of the diagram itself; on the other hand, although image recognition technology can process graphic content, it lacks the ability to understand the semantics of the diagram. In addition, existing methods are also difficult to effectively capture and utilize the relationship information between components in the architecture diagram, which limits the relevance and accuracy of the retrieval results.

[0004] Another challenge is that as the size of the system grows, the number and complexity of architecture diagrams also increase, making manual management and query increasingly difficult. Traditional methods usually require users to have considerable expertise to effectively locate the required architecture diagrams, which is a considerable obstacle for non-professional users. At the same time, with the development of technologies such as cloud computing and microservices, system architectures have become more dynamic and flexible, which also requires architecture diagram retrieval technology to adapt to the ever-changing environment and provide real-time and effective support.

[0005] In summary, due to the inability to fully understand and utilize the rich information in architecture diagrams, there are obvious limitations in dealing with large-scale, high-dimensional, evolving and changing architecture diagram data. Summary of the invention

[0006] The present application provides an architecture diagram retrieval method, device, electronic device, storage medium and product to improve the relevance and accuracy of retrieval results.

[0007] In a first aspect, an embodiment of the present application provides an architecture diagram retrieval method, comprising:

[0008] Obtaining a search request and determining a query vector corresponding to the search request;

[0009] Calculating the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to a fusion feature of text features, image features, and relationship features of the architecture diagram;

[0010] According to the similarity between the query vector and the architecture diagram vector of each architecture diagram, a retrieval result of the architecture diagram is determined.

[0011] In a second aspect, an embodiment of the present application further provides an architecture diagram retrieval device, comprising:

[0012] A conversion module, used to obtain a search request and determine a query vector corresponding to the search request;

[0013] A calculation module, used to calculate the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to the fusion features of the text features, image features and relationship features of the architecture diagram;

[0014] The retrieval module is used to determine the retrieval result of the architecture diagram according to the similarity between the query vector and the architecture diagram vector of each architecture diagram.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0016] one or more processors;

[0017] A storage device for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the architecture diagram retrieval method as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the architecture diagram retrieval method as described in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program and / or instructions, which, when executed by a processor, implements the architecture diagram retrieval method as described in any of the above embodiments.

[0021] The embodiments of the present application provide an architecture diagram retrieval method, device, electronic device, storage medium and product, the architecture diagram retrieval method comprising: obtaining a retrieval request, and determining a query vector corresponding to the retrieval request; calculating the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined based on the fusion features of the text features, image features and relationship features of the architecture diagram; and determining the retrieval results of the architecture diagram based on the similarity between the query vector and the architecture diagram vector of each architecture diagram. The above technical scheme can accurately capture the comprehensive information of the architecture diagram by determining the architecture diagram vector using the fusion features of the text features, image features and relationship features of the architecture diagram, thereby retrieving the architecture diagram vector similar to the query vector, and improving the relevance and accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 A flowchart of an architecture diagram retrieval method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the structure of an architecture diagram retrieval device provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0027] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] It should be noted that the concepts of "first", "second", etc. mentioned in the embodiments of the present application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order or interdependence of the functions performed by these devices, modules, units or other objects.

[0029] In addition, the embodiments in the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0030] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the relevant content of the solution.

[0032] Figure 1 The present invention provides a flowchart of an architecture diagram retrieval method according to an embodiment of the present invention. The present embodiment can be applied to the case of retrieving an architecture diagram. Specifically, the architecture diagram retrieval method can be performed by an architecture diagram retrieval device, which can be implemented by software and / or hardware and integrated in an electronic device. The electronic device includes but is not limited to a computer, a smart phone, a server, or other devices with image processing capabilities.

[0033] like Figure 1 As shown, the method specifically comprises the following steps:

[0034] S110: Obtain a search request, and determine a query vector corresponding to the search request.

[0035] In this embodiment, the architecture diagram mainly refers to a graphical representation that describes entities and the relationships between entities in an abstract way, and can also be understood as a tool for graphically representing the composition and structure of a system. The architecture diagram includes the relationships and interactions between different components. The retrieval request can also be understood as a query request, which includes a specified architecture diagram, and the retrieval request is used to retrieve one or more architecture diagrams similar to the specified architecture diagram from multiple architecture diagrams. The architecture diagram specified in the retrieval request can be represented in the form of a vector, i.e., a query vector. For example, the retrieval request can be vectorized based on natural language processing (Natural Language Processing, NLP), for example, the architecture diagram specified in the retrieval request is converted into a query vector using a language representation model, and the language representation model can be a bidirectional encoder representation based on Transformer (Bidirectional Encoder Representations from Transformers) or a generative pre-trained Transformer model (Generative Pre-trained Transformer, GPT), etc.

[0036] S120, calculating the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to a fusion feature of text features, image features, and relationship features of the architecture diagram;

[0037] In this embodiment, the architecture library can be understood as a collection of architecture diagrams, which includes multiple pre-collected architecture diagrams for retrieval. Each architecture diagram can have a corresponding description text (used to describe the main content and key components of the architecture diagram), and the components in the architecture diagram can have corresponding labels. On this basis, the architecture library can be a data set containing the original architecture diagram, the corresponding description text, and the label information of each component.

[0038] Each architecture diagram corresponds to an architecture diagram vector, which can be predetermined based on the text features, image features, and relationship features of the architecture diagram. Among them, the text features provide a semantic description of the architecture diagram to help understand the content and function of the architecture diagram; the image features provide visual information of the architecture diagram to help identify and classify patterns and structures in the image; the relationship features provide the connection relationship between the components corresponding to the architecture diagram to help understand the topology and dependency of the system architecture. The text features, image features, and relationship features of each architecture diagram are interrelated. For example, the text description and image features of the same architecture diagram can be connected together through an identifier (ID). On this basis, the architecture diagram vector can more comprehensively and accurately represent the comprehensive features of the architecture diagram.

[0039] The similarity between the query vector and each architecture diagram vector can represent the similarity between the structure diagram specified in the retrieval request and each architecture diagram in the architecture diagram library. The higher the similarity, the closer the two architecture diagrams are. The similarity can be calculated by cosine similarity, Manhattan distance, or Euclidean distance.

[0040] Optionally, the query vector corresponding to the search request may also be determined based on a fusion of text features, image features, and relationship features of the architecture diagram specified in the search request.

[0041] S130. Determine a retrieval result of the architecture diagram according to a similarity between the query vector and the architecture diagram vector of each architecture diagram.

[0042] In this embodiment, the search results may include architecture diagrams corresponding to one or more architecture diagram vectors that are most similar to the query vector, or architecture diagrams corresponding to one or more architecture diagram vectors whose similarity to the query vector is greater than or equal to a specified threshold, etc. Exemplarily, all architecture diagram vectors may be sorted according to the similarity between the query vector and each architecture diagram vector, with a higher score indicating greater similarity to the query vector.

[0043] An embodiment of the present application provides an architecture diagram retrieval method. Since the architecture diagram vector takes into account text, image and relationship features, it can accurately capture the comprehensive information of the architecture diagram, and retrieve architecture diagram vectors similar to the query vector. The architecture diagram in the retrieval results is similar to the architecture diagram specified in the retrieval request in terms of text semantics, visual information and connection structure, which can improve the relevance and accuracy of the retrieval results.

[0044] In one embodiment, the process of collecting the architecture diagram includes:

[0045] S1. Obtain a system function-oriented architecture diagram through manual, crawling technology, or application programming interface (API) methods; among them, manual is suitable for small-scale systems, crawling technology is suitable for situations where information can be extracted from existing data sources, and API is suitable for situations where the system provides a corresponding interface.

[0046] Taking crawling technology as an example, the process of collecting architecture diagrams may include:

[0047] 1) Determine the data source: Identify the data source that can provide architectural information, such as configuration files, log files, monitoring data, etc.

[0048] 2) Writing scripts: Use scripting languages ​​(such as Python and Shell) to write scripts for capturing and parsing these data;

[0049] 3) Parsing data: extracting key information from data sources, such as component names, IP addresses, ports, dependencies, etc.

[0050] 4) Generate architecture diagram: Use drawing libraries (such as Graphviz, NetworkX) or visualization tools (such as D3.js) to generate architecture diagrams.

[0051] 5) Verify and adjust: Verify that the generated architecture diagram is accurate and make adjustments as needed.

[0052] Furthermore, a database or file system is used to store the architecture diagram, and a unique ID is assigned to each architecture diagram. Taking the file system as an example, the process of storing the architecture diagram includes:

[0053] 1) Create a root directory to store all architecture diagrams;

[0054] 2) Each architectural diagram is saved as a separate file with a unique file name, such as a file name based on a Universally Unique Identifier;

[0055] 3) You can create subfolders to organize the architecture diagram as needed, such as grouping by date or project.

[0056] In one embodiment, the method further comprises:

[0057] S10. For each architecture diagram in the architecture diagram library, add description text to the architecture diagram and add labels to the components in the architecture diagram;

[0058] S20, extracting text features, image features, and relationship features of the architecture diagram based on the description text and the label, wherein the relationship features are represented based on an undirected graph, and each node in the undirected graph corresponds to a label of a component;

[0059] S30: Input the text features and the image features into the multimodal model to determine the fusion features of the architecture diagram through the multimodal model.

[0060] In this embodiment, the label can be associated with the corresponding component (i.e., the node in the undirected graph). An undirected graph is a graph with no direction on the edge, and the edge represents the relationship between two nodes. Each node has a unique label. By storing the label information in association with the architecture diagram ID and the component node, it is ensured that each label can be traced back to its corresponding architecture diagram and component, and then the text features and image features are input into the pre-trained multimodal model, and the pre-trained multimodal model learns the association between the data of the two modes through collaborative training.

[0061] In one embodiment, when determining the architecture graph vector, the advantages of Graph Neural Network (GNN) and multimodal learning can be combined to construct a multimodal model, namely, Multi-modal Graph Neural Network (MMGNN) for processing graph structure data, and capable of processing text, image and relationship features at the same time. Through the message passing mechanism, MM-GNN can propagate information between nodes, further enhance node representation, and capture more complex cross-modal interactions. In addition, the existing system knowledge graph can be used to enhance the representation of the architecture graph. The entity and relationship information in the knowledge graph is integrated into the MM-GNN as additional input, thereby providing more background knowledge and helping the model better understand the components and their relationships in the architecture graph. In addition, the MM-GNN model can be optimized through tasks such as link prediction, so that the model can not only perform retrieval, but also predict potential relationships.

[0062] In one embodiment, adding description text to the architecture diagram and adding labels to the components in the architecture diagram include:

[0063] S1010, recognizing text in the architecture diagram based on optical character recognition (OCR) and adding description text;

[0064] S1020. Segment the architecture diagram based on computer vision to identify components, add a label to each component, and associate the label with the identifier of the architecture diagram for storage.

[0065] In this embodiment, a description text is generated from the architecture diagram for system functions, and the description text is associated with the corresponding architecture diagram ID and stored. Specifically, text labels are extracted from the architecture diagram based on OCR technology; the architecture diagram is segmented using computer vision technology to identify each component, and labels are added to the components, the relationship types between the components are marked, the connection lines are classified, and the label information is associated with the architecture diagram ID and stored.

[0066] Optionally, extract text labels from the architecture diagram based on OCR technology, including: selecting an OCR tool, such as Tesseract OCR, Google Cloud Vision API, or Azure Computer Vision, applying OCR technology to read all text in the architecture diagram, obtaining the position and content of each text block, and post-processing the extracted text, such as removing irrelevant characters and splicing segmented text, to obtain text labels.

[0067] Optionally, use computer vision technology to segment the architecture diagram, including: using image segmentation algorithms, such as edge detection, region growing, clustering methods, or deep learning models (such as Mask R-CNN) to identify different components. Different elements, such as rectangular boxes, circular icons, etc., are distinguished based on features such as color, shape, and texture to identify individual components.

[0068] Optionally, labeling the components includes: for each identified component, labeling it using text information obtained through OCR. If a component does not have a directly corresponding text label, it can be considered to infer possible labels based on its position and context.

[0069] Optionally, the relationship types between components are annotated and the connection lines are classified, including: identifying the connection lines between components, which can be completed by a line detection algorithm. Analyzing the components connected at both ends of each connection line to determine the relationship type between them (such as data flow, control flow or dependency, etc.).

[0070] Optionally, the tag information is associated with the architecture diagram ID and stored, including: creating a table in a database to store the architecture diagram information and associated component information.

[0071] The table structure can contain the following fields:

[0072] diagram_id (foreign key, reference to the main table of architecture diagrams);

[0073] component_id (component unique identifier);

[0074] label (component label);

[0075] position (the position coordinates of the component in the diagram);

[0076] relationship_type (relationship type between components);

[0077] connection_lines (description of connection lines).

[0078] On this basis, by adding descriptive text to the architecture diagram and adding labels to the components in the architecture diagram, a basis is provided for extracting text features, image features and relationship features of the architecture diagram.

[0079] In one embodiment, text features, image features, and relationship features of the architecture diagram are extracted based on the description text and the tags, including:

[0080] S210. Based on the language representation model, generate a text vector of the architecture diagram according to the description text, and aggregate the generated text vectors as text features.

[0081] For example, when extracting text features, BERT can be used to generate text vectors and aggregate text vectors; this vector captures the semantic information of the text. For multiple text labels or description texts on an architecture diagram, the corresponding vectors can be aggregated (for example, by averaging or weighted averaging) to form a vector representing the text content of the entire architecture diagram as a text feature. The generated text content vector is associated with the architecture diagram ID and stored to ensure that each text vector can be traced back to its corresponding architecture diagram.

[0082] Optionally, use Hugging Face's transformers library to load a pre-trained BERT model, which can be a BERT base model that can generate word embeddings or sentence embeddings.

[0083] S220. Obtain a feature map of the architecture diagram based on the deep residual learning network model, and select the output of a specific layer as the image feature of the architecture diagram.

[0084] For example, when extracting image features, the image size of the architecture diagram can be adjusted first for normalization, and then the feature map is obtained by forward propagation through a deep residual learning network model (such as ResNet50), and the output of a specific layer is selected as the image feature, which reflects the visual information in the image. The generated image features are stored in association with the architecture diagram ID to ensure that each image feature can be traced back to its corresponding architecture diagram.

[0085] Optionally, use the models module in torchvision to load a pre-trained ResNet50 model. This model has been pre-trained on the ImageNet dataset.

[0086] By loading the pre-trained BERT language model and ResNet50 visual model, a network structure suitable for multimodal learning tasks can be constructed.

[0087] S230, reading the architecture diagram based on a computer vision library and converting it into a black and white image, generating a contour list of the black and white image, each contour in the contour list represents a component in the architecture diagram.

[0088] For example, use OpenCV to perform contour detection to obtain the location information of components in the architecture diagram. Specifically, use OpenCV to read the image of the architecture diagram and convert it into a grayscale image, then convert the grayscale image into a black and white image through binarization, find the contours in the black and white image, and return a list of contours, each of which represents a component in the architecture diagram.

[0089] S240, creating an undirected graph based on the network graph analysis library, and for each contour, adding node attributes to the nodes in the undirected graph, wherein the node attributes include bounding box information of the corresponding contour.

[0090] For example, use NetworkX to create an undirected graph, where nodes represent components and edges represent the connection relationship between components. Specifically, use NetworkX to create an undirected graph G, calculate the bounding box of each contour, and add the bounding box information as a node attribute to the undirected graph G. Each node represents a component in the architecture diagram, and the node attribute stores the bounding box information of the component. The generated relationship features (undirected graph structure and its attributes) are associated and stored with the architecture diagram ID to ensure that each relationship feature can be traced back to its corresponding architecture diagram.

[0091] S250, calculating the Euclidean distance between every two nodes. If the Euclidean distance between any two nodes is less than a threshold, adding an edge between the two nodes, and setting the weight of the added edge to the value of the Euclidean distance between the two nodes.

[0092] In this embodiment, the physical distance between components can be calculated based on the node positions to determine which components are interconnected and assign edge weights. Specifically, the Euclidean distance between every two nodes in the architecture diagram is calculated. If the distance between the two nodes is less than a certain threshold, an edge is added between the two nodes and the weight of the edge is set to the distance value.

[0093] The text features, image features, and relationship features of the architecture diagram obtained through the above process together constitute a comprehensive understanding of an architecture diagram. These three features can provide information about the architecture diagram from different perspectives. Combining these features, a powerful search system can be built that allows users to find related architecture diagrams based on text descriptions, visual features, or component relationships.

[0094] Optionally, when determining the architecture diagram vector of the architecture diagram, a context-based dynamic attention mechanism can be introduced to dynamically adjust the weights of text and image features according to the specific content of the query and the user's intention. For example, if the user's retrieval request contains more information about the visual structure, the weight of the image feature is increased; if the query focuses more on the functional description, the weight of the text feature is increased.

[0095] In one embodiment, the text features and the image features are input into the multimodal model to determine the fusion features of the architecture diagram through the multimodal model, including:

[0096] S310, respectively calculating the attention scores of the text features and the image features, and normalizing the attention scores of the text features and the image features to obtain the attention weights corresponding to the text features and the image features respectively;

[0097] S320, weighting the text features and the image features according to the attention weights corresponding to the text features and the image features respectively, to obtain a fusion feature;

[0098] S330, input the fusion features into the multimodal model, optimize the multimodal model according to the loss function, and obtain the fusion features of the architecture diagram.

[0099] In this embodiment, the attention mechanism can be used to fuse text features and image features. For example, the attention scores of text features and image features are calculated, the scores of the two modalities are combined for softmax normalization, the attention weights of the text and image are obtained respectively, the features are weighted according to the attention weights, and the weighted features are added to obtain the fused features; the attention mechanism dynamically adjusts the contribution of different modalities by calculating the attention score of each modality. It can better capture the interactive relationship between modalities and highlight important modal information.

[0100] Specifically, the process of fusing text features and image features using the attention mechanism includes:

[0101] 1) Calculate the query vector and key vector: Use a linear layer to convert text features and image features into query vectors and key vectors, respectively. It is assumed that text features and image features have the same dimension.

[0102] 2) Calculate the attention score: Calculate the similarity between the query vector and the key vector of the text feature through the dot product operation to obtain the attention score of the text feature.

[0103] 3) Calculate the similarity between the query vector and the key vector of the image feature to obtain the attention score of the image feature.

[0104] 4) Merge attention scores: concatenate the attention scores of text features and image features to form a joint attention score vector.

[0105] 5) Use softmax normalization: Apply the softmax function to the joint attention score vector to convert it into a probability distribution, namely the attention weight. Weight the features according to the attention weight: Use the attention weight to weight the text features and image features to obtain the weighted features. Add the weighted text features and image features to obtain the final fusion features.

[0106] In this embodiment, a fully connected layer can be used as an output layer to input the fused features into a multimodal model for the final classification or regression task. The multimodal model includes feature extraction, feature fusion, and output layers. Optionally, two fully connected layers are used with a ReLU activation function added in between to increase the nonlinear capability of the model.

[0107] In this embodiment, a loss function is defined to optimize the multimodal model; the multimodal model is trained in an end-to-end manner so that the multimodal model can automatically adjust the parameters of feature extraction, attention mechanism, and output layer while optimizing the final task goal.

[0108] For example, the training settings can use random initialization or load pre-trained weights. Select an optimizer, such as Adam optimizer or Stochastic Gradient Descent (SGD). When configuring hyperparameters, set the learning rate, batch size, and regularization coefficient.

[0109] The model training process includes:

[0110] Forward propagation: Feed a batch of data into the model and calculate the predicted output.

[0111] Calculate loss: Calculate the loss using the predicted output and the true label.

[0112] Backpropagation: Backpropagate the loss to calculate the gradient.

[0113] Update parameters: Use the optimizer to update all parameters of the model according to the gradients.

[0114] The above process is repeated until the predetermined number of iterations is reached or other stopping conditions are met.

[0115] Optionally, you can use a validation set to evaluate model performance and monitor overfitting. After training, evaluate the model on the test set to get a true representation of its generalization ability. If the model performs poorly, you may need to try a different model structure or fusion strategy. Adjust hyperparameters such as learning rate, batch size, etc. to find the best configuration. Apply dropout or other regularization techniques to prevent overfitting.

[0116] In one embodiment, the method further comprises:

[0117] S1310, converting the architecture diagram into a computer-processable form, wherein nodes represent entities in the architecture diagram and edges represent relationships between entities;

[0118] S1320, mapping nodes and edges in the architecture graph to a low-dimensional vector space based on graph embedding to obtain an architecture graph vector, and storing the architecture graph vector in a non-relational database or a vector database;

[0119] S1330. Use the vector database to construct an index for the architecture diagram vector.

[0120] In this embodiment, the architecture diagram can be converted into a computer-processable form, where nodes represent entities in the architecture diagram and edges represent relationships between nodes; nodes represent entities in the architecture, such as services, databases, or APIs, etc. Edges represent relationships between nodes, such as call relationships, dependency relationships, etc.;

[0121] Then, the nodes and edges in the architecture graph are mapped to a low-dimensional vector space through graph embedding. These vectors can retain the structural and semantic information of the nodes and edges in the graph. The graph embedding method can use Node2Vec, which generates node sequences through random walks and uses the Word2Vec model to train the architecture graph vector. The architecture graph vector obtained by graph embedding is stored, for example, using a NoSQL database or a vector database.

[0122] In addition, in order to efficiently query and retrieve similar architecture graph vectors, it is necessary to build an index structure. For example, a vector database (such as Faiss) is used to build an index based on the architecture graph vector to perform similarity query and retrieval. On this basis, by vectorizing the architecture graph and using the index structure to efficiently store and query these vectors, it is possible to perform fast similarity search and analysis on large-scale graph data, promoting architecture management and optimization.

[0123] Exemplarily, the similarity between the query vector and each architecture diagram vector can be calculated, and the search results can be sorted according to the similarity score. Specifically, the Euclidean distance method is used to measure the similarity between the query vector and each architecture diagram vector in the architecture diagram, and Faiss is used to build an index. The index is loaded and a similarity query is performed. The similarity score is returned for sorting, and the architecture diagram with the highest score is used as the search result.

[0124] In one embodiment, determining the query vector corresponding to the search request based on natural language processing includes: determining the query vector corresponding to the search request based on a language representation model and conditional random fields (CRF).

[0125] In this embodiment, NLP technology is used to parse the query text, extract key information, identify key entities and intentions, and determine the entities involved in the retrieval request (such as service name, function description, etc.) and the user's intentions (such as finding a specific service, obtaining documents, etc.).

[0126] Specifically, BERT+CRF is used to identify key entities and intents in retrieval requests, understand the specific meaning of the query, extract key information, and convert the preprocessed text into a numerical vector. The word embedding vector generated by BERT is used as input, and then the CRF layer is used for sequence labeling based on the word embedding generated by BERT to identify entities and intents in retrieval requests.

[0127] In one embodiment, the method further includes:

[0128] S100. Preprocess the input query text to obtain a cleaned and standardized input text.

[0129] Among them, the preprocessing process may include:

[0130] Word segmentation: Split the sentence into words or tokens.

[0131] Stop word removal: Remove common meaningless words, such as "de" (of), "shi" (is), etc.

[0132] Stemming or lemmatization: Convert the words to their basic forms.

[0133] Punctuation and special character removal: Clean unnecessary punctuation and special characters.

[0134] Case conversion: Unify the case format of the text.

[0135] The retrieval process of the architecture diagram is illustrated below through an example. The retrieval process of the architecture diagram includes the following steps:

[0136] Step A. Collect architecture diagrams, add descriptive text to the architecture diagrams, and add labels to the components of the architecture diagrams;

[0137] Step B. Preprocess the architecture diagrams, and then extract text features, image features, and relationship features;

[0138] Step C. Input the text features and image features into a pre-trained multimodal model, and let the pre-trained multimodal model learn the association between the two modalities through co-training;

[0139] Step D. Vectorize the architecture diagrams and store the architecture diagram vectors using an index structure;

[0140] Step E. Use NLP technology to understand the query intent, convert it to a query vector, calculate the similarity between the query vector and the architecture diagram vector, and sort the retrieval results according to the similarity score.

[0141] Among them, Step A specifically includes:

[0142] Step A-1. Obtain the architecture diagrams for system functions through manual, scraping technology, or API;

[0143] Step A-2. Store the architecture diagrams for system functions using a database or file system, and assign a unique ID to each architecture diagram;

[0144] Step A-3, generating a description text from the system function-oriented architecture diagram, and storing the description text in association with the corresponding architecture diagram ID;

[0145] Step A-4: Use OCR technology to extract text labels from the architecture diagram; use computer vision technology to segment the architecture diagram, identify each component, add labels to the components, mark the relationship types between components, classify the connection lines, and associate the label information with the architecture diagram ID for storage.

[0146] Wherein, step B specifically includes:

[0147] Step B-1, extract text features: BERT can be used to generate text vectors and aggregate text vectors;

[0148] Step B-2: extracting image features, specifically including:

[0149] Adjust the image size and normalize the image;

[0150] Obtain feature maps through forward propagation of the ResNet50 model and select the output of a specific layer as image features;

[0151] Step B-3: extracting relationship features, specifically including:

[0152] Use OpenCV to read the image and convert it to a grayscale image, convert the grayscale image to a black and white image through binarization, find the contours in the black and white image, and return a list of contours, each of which represents a component in the architecture diagram;

[0153] Use NetworkX to create an undirected graph G. For each contour, calculate its bounding box and add the bounding box information as a node attribute to the undirected graph G. Each node represents a component in the architecture diagram, and the node attribute stores the bounding box information of the component.

[0154] Calculate the Euclidean distance between each node, and if the distance is less than a certain threshold, add an edge between these nodes and set the weight of the edge to the distance value.

[0155] Wherein, step C specifically includes:

[0156] Step C-1: Load the pre-trained BERT language model and the pre-trained ResNet50 visual model.

[0157] Step C-2: Use the attention mechanism to fuse the text features and image features. When implementing, calculate the attention scores of the text features and image features, merge the scores of the two modalities together for softmax normalization, obtain the attention weights of the text and image respectively, weight the features according to the attention weights, and add the weighted features to obtain the fused features;

[0158] Step C-3, using the fully connected layer as the output layer, inputting the fused features into the multimodal model, the multimodal model including feature extraction, feature fusion and output layers;

[0159] Step C-4, defining a loss function to optimize the multimodal model;

[0160] Step C-5: Through end-to-end training, the multimodal model can automatically adjust the parameters of feature extraction, attention mechanism, and output layer while optimizing the final task objectives.

[0161] Wherein, step D specifically comprises:

[0162] Step D-1, converting the architecture diagram into a computer-processable form, wherein nodes represent entities in the architecture diagram and edges represent relationships between nodes;

[0163] Step D-2: Graph embedding maps the nodes and edges in the architecture graph into low-dimensional vector spaces that can preserve the structural and semantic information of the nodes and edges in the graph.

[0164] Step D-3: Store the architecture diagram vector obtained by embedding the diagram, using a NoSQL database or a vector database;

[0165] Step D-4: Use Faiss to build an index based on the architecture graph vector.

[0166] Wherein, step E comprises:

[0167] Step E-1, preprocessing the input query text to obtain cleaned and standardized input text;

[0168] Step E-2: Use BERT+CRF to identify key entities and intents in the query, understand the specific meaning of the query, extract key information, and convert the preprocessed text into a numerical vector.

[0169] Step E-3: Use the Euclidean distance method to measure the similarity between the query vector and the vectors of each architecture diagram in the architecture diagram, load the index built using Faiss and perform a similarity query, return the similarity score for sorting, and use the architecture diagram with the highest score as the retrieval result.

[0170] The architecture diagram retrieval method of the embodiment of the present application, by combining text features, image features and relationship features, and using a pre-trained multimodal model for collaborative training, can more accurately capture the comprehensive information of the architecture diagram, and fully understand the architecture diagram from multiple perspectives. It not only utilizes the visual information of the diagram, but also deeply explores its semantic content and structural relationships, thereby improving the relevance and accuracy of the retrieval results.

[0171] The context-based dynamic attention mechanism allows the weight distribution of different modal features to be automatically adjusted according to the specific content of the user's query. This means that for different query requirements, the model can flexibly highlight more relevant information sources (such as text or images), improving the model's flexibility in responding to diverse query requirements and ensuring that each search can obtain the best matching results. The use of efficient index structures such as Faiss to store and manage a large number of architectural graph vector representations significantly speeds up similarity searches on large-scale data sets, making real-time queries possible and greatly improving user experience and query efficiency.

[0172] By using natural language processing technologies such as BERT+CRF to understand query intent, it can accurately identify key entities and users' real needs, and convert complex architecture diagram information into a form that is easy to understand and search, reducing the requirements for user expertise. It can be applied to retrieval scenarios that contain rich semantic information and complex expressions.

[0173] Using pre-trained multimodal models for collaborative training makes the model more efficient when handling cross-modal tasks. This end-to-end learning method allows the model to learn feature representations directly from raw data without excessive manual feature engineering, which simplifies the development process and enhances the generalization ability of the system;

[0174] In addition, this method helps improve the maintainability and scalability of software projects by effectively managing and retrieving architecture diagrams, better organizing and utilizing existing architecture resources, promoting knowledge sharing, and supporting data analysis in the decision-making process.

[0175] Figure 2 A schematic diagram of the structure of an architecture diagram retrieval device provided in an embodiment of the present application. The architecture diagram retrieval device provided in this embodiment includes:

[0176] A conversion module 210, configured to obtain a search request and determine a query vector corresponding to the search request;

[0177] A calculation module 220, configured to calculate the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to a fusion feature of text features, image features, and relationship features of the architecture diagram;

[0178] The retrieval module 230 is used to determine the retrieval result of the architecture diagram according to the similarity between the query vector and the architecture diagram vector of each architecture diagram.

[0179] The device uses a neural network model to predetermine the coding parameters corresponding to different complexity levels, and selects appropriate target coding parameters according to the complexity level during the encoding process to improve the quality and efficiency of architecture diagram retrieval.

[0180] In one embodiment, the device further comprises:

[0181] An adding module, used for adding description text to each of the architecture diagrams in the architecture diagram library, and adding labels to the components in the architecture diagram;

[0182] An extraction module, configured to extract text features, image features, and relationship features of the architecture diagram according to the description text and the label, wherein the relationship features are represented based on an undirected graph, and each node in the undirected graph corresponds to a label of a component;

[0183] The vector determination module is used to input the text feature and the image feature into a multimodal model to determine the fusion feature of the architecture diagram through the multimodal model.

[0184] In one embodiment, a module is added, specifically for:

[0185] Recognize text in the architecture diagram based on optical character recognition and add descriptive text;

[0186] The architecture diagram is segmented based on computer vision to identify components, and a label is added to each component, and the label is associated with the identifier of the architecture diagram and stored.

[0187] In one embodiment, the extraction module includes:

[0188] A first extraction unit, configured to generate a text vector of the architecture diagram according to the description text based on a language representation model, and aggregate the generated text vectors;

[0189] A second extraction unit is used to obtain a feature map of the architecture diagram based on a deep residual learning network model, and select an output of a specific layer as an image feature of the architecture diagram;

[0190] A third extraction unit is used to read the architecture diagram based on a computer vision library and convert it into a black and white image, and generate a contour list of the black and white image, each contour in the contour list represents a component in the architecture diagram;

[0191] Creating an undirected graph based on a network graph analysis library, and for each of the contours, adding node attributes to nodes in the undirected graph, wherein the node attributes include bounding box information of the corresponding contour;

[0192] The Euclidean distance between every two nodes is calculated. If the Euclidean distance between any two nodes is less than a threshold, an edge is added between the two nodes, and the weight of the added edge is set to the value of the Euclidean distance between the two nodes.

[0193] In one embodiment, the vector determines the module, specifically because:

[0194] Calculating the attention scores of the text feature and the image feature respectively, and normalizing the attention scores of the text feature and the image feature to obtain the attention weights corresponding to the text feature and the image feature respectively;

[0195] According to the attention weights respectively corresponding to the text features and the image features, the text features and the image features are weighted to obtain fusion features;

[0196] The fusion features are input into a multimodal model, and the multimodal model is optimized according to a loss function to obtain the fusion features of the architecture diagram.

[0197] In one embodiment, the device further includes: an index building module, configured to:

[0198] Converting the architecture diagram into a computer-processable form, wherein nodes represent entities in the architecture diagram and edges represent relationships between entities;

[0199] Mapping nodes and edges in the architecture graph to a low-dimensional vector space based on graph embedding to obtain an architecture graph vector, and storing the architecture graph vector in a non-relational database or a vector database;

[0200] An index is constructed for the architecture graph vector using a vector database.

[0201] In one embodiment, the conversion module 210 is specifically used for:

[0202] A query vector corresponding to the retrieval request is determined based on a language representation model and a conditional random field.

[0203] The architecture diagram retrieval device provided in the embodiment of the present application can be used to execute the architecture diagram retrieval method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0204] Figure 3A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, user equipment, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0205] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0206] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, wireless networks.

[0207] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above.

[0208] In some embodiments, the method of the above embodiment may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform any of the above embodiment methods in any other appropriate manner (e.g., by means of firmware).

[0209] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0210] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0211] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0212] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device 10 having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device 10. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0213] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0214] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0215] An embodiment of the present application also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the architecture diagram retrieval method as described in any of the above embodiments.

[0216] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0217] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for retrieving an architecture diagram, characterized in that: include: Obtaining a search request and determining a query vector corresponding to the search request; Calculating the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to a fusion feature of text features, image features, and relationship features of the architecture diagram; According to the similarity between the query vector and the architecture diagram vector of each architecture diagram, a retrieval result of the architecture diagram is determined.

2. The method according to claim 1, characterized in that Also includes: For each of the architecture diagrams in the architecture diagram library, add description text to the architecture diagram and add labels to the components in the architecture diagram; Extracting text features, image features, and relationship features of the architecture diagram according to the description text and the label, wherein the relationship features are represented based on an undirected graph, and each node in the undirected graph corresponds to a label of a component; The text features and the image features are input into a multimodal model to determine fusion features of the architecture diagram through the multimodal model.

3. The method according to claim 2, characterized in that Adding description text to the architecture diagram and adding labels to the components in the architecture diagram includes: Recognize text in the architecture diagram based on optical character recognition and add descriptive text; The architecture diagram is segmented based on computer vision to identify components, and a label is added to each component, and the label is associated with the identifier of the architecture diagram and stored.

4. The method according to claim 2, characterized in that: Extracting text features, image features, and relationship features of the architecture diagram according to the description text and the label includes: Based on the language representation model, generate a text vector of the architecture diagram according to the description text, and aggregate the generated text vectors; Acquire a feature map of the architecture diagram based on a deep residual learning network model, and select an output of a specific layer as an image feature of the architecture diagram; Reading the architecture diagram based on a computer vision library and converting it into a black and white image, generating a contour list of the black and white image, each contour in the contour list representing a component in the architecture diagram; Creating an undirected graph based on a network graph analysis library, and for each of the contours, adding node attributes to nodes in the undirected graph, wherein the node attributes include bounding box information of the corresponding contour; The Euclidean distance between every two nodes is calculated. If the Euclidean distance between any two nodes is less than a threshold, an edge is added between the two nodes, and the weight of the added edge is set to the value of the Euclidean distance between the two nodes.

5. The method according to claim 2, characterized in that: Inputting the text feature and the image feature into a multimodal model to determine the fusion feature of the architecture diagram through the multimodal model includes: Calculating the attention scores of the text feature and the image feature respectively, and normalizing the attention scores of the text feature and the image feature to obtain the attention weights corresponding to the text feature and the image feature respectively; According to the attention weights corresponding to the text features and the image features respectively, the text features and the image features are weighted to obtain fusion features; The fusion features are input into a multimodal model, and the multimodal model is optimized according to a loss function to obtain the fusion features of the architecture diagram.

6. The method according to claim 1, characterized in that Also includes: Converting each of the architecture diagrams into a computer-processable form, wherein nodes represent entities in the architecture diagram and edges represent relationships between entities; Mapping nodes and edges in the architecture graph to a low-dimensional vector space based on graph embedding to obtain an architecture graph vector, and storing the architecture graph vector in a non-relational database or a vector database; An index is constructed for the architecture graph vector using a vector database.

7. An architecture diagram retrieval device, characterized in that: include: A conversion module, used to obtain a search request and determine a query vector corresponding to the search request; A calculation module, used to calculate the similarity between the query vector and the architecture diagram vector of each architecture diagram in the architecture diagram library, wherein the architecture diagram vector of each architecture diagram is determined according to the fusion features of the text features, image features and relationship features of the architecture diagram; The retrieval module is used to determine the retrieval result of the architecture diagram according to the similarity between the query vector and the architecture diagram vector of each architecture diagram.

8. An electronic device, characterized in that: include: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the architecture diagram retrieval method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the architecture diagram retrieval method as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or instructions are executed by a processor, the architecture diagram retrieval method as described in any one of claims 1 to 6 is implemented.

Images