A semantic representation learning method and system for engineering drawings based on cross-modal contrastive learning

Abstract

This invention provides a method and system for learning semantic representations of engineering drawings based on cross-modal contrastive learning. The method includes: encoding the engineering drawings to extract global visual features; encoding the engineering semantic text to extract engineering semantic information; generating image embedding vectors for the engineering drawings using an image projection head; generating engineering semantic text embedding vectors using a text projection head; mapping the image embedding vectors and engineering semantic text embedding vectors to the same shared semantic space through cross-modal alignment training; and performing image search based on the engineering drawings and text search based on the engineering semantic text within the shared semantic space. By constructing a joint representation space between engineering drawings and engineering semantic descriptions, this invention supports image search based on engineering drawings and text search based on engineering semantic text, enabling historical drawing retrieval and knowledge reuse oriented towards the manufacturing semantics of engineering drawings.

Description

Technical Field

[0001] This invention belongs to the field of industrial artificial intelligence and intelligent manufacturing technology, specifically relating to a semantic representation learning method and system for engineering drawings based on cross-modal contrastive learning. Background Technology

[0002] Two-dimensional engineering drawings are currently widely used in fields such as machinery manufacturing, equipment manufacturing, aerospace and industrial design, and are important basic data for product design, process planning, quality inspection and manufacturing.

[0003] Existing intelligent methods for engineering drawings mainly rely on: OCR character recognition; rule-based parameter parsing; image feature matching; and general visual language models.

[0004] However, the above method has the following problems: (1) Difficulty in understanding complex engineering semantics: For example, engineering information such as dimensions, tolerances, roughness, geometric tolerances, and technical requirements has strong professional semantics, and relying solely on character recognition cannot establish the true engineering meaning.

[0005] (2) Lack of a unified semantic space between text and graphics: Existing methods cannot effectively establish the relationship between “drawing structural features” and “process description”, “processing semantics”, and “manufacturing knowledge”.

[0006] (3) It is difficult to support manufacturing reuse scenarios.

[0007] (4) Traditional image retrieval methods mainly rely on visual similarity, which cannot meet the needs of industrial scenarios such as reuse of similar parts, reference of process routes, matching of historical cases, and tracing of quality problems.

[0008] Therefore, a cross-modal representation learning method is needed that can jointly model the visual features of engineering drawings and engineering semantic information. Summary of the Invention

[0009] The purpose of this invention is to provide a semantic representation learning method and system for engineering drawings based on cross-modal contrastive learning. By constructing a joint representation space between engineering drawings and engineering semantic descriptions, the invention enables semantic understanding and cross-modal association modeling of engineering drawings, supporting industrial scenarios such as drawing semantic retrieval, similar part matching, process semantic classification, manufacturing experience reuse, drawing clustering, and process recommendation.

[0010] This invention provides a semantic representation learning method for engineering drawings based on cross-modal contrastive learning, comprising the following steps: Step 1: Encode the engineering drawings to extract global visual features from the drawings, and encode the engineering semantic text to extract engineering semantic information from the text; Step 2: Generate an engineering drawing image embedding vector using the image projection head, and generate an engineering semantic text embedding vector using the text projection head; Step 3: Through cross-modal alignment training of images and text, map the image embedding vectors of engineering drawings and the embedding vectors of engineering semantic text to the same shared semantic space; Step 4: Perform image search based on engineering drawings and text search based on engineering semantic text within the shared semantic space.

[0011] Furthermore, the global visual features mentioned in step 1 include local engineering semantic features and process-related visual features; the engineering semantic information includes parameter semantics, process semantics, manufacturing knowledge, and technical requirements.

[0012] Furthermore, in step 1, RoPE is used for drawing location coding and text location coding.

[0013] Furthermore, after encoding and extracting global visual features from the engineering drawings in step 1, process semantic classification and process category prediction are also performed.

[0014] Furthermore, a process semantic classification head is used to classify the engineering drawings into multi-label processes, and a process category prediction head is used to predict the processing process category corresponding to the engineering drawings, and the process category prediction results and multi-label probabilities are output.

[0015] Furthermore, the graph search described in step 4 includes: Obtain engineering drawings, generate corresponding image embedding vectors, and retrieve historical drawings with similar structural features, process types, key parameters, and manufacturing semantics from the historical drawing embedding vector library.

[0016] Furthermore, the text search image mentioned in step 4 includes: Obtain the semantic text of the project, generate the corresponding text embedding vector, and directly calculate the similarity with the embedding vector of the historical drawings to obtain the semantically matched historical drawings.

[0017] Furthermore, step 4 also includes: By employing an embedded vector semantic recall mechanism, similar historical drawings are recalled, and similar reasons and manufacturing experience recommendations are generated by combining part type, key structure, dimensional parameters, tolerance grade, surface roughness, technical requirements, process category, historical process route and quality problem records.

[0018] This invention also provides a semantic representation learning system for engineering drawings based on cross-modal contrastive learning, comprising: An image encoder is used to encode engineering drawings and extract global visual features from them. A text encoder is used to encode engineering semantic text and extract engineering semantic information from the text; Image projection head, used to generate image embedding vectors for engineering drawings; A text projection head is used to generate engineering semantic text embedding vectors; The image-text cross-modal alignment training module is used to map the embedding vectors of engineering drawing images and the embedding vectors of engineering semantic text to the same shared semantic space, and to perform image search based on engineering drawings and text search based on engineering semantic text within the same shared semantic space.

[0019] Furthermore, the system also includes: The process semantic classification header is used to classify engineering drawings according to their processes. The process category prediction header is used to predict the processing process category corresponding to the engineering drawings.

[0020] Compared with the prior art, the beneficial effects of the present invention are: 1) Improve the accuracy of historical drawing retrieval: This invention utilizes cross-modal semantic alignment of text and images to map drawing structure, process category, key parameters, and engineering semantics into a unified vector space, enabling more accurate discovery of historical drawings with the same or similar manufacturing semantics. It solves the problems of traditional drawing retrieval, which typically relies on filenames, drawing numbers, manual tags, or simple visual similarity. These issues often lead to retrieval failures due to non-standard filenames, difficulties in searching due to complex drawing number rules, missed detections due to incomplete manual tags, and the identification of drawings with similar appearances but different manufacturing semantics, or similar manufacturing semantics but significant differences in appearance.

[0021] 2) Supports both image search and text search simultaneously, improving usability: This invention supports users initiating searches in different ways: uploading a drawing to search for similar historical drawings; users do not need to accurately remember drawing numbers, file names, or directory locations, but only need to upload the drawing or enter a natural language engineering description to retrieve relevant historical drawings. This lowers the barrier to historical drawing retrieval and improves the convenience for R&D, process, and quality personnel in using historical drawing assets.

[0022] 3) Improve the efficiency of reusing similar parts and reduce redundant design: This invention utilizes semantic retrieval to help R&D personnel quickly discover existing drawings of similar parts, especially those with similar structures, processes, parameters, or quality risks. This facilitates the reuse of existing designs, reduces redundant modeling and drawing, improves the reuse efficiency of standard parts, series parts, and similar parts, and shortens the design cycle of new products.

[0023] 4) Promote the accumulation and reuse of manufacturing knowledge: The present invention associates engineering drawings, process routes, etc., supports multimodal retrieval (text-to-image search, image-to-image search), making historical drawings no longer just static files but becoming retrievable, interpretable, and reusable manufacturing knowledge assets. This helps enterprises form a drawing knowledge base, a similar parts experience base, a process route knowledge base, a quality problem case base, and a parameter recommendation knowledge base, thereby enhancing the enterprise's engineering knowledge management and manufacturing experience reuse capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the engineering drawing semantic representation learning method based on cross-modal contrast learning of the present invention; Figure 2 It is a structural block diagram of the image end of the present invention; Figure 3 It is a structural block diagram of the text end of the present invention; Figure 4 It is a schematic diagram of the cross-modal unified semantic space of the present invention; Figure 5 It is a schematic diagram of the application of the process semantic classification head of the present invention; Figure 6 It is a structural diagram of the process semantic classification head of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The present invention will be described in detail below in conjunction with the embodiments shown in the drawings. However, it should be noted that these embodiments are not limitations on the present invention. Any equivalent transformation or substitution in terms of function, method, or structure made by those of ordinary skill in the art based on these embodiments shall fall within the protection scope of the present invention.

[0026] Refer Figure 1 As shown, this embodiment provides an engineering drawing semantic representation learning method based on cross-modal contrast learning, including the following steps: Step S1, encode the engineering drawings to extract the global visual features in the drawings, and encode the engineering semantic text to extract the engineering semantic information in the text; Step S2, generate an engineering drawing image embedding vector through an image projection head, and generate an engineering semantic text embedding vector through a text projection head; Step S3, through cross-modal alignment training of text and image, map the engineering drawing image embedding vector and the engineering semantic text embedding vector to the same shared semantic space; Step S4, perform image-to-image search based on the engineering drawings and text-to-image search based on the engineering semantic text within the shared semantic space.

[0027] In this embodiment, after encoding and extracting global visual features from the engineering drawings in step S1, process semantic classification and process category prediction are also performed. A process semantic classification head is used to perform multi-label process classification on the engineering drawings, and a process category prediction head is used to predict the corresponding processing technology category of the engineering drawings, outputting the process category prediction results and multi-label probabilities. This embodiment also discloses a system for implementing the above method, including: An image encoder is used to encode engineering drawings and extract global visual features from them. A text encoder is used to encode engineering semantic text and extract engineering semantic information from the text; The process semantic classification header is used to classify engineering drawings according to their processes. The process category prediction head is used to predict the processing process category corresponding to the engineering drawings. Image projection head, used to generate image embedding vectors for engineering drawings; A text projection head is used to generate engineering semantic text embedding vectors; The image-text cross-modal alignment training module is used to map the embedding vectors of engineering drawing images and the embedding vectors of engineering semantic text to the same shared semantic space, and to perform image search based on engineering drawings and text search based on engineering semantic text within the same shared semantic space.

[0028] This invention proposes a text-image embedding and alignment method oriented towards the manufacturing semantics of engineering drawings. Unlike general image-text alignment methods, the image-side input of this invention is a two-dimensional engineering drawing, and the text-side input is engineering semantic text, including dimensional parameter descriptions, tolerance and fit descriptions, surface roughness descriptions, process requirements, technical requirements, manufacturing knowledge, and engineering Q&A. The system extracts the drawing layout, symbol annotations, dimensional relationships, structural features, and manufacturing semantic information from the engineering drawing through an image encoder, and extracts parametric semantics, process semantics, and manufacturing knowledge from the engineering semantic text through a text encoder. Through cross-modal alignment training, the image embedding vector of the engineering drawing and the embedding vector of the engineering semantic text are mapped to the same shared semantic space, enabling the engineering elements in the drawing to establish a correspondence with the engineering semantic descriptions in the text. Thus, the system can support image search based on engineering drawings and text search based on engineering semantic text, realizing historical drawing retrieval and knowledge reuse oriented towards the manufacturing semantics of engineering drawings.

[0029] This invention further proposes a multi-task semantic learning mechanism for engineering drawings that integrates process category prediction. Following the global drawing features output by the image encoder, a process semantic classification head, a process category prediction head, and an image projection head are simultaneously set. The process semantic classification head is used for process classification, and the process category prediction head is used to predict the processing technology category corresponding to the engineering drawing. The image projection head is used to generate the engineering drawing image embedding vector and participates in cross-modal alignment between the engineering drawing image and the engineering semantic text.

[0030] The process categories include, but are not limited to, turning, milling, drilling, boring, reaming, grinding, welding, heat treatment, and surface treatment. Because drawings may involve multiple machining processes, the process category prediction head can use a Sigmoid function to output multi-label prediction results.

[0031] By introducing a process category prediction task, the model can explicitly learn the manufacturing process semantics in engineering drawings. This enables the generated drawing embedding vectors to not only express the visual structure and annotation semantics of the drawings, but also to further express the processing methods, process constraints, and manufacturing knowledge, thereby improving the accuracy of engineering drawing semantic retrieval, process similarity judgment, and reuse of historical process experience.

[0032] The present invention will now be described in further detail.

[0033] I. Detailed Description of Image Branch The image processing unit is used for visual semantic modeling of engineering drawings, extracting local engineering semantic features and process-related visual features from the drawings, and generating a unified visual semantic vector for the engineering drawings. The overall structure is as follows: Figure 2 As shown.

[0034] (1) Input of Full Drawing The input is a complete 2D engineering drawing, sourced from PDF, PNG, JPG, and other formats, all uniformly processed into JPG format. The drawing content includes dimensions, tolerances, geometric tolerances, surface roughness, thread symbols, welding symbols, technical requirements, part structural outline, and view layout.

[0035] (2) Image Preprocessing Because engineering drawings are characterized by high resolution, dense small targets, numerous fine line structures, and a mixture of text and geometry, specialized preprocessing is required. ① Image normalization: unify the input drawings to a fixed size, such as 1024×1024.

[0036] ② Data augmentation, including random scaling, slight rotation, brightness perturbation, contrast adjustment, and local masking, is used to enhance generalization ability.

[0037] (3) Image Encoder Image encoders are used to extract visual semantic features from engineering drawings.

[0038] The encoder can employ one or more of the following architectures: Vision Transformer (ViT), ConvNeXt, ResNet, and a hybrid CNN-Transformer architecture. This implementation uses RoPE's Vision Transformer (ViT).

[0039] 3.1 Patch Embedding Enter the engineering drawings: I ∈ R^(H × W × C) Where: H is the image height; W is the image width; C is the number of image channels.

[0040] The input drawings are divided into: N = HW / P² number of image patches.

[0041] Where: P is the image patch size; N is the number of image patches.

[0042] Each image patch is mapped as: x_i ∈ R^D; where D is the embedding dimension.

[0043] Then, the patch token (the vector itself obtained after embedding) is obtained through linear projection: h_i = W_p x_i; Where: W_p is the learnable projection matrix.

[0044] To enhance the ability to model spatial structural relationships in engineering drawings, this embodiment uses Rotary Position Embedding (RoPE) for position encoding.

[0045] Unlike traditional positional encoding: z_i = x_i + p_i, this embodiment does not directly add a fixed position vector to the token. Instead, it rotates and transforms the token features according to the spatial position of the token.

[0046] set up: The two-dimensional spatial position of the i-th token is: (x_i, y_i) The corresponding rotation angle is: θ_i but: Before attention calculation, the Query and Key features are rotated and encoded separately: Q_i' = R(θ_i)Q_i K_i' = R(θ_i)K_i Where: Q_i is the Query feature; K_i is the Key feature; R(θ_i) is the rotation matrix.

[0047] For a two-dimensional feature vector [u, v], the rotated feature representation is as follows: [ u cosθ_i - v sinθ_i , u sinθ_i + v cosθ_i ]; The rotation angle is determined by the token's spatial position: θ_i = pos_i / 10000^(2k / D); Where: pos_i represents the token position; k represents the embedding dimension index; D represents the total embedding dimension.

[0048] By using Rotary Position Embedding, the model can automatically learn the relative spatial relationships between patches, the relationship between parameter regions and structural regions, the relationship between engineering symbols and dimension lines, and the relationship between GD&T and the baseline during the attention calculation process, thereby enhancing the spatial semantic modeling capability of engineering drawings.

[0049] Subsequently, the rotated Query and Key are input into the multi-head self-attention module: Attention(Q,K,V) = Softmax( Q'K'^T / √D )V; Ultimately, this achieves the visual feature modeling of engineering drawings with relative spatial awareness.

[0050] 3.2 Transformer Encoding Patch token input to a multi-layer Transformer Encoder: z^(l+1) = TransformerBlock(z^l); Each layer includes: Multi-head Self Attention; Feed Forward Network; LayerNorm; Residual Connection.

[0051] (4)Global Feature Extraction The final global visual features are obtained: f_I(I) ∈ R^D; This feature also includes: part structure, parameter layout, local engineering semantics, and process vision mode.

[0052] (5) Projection Head To map text features to a unified semantic space, a projection head is added: v = W_I f_I(I); Where: W_I is the learnable projection matrix; v is the final image embedding.

[0053] Then normalization is performed: v_hat = v / ||v||; The final result is the Image Embedding, which is used for cross-modal alignment.

[0054] II. Text Branch (Detailed Description) The text layer is used to encode engineering semantics, extracting engineering semantic information such as parameter semantics, process semantics, manufacturing knowledge, and technical requirements. The overall structure is as follows: Figure 3 As shown: (1) Engineering Text Input The input text is engineering semantic text.

[0055] include: Parameter semantics, for example: "10mm diameter, H7 mating hole" Technological semantics, for example: The main machining methods are turning and drilling. Creating knowledge, for example: "Critical surfaces require grinding." Technical requirements, such as: "Unmarked chamfer C1" Engineering Q&A, for example: Does this part contain high-precision mating holes? (2) Text Tokenization Tokenization of text: T = [w1, w2, ..., wn] Where: w_i is the token.

[0056] Use the SentencePiece method.

[0057] (3)Text Encoder with Text RoPE The text encoder adopts the Transformer Encoder architecture, specifically BERT, RoBERTa, CLIPText Encoder, or an industrial-domain pre-trained language model. In a preferred embodiment, RoBERTa-wwm-ext is used as the initial model, and further pre-training is performed based on engineering drawing parameter descriptions, process descriptions, technical requirement texts, and engineering question-and-answer data to enable it to model engineering symbols, process terms, and manufacturing semantics.

[0058] (3.1) Text Embedding (using Text RoPE) Token embedding: e_i = E(w_i) Where: w_i is the i-th text token; E(·) is the token embeddinglookup table (a part of RoBERTa-wwm-ext).

[0059] e_i ∈ R^D.

[0060] Unlike traditional text position encoding h_i = e_i + p_i, this embodiment uses Rotary PositionEmbedding (RoPE) for text position modeling. Instead of directly adding position vectors to the token embedding, the token position information is encoded into the Query and Key features during the self-attention calculation process.

[0061] Specifically, the text token embedding is linearly transformed to obtain the Query, Key, and Value: Q_i = W_Q e_i; K_i = W_K e_i; V_i = W_V e_i; Where: W_Q, W_K, W_V are learnable parameter matrices; Q_i, K_i, V_i represent the Query, Key, and Value features of the i-th token, respectively.

[0062] Let the position of the i-th token in the text sequence be pos_i. Calculate the rotation angle based on this position: θ_(i,k) = pos_i / 10000^(2k / D); Where: k represents the index of the feature dimension pair; D represents the hidden dimension; θ_(i,k) represents the rotation angle of the i-th token on the k-th dimension pair.

[0063] For any two-dimensional feature pair in the Query or Key: [a_(2k), a_(2k+1)] Rotation transformation based on position angle: a'_(2k) = a_(2k) cosθ_(i,k) - a_(2k+1) sinθ_(i,k); a'_(2k+1) = a_(2k) sinθ_(i,k) + a_(2k+1) cosθ_(i,k); This yields the Query and Key containing location information: Q_i' = RoPE(Q_i, pos_i); K_i' = RoPE(K_i, pos_i); Then, self-attention is calculated using the rotated Query and Key: Attention(Q,K,V) = Softmax( Q'K'^T / √d )V; Where: d is the dimension of a single attention head; Q' and K' are the Query and Key after RoPE processing; V is not rotated.

[0064] Text RoPE enables text encoders to naturally perceive the relative positional relationships of tokens during attention computation, thereby enhancing their ability to model sequential relationships, logical relationships, and long-distance dependencies in engineering semantic text.

[0065] For example, for engineering semantic text: "Rough machining followed by grinding" and: "Grinding first, then rough machining" Both contain the same words, but the process order is different. By using Text RoPE, the model can better distinguish the semantic differences in the process corresponding to different token orders.

[0066] The Text RoPE in this embodiment is particularly suitable for the sequence of process steps, parameter modification relationships, technical requirement context, tolerance and processing method association, and long text engineering descriptions in the semantic description of engineering drawings.

[0067] (3.2) Transformer Encoding Input multi-level Transformer: h^(l+1) = TransformerBlock(h^l) Learning: the logical relationships of processes; the connections between manufacturing knowledge.

[0068] (3.3)Global Semantic Feature The final text semantic features are obtained as follows: f_T(T) ∈ R^D It includes: process semantics; manufacturing rules; engineering knowledge.

[0069] (4) Projection Head Through projection head: t = W_T f_T(T) Where: W_T is the projection matrix; t is the final text embedding.

[0070] Normalization: t_hat = t / ||t||; The final result is a Text Embedding, which is used for cross-modal semantic alignment.

[0071] III. Cross-modal unified semantic space like Figure 4 As shown, the image embedding v_hat and the text embedding t_hat are ultimately mapped into a unified semantic space.

[0072] The distance between matching image and text pairs decreases: cos(v_hat, t_hat) ↑ The distance between non-matching image / text pairs increases: cos(v_hat, t_hat) ↓ This enables semantic understanding of engineering drawings, retrieval of similar parts, semantic matching of processes, and reuse of manufacturing knowledge.

[0073] IV. Process Classification Head The Process Classification Head is used to enhance the model's ability to model the process semantics of engineering drawings.

[0074] In this embodiment, the process semantic classification head is positioned after the global visual features output by the image encoder. The overall structure is as follows: Figure 5 As shown: Compared to directly using Image Embedding for process classification, using Global Feature for process semantic supervision can retain more engineering structural features, parameter layout features, and local manufacturing semantic information.

[0075] The specific structure of the process semantic classification head is as follows: Figure 6 As shown: Where: D is the dimension of global image features; C is the number of process categories.

[0076] In some implementation methods: Because a single drawing may correspond to multiple process categories, a Sigmoid multi-label classification method is used. This embodiment preferably adopts the multi-label process classification method.

[0077] For example: The same part drawing may include multiple processes such as turning, drilling, grinding, heat treatment, and surface treatment.

[0078] Suppose the global visual features output by the image encoder are: g_i = f_I(I_i); The process classification header output is: o_i = W_2 σ(W_1 g_i + b_1) + b_2 in: W_1 and W_2 are learnable weight matrices; b_1 and b_2 are bias terms; σ(·) represents a nonlinear activation function.

[0079] The multi-label process classification output is as follows: ŷ_i = sigmoid(o_i); The process semantic supervision loss function is defined as: L_process = -(1 / N) Σ Σ [ y_(i,c)log(ŷ_(i,c)) + (1-y_(i,c))log(1-ŷ_(i,c)) ]; in: y_(i,c) represents the true label of the i-th drawing in the c-th process category; ŷ_(i,c) represents the model's predicted probability; N represents the batch size.

[0080] By introducing a process semantic classification head, the model not only learns the visual structural features of engineering drawings, but also the semantic features of manufacturing processes, thereby enhancing the cross-modal semantic representation capability of engineering drawings.

[0081] The process categories in this embodiment include, but are not limited to: turning, milling, drilling, grinding, tapping, welding, heat treatment, surface treatment, sheet metal, casting and other industrial manufacturing processes.

[0082] V. Learning Loss Function for Cross-Modal Representation of Engineering Drawings This embodiment provides a complete loss function definition for the cross-modal representation learning method of engineering drawings, including global image-text alignment, parameter region-level semantic alignment, process semantic supervision, engineering rule constraints, and multi-granularity consistency constraints.

[0083] 1. Global image-text alignment loss L_clip Suppose there are N image-text pairs in a batch of samples: (I_i, T_i), i = 1, ..., N; Image encoder output: v_i = f_I(I_i); Text encoder output: t_i = f_T(T_i); Image-text similarity is defined as: s_ij = (v_i^T t_j) / τ; Image-to-text loss: L_i2t = -(1 / N) Σ log( exp(s_ii) / Σ exp(s_ij) ); Loss of text to image conversion: L_t2i = -(1 / N) Σ log( exp(s_ii) / Σ exp(s_ji) ); Final global image-text alignment loss: L_clip = 0.5 × (L_i2t + L_t2i).

[0084] 2. Process semantic classification loss L_process Used to enable embeddings to learn process semantics.

[0085] Let the number of process categories be C.

[0086] Drawing process label: y_i ∈ {1,2,...,C}; Classified output: ŷ_i = softmax(Wv_i + b); Cross-entropy loss: L_process = -(1 / N)ΣΣ 1(y_i=c) log(ŷ_(i,c)); If multi-label classification is used, then the BCE loss is employed. L_process = -(1 / N) ΣΣ [ y_(i,c)log(ŷ_(i,c)) + (1-y_(i,c))log(1-ŷ_(i,c)) ] 3. Final total loss function L = L_clip + λ2·L_process.

[0087] in: L_clip: Image-text semantic alignment loss; L_process: Process semantic supervision loss.

[0088] VI. Examples of Training Samples The following are examples of training samples based on multi-label process semantic classification in cross-modal representation learning of engineering drawings.

[0089] During training, process labels are used for L_process, and drawings and engineering semantic text are used for L_clip.

[0090] Sample 1: Flange-type parts Input drawing: 2D engineering drawing of flange Engineering semantic text: This part is a flange-type machined component, which includes a center hole, evenly distributed mounting holes, and partially countersunk holes. It mainly involves turning, drilling, milling, and surface treatment processes.

[0091] Process label: { "turning":1, "drilling":1, "milling":1, "grinding":0, "tapping":0, "heat treatment":0, "surface treatment":1}.

[0092] Sample 2: Shaft-type parts Input drawing: Stepped shaft 2D engineering drawing Engineering semantic text: This part is a shaft-type rotating component, which includes multi-level steps, outer cylindrical mating surfaces, and keyway structures. It mainly involves turning, milling, grinding, and heat treatment processes.

[0093] Process label: { "turning":1, "drilling":0, "milling":1, "grinding":1, "tapping":0, "heat treatment":1, "surface treatment":0}.

[0094] Sample 3: Bracket with threaded hole Input drawing: 2D engineering drawing of bracket-type parts Engineering semantic text: This part is a bracket-type machined part, which includes a mounting surface, multiple through holes and threaded holes, and mainly involves milling, drilling, tapping and surface treatment processes.

[0095] Process label: { "turning":0, "drilling":1, "milling":1, "grinding":0, "tapping":1, "heat treatment":0, "surface treatment":1}.

[0096] Sample 4: High-precision mating hole parts Input drawing: Drawing of plate-type parts containing H7 mating holes. Engineering semantic text: This part includes high-precision mating holes and positioning reference surfaces. The hole diameter tolerance grade is H7, and the critical surface roughness requirements are high. It mainly involves milling, drilling, reaming and grinding processes.

[0097] Process label: { "turning":0, "drilling":1, "milling":1, "reaming":1, "grinding":1, "tapping":0, "heat treatment":0, "surface treatment":0}.

[0098] Sample 5: Welded structural components Input drawing: 2D engineering drawing of welding support Engineering semantic text: This part is a welded structural component, which includes a plate welded structure, weld symbols and assembly reference holes, and mainly involves material cutting, welding, drilling and surface treatment processes.

[0099] Process label: { "Blanking":1, "Welding":1, "Drilling":1, "Milling":0, "Turn":0, "Grinding":0, "Heat Treatment":0, "Surface Treatment":1}.

[0100] VII. Multimodal Search Because this embodiment introduces a cross-modal alignment mechanism between image and text during the training phase, the drawing embedding generated by the image encoder and the text embedding generated by the text encoder are mapped to the same shared semantic space. Therefore, the system supports not only image search retrieval with drawings as query input, but also text search image retrieval with engineering semantic text as query input.

[0101] In the image search scenario, a user uploads an engineering drawing, the system generates an image embedding of the drawing, and searches the historical drawing embedding library for historical drawings with similar structural features, process types, key parameters, and manufacturing semantics.

[0102] In text search image scenarios, users input engineering semantic text, such as "find flange parts drawings containing H7 mating holes and Ra1.6 sealing surfaces" or "find parts drawings whose main processing methods are turning and drilling". The system generates corresponding text embeddings and directly calculates the similarity with historical drawing embeddings, thereby returning historical drawings that match the semantics.

[0103] Through the above methods, the present invention can break through the traditional retrieval methods based on file name, drawing number or manual tags, realize image search and text search based on manufacturing semantics, improve the efficiency of historical drawing reuse, and promote the accumulation and reuse of process knowledge, processing experience and quality problem cases.

[0104] Image search: Input a drawing, the system generates a drawing embedding, and searches for similar drawings in the historical drawing embedding library.

[0105] In an image search scenario, a user uploads a new engineering drawing (I_query). The system generates the image embedding of the drawing using an image encoder and an image projection head. v_query = f_I(I_query) Then, semantically similar drawings are retrieved from the historical drawing embedding database: sim(v_query, v_j) = (v_query^T v_j) / (||v_query|| ||v_j||); Where v_query represents the image embedding corresponding to the user-uploaded drawing, and v_j represents the image embedding of the j-th historical drawing.

[0106] Because this invention trains image embeddings across modalities through image-text alignment, the image embeddings not only contain visual layout information of the drawing, but also represent high-level semantic information such as part structure, process type, key parameters, and manufacturing semantics. Therefore, the system can retrieve historical drawings with similar manufacturing semantics based on the drawing embeddings.

[0107] The system returns the Top-K candidate historical drawings based on similarity.

[0108] For example, a user uploads a new drawing of a flange-type part, which includes a center hole, evenly distributed mounting holes, sealing end faces, and several finishing requirements. The system generates an image embedding for this drawing and searches the historical drawing embedding database to return semantically similar historical drawings.

[0109] Text search for images: Input a piece of engineering semantic text, the system generates a text embedding, and retrieves semantically matching drawings from the historical drawing embedding library.

[0110] In a text search image scenario, the user inputs a piece of engineering semantic text T_query. The system generates the text embedding of this text using a text encoder and a text projection head: t_query = f_T(T_query) Then, semantically matching drawings are retrieved from the historical drawing embedding database: sim(t_query, v_j) = (t_query^T v_j) / (||t_query|| ||v_j||) Where v_j represents the image embedding of the j-th historical drawing, and t_query represents the text embedding corresponding to the user-input engineering semantic text.

[0111] Because this invention maps text embeddings and image embeddings to the same shared semantic space through cross-modal alignment training, the semantic similarity between text queries and historical drawings can be directly calculated.

[0112] The system returns the Top-K candidate historical drawings based on similarity.

[0113] For example, the user inputs: Find drawings of flange-type parts that include H7 mating holes and Ra1.6 sealing surfaces.

[0114] The system generates a text embedding for this text and searches the historical drawing embedding database, returning historical drawings with similar semantics, such as: Historical flanges with center holes marked H7; End cap parts with a sealing surface roughness requirement of Ra1.6; It also includes historical part drawings of high-precision mating holes and precision-machined sealing surfaces.

[0115] This invention constructs image encoding and text encoding branches for engineering drawings. Through cross-modal alignment training, it maps the drawing image embedding and the engineering semantic text embedding into the same shared semantic space, enabling the system to simultaneously support image search and retrieval based on drawings and text search and image retrieval based on engineering semantic text. Furthermore, this invention employs an embedding semantic recall mechanism. While recalling similar historical drawings, it combines part type, key structure, dimensional parameters, tolerance grade, surface roughness, technical requirements, process category, historical process route, and quality problem records to generate similarity reasons and manufacturing experience recommendations.

[0116] Compared to traditional engineering drawing retrieval methods based on filenames, drawing numbers, manual tags, or purely visual similarity, this invention can discover historically similar drawings from a manufacturing semantic perspective. It supports users in uploading drawings or inputting engineering semantic text for retrieval, improving the accuracy and flexibility of historical drawing searches. Simultaneously, this invention can output reasons for similarity, recommended process routes, historical processing parameters, quality problem cases, and inspection focus items, which helps improve the reuse efficiency of similar parts, reduce redundant design, accelerate process planning, and promote the accumulation and reuse of enterprise manufacturing knowledge and process experience. Specific technical effects are as follows: 1) Improve the accuracy of historical drawing retrieval: This invention utilizes cross-modal semantic alignment of text and images to map drawing structure, process category, key parameters, and engineering semantics into a unified vector space, enabling more accurate discovery of historical drawings with the same or similar manufacturing semantics. It solves the problems of traditional drawing retrieval, which typically relies on filenames, drawing numbers, manual tags, or simple visual similarity. These issues often lead to retrieval failures due to non-standard filenames, difficulties in searching due to complex drawing number rules, missed detections due to incomplete manual tags, and the identification of drawings with similar appearances but different manufacturing semantics, or similar manufacturing semantics but significant differences in appearance.

[0117] 2) Supports both image search and text search simultaneously, improving usability: This invention supports users initiating searches in different ways: uploading a drawing to search for similar historical drawings; users do not need to accurately remember drawing numbers, file names, or directory locations, but only need to upload the drawing or enter a natural language engineering description to retrieve relevant historical drawings. This lowers the barrier to historical drawing retrieval and improves the convenience for R&D, process, and quality personnel in using historical drawing assets.

[0118] 3) Improve the efficiency of reusing similar parts and reduce redundant design: This invention utilizes semantic retrieval to help R&D personnel quickly discover existing drawings of similar parts, especially those with similar structures, processes, parameters, or quality risks. This facilitates the reuse of existing designs, reduces redundant modeling and drawing, improves the reuse efficiency of standard parts, series parts, and similar parts, and shortens the design cycle of new products.

[0119] 4) Promote the accumulation and reuse of manufacturing knowledge: This invention links engineering drawings and process routes, supporting multimodal retrieval (text-to-image and image-to-image search). This transforms historical drawings from static files into searchable, interpretable, and reusable manufacturing knowledge assets. This helps enterprises build knowledge bases for drawings, similar parts, process routes, quality issues, and parameters, thereby enhancing their engineering knowledge management and manufacturing experience reuse capabilities.

[0120] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

Claims

1. A semantic representation learning method for engineering drawings based on cross-modal contrastive learning, characterized in that, Includes the following steps: Step 1: Encode the engineering drawings to extract global visual features from the drawings, and encode the engineering semantic text to extract engineering semantic information from the text; Step 2: Generate an engineering drawing image embedding vector using the image projection head, and generate an engineering semantic text embedding vector using the text projection head; Step 3: Through cross-modal alignment training of images and text, map the image embedding vectors of engineering drawings and the embedding vectors of engineering semantic text to the same shared semantic space; Step 4: Perform image search based on engineering drawings and text search based on engineering semantic text within the shared semantic space.

2. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 1, characterized in that, The global visual features mentioned in step 1 include local engineering semantic features and process-related visual features; The engineering semantic information includes parameter semantics, process semantics, manufacturing knowledge, and technical requirements.

3. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 1, characterized in that, In step 1, RoPE is used to encode the drawing location and the text location.

4. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 1, characterized in that, In step 1, after encoding the engineering drawings and extracting the global visual features from the drawings, process semantic classification and process category prediction are also performed.

5. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 4, characterized in that, A process semantic classification head is used to classify engineering drawings into multi-label processes, and a process category prediction head is used to predict the corresponding processing process category of the engineering drawings. The process category prediction results and multi-label probabilities are then output.

6. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 1, characterized in that, The image search described in step 4 includes: Obtain engineering drawings, generate corresponding image embedding vectors, and retrieve historical drawings with similar structural features, process types, key parameters, and manufacturing semantics from the historical drawing embedding vector library.

7. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 6, characterized in that, The text search image mentioned in step 4 includes: Obtain the semantic text of the project, generate the corresponding text embedding vector, and directly calculate the similarity with the embedding vector of the historical drawings to obtain the semantically matched historical drawings.

8. The semantic representation learning method for engineering drawings based on cross-modal contrastive learning according to claim 7, characterized in that, Step 4 also includes: By employing an embedded vector semantic recall mechanism, similar historical drawings are recalled, and similar reasons and manufacturing experience recommendations are generated by combining part type, key structure, dimensional parameters, tolerance grade, surface roughness, technical requirements, process category, historical process route and quality problem records.

9. A semantic representation learning system for engineering drawings based on cross-modal contrastive learning, characterized in that, include: An image encoder is used to encode engineering drawings and extract global visual features from them. A text encoder is used to encode engineering semantic text and extract engineering semantic information from the text; Image projection head, used to generate image embedding vectors for engineering drawings; A text projection head is used to generate engineering semantic text embedding vectors; The image-text cross-modal alignment training module is used to map the embedding vectors of engineering drawing images and the embedding vectors of engineering semantic text to the same shared semantic space, and to perform image search based on engineering drawings and text search based on engineering semantic text within the same shared semantic space.

10. The engineering drawing semantic representation learning system based on cross-modal contrastive learning according to claim 9, characterized in that, Also includes: The process semantic classification header is used to classify engineering drawings according to their processes. The process category prediction header is used to predict the processing process category corresponding to the engineering drawings.

Images