Engineering drawing filing method and system based on image segmentation and image processing

Abstract

The invention provides an engineering drawing filing method and system based on image segmentation and image processing. The method comprises the steps of obtaining an engineering drawing image; performing preprocessing based on the engineering drawing image to obtain a pure-color drawing image; segmenting the pure color drawing image through a semantic segmentation model to obtain image data information of the pure color drawing image; the image data information comprises an image background, a CAD image part, an image label and a handwriting part; performing feature extraction based on the image data information to obtain a complete image signature table block diagram; and performing character recognition based on the image label table block diagram, and archiving the content associated with the engineering drawing according to the recognition result. According to the method, a large number of drawings can be quickly processed, and key archiving information is extracted; the risk of human errors is reduced while a large amount of time and labor are saved; and after archiving, the user can quickly obtain the required information, so that the cooperation and decision-making efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering data management, and relates to an engineering drawing archiving method, and in particular to an engineering drawing archiving method and system based on image segmentation and image processing. Background Art

[0002] Engineering drawings are an indispensable part of engineering construction projects. However, for a long time, the traditional method of archiving drawings often requires a lot of time and human resources to manually classify, organize and store these drawings. The emergence of image segmentation and image processing technology has brought revolutionary changes to the archiving of engineering drawings.

[0003] At present, in the archiving and management of engineering drawings, due to the refinement of engineering drawing classification and the diversity of drawing types, the traditional manual sorting and classification methods are far from meeting the current needs of engineering drawing management. Moreover, in the process of manually sorting and classifying drawings, there is a risk of human error, which brings inconvenience to drawing management; and it is impossible to process a large number of drawings at a very fast speed and extract key archiving information, resulting in a large waste of manpower and time, and reducing efficiency. At the same time, after the engineering drawings are archived, users cannot quickly, effectively and easily search and browse the drawing database by keywords, which makes it even more inconvenient for engineers, designers and other project-related personnel to quickly obtain the required information, which greatly reduces the efficiency of collaboration and decision-making. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a method and system for archiving engineering drawings based on image segmentation and image processing, which is used to solve the problems in the prior art of implementing the engineering drawing archiving management work technology, such as the risk of human error caused by manual management of drawing archiving, the inability to process a large number of drawings at a very fast speed, and the inability to extract key archiving information, resulting in a large amount of manpower and time being wasted, reducing efficiency, and after the engineering drawings are archived, users are unable to quickly and effectively search and browse the drawing database by keywords, thereby reducing the efficiency of collaboration and decision-making.

[0005] To achieve the above-mentioned purpose and other related purposes, in a first aspect, the present application provides an engineering drawing archiving method based on image segmentation and image processing, comprising the following steps: acquiring an engineering drawing image; performing preprocessing based on the engineering drawing image to obtain a pure color drawing image; segmenting the pure color drawing image through a semantic segmentation model to obtain image data information of the pure color drawing image; wherein the image data information includes an image background, a CAD image part, an image label and a handwritten part; performing feature extraction based on the image data information to obtain a complete image label table frame diagram; performing text recognition based on the image label table frame diagram, and archiving the content associated with the engineering drawing according to the recognition result.

[0006] In an implementation of the first aspect, preprocessing is performed based on the engineering drawing image to obtain a pure color drawing image, including the following steps: grayscale processing is performed on the engineering drawing image to obtain an engineering drawing grayscale image; and binarization processing is performed based on the engineering drawing grayscale image to obtain a pure color drawing image.

[0007] In an implementation of the first aspect, the semantic segmentation model includes an encoder, a Transformer encoder and a decoder; the pure color drawing image is segmented by the semantic segmentation model to obtain the image data information of the pure color drawing image, including the following steps: based on the semantic segmentation algorithm, the pure color drawing image is subjected to feature extraction to obtain a plurality of drawing image sequences; the drawing image sequence is subjected to the Transformer encoder to obtain the weighted feature image of the drawing image sequence; the weighted feature image is subjected to pixel-level prediction and segmentation based on the self-attention mechanism to obtain the image data information of the pure color drawing image.

[0008] In an implementation of the first aspect, performing feature extraction on the pure color drawing image based on a semantic segmentation algorithm to obtain a plurality of drawing image sequences includes the following steps: performing feature extraction on the pure color drawing image based on a convolutional neural network to obtain a plurality of feature images at different scales as a drawing image sequence.

[0009] In an implementation of the first aspect, obtaining a weighted feature image of the drawing image sequence through a Transformer encoder includes the following steps: calculating the attention scores between pixels of the feature image of the drawing image sequence and other pixels based on a self-attention mechanism to reduce the sequence length of the drawing image sequence; and obtaining the weighted feature image of the drawing image sequence based on a Mix-FFN module.

[0010] In an implementation of the first aspect, the self-attention mechanism formula is:

[0011]

[0012] Among them, Q represents the calculation of the attention score between the pixel and other pixels; K and V represent the actual weighted aggregation; It is expressed as the weighted value between pixels of Q, K, and T.

[0013] In an implementation manner of the first aspect, the feature extraction includes any one or more combinations of image erosion, image dilation, edge detection and contour drawing.

[0014] In an implementation of the first aspect, performing feature extraction based on the image data information to obtain a complete image signature table frame diagram includes the following steps: processing the image signature in the image data information by image erosion to obtain an image signature table frame diagram with reduced boundaries; performing image dilation based on the image signature table frame diagram with reduced boundaries to obtain an expanded image signature table frame diagram; performing feature extraction on the expanded image signature table frame diagram using an edge detection algorithm to obtain an image signature table frame diagram with an edge area; and performing contour drawing based on the image signature table frame diagram with an edge area to obtain a complete image signature table frame diagram.

[0015] In the second aspect, the present application provides an engineering drawing archiving system based on image segmentation and image processing, including: an acquisition module for acquiring engineering drawing images; a data processing module for performing preprocessing based on the engineering drawing images to obtain pure color drawing images; a segmentation module for segmenting the pure color drawing images through a semantic segmentation model to obtain image data information of the pure color drawing images; wherein the image data information includes image background, CAD image part, image label and handwritten part; a feature extraction module for performing feature extraction based on the image data information to obtain an image label table frame diagram; a recognition module for performing text recognition based on the image label table frame diagram, and archiving the content associated with the engineering drawing according to the recognition result.

[0016] In the last aspect, the present application provides an engineering drawing archiving device based on image segmentation and image processing, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the engineering drawing archiving device based on image segmentation and image processing executes the engineering drawing archiving method based on image segmentation and image processing.

[0017] As described above, the engineering drawing archiving method and system based on image segmentation and image processing of the present invention have the following beneficial effects:

[0018] This application provides an engineering drawing archiving method based on image segmentation and image processing. The self-attention mechanism used to achieve pixel-level prediction can better capture contextual information and improve the accuracy of segmentation results, so that SegFormer can accurately process large-scale images while reducing computing and memory consumption; at the same time, it greatly improves the efficiency and speed of archiving, saves a lot of time and labor, and reduces the risk of human error. Furthermore, after archiving salary drawings, users can easily search and browse drawing data by keywords, so as to quickly obtain the required information and improve the efficiency of collaboration and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a flow chart of an embodiment of an engineering drawing archiving method based on image segmentation and image processing of the present invention.

[0020] Figure 2 Shown is a schematic diagram of an application scenario of an engineering drawing archiving method based on image segmentation and image processing in one embodiment of the present invention.

[0021] Figure 3 It is a schematic flow chart of S12 in the engineering drawing archiving method based on image segmentation and image processing of the present invention.

[0022] Figure 4 It is a schematic flow chart of S13 in the engineering drawing archiving method based on image segmentation and image processing of the present invention.

[0023] Figure 5 It is a schematic flow chart of S14 in the engineering drawing archiving method based on image segmentation and image processing of the present invention.

[0024] Figure 6 It is a schematic diagram showing the principle structure of an engineering drawing archiving system based on image segmentation and image processing in one embodiment of the present invention.

[0025] Figure 7 It is a schematic diagram showing the principle structure of an engineering drawing archiving device based on image segmentation and image processing in one embodiment of the present invention.

[0026] Component number description

[0027] 61 Get Module

[0028] 62 data processing modules

[0029] 63 split module

[0030] 64 Feature Extraction Module

[0031] 65 Identification Module

[0032] 71 processors

[0033] 72 Memory

[0034] Steps S11 to S15 DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0037] The engineering drawing archiving method based on image segmentation and image processing provided in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0038] See also Figure 1 and Figure 2 , respectively showing a flow chart of an engineering drawing archiving method based on image segmentation and image processing of the present invention in one embodiment and a schematic diagram of an application scenario of an engineering drawing archiving method based on image segmentation and image processing of the present invention in one embodiment. Figure 1 and Figure 2 As shown, this embodiment provides an engineering drawing archiving method based on image segmentation and image processing.

[0039] The engineering drawing archiving method based on image segmentation and image processing specifically comprises the following steps:

[0040] S11, obtaining an engineering drawing image.

[0041] In this embodiment, the target drawing is scanned to obtain a scanned engineering drawing image.

[0042] S12, preprocessing the engineering drawing image to obtain a pure color drawing image. Figure 3 , which is a flow chart of S12 in the engineering drawing archiving method based on image segmentation and image processing of the present invention. Figure 3As shown, the S12 comprises the following steps:

[0043] Since the scanned engineering drawings often have a bluish color due to scanner problems, the engineering drawing images are pre-processed first.

[0044] S121, grayscale processing is performed on the engineering drawing image to obtain an engineering drawing grayscale image.

[0045] Specifically, the format conversion of the rectangular frame of the engineering drawing image is because the formats of the sample images obtained by different devices may be quite different. Therefore, it is necessary to convert the sample images of different formats into PNG format.

[0046] The engineering drawing image is converted into a grayscale image using a grayscale calculation formula. The grayscale calculation formula is:

[0047]

[0048] Among them, Gray represents the grayscale value of the grayscale image of the engineering drawing image obtained after grayscale processing; R, G, and B respectively represent the color values ​​of the three primary colors in the PNG format of the engineering drawing image.

[0049] S122, performing binarization processing based on the engineering drawing grayscale image to obtain a pure color drawing image.

[0050] The engineering drawing grayscale image is then binarized to obtain a black and white pure color drawing image of the engineering drawing image.

[0051] Specifically, a global threshold is selected; each pixel in the grayscale image of the engineering drawing is traversed, and the grayscale value of each pixel in the grayscale image of the engineering drawing is compared with the reference threshold; if the pixel grayscale value is greater than the threshold, it is set to white; if the pixel grayscale value is less than or equal to the threshold, it is set to black. All pixel values ​​in the grayscale image of the engineering drawing are compared in sequence until all pixels are processed, and finally a black and white pure color drawing image is obtained. At the same time, the information corresponding to the image is annotated.

[0052] In addition, the following methods may be used to select the threshold, including but not limited to: Otsu method (OTSU), maximum entropy method (MaxEntropy), least square method (LeastSquare), local adaptive threshold method (LocalAdaptiveThreshold) and other methods.

[0053] S13, segmenting the pure color drawing image through a semantic segmentation model to obtain image data information of the pure color drawing image. The image data information includes but is not limited to: image background, CAD image part, image label and handwritten part. Figure 4 , which is a flow chart of S13 in the engineering drawing archiving method based on image segmentation and image processing of the present invention. Figure 4 As shown, the S13 comprises the following steps:

[0054] The semantic segmentation model includes: an encoder, a Transformer encoder and a decoder. The encoder includes: a plurality of convolutional layers, a batch normalization layer, a pooling layer, a flattening layer and a fully connected layer. The convolutional layers are connected to each other through the batch normalization layer and the activation function to extract image features; the pooling layer is connected to the convolutional layer to reduce the image feature dimension to generate image feature data; the flattening layer is used to flatten the image feature data; and the fully connected layer is used to generate image feature encoding data.

[0055] S131, extracting features of the pure color drawing image based on a semantic segmentation algorithm to obtain a plurality of drawing image sequences.

[0056] In this embodiment, feature extraction is performed on the pure color drawing image based on the Segformer semantic segmentation algorithm to obtain feature images at several different scales as a drawing image sequence.

[0057] Specifically, the pure color drawing image is input into the semantic segmentation model, and the encoder of the ResNet convolutional neural network extracts features from the pure color drawing image through a series of convolutional layers and pooling layers. The pure color drawing image is divided into a series of small blocks of the same size, each of which is an image patch; each image patch is regarded as an image drawing sequence. These layers in the convolutional neural network will gradually reduce the size of the feature map but increase the number of channels. Such a design can extract features at different scales, thereby capturing semantic information at different levels. For example: deeper feature maps may be more suitable for capturing the overall semantic information of the image, while shallower feature maps may be more suitable for capturing the details of the image. Therefore, the output of this encoding layer is a series of feature maps, each of which contains features of different scales and levels. These feature maps will be passed to the Transformer encoding to capture long-range dependencies and contextual information between pixels.

[0058] It should be noted that the convolutional neural network model can be ResNet convolutional neural network, or it can be processed by one or more methods such as ResNet, AlexNet, VGG, GoogLeNet, LeNet-5, etc. The semantic segmentation model can also be processed by one or more methods such as FCN, UNet, DeepLab, etc.

[0059] S132, obtaining a weighted feature image of the drawing image sequence through a Transformer encoder.

[0060] In this embodiment, the attention scores between pixels of the feature image of the drawing image sequence and other pixels are calculated based on the self-attention mechanism to reduce the sequence length of the drawing image sequence; and the weighted feature image of the drawing image sequence is obtained based on the Mix-FFN module.

[0061] Specifically, the input drawing image sequence is feature extracted using Transformer encoder layers, which help learn spatial relationships and contextual information in the image. An optional position encoder is used to provide position information to help the model capture the geometric structure in the image.

[0062] Because in the traditional self-attention mechanism, the features of each pixel are linearly projected into three different representation spaces: query Q, key K and value V; query Q represents the attention score between the pixel and other pixels, key K represents and value V represents the actual weighted aggregation. Their feature lengths can be different. When several drawing image sequences are input into the Transformer encoder, Q, K, and V have the same dimensions. The calculation of the three values ​​can obtain the correlation or similarity distribution between each pixel and other pixels, that is, the weighted value. By multiplying the value V, the weighted sum of the pixels is obtained.

[0063] The self-attention mechanism formula is:

[0064]

[0065] Among them, Q represents the calculation of the attention score between the pixel and other pixels; K and V represent the actual weighted aggregation; It is expressed as the weighted value between pixels of Q, K, and T.

[0066] Its complexity is: O(N 2 ), which will limit the speed when processing large-scale high-definition images.

[0067] The efficient self-attention proposed by Segformer uses a reduction factor R to reduce the sequence length:

[0068]

[0069] Therefore, the dimension of K becomes: The computational complexity becomes:

[0070] Next, the Transformer encoder inputs the feature maps of different scales and levels output by the previous encoder, and passes through the Mix-FFN module, which analyzes the position information carried in the feature map and outputs the weighted feature map through an efficient self-attention mechanism for subsequent decoder input.

[0071] Specifically, Mix-FFN is the feed-forward layer used in the SegFormer architecture. Load the trained Mix-FFN model, input feature maps of different scales and levels, perform forward propagation, generate feature images, and then extract feature images from the output of the Mix-FFN model. Visualize the extracted feature images, observe the visualized feature images, and analyze the location information carried in them; you can observe the color distribution, texture changes, and other visualization features in the feature images to determine whether they contain information about the location.

[0072] S133, performing pixel-level prediction and segmentation on the weighted feature image based on a self-attention mechanism to obtain image data information of the pure color drawing image.

[0073] In this embodiment, SegFormer uses a self-attention mechanism to make pixel-level predictions. The self-attention mechanism allows the model to consider the relevance of other pixels in the image when predicting each pixel. This attention mechanism enables SegFormer to better capture contextual information, improve the accuracy of segmentation results, and enable SegFormer to accurately process large-scale images while reducing computation and memory consumption.

[0074] Specifically, the All-MLP decoder in the Segformer semantic segmentation algorithm performs pixel-level prediction on the weighted feature image. MLP (Multi-Layer Perceptron) is a fully connected layer commonly used in deep learning, consisting of multiple neurons, each of which is connected to all neurons in the previous layer to form a fully connected structure. The implementation steps of the All-MLP decoder are as follows:

[0075] (1) The weighted feature image is input into the MLP layer of the All-MLP decoder. At this time, the multi-level features will unify the channel dimension through an MLP layer.

[0076] For example: Assuming the input image size is 224x224 and has 3 channels:

[0077] input_image=torch.randn(1,3,224,224);

[0078] Pass the input image through the encoder to get the feature map:

[0079] input_feature=model.encoder(input_image);

[0080] Here we assume that the input image is a tensor of size [1,3,224,224], where the first dimension represents the batch size, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width. You can adjust the size and number of channels of the input image according to the actual situation.

[0081] (2) These features are then upsampled to one-quarter and concatenated, and then fused through an MLP layer. The last MLP fully connected layer predicts the segmentation mask using the fused features.

[0082] For example, passing the feature map to the decoder for pixel-level prediction and segmentation:

[0083] output_segmentation=model(input_feature).

[0084] Then, the argmax operation is performed on the output result to obtain the pixel-level segmentation result.

[0085] S14, extracting features based on the image data information to obtain a complete image label table diagram. The feature extraction includes any one or more combinations of image erosion, image dilation, edge detection and contour drawing. Figure 5 , which is a flow chart of S14 in the engineering drawing archiving method based on image segmentation and image processing of the present invention. Figure 5 As shown, the S14 comprises the following steps:

[0086] In this embodiment, the pure color drawing image is segmented in step S13, and each engineering drawing image is segmented into four parts, including: image background, CAD image part, image label and handwriting part. In this embodiment, the image label part in the segmented image data information is mainly processed.

[0087] S141, processing the image signature in the image data information by image erosion to obtain a frame diagram of the image signature table with reduced boundaries.

[0088] In this implementation, image erosion technology is used to reduce the boundary of the target object, remove noise and small isolated points from the image label, and make the target object more compact.

[0089] Specifically, a convolution kernel is defined first, with a length of the image length divided by a coefficient (preferably 15), and a width of 1. Using this convolution kernel, the image signature and the convolution kernel are used as inputs, and the image is convolved, and the value of the pixel is the minimum value in the convolution kernel.

[0090] For example: first read the image label to be processed, determine the size and format of the image label; import the library for image label processing and convolution; load the image to be convolved, define the convolution kernel to be convolved, and use the convolution kernel to convolve the image; at this time, after completing the erosion operation on the image, the target object can be gradually "eroded" to reduce the boundary of the image label, remove edge noise, fill holes, etc.; finally, display the processed image to obtain the image label table frame diagram with reduced boundaries for the next step of processing.

[0091] S142, performing image expansion based on the image signature table frame diagram with reduced boundaries to obtain an expanded image signature table frame diagram.

[0092] In this embodiment, an image expansion processing technology is used to expand the boundary of the target object, fill holes and connect discontinuous areas, so that the target object is more complete.

[0093] Specifically, the image tag processed by the image erosion step S141 and the convolution kernel are used as inputs to convolve the image, and the pixel value is the maximum value in the convolution kernel. For example, based on the image tag table frame diagram with reduced boundaries and the defined convolution kernel, the image tag table frame diagram is convolved.

[0094] After completing the above image erosion and image dilation processing, the vertical lines in the original image label will be removed, leaving only all horizontal lines. Then redefine a new convolution kernel with a length of 1 and a width equal to the width of the label area divided by a coefficient (preferably 15). Repeat the erosion and dilation operations to obtain all the vertical lines of the table. Finally, the expanded image label table frame diagram is obtained.

[0095] S143, using an edge detection algorithm to extract features from the expanded image signature table frame diagram to obtain an image signature table frame diagram with an edge area.

[0096] In this implementation, an edge detection algorithm is used to perform gradient calculation, threshold processing, non-maximum suppression and other steps on the expanded image signature table frame diagram to achieve a feature extraction process, thereby obtaining an image signature table frame diagram with an edge area.

[0097] Edge detection is a common image processing technique used to find boundary information in an image. Edges usually represent the outline and important features of the target object.

[0098] Specifically, the expanded image signature table frame diagram is smoothed and filtered based on the Gaussian function to obtain a smoothed image signature table frame diagram; then the local gradient amplitude value and edge direction of each pixel point in the smoothed image signature table frame diagram are calculated; the gradient is non-maximum suppressed; finally, an image signature table frame diagram with an edge area is obtained.

[0099] It should be noted that commonly used edge detection algorithms include but are not limited to: Sobel, Canny, Laplacian and other technologies for processing.

[0100] S144, performing contour drawing based on the image label table frame diagram with the edge area to obtain a complete image label table frame diagram.

[0101] In this implementation, the discrete edge points in the image tag table frame with edge areas are connected to generate a complete outline of the target object. The connection relationship and distance of the pixels are analyzed and processed to generate a smooth contour line.

[0102] Contour drawing is the process of converting edge information into continuous contour lines.

[0103] Specifically, in the above step S143, the edge portion of the image signature table frame is extracted by an edge detection algorithm, and feature enhancement is performed on this basis. For example, morphological operations (such as dilation and erosion), Fourier transform or wavelet transform are used. These methods can improve the visualization effect of the contour, making it easier to identify and draw.

[0104] Then, contour refinement is performed. This can be achieved through various algorithms, such as pixel-based refinement, line-based refinement, or region-based refinement. The goal of the refinement algorithm is to reduce unnecessary contour branches and eliminate redundant details, thereby obtaining a more concise and clear contour.

[0105] Next, the image outline is output. The processed outline can be output in various formats, such as vector format (such as SVG) or bitmap format (such as PNG). It is very important to choose the appropriate output format according to the application requirements. For vector format, the outline is represented as a series of mathematical formulas (such as Bezier curves), which is suitable for lossless enlargement and reduction of graphic design. The bitmap format directly pixelates the outline, which is suitable for image display that requires high resolution and realism.

[0106] Finally, the contour is drawn and the output contour is drawn onto the target image to generate a complete image label table frame diagram. This step can be implemented through a variety of drawing software or programming languages, such as Adobe Illustrator, AutoCAD or OpenGL. During the drawing process, the position, color, thickness and other attributes of the contour need to be adjusted as needed to make it perfectly integrated with the target image.

[0107] In summary, the erosion and dilation operations of the image are used to remove noise and small areas, and then the edge detection algorithm is used to extract the boundary information of the image. Finally, the contour drawing algorithm is used to convert the edge information into continuous contour lines.

[0108] S15, performing text recognition based on the image label table frame diagram, and archiving the contents associated with the engineering drawing according to the recognition result.

[0109] In this embodiment, text recognition is used to identify the filing information in the cells after the table is segmented, such as the drawing number, the approver, etc. According to the identified filing information, each part associated with the drawing is filed.

[0110] Specifically, the PaddleOCR Chinese text extraction model is used for recognition. According to the recognized archive information, the drawing archive information is automatically filled in. For example: the text in the image label table frame diagram is recognized to obtain text data; the text data is filtered and extracted to obtain the keywords in this string of data, such as drawing number, drawing name, scale, engineering design unit, review unit, preparation unit, review date and other information content; finally, according to the recognized archive information, the various parts associated with the drawing are archived.

[0111] The engineering drawing archiving method based on image segmentation and image processing provided by this application can better capture contextual information and improve the accuracy of segmentation results by using the self-attention mechanism to achieve pixel-level prediction, so that SegFormer can accurately process large-scale images while reducing computing and memory consumption; at the same time, it greatly improves the efficiency and speed of archiving, saves a lot of time and labor, and reduces the risk of human error. Furthermore, it also enables users to easily search and browse drawing data by keywords after archiving salary drawings, so as to quickly obtain the required information and improve the efficiency of collaboration and decision-making.

[0112] The protection scope of the engineering drawing archiving method based on image segmentation and image processing described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0113] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 1 The engineering drawing archiving method based on image segmentation and image processing.

[0114] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0115] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0116] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives a computer-readable program instruction from the network, and forwards the computer-readable program instruction for storage in the computer-readable storage medium in each computing / processing device. The computer program instruction for performing the operation of the present application can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, an integrated circuit configuration data, or a source code or object code written in any combination of one or more programming languages, wherein the programming language includes an object-oriented programming language, such as Smalltalk, C++, etc., and a procedural programming language, such as "C" language or similar programming language. Computer readable program instructions can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In certain embodiments, by using the state information of computer readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0117] An embodiment of the present application also provides an engineering drawing archiving system based on image segmentation and image processing. The engineering drawing archiving system based on image segmentation and image processing can implement the engineering drawing archiving method based on image segmentation and image processing described in the present application. However, the implementation device of the engineering drawing archiving method based on image segmentation and image processing described in the present application includes but is not limited to the structure of the engineering drawing archiving system based on image segmentation and image processing listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0118] The engineering drawing archiving system based on image segmentation and image processing provided by this embodiment will be described in detail below with reference to the diagrams.

[0119] This embodiment provides an engineering drawing archiving system based on image segmentation and image processing, including:

[0120] See also Figure 6 , which is a schematic diagram of the principle structure of an engineering drawing archiving system based on image segmentation and image processing in one embodiment of the present invention. Figure 6 As shown, the engineering drawing archiving system based on image segmentation and image processing includes: an acquisition module 61, a data processing module 62, a segmentation module 63, a feature extraction model 64 and a recognition module 65.

[0121] The acquisition module 61 is used to acquire engineering drawing images.

[0122] In this embodiment, the target drawing is scanned to obtain a scanned engineering drawing image.

[0123] The data processing module 62 is used to perform preprocessing based on the engineering drawing image to obtain a pure color drawing image.

[0124] Since the scanned engineering drawings often have a bluish color due to scanner problems, the engineering drawing images are pre-processed first.

[0125] The engineering drawing image is grayscaled to obtain an engineering drawing grayscale image.

[0126] Specifically, the format conversion of the rectangular frame of the engineering drawing image is because the formats of the sample images obtained by different devices may be quite different. Therefore, it is necessary to convert the sample images of different formats into PNG format.

[0127] Binarization is performed based on the engineering drawing grayscale image to obtain a pure color drawing image.

[0128] The engineering drawing grayscale image is then binarized to obtain a black and white pure color drawing image of the engineering drawing image.

[0129] Specifically, a global threshold is selected; each pixel in the grayscale image of the engineering drawing is traversed, and the grayscale value of each pixel in the grayscale image of the engineering drawing is compared with the reference threshold; if the pixel grayscale value is greater than the threshold, it is set to white; if the pixel grayscale value is less than or equal to the threshold, it is set to black. All pixel values ​​in the grayscale image of the engineering drawing are compared in sequence until all pixels are processed, and finally a black and white pure color drawing image is obtained. At the same time, the information corresponding to the image is annotated.

[0130] The segmentation module 63 is used to segment the pure color drawing image through a semantic segmentation model to obtain image data information of the pure color drawing image; wherein the image data information includes image background, CAD image part, image label and handwritten part.

[0131] The semantic segmentation model includes: an encoder, a Transformer encoder and a decoder. The encoder includes: a plurality of convolutional layers, a batch normalization layer, a pooling layer, a flattening layer and a fully connected layer. The convolutional layers are connected to each other through the batch normalization layer and the activation function to extract image features; the pooling layer is connected to the convolutional layer to reduce the image feature dimension to generate image feature data; the flattening layer is used to flatten the image feature data; and the fully connected layer is used to generate image feature encoding data.

[0132] Based on the semantic segmentation algorithm, features of the pure color drawing image are extracted to obtain several drawing image sequences.

[0133] In this embodiment, feature extraction is performed on the pure color drawing image based on the Segformer semantic segmentation algorithm to obtain feature images at several different scales as a drawing image sequence.

[0134] Specifically, the pure color drawing image is input into the semantic segmentation model, and the encoder of the ResNet convolutional neural network extracts features from the pure color drawing image through a series of convolutional layers and pooling layers. The pure color drawing image is divided into a series of small blocks of the same size, each of which is an image patch; each image patch is regarded as an image drawing sequence. These layers in the convolutional neural network will gradually reduce the size of the feature map but increase the number of channels. This design can extract features at different scales, thereby capturing semantic information at different levels.

[0135] The weighted feature image of the drawing image sequence is obtained by using a Transformer encoder for the drawing image sequence.

[0136] In this embodiment, the attention scores between pixels of the feature image of the drawing image sequence and other pixels are calculated based on the self-attention mechanism to reduce the sequence length of the drawing image sequence; and the weighted feature image of the drawing image sequence is obtained based on the Mix-FFN module.

[0137] Specifically, the input drawing image sequence is feature extracted using Transformer encoder layers, which help learn spatial relationships and contextual information in the image. An optional position encoder is used to provide position information to help the model capture the geometric structure in the image.

[0138] Because in the traditional self-attention mechanism, the features of each pixel are linearly projected into three different representation spaces: query Q, key K and value V; query Q represents the attention score between the pixel and other pixels, key K represents and value V represents the actual weighted aggregation. Their feature lengths can be different. When several drawing image sequences are input into the Transformer encoder, Q, K, and V have the same dimensions. The calculation of the three values ​​can obtain the correlation or similarity distribution between each pixel and other pixels, that is, the weighted value. By multiplying the value V, the weighted sum of the pixels is obtained.

[0139] Next, the Transformer encoder inputs the feature maps of different scales and levels output by the previous encoder, and passes through the Mix-FFN module, which analyzes the position information carried in the feature map and outputs the weighted feature map through an efficient self-attention mechanism for subsequent decoder input.

[0140] Specifically, Mix-FFN is the feed-forward layer used in the SegFormer architecture. Load the trained Mix-FFN model, input feature maps of different scales and levels, perform forward propagation, generate feature images, and then extract feature images from the output of the Mix-FFN model. Visualize the extracted feature images, observe the visualized feature images, and analyze the location information carried in them; you can observe the color distribution, texture changes, and other visualization features in the feature images to determine whether they contain information about the location.

[0141] Based on the self-attention mechanism, the weighted feature image is predicted and segmented at the pixel level to obtain the image data information of the pure color drawing image.

[0142] In this embodiment, SegFormer uses a self-attention mechanism to perform pixel-level prediction.

[0143] Specifically, the All-MLP decoder in the Segformer semantic segmentation algorithm performs pixel-level prediction on the weighted feature image.

[0144] The feature extraction module 64 is used to extract features based on the image data information to obtain an image signature table diagram. The feature extraction includes any one or more combinations of image erosion, image expansion, edge detection and contour drawing.

[0145] In this embodiment, by segmenting the pure color drawing image, each engineering drawing image is segmented into four parts, including: image background, CAD image part, image label and handwriting part. In this embodiment, the image label part in the segmented image data information is mainly processed.

[0146] The image signature in the image data information is processed by image erosion to obtain an image signature table frame diagram with reduced boundaries.

[0147] In this implementation, image erosion technology is used to reduce the boundary of the target object, remove noise and small isolated points from the image label, and make the target object more compact.

[0148] Specifically, a convolution kernel is defined first, with a length equal to the image length divided by the coefficient and a width of 1. Using this convolution kernel, the image label and the convolution kernel are used as input, and the image is convolved, and the value of the pixel is the minimum value in the convolution kernel.

[0149] The image is expanded based on the image label table frame diagram with reduced boundaries to obtain an expanded image label table frame diagram.

[0150] In this embodiment, an image expansion processing technology is used to expand the boundary of the target object, fill holes and connect discontinuous areas, so that the target object is more complete.

[0151] Specifically, the image label processed by the image erosion step and the convolution kernel are used as input, the image is convolved, and the value of the pixel point is the maximum value in the convolution kernel.

[0152] After completing the above image erosion and image dilation processing, the vertical lines in the original image label will be cleared, leaving only all horizontal lines. Then redefine a new convolution kernel with a length of 1 and a width equal to the width of the label area divided by the coefficient. Repeat the erosion and dilation operations to obtain all the vertical lines of the table. Finally, the expanded image label table frame diagram is obtained.

[0153] An edge detection algorithm is used to extract features from the expanded image signature table frame diagram to obtain an image signature table frame diagram with an edge area.

[0154] In this implementation, an edge detection algorithm is used to perform gradient calculation, threshold processing, non-maximum suppression and other steps on the expanded image signature table frame diagram to achieve a feature extraction process, thereby obtaining an image signature table frame diagram with an edge area.

[0155] Specifically, the expanded image signature table frame diagram is smoothed and filtered based on the Gaussian function to obtain a smoothed image signature table frame diagram; then the local gradient amplitude value and edge direction of each pixel point in the smoothed image signature table frame diagram are calculated; the gradient is non-maximum suppressed; finally, an image signature table frame diagram with an edge area is obtained.

[0156] Based on the image label table frame diagram with the edge area, contour drawing is performed to obtain a complete image label table frame diagram.

[0157] In this implementation, the discrete edge points in the image tag table frame with edge areas are connected to generate a complete outline of the target object. The connection relationship and distance of the pixels are analyzed and processed to generate a smooth contour line.

[0158] Specifically, the edge part of the image label table frame diagram is extracted through the edge detection algorithm, and feature enhancement is performed on this basis. Then the contour is refined. Next, the image contour is output. The processed contour can be output in various formats. Finally, the contour is drawn, and the output contour is drawn on the target image to generate a complete image label table frame diagram.

[0159] In summary, the erosion and dilation operations of the image are used to remove noise and small areas, and then the edge detection algorithm is used to extract the boundary information of the image. Finally, the contour drawing algorithm is used to convert the edge information into continuous contour lines.

[0160] The recognition module 65 is used to perform text recognition based on the image label table frame diagram, and archive the content associated with the engineering drawing according to the recognition result.

[0161] In this embodiment, text recognition is used to archive information in the cells after the table is segmented. According to the recognized archive information, each part associated with the drawing is archived.

[0162] Specifically, the PaddleOCR Chinese text extraction model is used for recognition. Based on the recognized archive information, the drawing archive information is automatically filled in. The text in the image label table frame is recognized to obtain text data; the text data is filtered and extracted to obtain the keywords in this string of data; finally, based on the recognized archive information, the various parts associated with the drawing are archived.

[0163] The engineering drawing archiving model based on image segmentation and image processing has greatly improved the efficiency of archiving. It can process a large number of drawings at a very fast speed and extract key archiving information; it saves a lot of time and labor and reduces the risk of human error. After archiving, users can easily search and browse the drawing database by keyword. This convenience enables engineers, designers and other project-related personnel to quickly obtain the required information and improve the efficiency of collaboration and decision-making.

[0164] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0165] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0166] See also Figure 7 , which is a schematic diagram showing the principle structure of an engineering drawing archiving device based on image segmentation and image processing in one embodiment of the present invention. Figure 7 As shown, this embodiment provides an engineering drawing archiving device based on image segmentation and image processing, and the engineering drawing archiving device based on image segmentation and image processing includes: a processor 71 and a memory 72; the memory 72 is used to store a computer program; the processor 71 is connected to the memory 72, and is used to execute the computer program stored in the memory 72, so that the engineering drawing archiving device based on image segmentation and image processing performs each step of the engineering drawing archiving method based on image segmentation and image processing as described above.

[0167] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0168] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0169] In summary, the engineering drawing archiving method, system, medium and device based on image segmentation and image processing provided by the present application have the following beneficial effects:

[0170] The engineering drawing archiving method based on image segmentation and image processing provided by this application uses a self-attention mechanism to achieve pixel-level prediction, which can better capture contextual information and improve the accuracy of segmentation results, so that SegFormer can accurately process large-scale images while reducing computing and memory consumption; at the same time, it greatly improves the efficiency and speed of archiving, saves a lot of time and labor, and reduces the risk of human error. Furthermore, it also enables users to easily search and browse drawing data by keywords after archiving salary drawings, so as to quickly obtain the required information and improve the efficiency of collaboration and decision-making.

[0171] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for archiving engineering drawings based on image segmentation and image processing, characterized in that: The following steps are involved: Acquire engineering drawing images; Performing preprocessing based on the engineering drawing image to obtain a pure color drawing image; The pure color drawing image is segmented by a semantic segmentation model to obtain image data information of the pure color drawing image; wherein the image data information includes an image background, a CAD image part, an image label and a handwritten part; Perform feature extraction based on the image data information to obtain a complete image signature table frame diagram; Text recognition is performed based on the image label table frame diagram, and the content associated with the engineering drawing is archived according to the recognition result.

2. The engineering drawing archiving method based on image segmentation and image processing according to claim 1 is characterized in that: Preprocessing the engineering drawing image to obtain a pure color drawing image includes the following steps: Gray-scale the engineering drawing image to obtain an engineering drawing grayscale image; Binarization is performed based on the engineering drawing grayscale image to obtain a pure color drawing image.

3. The engineering drawing archiving method based on image segmentation and image processing according to claim 1 is characterized in that: The semantic segmentation model includes an encoder, a Transformer encoder and a decoder; segmenting the pure color drawing image by the semantic segmentation model to obtain the image data information of the pure color drawing image includes the following steps: Extracting features of the pure color drawing image based on a semantic segmentation algorithm to obtain a plurality of drawing image sequences; Obtaining a weighted feature image of the drawing image sequence through a Transformer encoder for the drawing image sequence; Based on the self-attention mechanism, the weighted feature image is predicted and segmented at the pixel level to obtain the image data information of the pure color drawing image.

4. The engineering drawing archiving method based on image segmentation and image processing according to claim 3 is characterized in that: Extracting features from the pure color drawing image based on a semantic segmentation algorithm to obtain a plurality of drawing image sequences includes the following steps: Based on the convolutional neural network, features of the pure color drawing image are extracted to obtain feature images at several different scales as a drawing image sequence.

5. The engineering drawing archiving method based on image segmentation and image processing according to claim 3 is characterized in that: Obtaining a weighted feature image of the drawing image sequence through a Transformer encoder comprises the following steps: Calculating the attention scores between pixels of the feature image of the drawing image sequence and other pixels based on the self-attention mechanism to reduce the sequence length of the drawing image sequence; Based on the Mix-FFN module, a weighted feature image of the drawing image sequence is obtained.

6. The engineering drawing archiving method based on image segmentation and image processing according to claim 5 is characterized in that: The self-attention mechanism formula is: Among them, Q represents the calculation of the attention score between the pixel and other pixels; K and V represent the actual weighted aggregation; It is expressed as the weighted value between pixels of Q, K, and T.

7. The engineering drawing archiving method based on image segmentation and image processing according to claim 1 is characterized in that: The feature extraction includes any one or more combinations of image erosion, image dilation, edge detection and contour drawing.

8. The engineering drawing archiving method based on image segmentation and image processing according to claim 1 or 7, characterized in that: Based on the image data information, feature extraction is performed to obtain a complete image label table frame diagram, which includes the following steps: Processing the image signature in the image data information by image erosion to obtain a frame diagram of the image signature table with reduced boundaries; Perform image expansion based on the image label table frame diagram with reduced boundaries to obtain an expanded image label table frame diagram; Using an edge detection algorithm to extract features from the expanded image signature table frame diagram to obtain an image signature table frame diagram with an edge area; Based on the image label table frame diagram with the edge area, contour drawing is performed to obtain a complete image label table frame diagram.

9. An engineering drawing archiving system based on image segmentation and image processing, characterized in that: include: An acquisition module, used for acquiring engineering drawing images; A data processing module, used for preprocessing the engineering drawing image to obtain a pure color drawing image; A segmentation module, used to segment the pure color drawing image through a semantic segmentation model to obtain image data information of the pure color drawing image; wherein the image data information includes an image background, a CAD image part, an image label and a handwritten part; A feature extraction module, used to extract features based on the image data information to obtain an image signature table frame diagram; The recognition module is used to perform text recognition based on the image label table frame diagram, and archive the content associated with the engineering drawing according to the recognition result.

10. An engineering drawing archiving device based on image segmentation and image processing, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the engineering drawing archiving device based on image segmentation and image processing can execute the engineering drawing archiving method based on image segmentation and image processing as described in any one of claims 1 to 8.

Images