Picture information extraction method and device and computer readable storage medium

Abstract

The present application relates to the technical field of image processing, in particular to a picture information extraction method, a picture information extraction device and a computer readable storage medium. The picture information extraction method provided in the technical scheme of the present application comprises: a text detection step, detecting text elements in a picture through a character detection model, performing text detection marking, and segmenting the picture according to the coordinates of the detected text elements to obtain a text picture; a text recognition step, recognizing text elements in the text picture through a character recognition model, performing text recognition marking on the text elements, and performing special text recognition marking on specific character elements in the text elements; an information extraction step, acquiring information corresponding to the text elements in the picture through a multi-modal model according to the picture, the content, coordinates and marking information of the text elements in the picture, and a preset special character rule.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image information extraction method, an image information extraction device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development and maturation of computer-related hardware and software technologies, the medium for computer-aided design of industrial products such as parts is shifting from 2D drawings to 3D models. However, a large number of 2D part drawings still exist in the industry in the form of paper or scanned images. These 2D part drawings not only contain view information that constrains the shape of the part, but also a large amount of dimensional, feature, design, and production information presented in the form of text annotations. In current technologies, when building 3D models based on 2D part drawings, contour recognition is typically used to extract the shape view information from the 2D part drawings, and the approximate structure of the part is reproduced through 3D reconstruction. However, since machine reconstruction often calculates the actual dimensions based on the pixels and scale of the drawing, and the multiple views of industrial product drawings are not necessarily drawn according to the true scale, there will be a certain degree of reconstruction discrepancy. At the same time, the lack of information about processes and manufacturing precision in the drawings limits the scope of business that can be supported.

[0003] Alternatively, existing technologies may employ manual identification of text and / or symbol annotations in images to assist in 3D model reconstruction. However, manual identification relies on professionals with background knowledge in mechanical engineering or engineering graphics, which presents problems such as high labor costs, low reading efficiency, and unreliable reading accuracy. Summary of the Invention

[0004] To address the above problems, this invention provides an image information extraction method, an image information extraction device, and a computer-readable storage medium. The image information extraction method can identify different text information such as position and orientation, as well as various forms of symbols and annotations in part engineering drawings, and extract the above information by combining view information and text information, and then realize applications such as 3D modeling based on the extracted and identified information.

[0005] The present invention provides an image information extraction method, comprising: a text detection step, wherein text elements in an image are detected by a text detection model, and the image is segmented according to the coordinates of the detected text elements to obtain a text image; a text recognition step, wherein text elements in the text image are recognized by a text recognition model, and the text elements are labeled; and an information extraction step, wherein information corresponding to the text elements in the image is obtained by a multimodal model according to the image, the content, coordinates, and label information of the text elements in the image, as well as the rules of preset special characters in the text elements.

[0006] According to the technical solution of the present invention, since text or annotation information in an image may be set in different formats and orientations at different locations in the image, the position of text elements in the image is first identified by a text detection model, and the image is segmented according to coordinates. After the image is segmented into multiple text image units, the text content is identified by a text recognition model. Furthermore, the image, the content, coordinates, and label information of the text elements in the image, as well as the rules of preset special characters in the text elements, are input into a multimodal model. Through the intelligent recognition of image information and text information by the multimodal model, even when the text information in the image is chaotic and complex, with different positions and orientations and containing multiple special characters, efficient, accurate, and automated extraction of text content and target entities within the text content can still be achieved. On this basis, more comprehensive and accurate information can be obtained from the image, enabling further information data processing and calculation, such as automated and accurate construction and size correction of three-dimensional models based on the extracted information.

[0007] According to the technical solution of the present invention, in addition to the view information that defines the structure of the part, the part engineering drawing also contains a large amount of text information such as the model information and size information of the part. The above text information involves a large number of feature symbols representing the illustrated structure and annotation information of the size of the part in the drawing, and there is text information arranged vertically or in other non-directional angles. The above image information extraction method can accurately identify and extract the above text information.

[0008] Preferably, in the technical solution of the present invention, the image information extraction method further detects and extracts the dimension annotation graphics in the image during the text detection step.

[0009] According to the technical solution of the present invention, image information and text information in the image are extracted simultaneously as a reference to obtain more comprehensive and accurate data.

[0010] Preferably, in the technical solution of the present invention, the text detection model in the image information extraction method is composed of a residual network and a densely connected convolutional network, and the text recognition model is an encoder-decoder structure.

[0011] According to the technical solution of this invention, a residual network (ResNet) and a densely connected convolutional network (DenseNet) are combined. ResNet can inherit and reuse shallow features multiple times, while DenseNet focuses on using previous features to obtain new features. By passing the results of both through upsampling layers and a series of long and short skip connections, the semantic gap is reduced. Finally, the features are input into a grid map and the model is fused to correct the segmentation results, thereby achieving less computation and better training speed. The encoding-decoding method of the text recognition model abandons the traditional preprocessing step of labeling, making the entire recognition process simpler and more efficient.

[0012] Preferably, in the information extraction step of the image information extraction method provided by the technical solution of the present invention, a sample file is generated by comparing the coordinates of the text element and the coordinates of the label corresponding to the tag.

[0013] According to the technical solution of the present invention, by comparing the coordinates of the text elements obtained in the above steps with the coordinates of the labeling boxes corresponding to the labels, it is possible to determine which text content contains labels, thereby generating a training sample file. This training sample file contains the coordinates of the text, the text content, and the text labels. Furthermore, special characters can be retrieved through regular expression matching or other character matching methods, thus obtaining the feature symbols and dimension annotations in the part engineering drawing.

[0014] Preferably, in the technical solution of the present invention, the image information extraction method further includes a stitching size step, which includes a labeling form step, in which the corresponding size labeling form is determined according to the shape type of each unit in the model corresponding to the image; a labeling line step, in which labeling lines are found in the image and stored according to the size labeling form; and a size matching step, in which size information is obtained according to the shape type and the corresponding size labeling rules, and the size of each unit is adjusted.

[0015] According to the technical solution of the present invention, different units in the image correspond to different annotation types, such as double arrows, broken line arrows, etc. After confirming the size annotation form corresponding to each unit in the 3D model corresponding to the image, the corresponding size annotation content is read, and different extraction methods are provided for each size annotation form. Finally, the size of the unit is adjusted according to the read information, which is convenient, efficient and accurate.

[0016] Furthermore, in the technical solution of the present invention, the image information extraction method further includes a size accuracy acquisition step, which involves acquiring the size information corresponding to all special characters, finding the size with the most decimal places and its corresponding number of decimal places, where the number of decimal places is the size accuracy; and / or, acquiring the accuracy information in the text elements, and obtaining the size accuracy based on the accuracy information.

[0017] According to the technical solution of the present invention, the size accuracy of the view in the image is confirmed based on the accuracy information read from the text information of the image itself. The operation is convenient and the content is accurate. The above two accuracy confirmation methods can be mutually verified or selected as the best option.

[0018] Preferably, in the technical solution of the present invention, the image information extraction method further includes a preprocessing step, which includes: a specification adjustment step, adjusting the original image by one or more combinations of cropping, scaling, and rotation; a binarization step, performing binarization processing on the image obtained after the specification adjustment step; and a sharpness adjustment step, performing multiple rounds of erosion and / or dilation processing on the image obtained after the binarization step to obtain a preprocessed image.

[0019] According to the technical solution of the present invention, the above preprocessing steps first standardize the specifications of the original image, then adjust the image after specification adjustment to black and white through a binarization step, and then reduce or eliminate problems such as rough edges, noise, and unclearness in the original image through multiple rounds of erosion and / or dilation processing, so as to ensure the clarity and smoothness of the image for subsequent information extraction.

[0020] Furthermore, in the technical solution of the present invention, the image information extraction method also includes an angle adjustment step, which uses a convolutional neural network algorithm model to identify and adjust the orientation of the preprocessed image.

[0021] According to the technical solution of the present invention, the image obtained after the above preprocessing steps may have multiple angles relative to the preset direction, such as 0°, 90°, 180°, 270°, etc. The angle recognition problem can be transformed into a classification problem through the convolutional neural network algorithm model, which can easily calculate the angle of the current image and rotate and adjust the image angle so that the images for subsequent information extraction are in a uniform angle direction, thereby improving the efficiency and accuracy of subsequent information extraction.

[0022] The technical solution of the present invention also provides an image information extraction device, including a text detection module that detects text elements in an image and segments the image to obtain a text image based on the coordinates of the text elements; a text recognition module that identifies text elements in the text image using a text recognition model and labels the text elements; and an information extraction module that obtains information corresponding to the text elements in the image using a multimodal model based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules for pre-set special characters in the text elements.

[0023] The technical solution of this invention provides an image information extraction device that can identify the position of text elements in an image through a text detection module and segment the image according to coordinates. After segmenting the image into multiple text image units, the text content is recognized by a text recognition module. Furthermore, the image, the content, coordinates, and label information of the text elements, as well as the rules for pre-defined special characters in the text elements, are input into a multimodal model. Through the intelligent recognition of image and text information by the multimodal model, efficient, accurate, and automated text content extraction can be achieved even when the text information in the image is chaotic, complex, and contains various special characters. Based on this, the intelligent recognition of image and text information allows for the acquisition of more comprehensive and accurate information from the image, enabling further information data processing and calculation, such as the automated and accurate construction and size correction of a 3D model based on the extracted information.

[0024] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described image information extraction method. Attached Figure Description

[0025] Figure 1 This is a flowchart of an image information extraction method provided in an embodiment of the present invention;

[0026] Figure 2 This is a part engineering drawing provided in an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of a preferred image information extraction method provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of an image information extraction device provided in an embodiment of the present invention.

[0029] Figure labeling: 1-Text detection module, 2-Text recognition module, 3-Information extraction module. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0031] Figure 1This is a flowchart of an image information extraction method provided in an embodiment of the present invention.

[0032] like Figure 1 As shown, in an embodiment of the present invention, an image information extraction method is provided, including: a text detection step S1, which detects text elements in an image using a text detection model and segments the image to obtain a text image based on the coordinates of the detected text elements; a text recognition step S2, which identifies text elements in the text image using a text recognition model and labels the text elements; and an information extraction step S3, which obtains information corresponding to the text elements in the image using a multimodal model based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules for pre-defined special characters in the text elements.

[0033] In embodiments of the present invention, the images in the image information extraction method can be engineering drawings, design drawings, etc. of parts or other components, and are not limited herein.

[0034] Figure 2 This is a part engineering drawing provided in an embodiment of the present invention.

[0035] like Figure 2 As shown, in this embodiment, the image is taken as an engineering drawing of the part. In the prior art, according to... Figure 2 When creating 3D models from engineering drawings, only the shape view information in the image is typically referenced. However, part engineering drawings, in addition to the view information defining the part's structure, also contain a large amount of text information such as part model information, dimension information, technical information, and drawing information. All of this text information collectively constitutes a description of the part engineering drawing, but the format, font, arrangement, and content of the text in each group of text information are different. Furthermore, part engineering drawings and other drawings often contain many special characters and multi-angled slanted text that are difficult to read.

[0036] In an embodiment of the present invention, a two-stage OCR model is designed, including a text detection model and a text recognition model.

[0037] First, the text recognition step S1 is performed. Based on the hidden features of the image obtained by using convolutional layers, a residual network (ResNet) and a densely connected convolutional network (DenseNet) are combined. The residual network (ResNet) can inherit and reuse shallow features multiple times, while the densely connected convolutional network (DenseNet) focuses on using previous features to obtain new features. By passing the results of both through upsampling layers and a series of long and short skip connections, the semantic gap is reduced. Finally, the features are input into the grid map and the model is fused to correct the segmentation results, so as to achieve less computation and better training speed.

[0038] After detecting the text elements in the picture through the above-mentioned text detection model, the picture is segmented into text pictures according to coordinates for further text recognition.

[0039] In the text recognition step S2, the text recognition model adopts the encoder-decoder (encoding-decoding) method, and at the same time adopts the image transformer and the text transformer. In the encoder encoding stage, it adopts the model structure of the ViT mode (Vision Transformer, visual self-attention model), cuts the picture into fixed-size patches and performs position encoding, and obtains the intermediate hidden state after encoding through the multi-head attention layer and the feed-forward network of the image. The hidden state needs to go through the decoding attention layer, the multi-head self-attention layer and the feed-forward network when performing text decoding. This encoding-decoding method abandons the preprocessing of traditional markers and is more simple and efficient.

[0040] In the text recognition step S2, it is also necessary to label the corresponding labels for the text elements. Among them, ordinary text can obtain the label of normal text, such as the character "标", but special characters require special labels, and the positions with labels are framed in the picture by a marking box ( Figure 2 the dotted box in). It is worth mentioning that in Figure 2 only some labels and marking boxes are marked exemplarily, and the forms of labels and marking boxes can be adjusted by the designers themselves in actual applications.

[0041] Specifically, in this embodiment, the characteristic symbols representing the graphic structure in the text information are defined as special characters. Special characters can be mechanical engineering drawing symbols, as shown in the following table:

[0042]

[0043]

[0044] Only several common characteristic symbols in engineering drawings are illustrated by way of enumeration in the above table. In the actual application of the present invention, special characters are usually characters that are not common in general text content or have special meanings in themselves. The types and quantities of special characters and their corresponding meanings can be defined by the user himself, and are not limited here.

[0045] It is worth mentioning that there may also be some special marked texts in the part engineering drawing, such as This dimension format, which defines the upper and lower limits of the dimension, conveys information at the same level. Since this format is frequently used in drawings, it was not labeled as a single block according to the standard text recognition rules, but rather as D<>+0.05<>+0.01. Other characters with significantly different sizes were labeled in different blocks according to standard annotation methods.

[0046] In the text detection step S1 and text recognition step S2 described above, the content and coordinates of text elements in the image can be obtained through the two-stage OCR model, including the mapping of special characters.

[0047] Next, after completing the above text recognition and image labeling process, the information extraction step S3 is performed. Based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules of the preset special characters in the text elements, the information corresponding to the text elements in the image is obtained through a multimodal model.

[0048] In an embodiment of the present invention, the coordinates of the text element and the coordinates of the tag box corresponding to the tag are compared to determine which texts have tags, i.e., texts containing special characters. The final training sample is a JSON file, which contains the coordinates of the text, the text content, and the tag corresponding to the text.

[0049] Specifically, during the pre-training phase, the multimodal model primarily integrates image and text information by using a text-image association task (predicting whether an image fragment corresponding to uncovered text is covered). In terms of structure, the main structure of the multimodal model references the transformer. The transformer's structure involves the encoded data first passing through a word embedding layer and a position encoding layer, then sequentially through an encoder and a decoder. Each encoder contains a multi-head attention layer and a feedforward layer, and each decoder contains two multi-head attention layers and a feedforward layer. Finally, the output passes through a linear layer and a softmax layer.

[0050] Specifically, the multimodal model first needs to scale the image to a fixed size and encode the image and text separately. The encoding is the sum of the initial embedding, 1D positional encoding and 2D positional encoding.

[0051] The initial embedding of the text uses the embedding word vectors of the Roberta pre-trained model, which can reflect the similarity relationship between texts by calculating the similarity between vectors;

[0052] One-dimensional position encoding is a one-dimensional position embedding obtained by passing the sequential number of each bounding box (bbox) through a one-dimensional position encoding layer. Different markers in the same bounding box share the same one-dimensional position encoding number, that is, the one-dimensional position encoding represents the sequential number of the bounding box.

[0053] Two-dimensional position encoding is the result of the coordinate embedding layer obtained by the specific coordinates of each bounding box (bbox). The original coordinate information includes the top left corner (e.g., point value coordinates x1, y1), bottom right corner (e.g., point value coordinates x2, y2), width (e.g., width value 100), and height (e.g., height value 200) of the bounding box.

[0054] The initial image encoding adopts the ViT approach, which divides the image into several patches and performs flattening and embedding layer mapping to achieve dimensionality reduction. The one-dimensional and two-dimensional positional encoding methods can refer to the text corresponding positional encoding method. The patches at the image level only need to be treated as bounding boxes. That is, the one-dimensional positional encoding represents the sequential number of the patch, and the two-dimensional positional encoding is the result obtained by each specific coordinate through the coordinate embedding layer. The original coordinate information includes the top left corner, bottom right corner, width, and height of the patch.

[0055] The main structure after the encoding layer is based on the transformer, with slight modifications to the self-attention layer in the multi-head attention layer. When calculating the self-attention layer, the Q and K matrices need to be multiplied to obtain the attention score of each token to other tokens, and the V matrix is ​​weighted by the score. That is, Q represents a matrix composed of a set of queries, K represents a matrix composed of a set of keys, and V represents a matrix composed of a set of values.

[0056] In an embodiment of the present invention, the multimodal model adds a bias term to the score after the QK matrix operation:

[0057]

[0058] The last three terms represent the position offset terms obtained from one-dimensional and two-dimensional position information, thereby improving the calculation accuracy and efficiency through the above formula.

[0059] Finally, the V matrix is ​​weighted by taking an exponential average of the biased scores:

[0060]

[0061] The above model yields image patches in vector form and vector representations of each token. Classifying these token vector representations allows us to obtain the label representation for each token. Finally, combining the tokens with the original text recognition data yields the label corresponding to the bounding box (bbox). Specifically, if the text belongs to the entity to be extracted, the multimodal model needs to additionally label the bbox with entity tags, such as "geometric tolerances".

[0062] Through the preceding steps S1 and S2, the text in the drawing that needs to be extracted, describing information such as surface roughness, parallelism, and perpendicularity dimensions, can be labeled. After obtaining the text content, location, and labels, the aforementioned multimodal model recalls special characters (such as ⊥) through regular expression matching, and finds the text information (such as 0.02A) that needs to be combined with the special characters and the special requirements mapped into textual expressions according to rules.

[0063] Preferably, in an embodiment of the present invention, the image information extraction method further detects and extracts dimension annotation graphics, such as leader lines and arrow lines, in the text detection step.

[0064] Figure 3 This is a flowchart of a preferred image information extraction method provided in an embodiment of the present invention.

[0065] like Figure 3 As shown, in an embodiment of the present invention, the image information extraction method further includes a stitching size step S4, which includes a labeling form step, which determines the corresponding size labeling form based on the shape type of each unit in the image; a labeling line step, which finds and stores labeling lines in the image based on the size labeling form; and a size matching step, which adjusts the size of each unit based on the labeling lines and the corresponding size labels.

[0066] Among them, shape type refers to the actual shape and structure of the part, such as face, chamfer, round hole, etc.; dimensioning form refers to the form in which dimensions are marked in the drawing, including: leader line dimensioning, where the leader line is horizontal or vertical, such as... Figure 2 The numbers 29, 30, etc., are marked in the middle; arrow lines are used for marking, where the arrow lines are broken lines, such as... Figure 2 The letters M, R, d, etc., are used to label these areas. Leader lines and arrow lines are collectively referred to as label lines.

[0067] The correspondence between shape type and dimension annotation format in the annotation process is shown in the table below:

[0068]

[0069]

[0070] The table above only illustrates several common shape types and their corresponding dimensioning formats in engineering drawings. In the actual application of this invention, the shape types and their corresponding dimensioning formats can be defined by the user and are not restricted here.

[0071] After obtaining the dimensioning format corresponding to the shape type of each unit through the above correspondence, the annotation line step is performed: find the arrow that annotates the structure according to its respective rules. Starting from the coordinates of the found arrow, perform a depth traversal of pixels to find the complete annotation line, and construct an annotation line object for storage.

[0072] During depth traversal, if a dimension line intersects with a solid line (i.e., a part outline), the dimension line needs to be extended in the direction with the smallest angle to the line before the intersection point. When a dimension line reaches another arrow or there are no more black pixels to traverse ahead, the traversal of that dimension line is complete and it is stored as an object.

[0073] Finally, a size matching step is performed, where size information is obtained based on the shape type and corresponding dimensioning rules, and the size of each unit is adjusted.

[0074] The correspondence between shape type and annotation matching rules and size matching rules is shown in the table below:

[0075]

[0076] For example, Figure 2 As shown in the dashed box, the dimension line is a double arrow line, and the corresponding dimension is an arrow line (with information on one side of the line), the information being the dimension 30.

[0077] In summary, in step S4 of the stitching dimension process, the dimension information corresponding to the dimension line of each unit in the image is obtained, which is the dimension information of each unit. Based on this dimension, the modeling dimension correction can be completed.

[0078] In the embodiments of the present invention, different units in the image correspond to different annotation types, such as double arrows, broken line arrows, etc. After confirming the size annotation form corresponding to each unit, the corresponding size annotation content is read, and different extraction methods are provided for each size annotation form. Finally, the size of the unit is adjusted according to the read information, which is convenient, efficient and highly accurate.

[0079] Furthermore, in an embodiment of the present invention, the image information extraction method further includes a size accuracy acquisition step S5, which involves acquiring size information corresponding to all special characters, selecting the size with the most decimal places and its corresponding number of decimal places as the size accuracy; and / or acquiring accuracy information in text elements, and obtaining the size accuracy based on the accuracy information.

[0080] refer to Figure 2 The data contains size information with different decimal places: 29, 6.5, and 0.02. The size with the most decimal places (0.02) and its corresponding two decimal places are selected as the size precision. Alternatively, if the text extracted from the image contains precision information such as "size precision is 0.01", the 0.01 stated in the text can also be used as the size precision.

[0081] In the embodiments of the present invention, the size accuracy of the view in the image is confirmed based on the accuracy information read from the text information of the image itself. This method is convenient to operate and accurate. The two accuracy confirmation methods can be mutually verified or selected as the best option.

[0082] refer to Figure 3 In an embodiment of the present invention, the image information extraction method further includes a preprocessing step S01, which includes: a specification adjustment step, adjusting the original image by one or more combinations of cropping, scaling, and rotation; a binarization step, performing binarization processing on the image obtained after the specification adjustment step; and a sharpness adjustment step, performing multiple rounds of erosion and / or dilation processing on the image obtained after the binarization step to obtain a preprocessed image.

[0083] In the embodiments of the present invention, the above-mentioned preprocessing step S01 first standardizes the specifications of the original image, then adjusts the image after specification adjustment to black and white through a binarization step, and then reduces or eliminates problems such as rough edges, noise, and unclearness in the original image through multiple rounds of erosion and / or dilation processing, so as to ensure the clarity and smoothness of the image for subsequent information extraction.

[0084] Furthermore, in an embodiment of the present invention, the image information extraction method further includes an angle adjustment step S02, which uses a convolutional neural network algorithm model to identify and adjust the orientation of the preprocessed image.

[0085] In the embodiments of the present invention, the image obtained after the above preprocessing steps may have multiple angles relative to the preset direction, such as 0°, 90°, 180°, 270°, etc. The convolutional neural network algorithm model can transform the angle recognition problem into a classification problem, easily calculate the angle of the current image, and rotate and adjust the image angle so that the images for subsequent information extraction are in a uniform angle direction, thereby improving the efficiency and accuracy of subsequent information extraction.

[0086] In summary, in the embodiments of the present invention, since text or annotation symbols in images such as part engineering drawings may be set in different formats and orientations at different locations in the image, the text detection model is first used to identify the position of text elements in the image and the image is segmented according to coordinates. After the image is segmented into multiple text image units, the text content is identified by the text recognition model, and further, preset special characters are captured and recognized. Thus, even when the text information in the image is chaotic and complex, with different positions and orientations and containing multiple special characters, efficient, accurate, and automated text content extraction can still be achieved. On this basis, the intelligent recognition of image information and text information is combined to obtain more comprehensive and accurate information from the image, which can then be further processed and calculated, such as achieving automated and accurate construction and dimensional correction of 3D models based on the extracted information.

[0087] Figure 4 This is a schematic diagram of an image information extraction device provided in an embodiment of the present invention.

[0088] like Figure 4 As shown, in an embodiment of the present invention, an image information extraction device is also provided, including a text detection module 1, which detects text elements in an image and segments the image according to the coordinates of the text elements to obtain a text image; a text recognition module 2, which recognizes text elements in the text image through a text recognition model and labels the text elements; and an information extraction module 3, which obtains the information corresponding to the text elements in the image through a multimodal model based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules of preset special characters in the text elements.

[0089] The text detection module 1, text recognition module 2, and information extraction module 3 can be independent hardware and software structures or integrated hardware and software structures; no restrictions are imposed here. The image information extraction device can be a combination of one or more physical devices such as scanners, mobile terminals, and computers with software programs.

[0090] In an embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above-described image information extraction method.

[0091] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for extracting image information, characterized in that, include: The text detection step involves detecting text elements in an image using a text detection model, segmenting the image based on the coordinates of the detected text elements to obtain a text image, and detecting and extracting dimension annotation graphics from the image. The text recognition step involves identifying the text elements in the text image using a text recognition model and labeling the text elements. The information extraction step involves obtaining the information corresponding to the text elements in the image through a multimodal model, based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules for the preset special characters in the text elements. The splicing size steps include: The annotation process involves determining the corresponding dimension annotation format based on the shape type of each unit in the model corresponding to the image. The annotation line step involves finding and storing annotation lines in the image according to the dimension annotation format. The size matching step involves obtaining size information based on the shape type and the corresponding size annotation rules, and adjusting the size of each unit.

2. The image information extraction method as described in claim 1, characterized in that, The text detection model consists of a residual network and a densely connected convolutional network, and the text recognition model is an encoder-decoder structure.

3. The image information extraction method as described in claim 2, characterized in that, In the information extraction step, a sample file is generated by comparing the coordinates of the text element with the coordinates of the tag corresponding to the label.

4. The image information extraction method as described in claim 1, characterized in that, It also includes the step of obtaining dimensional accuracy. Obtain the size information corresponding to all the special characters, find the size with the most decimal places and its corresponding number of decimal places, where the number of decimal places is the size precision of the image; And / or, Obtain the precision information from the text element, and obtain the size precision of the image based on the precision information.

5. The image information extraction method as described in claim 1, characterized in that, It also includes a preprocessing step, which includes: The specification adjustment steps involve adjusting the original image using one or more combinations of cropping, scaling, and rotation. The binarization step involves performing binarization processing on the image obtained after the specification adjustment step. The sharpness adjustment step involves performing multiple rounds of erosion and / or dilation processing on the image obtained after the binarization step to obtain a preprocessed image.

6. The image information extraction method as described in claim 5, characterized in that, It also includes an angle adjustment step, which uses a convolutional neural network algorithm model to identify and adjust the orientation of the image obtained after the preprocessing step.

7. An image information extraction device, characterized in that, include The text detection module detects text elements in an image using a text detection model, segments the image to obtain a text image based on the coordinates of the detected text elements, and detects and extracts dimension annotation graphics from the image. The text recognition module identifies the text elements in the text image using a text recognition model and tags the text elements. The information extraction module obtains the information corresponding to the text elements in the image through a multimodal model based on the image, the content, coordinates, and label information of the text elements in the image, as well as the rules of the preset special characters in the text elements. The splicing size module includes: The annotation format submodule determines the corresponding dimension annotation format based on the shape type of each unit in the model corresponding to the image; The annotation line submodule locates and stores annotation lines in the image according to the dimension annotation format. The size matching submodule obtains size information based on the shape type and the corresponding size annotation rules, and adjusts the size of each unit.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image information extraction method as described in any one of claims 1-6.

Images