Drawing identification method, device and equipment and readable storage medium

Abstract

The application provides a kind of drawing identification method, device, equipment and readable storage medium.The method comprises: obtaining the text content in the drawing image to be identified and the coordinate information of the text content;Obtain the text box information in the drawing image, the text box information includes the coordinate information of text box, or, including the coordinate information of text box and line type;According to the coordinate information of the text box and the coordinate information of the text content, determine the correlation of the text box and the text content;According to the correlation, text box information and text content, determine the identification result of the drawing image.The identification result in the embodiment of the application not only includes text content, but also contains text box information and the correlation of text content and the text box information, so that the text content in each text box can also be determined based on the correlation, and then the comprehensive information in the drawing image to be identified can be obtained based on the identification result.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, device, and readable storage medium for drawing recognition. Background Technology

[0002] Drawings contain a wealth of information, and content recognition in drawings can convert this information into text format, making them easier to edit and manage. This has broad practical application value. On one hand, when drawings are used in practice, their content needs to be reviewed to avoid errors, and obtaining information from the drawings is more conducive to content review. On the other hand, managing real-world scenarios also requires significant manpower and time, and obtaining drawing content information corresponding to those scenarios can enable intelligent management. Therefore, there is a substantial demand for content recognition in drawings. However, current drawing recognition technologies often struggle to obtain comprehensive and systematic drawing content information. Summary of the Invention

[0003] The purpose of this invention is to provide a drawing recognition method, apparatus, device, and readable storage medium to solve the problem of how to obtain comprehensive drawing content information from a system.

[0004] To achieve the above objectives, embodiments of the present invention provide a drawing recognition method, comprising:

[0005] Obtain the text content and coordinate information of the text content in the drawing image to be identified;

[0006] Obtain text box information from the drawing image, wherein the text box information includes the coordinate information of the text box, or includes the coordinate information and line type of the text box;

[0007] The association between the text box and the text content is determined based on the coordinate information of the text box and the coordinate information of the text content.

[0008] Based on the aforementioned association, text box information, and text content, the recognition result of the drawing image is determined.

[0009] Optionally, the text content in the drawing image to be identified is obtained, including:

[0010] Extract the visual features of the text content in the text area of ​​the drawing image;

[0011] Based on the visual features, extract the semantic features of the text content in the text region;

[0012] The visual features and semantic features are fused to obtain the text content of the drawing image to be identified.

[0013] Optionally, obtaining the coordinate information of the text content includes:

[0014] Identify the text region of the drawing image and determine the coordinate information of the text region;

[0015] Based on the coordinate information of the text region, obtain the coordinate information of the text content.

[0016] Optionally, based on the visual features, semantic features of the text content in the text region are extracted, including:

[0017] Based on the visual features, extract visual sequence features containing contextual information;

[0018] The visual sequence features are converted into semantic features.

[0019] Optionally, the method of obtaining text box information in the drawing image to be identified includes:

[0020] Generate a predicted feature map of the drawing image;

[0021] Extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box.

[0022] Based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained.

[0023] Optionally, generating a predicted feature map of the drawing image includes:

[0024] The target multi-scale features of the drawing image are extracted based on the Feature Pyramid Network (FPN).

[0025] Feature map prediction is performed based on the target's multi-scale features to obtain multi-scale predicted feature maps.

[0026] Optionally, based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained, including:

[0027] Based on the coordinates of the predicted text boxes, the overlapping predicted text boxes are filtered to obtain the filtered predicted text boxes.

[0028] Based on the target content of the predicted text boxes after filtering, obtain the text box information in the drawing image.

[0029] Optionally, based on the target content of the predicted feature map, the text box information in the drawing image to be identified is obtained, including:

[0030] Obtain a set of target predicted text boxes, wherein the overlap between the predicted text boxes and the real text boxes in the set of target predicted text boxes is greater than a preset threshold;

[0031] Based on the coordinates and line type of the predicted text boxes in the target predicted text box set, obtain the text box information in the drawing image.

[0032] Optionally, embodiments of the present invention also provide a drawing recognition device, comprising:

[0033] The first acquisition module is used to acquire the text content in the drawing image to be identified and the coordinate information of the text content;

[0034] The second acquisition module is used to acquire text box information in the drawing image. The text box information includes the coordinate information of the text box, or it includes the coordinate information and line type of the text box.

[0035] The third acquisition module is used to determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content;

[0036] The fourth acquisition module is used to determine the recognition result of the drawing image based on the association, text box information and text content.

[0037] Optionally, embodiments of the present invention also provide a drawing recognition device, including: a transceiver and a processor;

[0038] The processor is used to acquire text content and coordinate information of the text content in a drawing image to be recognized; acquire text box information in the drawing image, the text box information including coordinate information of the text box, or including coordinate information and line type of the text box; determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content; and determine the recognition result of the drawing image based on the association, the text box information, and the text content.

[0039] This invention also provides a drawing recognition device, comprising: a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the steps of the drawing recognition method as described above.

[0040] This invention also provides a readable storage medium storing a program or instructions thereon, characterized in that the program or instructions, when executed by a processor, implement the steps of the drawing recognition method as described above.

[0041] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0042] In this embodiment of the invention, text content and coordinate information of the text content in a drawing image to be identified are obtained; text box information in the drawing image is obtained; the association between the text boxes and the text content is determined based on the coordinate information of the text boxes and the coordinate information of the text content; and the recognition result of the drawing image is determined based on the association, the text box information, and the text content. The recognition result in this embodiment of the invention includes not only the text content but also the text box information and the association between the text content and the text box information. Based on this association, the text content within each text box can be determined, and thus, based on the recognition result, comprehensive system information in the drawing image to be identified can be obtained. Attached Figure Description

[0043] Figure 1 This is one of the flowcharts for the drawing recognition method according to an embodiment of the present invention;

[0044] Figure 2 The illustrated network structure diagram is shown in the embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the semantic feature module in an embodiment of the present invention;

[0046] Figure 4 This is a second flowchart of the drawing recognition method according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the drawing recognition device according to an embodiment of the present invention;

[0048] Figure 6 This is one of the structural block diagrams of the drawing recognition device according to an embodiment of the present invention;

[0049] Figure 7 This is a second structural block diagram of the drawing recognition device according to an embodiment of the present invention. Detailed Implementation

[0050] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0051] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0052] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0053] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0054] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0055] In practical drawings, text and rectangles are often combined to represent actual objects. Rectangles indicate the size and position of the object, and different line types can represent different states of the object. The text within the rectangles represents the object's name, thus depicting the actual layout on the drawing. Furthermore, since there are often many objects in reality, the rectangles in the drawings tend to be densely packed. These drawings contain rich information, and content recognition can convert this information into text, making it easier to edit and manage. This has broad practical application value. On the one hand, drawings need to be reviewed to avoid errors before being used in practice; obtaining information from the drawings facilitates this review. On the other hand, managing real-world scenarios requires significant manpower and time; obtaining the corresponding drawing content information enables intelligent management. Therefore, there is a significant demand for content recognition in drawings. With the development of computer vision and artificial intelligence, automatic drawing recognition can be achieved, greatly improving work efficiency, reducing labor costs, and is widely used in drawing design, construction, and management applications.

[0056] Currently, drawing recognition uses OCR (optical character recognition) technology, which includes two steps: text region detection and localization, and text recognition. Based on the feature extraction method, it can be divided into traditional OCR methods and deep learning-based OCR methods.

[0057] 1. Traditional OCR methods: Connectivity analysis is used to locate text regions, and then image processing techniques such as binarization and projection analysis are used to segment text lines. Text recognition uses traditional methods such as statistical machine learning support vector machines (SVM) classifiers to recognize the text content of drawings.

[0058] 2. Deep Learning-Based OCR Methods: These methods include image preprocessing, text detection, and text recognition. They utilize convolutional neural networks to automatically extract image features for text detection and recognition. Text detection algorithms primarily include regression-based and segmentation-based methods to locate the text region. Commonly used detection frameworks include DBNet, CTPN, and EAST. Text recognition algorithms mainly include feature extraction, context sequence feature extraction, and character prediction to identify the text content within the region. Commonly used recognition frameworks include CRNN and RARA.

[0059] Current drawing recognition only recognizes the text content in drawings. Drawings contain many rectangles and corresponding text names. Most existing technologies only recognize the text and do not associate the text with the corresponding rectangles. Therefore, it is difficult to obtain the matching relationship between the rectangle positions and the text, making it difficult to obtain the system's drawing content information.

[0060] like Figure 1 As shown, an embodiment of the present invention provides a drawing recognition method, including:

[0061] Step 101: Obtain the text content and coordinate information of the text content in the drawing image to be identified.

[0062] In this embodiment of the invention, the drawing to be identified is first converted into an image format to obtain the image of the drawing to be identified.

[0063] Step 102: Obtain the text box information in the drawing image. The text box information includes the coordinate information of the text box, or it includes the coordinate information and line type of the text box.

[0064] The text box here can be a rectangle. Line types include, but are not limited to, solid lines and dashed lines.

[0065] Step 103: Determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content;

[0066] In this step, the text box and the text content are matched based on the coordinate information of the text box and the coordinate information of the text content to determine the association between the text box and the text content.

[0067] Step 104: Determine the recognition result of the drawing image based on the association, text box information, and text content.

[0068] In this step, based on the above-mentioned relationships, the text content in the text box can be obtained. The length and width information of the text box can be obtained based on the coordinate information of the text box. The shape of the text box, such as a square or a rectangle, can be determined based on the length and width information. The row and column number of the text box can be calculated based on the coordinate information of the text box. Finally, the above-mentioned information is combined to output all the attribute content in the drawing, that is, the recognition result.

[0069] In this embodiment of the invention, text content and coordinate information of the text content in a drawing image to be identified are obtained; text box information in the drawing image is obtained; the association between the text boxes and the text content is determined based on the coordinate information of the text boxes and the coordinate information of the text content; and the recognition result of the drawing image is determined based on the association, the text box information, and the text content. The recognition result in this embodiment of the invention includes not only the text content but also the text box information and the association between the text content and the text box information. Based on this association, the text content within each text box can be determined, and thus, based on the recognition result, comprehensive system information in the drawing image to be identified can be obtained.

[0070] Optionally, the text content in the drawing image to be identified is obtained, including:

[0071] Extract visual features of the text content in the text region of the drawing image to be identified;

[0072] Based on the visual features, extract the semantic features of the text content in the text region;

[0073] The visual features and semantic features are fused to obtain the text content of the drawing image.

[0074] As an optional implementation method, such as Figure 2 As shown, a text recognition network based on fused semantic features identifies the text content in a drawing image. This text recognition network includes a text detection module and a semantic text recognition module. The text detection module is used to determine the text region, and the semantic text recognition module is used to identify the text content in the text region.

[0075] like Figure 2As shown, this semantic text recognition module uses a convolutional recurrent neural network (CRNN) as the basic framework and MobileNet v3 as the backbone network. Visual features can be extracted through the convolutional layer. The semantic text recognition module also includes a semantic feature module for extracting semantic features, making the extracted features more robust. Finally, the visual features and semantic features are used together through a softmax transcription layer to obtain the recognized text content.

[0076] In this embodiment of the invention, more accurate textual semantic information can be obtained based on visual features and semantic features, thereby obtaining accurate text content.

[0077] Optionally, before acquiring the visual features of the text content in the text area of ​​the drawing image, the method further includes:

[0078] The text region of the drawing image to be identified is identified, and the coordinate information of the text region is determined.

[0079] Optionally, a text detection module identifies text regions and their coordinate information. This text detection module uses the EAST text detection framework and MobileNet v3 as the backbone network to extract features. After passing through convolutional layers, feature maps of different scales are obtained, extracting text features at different scales. Then, through a feature fusion module, the multi-scale features are fused using upsampling and concat methods. Finally, the probability distribution map of whether a pixel belongs to the text and the coordinates of the text region are output.

[0080] Here, the text region in the drawing image to be identified is identified so that the text content can be identified based on the text region in the future, and the coordinate information of the text content can be determined based on the coordinate information of the text region.

[0081] Optionally, obtaining the coordinate information of the text content includes:

[0082] Based on the coordinate information of the text region, obtain the coordinate information of the text content.

[0083] Here, the coordinate information of the text content is determined based on the coordinate information of the text area, so that the text content and the text box can be matched in the future based on the coordinate information of the text content and the coordinate information of the text box.

[0084] Optionally, based on the visual features, semantic features of the text content in the text region are extracted, including:

[0085] Based on the visual features, extract visual sequence features containing contextual information;

[0086] The visual sequence features are converted into semantic features.

[0087] Specifically, after extracting visual features through the aforementioned convolutional layer, visual sequence features containing contextual information are obtained through a BiLSTM structure, and then the visual sequence features are converted into semantic features based on the semantic feature module.

[0088] The specific implementation process of this semantic feature module is as follows:

[0089] The semantic feature module includes a visual-semantic embedding module and a semantic reasoning module, with the following structure: Figure 3 As shown, the visual-semantic embedding module generates approximate text embedding information from visual features. First, the visual features G are input into a fully connected layer activated by softmax, and then compared with the ground truth value g. t By comparison, the embedding loss L is obtained using cross-entropy loss. e As shown in Formula 1, this makes them more focused on the target character.

[0090]

[0091] Where N is the number of predicted values, y t It is the predicted value at time step t, g t Let p be the true label value at the t-th time step, and p represent the probability.

[0092] Furthermore, the visual feature G is further processed through argmax operations and an embedding layer, based on its relationship with g. t The embedding vector e is calculated from the closest predicted output character. t ′ The semantic reasoning module is used to implement global semantic reasoning, taking the previously obtained embedding vector e... t ′ The input is fed into the converter unit to obtain global contextual information. Multi-layered textual information can be implicitly modeled by multiple converter units, ultimately outputting semantic features S. Inference loss L... r It can be defined as:

[0093]

[0094] Among them, s t It is the semantic feature of the t-th time step.

[0095] By using semantic information to calculate cross-entropy loss to optimize the target probability, the convergence time is reduced, and more accurate semantic information of the text is learned, which is more conducive to the recognition of text content.

[0096] Optionally, obtaining text box information in the drawing image to be identified includes:

[0097] Generate a predicted feature map of the drawing image;

[0098] Extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box.

[0099] Based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained.

[0100] In embodiments of the present invention, such as Figure 2 As shown, text box information in a drawing image to be identified can be obtained based on a dense object detection network. This dense object detection network can be based on a RetinaNet object detection network with a dense detection strategy added as a framework. The drawing image to be identified is input into the dense object detection network to obtain the predicted feature maps mentioned above. Each predicted feature map has three output branches: a detection branch, which outputs the coordinates of the detected object (i.e., the coordinates of the predicted text box); a classification branch, which outputs the class probability of the detected object (i.e., the line type of the predicted text box); and a Soft-IoU branch, which outputs the IoU value between the predicted box and the ground truth box (the overlap between the predicted text box and the ground truth text box). Based on the contents of these three output branches, it is possible to ensure that the most accurately located predicted box (i.e., the predicted text box) is found among the dense objects.

[0101] Optionally, generating a predicted feature map of the drawing image includes:

[0102] The target multi-scale features of the drawing image are extracted based on the Feature Pyramid Network (FPN).

[0103] Based on feature map prediction using the multi-scale features of the target, a multi-scale predicted feature map is obtained.

[0104] As an optional implementation method, it is possible to obtain Figure 2 As shown, the image of the drawing to be identified is input into a dense object detection network, using ResNet50 as the backbone. Then, FPN is used to extract multi-scale features of the target, forming multi-scale prediction feature maps. Based on these multi-scale prediction feature maps, the predicted feature map of the drawing image is obtained. The target content of the multi-scale prediction feature maps is fused to obtain the target content of the final prediction feature map, and based on this, the text box information in the drawing image to be identified is obtained. The multi-scale prediction feature map can accurately obtain the predicted bounding boxes in the drawing image to be identified.

[0105] Optionally, based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained, including:

[0106] Filter the overlapping predicted text boxes according to the coordinates of the predicted text boxes, and obtain the predicted text boxes after the filtering process;

[0107] Obtain the text box information in the to-be-recognized drawing image according to the target content of the predicted text boxes after the filtering process.

[0108] Optionally, use the EM-Merger unit to filter the overlapping predicted text boxes.

[0109] Since the rectangular boxes to be drawn in general drawings are often relatively dense, use the EM-Merger unit to filter the overlapping predicted boxes to ensure more accurate predicted boxes in dense targets. Finally, obtain the position coordinates and line type types of each rectangular box densely distributed in the drawing. The rectangular boxes with different line types are regarded as different target categories in the detection.

[0110] In the current mainstream detection methods, there will be a large number of overlaps in the predicted boxes when detecting densely with similar targets and close positions. The predicted boxes of similar objects with close positions are inaccurate. In the dense detection strategy adopted by the present invention, the Soft-IoU branch and the EM-Merger unit can solve the problem of overlapping predicted boxes in dense detection.

[0111] The specific implementation processes of the Soft-IoU branch and the EM-Merger unit are as follows:

[0112] The Soft-IoU branch calculates the IoU between the detection target and the ground truth annotation, and outputs the Soft-IoU score as a supplementary optimization item for object detection to increase the accuracy of the predicted target position. The specific steps of the EM-Merger unit are as follows: First, convert the target boxes bbox detected in each image into Gaussian distributions (N). All the bbox on one image are represented as a mixture Gaussian distribution. Then, use K new Gaussian distributions to form a new mixture distribution to represent the mixture distribution composed of the original Gaussian distributions (K << N), ensuring that the two are similar enough. Use the KL divergence to represent the similarity between the two. Finally, convert the K Gaussian distributions back to bbox, which is the final predicted box, so as to reduce a large number of overlapping predicted boxes and obtain more accurate predictions. Use binary cross-entropy as its loss function, as shown in formula (3):

[0113]

[0114] where n represents the number of predicted boxes, IoU i represents the IoU between the predicted box and the ground truth box, represents the predicted IoU score.

[0115] The loss function of the entire dense object detection network is as shown in formula 4:

[0116] L = Lclassification +L regression +L sIoU ;Formula 4

[0117] Where L classification L represents the classification loss function. regression Represents the regression loss function;

[0118] Optionally, based on the target content of the predicted feature map, the text box information in the drawing image to be identified is obtained, including:

[0119] Obtain a set of target predicted text boxes, wherein the overlap between the predicted text boxes and the real text boxes in the set of target predicted text boxes is greater than a preset threshold;

[0120] Based on the coordinates and line type of the predicted text boxes in the target predicted text box set, the text box information in the drawing image to be identified is obtained.

[0121] Here, based on the aforementioned overlap, predicted text boxes with an overlap greater than a preset threshold are obtained as text boxes in the drawing image to be identified, thereby improving the accuracy of locating text boxes.

[0122] In this embodiment of the invention, after obtaining the text box information and text content, by Figure 2 The matching module in the drawing takes the text content and position coordinates recognized by the text recognition network and the position coordinates of the rectangle obtained by the dense object detection network as input. First, based on the positional relationship between the rectangle coordinates and the text coordinates, the rectangle and the corresponding text are matched to obtain the association information between the rectangle and the corresponding text. Then, based on the position coordinates of the rectangle, the length and width of the rectangle are obtained to determine the shape of the drawn rectangle (square / rectangle), and the row and column numbers of the rectangle are calculated based on the coordinate sorting. Finally, the obtained association information is combined and matched to output all attributes in the drawing.

[0123] The drawing recognition method of this invention can identify information in a drawing where text located inside a rectangular frame corresponds one-to-one with the rectangular frame. It associates and matches the text with its corresponding rectangular frame, obtaining various information, including the row and column number of the rectangular frame, the shape of the rectangular frame (rectangle / square), the text inside the rectangular frame, and the line type of the rectangular frame (dashed / solid line). The specific process of the method proposed in this invention is as follows: Figure 4As shown, the drawing to be recognized is first converted into an image format (the image of the drawing to be recognized), which is then used as input. This image is fed into a text recognition network that integrates semantic features and a dense object detection network, respectively, to obtain the text content and coordinates, rectangle coordinates, and line type of the drawing. Finally, the text coordinates and rectangle coordinates from the two modules are matched to obtain the final drawing content information corresponding to the text and rectangles. This invention combines text recognition and object detection methods in drawing recognition, obtaining information on the one-to-one correspondence between text and rectangles, making the recognized drawing content more systematic and facilitating drawing review and intelligent management. Furthermore, using a text recognition module with semantic feature extraction to recognize the text content of the drawing can combine the contextual semantic information of the text content for recognition, and fuse it with visual features for joint recognition of text content, improving the accuracy of text recognition in drawings.

[0124] like Figure 5 As shown, this embodiment of the invention also provides a drawing recognition device 500, comprising:

[0125] The first acquisition module 501 is used to acquire the text content in the drawing image to be identified and the coordinate information of the text content;

[0126] The second acquisition module 502 is used to acquire text box information in the drawing image to be identified. The text box information includes the coordinate information of the text box, or it includes the coordinate information and line type of the text box.

[0127] The third acquisition module 503 is used to determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content;

[0128] The fourth acquisition module 504 is used to determine the recognition result of the drawing image based on the association, text box information and text content.

[0129] Optionally, the first acquisition module includes:

[0130] The first acquisition submodule is used to extract the visual features of the text content in the text area of ​​the drawing image to be identified;

[0131] The second acquisition submodule is used to extract semantic features of the text content in the text region based on the visual features;

[0132] The third acquisition submodule is used to fuse the visual features and the semantic features to obtain the text content of the drawing image to be identified.

[0133] Optionally, the first acquisition module further includes:

[0134] The processing submodule is used to identify the text region of the drawing image to be identified and determine the coordinate information of the text region;

[0135] The fourth acquisition submodule is used to acquire the coordinate information of the text content based on the coordinate information of the text region.

[0136] Optionally, the second acquisition submodule includes:

[0137] The first acquisition unit is used to extract visual sequence features containing contextual information based on the visual features;

[0138] A conversion unit is used to convert the visual sequence features into semantic features.

[0139] Optionally, the second acquisition module includes:

[0140] The fifth acquisition submodule is used to generate a predicted feature map of the drawing image to be identified;

[0141] The sixth acquisition submodule is used to extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box.

[0142] The seventh acquisition submodule is used to obtain the text box information in the drawing image to be identified based on the target content of the predicted text box.

[0143] Optionally, the fifth acquisition submodule includes:

[0144] The second acquisition unit is used to extract target multi-scale features of the drawing image based on the Feature Pyramid Network (FPN).

[0145] The third acquisition unit is used to predict the feature map based on the multi-scale features of the target to obtain a multi-scale predicted feature map.

[0146] Optionally, the seventh acquisition submodule includes:

[0147] The fourth acquisition unit is used to filter overlapping prediction text boxes based on the coordinates of the prediction text boxes to obtain filtered prediction text boxes.

[0148] The fifth acquisition unit is used to acquire the text box information in the drawing image to be identified based on the target content of the predicted text box after filtering.

[0149] Optionally, the seventh acquisition submodule includes:

[0150] The sixth acquisition unit is used to acquire a set of target predicted text boxes, wherein the overlap between the predicted text boxes and the real text boxes in the set of target predicted text boxes is greater than a preset threshold.

[0151] The seventh acquisition unit is used to acquire the text box information in the drawing image to be identified based on the coordinates and line type of the predicted text boxes in the target predicted text box set.

[0152] In this embodiment of the invention, text content and coordinate information of the text content in a drawing image to be identified are obtained; text box information in the drawing image is obtained; the association between the text boxes and the text content is determined based on the coordinate information of the text boxes and the coordinate information of the text content; and the recognition result of the drawing image is determined based on the association, the text box information, and the text content. The recognition result in this embodiment of the invention includes not only the text content but also the text box information and the association between the text content and the text box information. Based on this association, the text content within each text box can be determined, and thus, based on the recognition result, comprehensive system information in the drawing image to be identified can be obtained.

[0153] like Figure 6 As shown, this embodiment of the invention also provides a drawing recognition device, including: a transceiver 620 and a processor 610;

[0154] The processor 610 is used to acquire text content and coordinate information of the text content in a drawing image to be recognized; acquire text box information in the drawing image, the text box information including coordinate information of the text box, or including coordinate information and line type of the text box; determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content; and determine the recognition result of the drawing image based on the association, the text box information, and the text content.

[0155] It should be noted that this device is the device corresponding to the above method embodiments. All implementations of the above method embodiments can be applied to this device embodiment and can achieve the same technical effect, which will not be repeated here.

[0156] like Figure 7 As shown, this embodiment of the invention also provides a dialog text classification device, including: a transceiver 710, a processor 700, a memory 720, and a program or instructions stored in the memory 720 and executable on the processor 700; when the processor 700 executes the program or instructions, it implements the steps of the drawing recognition method described above.

[0157] The transceiver 710 is used to receive and send data under the control of the processor 700.

[0158] Among them, Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 700 and memory represented by memory 720 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 710 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 730 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0159] The processor 700 is responsible for managing the bus architecture and general processing, while the memory 720 can store the data used by the processor 700 during operation.

[0160] An embodiment of the present invention provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the drawing recognition method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0161] The processor mentioned above is the processor in the drawing recognition device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] It should be further noted that the terminals described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the functional components described are referred to as modules in order to more specifically emphasize the independence of their implementation.

[0163] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0164] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0165] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0166] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0167] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A drawing recognition method, characterized in that, include: Obtain the text content and coordinate information of the text content in the drawing image to be identified; Obtain text box information from the drawing image, wherein the text box information includes the coordinate information of the text box, or includes the coordinate information and line type of the text box; The association between the text box and the text content is determined based on the coordinate information of the text box and the coordinate information of the text content. Based on the aforementioned association, text box information, and text content, the recognition result of the drawing image is determined; The step of obtaining the text box information in the drawing image to be identified includes: The drawing image to be identified is input into a dense object detection network to generate a predicted feature map of the drawing image; wherein each predicted feature map has three output branches: a detection branch, a classification branch, and a Soft-IoU branch; Extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box. Based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained.

2. The drawing recognition method according to claim 1, characterized in that, Obtain the text content from the drawing image to be identified, including: Extract the visual features of the text content in the text area of ​​the drawing image; Based on the visual features, extract the semantic features of the text content in the text region; The visual features and semantic features are fused to obtain the text content of the drawing image to be identified.

3. The drawing recognition method according to claim 2, characterized in that, Obtaining the coordinate information of the text content includes: Identify the text region of the drawing image and determine the coordinate information of the text region; Based on the coordinate information of the text region, obtain the coordinate information of the text content.

4. The drawing recognition method according to claim 2, characterized in that, Based on the visual features, semantic features of the text content in the text region are extracted, including: Based on the visual features, extract visual sequence features containing contextual information; The visual sequence features are converted into semantic features.

5. The drawing recognition method according to claim 1, characterized in that, Generating a predicted feature map of the drawing image includes: The target multi-scale features of the drawing image are extracted based on the Feature Pyramid Network (FPN). Feature map prediction is performed based on the target's multi-scale features to obtain multi-scale predicted feature maps.

6. The drawing recognition method according to claim 1, characterized in that, Based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained, including: Based on the coordinates of the predicted text boxes, the overlapping predicted text boxes are filtered to obtain the filtered predicted text boxes. Based on the target content of the predicted text boxes after filtering, obtain the text box information in the drawing image.

7. The drawing recognition method according to claim 1, characterized in that, Based on the target content of the predicted feature map, the text box information in the drawing image to be identified is obtained, including: Obtain a set of target predicted text boxes, wherein the overlap between the predicted text boxes and the real text boxes in the set of target predicted text boxes is greater than a preset threshold; Based on the coordinates and line type of the predicted text boxes in the target predicted text box set, obtain the text box information in the drawing image.

8. A drawing recognition device, characterized in that, include: The first acquisition module is used to acquire the text content in the drawing image to be identified and the coordinate information of the text content; The second acquisition module is used to acquire text box information in the drawing image. The text box information includes the coordinate information of the text box, or it includes the coordinate information and line type of the text box. The third acquisition module is used to determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content; The fourth acquisition module is used to determine the recognition result of the drawing image based on the association, text box information and text content; The second acquisition module includes: The fifth acquisition submodule is used to input the drawing image to be identified into a dense target detection network to generate a predicted feature map of the drawing image; wherein each predicted feature map has three output branches: a detection branch, a classification branch and a Soft-IoU branch. The sixth acquisition submodule is used to extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box. The seventh acquisition submodule is used to obtain the text box information in the drawing image to be identified based on the target content of the predicted text box.

9. A drawing recognition device, characterized in that, include: Transceiver and processor; The processor is used to acquire the text content in the drawing image to be identified and the coordinate information of the text content; Obtain text box information from the drawing image, the text box information including the coordinate information of the text box, or including the coordinate information of the text box and the line type; determine the association between the text box and the text content based on the coordinate information of the text box and the coordinate information of the text content; determine the recognition result of the drawing image based on the association, the text box information and the text content; The step of obtaining the text box information in the drawing image to be identified includes: The drawing image to be identified is input into a dense object detection network to generate a predicted feature map of the drawing image; wherein each predicted feature map has three output branches: a detection branch, a classification branch, and a Soft-IoU branch; Extract the target content of the predicted text box corresponding to the predicted feature map. The target content includes the overlap between the predicted text box and the real text box, the coordinates of the predicted text box, and the line type of the predicted text box. Based on the target content of the predicted text box, the text box information in the drawing image to be identified is obtained.

10. A drawing recognition device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the steps of the drawing recognition method as described in any one of claims 1 to 7.

11. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the drawing recognition method as described in any one of claims 1 to 7.

Images