A home decoration product drawing recognition method and system combined with a visual model and a storage medium

Abstract

The application discloses a kind of house decoration product drawing identification method, system and storage medium combined visual model, method includes: obtaining house decoration product drawing PDF document, constructs page picture, obtains line segment element in page, text element;Through detection network, the CAD element and Table element of the page picture are detected, and the CAD element of excluding false detection is excluded;The line segment element is divided into two parts of table belonging line segment and free line segment;Free line segment around CAD element and text element are merged, and the final text element is determined;According to table format order, all text elements and CAD elements are reorganized, to complete document filing in the form of text table.The embodiment of the application can extract complete document information, improve the recognition effect of document and be widely applied in computer technology field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular to a home decoration product drawing recognition method and system combined with a visual model and a storage medium. BACKGROUND

[0002] In the field of home decoration manufacturing, there are a large number of PDF documents derived from CAD. The pages of such documents are only composed of line segments and text elements, so that the conventional document extraction method is difficult to process such documents.

[0003] The mainstream document processing method is mainly based on two categories. The first category is the drawing document recognition based on OCR. This method first intelligently recognizes the drawing content through OCR technology, and then arranges and archives the document content through a series of post-processing algorithms. However, this method basically completely depends on the OCR effect. On the one hand, the OCR effect cannot achieve 100% information extraction, and on the other hand, the method needs to face many and complex post-processing situations.

[0004] The second category is the drawing recognition technology based on deep learning. This method completes the drawing recognition task through some trained deep learning model. However, this method also severely depends on the model recognition result, and the effect of the model is relatively worse.

[0005] At present, other methods are mainly for a small amount of graphic element recognition on the drawing, and lack of complete document information extraction. SUMMARY

[0006] The main purpose of the embodiment of the present application is to provide a home decoration product drawing recognition method and system combined with a visual model, which can extract complete document information and improve the recognition effect of the document.

[0007] To achieve the above purpose, one aspect of the embodiment of the present application provides a home decoration product drawing recognition method combined with a visual model, comprising the following steps:

[0008] Obtain a home decoration product drawing PDF document, construct a page picture, and obtain line segment elements and text elements in the page;

[0009] Detect CAD elements and Table elements of the page picture through a detection network, and exclude misdetected CAD elements;

[0010] Divide the line segment elements into two parts of table belonging line segments and free line segments;

[0011] Merge the free line segments and text elements around the CAD elements to determine the final text elements;

[0012] Reorganize all text elements and CAD elements in the order of table format, and complete the document archiving in the form of a text table.

[0013] In some embodiments, the PDF document of the home decoration product drawing is obtained, and the page picture is constructed, and the line segment elements and the text elements in the page are obtained, including the following steps:

[0014] Obtain the PDF document of the home decoration product drawing.

[0015] According to the PDF document of the home decoration product drawing, initialize the line segment elements, the text elements and the page picture.

[0016] Traverse each page of the drawing, record all line segment elements to a line segment element set, record all text elements to a text element set, and record all page pictures to a page picture set.

[0017] In some embodiments, the CAD elements and the Table elements of the page picture are detected by detecting a network, including the following steps:

[0018] Call the target detection model Grounding DINO.

[0019] Initialize the CAD element, and confirm the data structure of the CAD element as a dictionary type.

[0020] Traverse each page of the drawing, and perform the following processing:

[0021] Obtain the current page picture.

[0022] Use the target detection model to respectively identify the "CAD" and "Table" elements therein, wherein the detection threshold of the target box is set to 0.35, and the text threshold is set to 0.25.

[0023] Obtain all detection results, each detection result including a category, a target box and a confidence.

[0024] Traverse all detection results of the CAD category, if the first target box of the detection result overlaps with a second target box of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target box, the detection result of the CAD category is deleted; and the other detection results are retained.

[0025] Record and store the detection box objects of all retained detection results of the CAD category.

[0026] In some embodiments, the line segment elements are divided into table belonging line segments and free line segments, including the following steps:

[0027] Initialize the line segments and free line segments belonging to the table, and the data structure is dictionary type;

[0028] Go through each sheet and perform the following steps:

[0029] Get the line segment element of the current page;

[0030] Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment;

[0031] Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, it is classified as the line segment belonging to the table;

[0032] Traverse all line segment elements and classify all line segment elements that do not belong to the line segment of the table as free line segments;

[0033] Get the text element of the current page.

[0034] In some embodiments, merging free lines and text elements around the CAD element to determine the final text element comprises the following steps:

[0035] Based on a custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range. The CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag. Specifically:

[0036] Go through each sheet and perform the following steps:

[0037] Define the threshold for merging free segments;

[0038] Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element;

[0039] Get the offline segment of the current page;

[0040] Traverse two by two to calculate the distance between the CAD object frame and the free line segment;

[0041] Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range;

[0042] After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection results of the current page are updated;

[0043] Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving;

[0044] The CAD object element is added to the text elements of the page as a special text label.

[0045] In some embodiments, the step of merging the free line segment into the CAD object range if the distance is less than the free line segment merging threshold value includes the following steps:

[0046] If the number of CAD objects with a distance less than the free line segment merging threshold value is 0, the free line segment is not merged into any CAD object;

[0047] If the number of CAD objects with a distance less than the free line segment merging threshold value is 1, the free line segment is merged into the unique CAD object;

[0048] If the number of CAD objects with a distance less than the free line segment merging threshold value is greater than 1, the free line segment is merged into the closest CAD object.

[0049] In some embodiments, the step of reorganizing all text elements and CAD elements in a table format sequence to complete document archiving in a text table form includes the following steps:

[0050] The position of each cell is calculated for the table belonging line segment of each page, all text elements are positioned in a specific cell, and the final extracted table information is obtained to complete archiving; specifically including:

[0051] Initialize the table object and configure the data structure as an array;

[0052] Iterate through each page of the drawing and perform the following steps:

[0053] Obtain the table belonging line segment of the current page;

[0054] Divide the table rows and columns according to the horizontal line segment elements and vertical line segment elements, iterate through each row and column, determine a cell and the bounding box corresponding to the cell;

[0055] Iterate through the text elements of the current page, determine whether the position of the text element is within the position of the cell according to the position relationship between the text element and the cell, and record the text element in the cell text collection;

[0056] Iterate through all cell text collections and sort all text elements in the cell text collections according to the vertical axis coordinate from large to small and the horizontal axis coordinate from small to small;

[0057] Sort the cell positions recorded by the cell text collection in ascending order and record all text elements in the table object;

[0058] The information extraction of the home decoration product drawing PDF document is completed, and the final extracted table information is obtained to complete archiving.

[0059] Another aspect of the present invention provides a home improvement product drawing recognition system in combination with a visual model, including:

[0060] The first module is used to obtain the PDF document of home decoration product drawings, construct page images, and obtain line elements and text elements within the page;

[0061] The second module is used to detect the CAD elements and Table elements of the page image through a detection network and exclude the CAD elements that are falsely detected;

[0062] The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments;

[0063] The fourth module is used to merge free lines and text elements around the CAD element to determine the final text element;

[0064] The fifth module is used to rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

[0065] To achieve the above object, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0066] To achieve the above object, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0067] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0068] Embodiments of the present invention include at least the following beneficial effects: The present invention provides a method for recognizing home improvement product drawings using a visual model. This method obtains a PDF document of a home improvement product drawing, constructs a page image, and obtains line elements and text elements within the page. Using a detection network, the method detects CAD elements and Table elements within the page image and eliminates falsely detected CAD elements. The line elements are divided into table-related line segments and free line segments. The free line segments and text elements surrounding the CAD element are merged to determine the final text element. All text elements and CAD elements are reorganized in a table format to complete document archiving in a text table format. This embodiment of the present invention can extract complete document information, improving document recognition effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0070] Figure 2 It is a flowchart of the overall steps provided by an embodiment of the present invention;

[0071] Figure 3 It is a flowchart of the implementation steps provided by an embodiment of the present invention;

[0072] Figure 4 This is a display format diagram of a PDF document of a home improvement product drawing provided by an embodiment of the present invention;

[0073] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0075] It will be understood that the terms "first," "second," and the like used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination."

[0076] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0078] The embodiment of the present invention provides a method, system and storage medium for recognizing home decoration product drawings in combination with visual models, which relate to the field of computer technology. The method for recognizing home decoration product drawings in combination with visual models provided in the embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method for recognizing home decoration product drawings in combination with visual models, etc., but is not limited to the above forms.

[0079] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0080] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to the network in a wireless or wired manner to complete data transmission and exchange.

[0081] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0082] In addition, server 101 can also be a node server in a blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0083] Terminal 102 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc. Terminal 102 may also be a vehicle-mounted terminal of the various device types described above, but is not limited thereto. Terminal 102 and server 101 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0084] Based on the example Figure 1In the implementation environment shown, an embodiment of the present invention provides a method for recognizing home decoration product drawings in combination with a visual model. The following is an example of applying the method for recognizing home decoration product drawings in combination with a visual model in the server 101. It can be understood that the method can also be applied to the terminal 102.

[0085] Reference Figure 2 , Figure 2 This is a flowchart of a method for recognizing home decoration product drawings by combining a visual model and applied to a server according to an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0086] Obtain PDF documents of home decoration product drawings, construct page images, and obtain line elements and text elements within the page;

[0087] Detecting the CAD elements and Table elements of the page image through a detection network, and eliminating falsely detected CAD elements;

[0088] Dividing the line segment elements into two parts: line segments belonging to the table and free line segments;

[0089] Merge the free lines and text elements around the CAD element to determine the final text element;

[0090] Rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

[0091] In some embodiments, the process of obtaining a PDF document of a home improvement product drawing, constructing a page image, and obtaining line elements and text elements within the page includes the following steps:

[0092] Get PDF documents of home decoration product drawings;

[0093] Initialize line elements, text elements, and page images based on the home improvement product drawing PDF document;

[0094] Traverse each page of the drawing, record all line segment elements into the line segment element collection, record all text elements into the text element collection, and record all page images into the page image collection.

[0095] In some embodiments, detecting the CAD element and the Table element of the page image by detecting the network includes the following steps:

[0096] Call the target detection model Grounding DINO;

[0097] Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type;

[0098] Traverse each page of the drawing and perform the following processing:

[0099] Get the current page image;

[0100] Use the object detection model to identify the "CAD" and "Table" elements respectively. The detection threshold of the target box is set to 0.35, and the text threshold is set to 0.25.

[0101] Get all detection results, each of which includes category, target box and confidence level;

[0102] Traverse all detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results;

[0103] All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

[0104] In some embodiments, dividing the line segment elements into line segments belonging to a table and free line segments comprises the following steps:

[0105] Initialize the line segments and free line segments belonging to the table, and the data structure is dictionary type;

[0106] Go through each sheet and perform the following steps:

[0107] Get the line segment element of the current page;

[0108] Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment;

[0109] Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, it is classified as the line segment belonging to the table;

[0110] Traverse all line segment elements and classify all line segment elements that do not belong to the line segment of the table as free line segments;

[0111] Get the text element of the current page.

[0112] In some embodiments, merging free lines and text elements around the CAD element to determine the final text element comprises the following steps:

[0113] Based on a custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range. The CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag. Specifically:

[0114] Go through each sheet and perform the following steps:

[0115] Define the threshold for merging free segments;

[0116] Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element;

[0117] Get the offline segment of the current page;

[0118] Traverse two by two to calculate the distance between the CAD object frame and the free line segment;

[0119] Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range;

[0120] After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection results of the current page are updated;

[0121] Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving;

[0122] Add the CAD object element as a special text tag to the text element of the page.

[0123] In some embodiments, merging free line segments whose distance is less than a free line segment merging threshold into the CAD object range comprises the following steps:

[0124] If the CAD object whose distance is less than the free line segment merging threshold is 0, the free line segment will not be merged into any CAD object;

[0125] If the CAD object with a distance less than the free segment merging threshold is 1, then the free segment is merged into this unique CAD object;

[0126] If the distance between CAD objects less than the free segment merge threshold is greater than 1, the free segment is merged onto the closest CAD object.

[0127] In some embodiments, the step of rearranging all text elements and CAD elements in a table format to complete document filing in a text table format includes the following steps:

[0128] Calculate the position of each cell for each line segment of the table on each page, locate all text elements in specific cells, obtain the final extracted table information, and complete the archiving; specifically including:

[0129] Initialize the table object and configure the data structure as an array;

[0130] Go through each sheet and perform the following steps:

[0131] Get the line segment to which the table on the current page belongs;

[0132] Divide the table into rows and columns according to the horizontal line segment elements and the vertical line segment elements, traverse each row and column, and determine a cell and the bounding box corresponding to the cell;

[0133] Traverse the text elements of the current page, determine whether the position of the text element is within the position of the cell based on the position relationship between the text element and the cell, and record it in the cell text set;

[0134] Traverse all cell text sets and sort all text elements in them in descending order of vertical axis coordinates and descending order of horizontal axis coordinates;

[0135] Sort the cell positions recorded in the cell text collection in positive order and record all text elements in the table object;

[0136] The information extraction of the home decoration product drawing PDF document is completed, the final extracted table information is obtained, and archiving is completed.

[0137] The following describes the specific implementation process of the present invention in detail using a specific application scenario as an example:

[0138] refer to Figure 3 The present invention provides a method for recognizing home improvement product drawings in combination with a visual model, comprising:

[0139] 1. Get the PDF document of the home decoration product drawing, traverse each page in the drawing, build the page image, and get the line segments and text information in the page; usually, the PDF document format of the home decoration product drawing is as follows Figure 4 shown.

[0140] 2. Obtain an image of each page and input it into the image recognition detection network. Detect the object detection frames of the "CAD" and "Table" objects. Based on the positional relationship between the two types of detection frames, calculate the "CAD" results that need to be excluded, and finally retain all "CAD" objects that have not been excluded.

[0141] 3. Get the line segment elements of each page, determine whether each line segment belongs to a table structure based on the positional relationship of the line segment elements, obtain the "table-belonging line segments" and "free line segments", and obtain the "text elements" of each page;

[0142] 4. Based on the custom element merging threshold, all "free lines" and "text elements" whose distance is less than the merging threshold are merged into the "CAD" object range. The "CAD" object range is recalculated, and the "CAD" object element is added as a special text tag to the "Text Elements" collection of the page.

[0143] 5. Calculate the position of each cell in the "table segment" of each page, locate all "text elements" in specific cells, obtain the final extracted table information, and complete archiving.

[0144] The first step is to obtain the PDF document of the home improvement product drawing, traverse each page in the drawing, build the page image, and obtain the line segments and text information within the page:

[0145] 1) Obtain PDF documents of home improvement product drawings;

[0146] 2) Initialize "line elements" Lines, "text elements" Texts, and "page images" Images. The data structures are all dictionary types;

[0147] 3) Traverse each page of the drawing and define the current page index as page:

[0148] a) Get all "line segment elements" and record them in Lines[page]. Each line segment element has a set of binary coordinates [x, y].

[0149] b) Get all "text elements" and record them in Texts[page];

[0150] c) Get all "page images" and record them in Images[page];

[0151] In the second step, we obtain the page image of each page and input it into the image recognition detection network to detect the target detection frames of the two types of objects, "CAD" and "Table". Based on the positional relationship between the two types of detection frames, we calculate the "CAD" results that need to be excluded, and finally retain all the "CAD" objects that have not been excluded:

[0152] 1) Call the target detection model Grounding DINO;

[0153] 2) Initialize "CAD elements" Elements, the data structure is a dictionary type;

[0154] 3) Traverse each page of drawings, define the current page index as page:

[0155] a) Get the current page picture Images[page];

[0156] b) Use the target detection model to identify the "CAD" and "Table" elements respectively, wherein the detection threshold of the target box is set to 0.35, and the text threshold is set to 0.25;

[0157] c) Obtain all detection results det_cad, each det_cad contains class, box, and confidence;

[0158] d) Traverse all detection results det_cad with class = "CAD":

[0159] i. If the detection result target box box1 overlaps with a detection result target box box2 with class = "Table", and the overlap area of box1 and box2 accounts for more than 90% of the area of box2, delete the detection result;

[0160] ii. Keep the rest of the detection results;

[0161] e) Only keep the detection results with class = "CAD", and record the box object of the detection result in Elements[page];

[0162] Third step, get the line segment elements of each page, according to the position relationship of the line segment elements, judge whether each line segment belongs to a table structure, obtain "table belonging line segment" and "free line segment", and get the "text element" of each page:

[0163] 1) Initialize "table belonging line segment" TableLines and "free line segment" FreeLines, both of which are dictionaries;

[0164] 2) Traverse each page of drawings, define the current page index as page:

[0165] a) Get the "line segment elements" Lines[page] of the current page;

[0166] b) Traverse all "line segment elements", and divide "horizontal line segment elements" hLines[page] and "vertical line segment elements" vLines[page] according to the relationship between the start and end point coordinates of the line segment;

[0167] c) Traverse the combination of "horizontal line segment elements" hLines[page] and "vertical line segment elements" vLines[page] in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, it is classified as the "table line segment" TableLines[page];

[0168] d) Traverse all "line segment elements" and classify all line segment elements that do not belong to "table-belonging line segments" as "free line segments" FreeLines[page];

[0169] e) Get the "text element" Texts{page} of the current page;

[0170] Step 4: Based on the custom element merging threshold, all "free lines" and "text elements" whose distance is less than the merging threshold are merged into the "CAD" object range. The "CAD" object range is recalculated, and the "CAD" object element is added as a special text tag to the "Text Elements" collection of the page:

[0171] 1) Traverse each page of the drawing and define the current page index as page:

[0172] a) Define the “free line segment” merging threshold θ;

[0173] b) Get the CAD detection results Elements[page] of the current page. Each result is the detection box corresponding to the CAD element;

[0174] c) Get the "free lines" of the current page FreeLines[page];

[0175] d) Traverse two by two and calculate the distance between the "CAD" object frame and the "free line segment";

[0176] e) Merge the “free line segments” whose distance is less than the “free line segment” merging threshold θ into the “CAD” object range. For each “free line segment”:

[0177] i. If the distance between any CAD objects and the merging threshold θ of the "free line segment" is 0, the "free line segment" is not merged into any CAD object;

[0178] ii. If the distance to the "CAD" object is less than the "free line segment" merging threshold θ, the "free line segment" is merged into this unique "CAD" object;

[0179] iii. If the number of CAD objects whose distance is less than the "free line segment" merging threshold θ is greater than 1, the "free line segment" is merged into the "CAD object" with the closest distance;

[0180] f) After merging, recalculate the detection box of the "CAD" object and update Elements[page];

[0181] g) Traverse all the detection boxes of the "CAD" objects in Elements[page], and according to the box, cut the image block from the drawing image by the target box and store it for archiving;

[0182] h) Add the "CAD" object element as a special text label to the "text element" Texts[page] of the page;

[0183] Step 5, calculate the position of each cell for each page of "table-owned line segment", position all "text elements" to specific cells, get the final extracted table information, and complete the archiving:

[0184] 1) Initialize the "table" object Tables, and the data structure is an array;

[0185] 2) Traverse each page of the drawing, define the current page index as page:

[0186] a) Get the "table-owned line segment" TableLines[page] of the current page;

[0187] b) According to the "horizontal line segment element" hLines[page] and "vertical line segment element" vLines[page], divide the table rows and columns, traverse each row and column, that is, a cell cell, which is expressed as a bounding box;

[0188] c) Traverse the "text element" Texts[page] of the current page, according to the position relationship between the "text element" and the cell, judge whether the position of the "text element" is in the position of the cell, record as CellTexts[page][position], position is the top left corner of the cell;

[0189] d) Traverse all CellTexts[page][position], sort all "text elements" in the order of descending vertical axis coordinate and ascending horizontal axis coordinate;

[0190] e) Sort the cell positions recorded by CellTexts[page] in ascending order, and record all "text elements" to Tables[page];

[0191] 3) After the information extraction of the home decoration product drawing PDF document is completed, the final extracted table information is obtained, and the archiving is completed.

[0192] In summary, compared to the existing technology, PDF documents exported from CAD are often difficult to efficiently identify and extract complete information and content, including text objects and CAD blocks, etc. The existing traditional text processing technology and the processing technology based entirely on deep learning each have certain problems, which are unable to be solved by each of them. Therefore, the embodiment of the present invention targets a specific type of home improvement product drawing PDF document, combines the deep target detection model with the traditional drawing processing technology, extracts the CAD elements and their surrounding elements separately through a specific algorithm logic, and expresses the entire document structure as a complete table structure. The information order in the document is organized through the characteristics of the table, and the extraction of the document content is completed relatively completely, providing support for subsequent document information technology.

[0193] Another aspect of the present invention provides a home improvement product drawing recognition system in combination with a visual model, including:

[0194] The first module is used to obtain the PDF document of home decoration product drawings, construct page images, and obtain line elements and text elements within the page;

[0195] The second module is used to detect the CAD elements and Table elements of the page image through a detection network and exclude the CAD elements that are falsely detected;

[0196] The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments;

[0197] The fourth module is used to merge free lines and text elements around the CAD element to determine the final text element;

[0198] The fifth module is used to rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

[0199] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0200] An embodiment of the present invention further provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the above-described method for recognizing home improvement product drawings using a visual model. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0201] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0202] See also Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0203] The processor 501 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0204] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called by the processor 501 to execute the home improvement product drawing recognition method combined with a visual model according to the embodiment of the present invention.

[0205] Input / output interface 503, used to implement information input and output;

[0206] Communication interface 504, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0207] Bus 505 , which transmits information between various components of the device (e.g., processor 501 , memory 502 , input / output interface 503 , and communication interface 504 );

[0208] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0209] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for recognizing home improvement product drawings in combination with a visual model.

[0210] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0211] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0212] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present invention will be obtained.

[0213] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0214] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0215] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0216] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0217] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0218] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0219] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0220] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0222] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store programs.

[0223] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the invention is not limited thereby. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the invention should be within the scope of the invention.

Claims

1. A method for recognizing home decoration product drawings in combination with a visual model, characterized in that: The following steps are involved: Obtain PDF documents of home decoration product drawings, construct page images, and obtain line elements and text elements within the page; Detecting the CAD elements and Table elements of the page image through the detection network, and eliminating the CAD elements that are falsely detected; Dividing the line segment elements into two parts: line segments belonging to the table and free line segments; Merge the free lines and text elements around the CAD element to determine the final text element; Rearrange all text elements and CAD elements in table format and complete document filing in text table form; The method of detecting the CAD element and the Table element of the page image by detecting the network includes the following steps: Call the target detection model Grounding DINO; Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type; Traverse each page of the drawing and perform the following processing: Get the current page image; Use the object detection model to identify the "CAD" and "Table" elements respectively. The detection threshold of the target box is set to 0.35, and the text threshold is set to 0.

25. Get all detection results, each of which includes category, target box and confidence level; Traverse all detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results; All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

2. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The method of obtaining a PDF document of a home decoration product drawing, constructing a page image, and obtaining line elements and text elements in the page includes the following steps: Get PDF documents of home decoration product drawings; Initialize line elements, text elements, and page images based on the home improvement product drawing PDF document; Traverse each page of the drawing, record all line segment elements into the line segment element collection, record all text elements into the text element collection, and record all page images into the page image collection.

3. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The dividing the line segment elements into line segments belonging to a table and free line segments comprises the following steps: Initialize the line segments and free line segments belonging to the table, and the data structure is dictionary type; Go through each sheet and perform the following steps: Get the line segment element of the current page; Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment; Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, it is classified as the line segment belonging to the table; Traverse all line segment elements and classify all line segment elements that do not belong to the line segment of the table as free line segments; Get the text element of the current page.

4. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The step of merging free lines and text elements around the CAD element to determine the final text element includes the following steps: Based on a custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range. The CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag. Specifically: Go through each sheet and perform the following steps: Define the threshold for merging free segments; Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element; Get the offline segment of the current page; Traverse two by two to calculate the distance between the CAD object frame and the free line segment; Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range; After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection results of the current page are updated; Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving; Add the CAD object element as a special text tag to the text element of the page.

5. The method for recognizing home decoration product drawings in combination with a visual model according to claim 4, characterized in that: The step of merging the free line segments whose distance is less than the free line segment merging threshold into the CAD object range includes the following steps: If the CAD object whose distance is less than the free line segment merging threshold is 0, the free line segment will not be merged into any CAD object; If the CAD object with a distance less than the free segment merging threshold is 1, then the free segment is merged into this unique CAD object; If the distance between CAD objects less than the free segment merge threshold is greater than 1, the free segment is merged onto the closest CAD object.

6. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The process of rearranging all text elements and CAD elements in a table format to complete document archiving in a text table format includes the following steps: Calculate the position of each cell for each line segment of the table on each page, locate all text elements in specific cells, obtain the final extracted table information, and complete the archiving; specifically including: Initialize the table object and configure the data structure as an array; Go through each sheet and perform the following steps: Get the line segment to which the table on the current page belongs; Divide the table into rows and columns according to the horizontal line segment elements and the vertical line segment elements, traverse each row and column, and determine a cell and the bounding box corresponding to the cell; Traverse the text elements of the current page, determine whether the position of the text element is within the position of the cell based on the position relationship between the text element and the cell, and record it in the cell text set; Traverse all cell text sets and sort all text elements in them in descending order of vertical axis coordinates and descending order of horizontal axis coordinates; Sort the cell positions recorded in the cell text collection in positive order and record all text elements in the table object; The information extraction of the home decoration product drawing PDF document is completed, the final extracted table information is obtained, and archiving is completed.

7. A home decoration product drawing recognition system combined with a visual model, characterized in that: include: The first module is used to obtain the PDF document of home decoration product drawings, construct page images, and obtain line elements and text elements within the page; The second module is used to detect the CAD elements and Table elements of the page image through a detection network and exclude the CAD elements that are falsely detected; The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments; The fourth module is used to merge free lines and text elements around the CAD element to determine the final text element; The fifth module is used to rearrange all text elements and CAD elements in table format and complete document archiving in text table form; The second module is specifically used for: Call the target detection model Grounding DINO; Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type; Traverse each page of the drawing and perform the following processing: Get the current page image; Use the object detection model to identify the "CAD" and "Table" elements respectively. The detection threshold of the target box is set to 0.35, and the text threshold is set to 0.

25. Get all detection results, each of which includes category, target box and confidence level; Traverse all detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results; All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images