Home decoration product drawing recognition method and system combined with visual model and storage medium

Abstract

The invention discloses a home decoration product drawing recognition method and system combined with a visual model and a storage medium. The method comprises the steps that a home decoration product drawing PDF document is acquired, a page picture is constructed, and line segment elements and text elements in a page are acquired; detecting CAD (Computer Aided Design) elements and Tab elements of the page picture through a detection network, and eliminating mistakenly detected CAD elements; dividing the line segment elements into two parts of line segments to which the table belongs and free line segments; combining the free line segments around the CAD elements and the text elements, and determining final text elements; and all text elements and CAD elements are rearranged according to a table format sequence, and document filing is completed in a text table form. According to the embodiment of the invention, complete document information can be extracted, the document recognition effect is improved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system and storage medium for identifying home decoration product drawings in combination with a visual model. Background Art

[0002] In the field of home furnishing manufacturing, there are a large number of PDF documents exported from CAD. The pages of such documents are composed only of line segments and text elements, so conventional document extraction methods are difficult to process such documents.

[0003] The mainstream document processing methods are mainly based on two categories. The first category is OCR-based drawing document recognition. This method first uses OCR technology to intelligently recognize the drawing content, and then organizes and archives the document content through a series of post-processing algorithms. However, this type of method basically relies entirely on OCR effects. On the one hand, OCR effects cannot achieve 100% information extraction. On the other hand, this method needs to face many complex post-processing situations.

[0004] The second method is the drawing recognition technology based on deep learning. This method uses some trained deep learning models to complete the drawing recognition task. However, this method also relies heavily on the model recognition results, and the effect of the model will be relatively worse.

[0005] At present, other methods are mainly aimed at recognizing a small number of graphic elements on drawings, and lack the extraction of complete document information. Summary of the invention

[0006] The main purpose of the embodiments of the present invention is to propose a method, system and storage medium for recognizing home decoration product drawings in combination with a visual model, which can extract complete document information and improve the recognition effect of the document.

[0007] To achieve the above purpose, one aspect of an embodiment of the present invention provides a method for recognizing home decoration product drawings in combination with a visual model, comprising the following steps:

[0008] Get the PDF document of home decoration product drawings, build page images, and get the line elements and text elements in the page;

[0009] Detecting the CAD elements and Table elements of the page image through the detection network, and excluding the misdetected CAD elements;

[0010] Dividing the line segment elements into two parts: line segments belonging to the table and free line segments;

[0011] Merge the free line segments and text elements around the CAD elements to determine the final text element;

[0012] Rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

[0013] In some embodiments, the process of obtaining a PDF document of a home improvement product drawing, constructing a page image, and obtaining line elements and text elements in the page includes the following steps:

[0014] Get the PDF document of home decoration product drawings;

[0015] Initialize line elements, text elements, and page images according to the home improvement product drawing PDF document;

[0016] Traverse each page of the drawing, record all line segment elements into the line segment element collection, record all text elements into the text element collection, and record all page images into the page image collection.

[0017] In some embodiments, detecting the CAD element and the Table element of the page image by detecting the network includes the following steps:

[0018] Call the target detection model Grounding DINO;

[0019] Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type;

[0020] Traverse each page of the drawing and perform the following processing:

[0021] Get the current page image;

[0022] Use the object detection model to identify the "CAD" and "Table" elements respectively, where the detection threshold of the target box is set to 0.35 and the text threshold is set to 0.25;

[0023] Get all detection results, each of which includes category, target box and confidence level;

[0024] Traverse all the detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results;

[0025] All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

[0026] In some embodiments, dividing the line segment elements into two parts, the line segments belonging to the table and the free line segments, comprises the following steps:

[0027] Initialize the line segments and free line segments to which the table belongs, and the data structure is all dictionary type;

[0028] Go through each sheet and perform the following steps:

[0029] Get the line segment element of the current page;

[0030] Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment;

[0031] Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, they are classified as the line segment to which the table belongs.

[0032] Traverse all line segment elements and classify all line segment elements that do not belong to the line segment to which the table belongs as free line segments;

[0033] Get the text element of the current page.

[0034] In some embodiments, the step of merging free lines and text elements around the CAD element to determine the final text element comprises the following steps:

[0035] According to the custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range, the CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag; specifically:

[0036] Go through each sheet and perform the following steps:

[0037] Define the threshold for merging free segments;

[0038] Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element;

[0039] Get the offline segments of the current page;

[0040] Traverse two by two to calculate the distance between the CAD object frame and the free line segment;

[0041] Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range;

[0042] After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection result of the current page is updated;

[0043] Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving;

[0044] Add the CAD object element as a special text tag to the text element of the page.

[0045] In some embodiments, merging the free line segments whose distance is less than the free line segment merging threshold into the CAD object range comprises the following steps:

[0046] If the CAD object whose distance is less than the threshold of merging free line segments is 0, the free line segment will not be merged into any CAD object;

[0047] If the CAD object whose distance is less than the free line segment merging threshold is 1, then merge the free line segment into this unique CAD object;

[0048] If the distance of the CAD objects less than the threshold of merging free segments is greater than 1, the free segment is merged to the closest CAD object.

[0049] In some embodiments, the step of rearranging all text elements and CAD elements in a table format to complete document filing in a text table format includes the following steps:

[0050] Calculate the position of each cell for each line segment of the table on each page, locate all text elements in specific cells, obtain the final extracted table information, and complete the archiving; specifically include:

[0051] Initialize the table object and configure the data structure as an array;

[0052] Go through each sheet and perform the following steps:

[0053] Get the line segment to which the table on the current page belongs;

[0054] Divide the table into rows and columns according to the horizontal line segment elements and the vertical line segment elements, traverse each row and column, and determine a cell and the bounding box corresponding to the cell;

[0055] Traverse the text elements of the current page, determine whether the position of the text element is within the position of the cell according to the position relationship between the text element and the cell, and record it in the cell text set;

[0056] Traverse all cell text sets and sort all text elements in them in the order of vertical coordinates from large to small and horizontal coordinates from small to large;

[0057] Sort the cell positions recorded in the cell text collection in positive order and record all text elements into the table object;

[0058] The information extraction of the home improvement product drawing PDF document is completed, the final extracted table information is obtained, and the archiving is completed.

[0059] Another aspect of the embodiment of the present invention further provides a home decoration product drawing recognition system combined with a visual model, including:

[0060] The first module is used to obtain the PDF document of the home decoration product drawing, construct the page image, and obtain the line elements and text elements in the page;

[0061] The second module is used to detect the CAD elements and Table elements of the page image through a detection network, and exclude the CAD elements that are misdetected;

[0062] The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments;

[0063] The fourth module is used to merge the free line segments and text elements around the CAD elements to determine the final text elements;

[0064] The fifth module is used to reorganize all text elements and CAD elements in a table format and complete document archiving in the form of text tables.

[0065] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0066] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0067] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0068] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method for recognizing home decoration product drawings in combination with a visual model, the solution obtains a PDF document of a home decoration product drawing, constructs a page image, obtains line segment elements and text elements in the page; detects CAD elements and Table elements of the page image through a detection network, and excludes misdetected CAD elements; divides the line segment elements into two parts: line segments belonging to the table and free line segments; merges free line segments and text elements around CAD elements to determine the final text elements; reorganizes all text elements and CAD elements in a table format order, and completes document archiving in the form of a text table. The embodiments of the present invention can extract complete document information and improve the recognition effect of documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0070] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0071] Figure 3 is a flowchart of implementation steps provided by an embodiment of the present invention;

[0072] Figure 4 It is a display form diagram of a PDF document of a home decoration product drawing provided by an embodiment of the present invention;

[0073] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0075] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0076] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0078] The invention provides a method, system and storage medium for identifying home decoration product drawings in combination with visual models, and relates to the field of computer technology. The method for identifying home decoration product drawings in combination with visual models provided in the invention can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method for identifying home decoration product drawings in combination with visual models, etc., but is not limited to the above forms.

[0079] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0081] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0082] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0083] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0084] Based on the example Figure 1In the implementation environment shown, an embodiment of the present invention provides a method for identifying home decoration product drawings combined with a visual model. The following is an example of applying the method for identifying home decoration product drawings combined with a visual model in a server 101. It can be understood that the method can also be applied to a terminal 102.

[0085] Reference Figure 2 , Figure 2 This is a flowchart of a method for recognizing home decoration product drawings in combination with a visual model and applied to a server according to an embodiment of the present invention. The execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0086] Get the PDF document of home decoration product drawings, build page images, and get the line elements and text elements in the page;

[0087] Detecting the CAD elements and Table elements of the page image through the detection network, and excluding the misdetected CAD elements;

[0088] Dividing the line segment elements into two parts: line segments belonging to the table and free line segments;

[0089] Merge the free line segments and text elements around the CAD elements to determine the final text element;

[0090] Rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

[0091] In some embodiments, the process of obtaining a PDF document of a home improvement product drawing, constructing a page image, and obtaining line elements and text elements in the page includes the following steps:

[0092] Get the PDF document of home decoration product drawings;

[0093] Initialize line elements, text elements, and page images according to the home improvement product drawing PDF document;

[0094] Traverse each page of the drawing, record all line segment elements into the line segment element collection, record all text elements into the text element collection, and record all page images into the page image collection.

[0095] In some embodiments, detecting the CAD element and the Table element of the page image by detecting the network includes the following steps:

[0096] Call the target detection model Grounding DINO;

[0097] Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type;

[0098] Traverse each page of the drawing and perform the following processing:

[0099] Get the current page image;

[0100] Use the object detection model to identify the "CAD" and "Table" elements respectively, where the detection threshold of the target box is set to 0.35 and the text threshold is set to 0.25;

[0101] Get all detection results, each of which includes category, target box and confidence level;

[0102] Traverse all the detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results;

[0103] All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

[0104] In some embodiments, dividing the line segment elements into two parts, the line segments belonging to the table and the free line segments, comprises the following steps:

[0105] Initialize the line segments and free line segments to which the table belongs, and the data structure is all dictionary type;

[0106] Go through each sheet and perform the following steps:

[0107] Get the line segment element of the current page;

[0108] Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment;

[0109] Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, they are classified as the line segment to which the table belongs.

[0110] Traverse all line segment elements and classify all line segment elements that do not belong to the line segment to which the table belongs as free line segments;

[0111] Get the text element of the current page.

[0112] In some embodiments, the step of merging free lines and text elements around the CAD element to determine the final text element comprises the following steps:

[0113] According to the custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range, the CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag; specifically:

[0114] Go through each sheet and perform the following steps:

[0115] Define the threshold for merging free segments;

[0116] Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element;

[0117] Get the offline segments of the current page;

[0118] Traverse two by two to calculate the distance between the CAD object frame and the free line segment;

[0119] Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range;

[0120] After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection result of the current page is updated;

[0121] Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving;

[0122] Add the CAD object element as a special text tag to the text element of the page.

[0123] In some embodiments, merging the free line segments whose distance is less than the free line segment merging threshold into the CAD object range comprises the following steps:

[0124] If the CAD object whose distance is less than the threshold of merging free line segments is 0, the free line segment will not be merged into any CAD object;

[0125] If the CAD object whose distance is less than the free line segment merging threshold is 1, then merge the free line segment into this unique CAD object;

[0126] If the distance of the CAD objects less than the threshold of merging free segments is greater than 1, the free segment is merged to the closest CAD object.

[0127] In some embodiments, the step of rearranging all text elements and CAD elements in a table format to complete document filing in a text table format includes the following steps:

[0128] Calculate the position of each cell for each line segment of the table on each page, locate all text elements in specific cells, obtain the final extracted table information, and complete the archiving; specifically include:

[0129] Initialize the table object and configure the data structure as an array;

[0130] Go through each sheet and perform the following steps:

[0131] Get the line segment to which the table on the current page belongs;

[0132] Divide the table into rows and columns according to the horizontal line segment elements and the vertical line segment elements, traverse each row and column, and determine a cell and the bounding box corresponding to the cell;

[0133] Traverse the text elements of the current page, determine whether the position of the text element is within the position of the cell according to the position relationship between the text element and the cell, and record it in the cell text set;

[0134] Traverse all cell text sets and sort all text elements in them in the order of vertical coordinates from large to small and horizontal coordinates from small to large;

[0135] Sort the cell positions recorded in the cell text collection in positive order and record all text elements into the table object;

[0136] The information extraction of the home improvement product drawing PDF document is completed, the final extracted table information is obtained, and the archiving is completed.

[0137] The following describes the specific implementation process of the present invention in detail by taking a specific application scenario as an example:

[0138] refer to Figure 3 The present invention provides a method for recognizing home decoration product drawings in combination with a visual model, comprising:

[0139] 1. Get the PDF document of the home decoration product drawing, traverse each page in the drawing, build the page image, and get the line segment and text information in the page; usually, the PDF document format of the home decoration product drawing is as follows Figure 4 shown.

[0140] 2. Get the page image of each page, input it into the image recognition detection network, detect the target detection frames of the two types of objects "CAD" and "Table", calculate the "CAD" results that need to be excluded based on the positional relationship between the two types of detection frames, and finally retain all the "CAD" objects that have not been excluded;

[0141] 3. Get the line segment elements of each page, determine whether each line segment belongs to a table structure according to the position relationship of the line segment elements, obtain the "table-belonging line segments" and "free line segments", and get the "text elements" of each page;

[0142] 4. According to the custom element merging threshold, merge all "free segments" and "text elements" whose distance is less than the merging threshold into the "CAD" object range, recalculate the "CAD" object range, and add the "CAD" object element as a special text tag to the "text element" collection of the page;

[0143] 5. Calculate the position of each cell for the "table segment" on each page, locate all "text elements" in specific cells, obtain the final extracted table information, and complete archiving.

[0144] The first step is to obtain the PDF document of the home improvement product drawing, traverse each page in the drawing, build the page image, and obtain the line segment and text information in the page:

[0145] 1) Obtain the PDF document of home decoration product drawings;

[0146] 2) Initialize "line elements" Lines, "text elements" Texts, and "page images" Images. The data structures are all dictionary types;

[0147] 3) Traverse each page of the drawing and define the current page index as page:

[0148] a) Get all "line segment elements" and record them in Lines[page]. Each line segment element has a set of binary coordinates [x, y].

[0149] b) Get all "text elements" and record them in Texts[page];

[0150] c) Get all "page images" and record them in Images[page];

[0151] The second step is to obtain the page image of each page and input it into the image recognition detection network to detect the target detection frames of the two types of objects, "CAD" and "Table". According to the positional relationship between the two types of detection frames, the "CAD" results that need to be excluded are calculated, and finally all "CAD" objects that have not been excluded are retained:

[0152] 1) Call the target detection model Grounding DINO;

[0153] 2) Initialize "CAD elements" Elements, the data structure is a dictionary type;

[0154] 3) Traverse each page of the drawing and define the current page index as page:

[0155] a) Get the current page image Images[page];

[0156] b) Use the object detection model to identify the “CAD” and “Table” elements respectively, where the detection threshold of the target box is set to 0.35 and the text threshold is set to 0.25;

[0157] c) Get all the detection results det_cad, each det_cad contains the category class, target box box, and confidence confidence;

[0158] d) Traverse all the detection results det_cad with class="CAD":

[0159] i. If the detection result target box box1 overlaps with a detection result target box box2 of a class="Table", and the overlapping area of ​​box1 and box2 accounts for more than 90% of the area of ​​box2, delete the detection result;

[0160] ii. Keep the remaining test results;

[0161] e) Only keep the detection results with class="CAD" and record the box object of the detection results into Elements[page];

[0162] The third step is to obtain the line segment elements of each page, and determine whether each line segment belongs to a table structure based on the position relationship of the line segment elements, obtain the "line segments belonging to the table" and "free line segments", and obtain the "text elements" of each page:

[0163] 1) Initialize "table-belonging line segments" TableLines and "free line segments" FreeLines, and the data structures are both dictionary types;

[0164] 2) Traverse each page of the drawing and define the current page index as page:

[0165] a) Get the "line element" Lines[page] of the current page;

[0166] b) Traverse all "line segment elements" and divide them into "horizontal line segment elements" hLines[page] and "vertical line segment elements" vLines[page] according to the coordinate relationship between the starting and ending points of the line segment;

[0167] c) Traverse the combination of "horizontal line segment elements" hLines[page] and "vertical line segment elements" vLines[page] in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, they are classified as "table line segments" TableLines[page];

[0168] d) Traverse all "line segment elements" and classify all line segment elements that do not belong to "line segments belonging to the table" as "free line segments" FreeLines[page];

[0169] e) Get the "text element" Texts{page} of the current page;

[0170] Step 4: Based on the custom element merging threshold, merge all "free segments" and "text elements" whose distance is less than the merging threshold into the "CAD" object range, recalculate the "CAD" object range, and add the "CAD" object element as a special text tag to the "text element" collection of the page:

[0171] 1) Traverse each page of the drawing and define the current page index as page:

[0172] a) define the “free line segment” merging threshold θ;

[0173] b) Get the CAD detection results Elements[page] of the current page. Each result is the detection box corresponding to the CAD element;

[0174] c) Get the "free lines" of the current page FreeLines[page];

[0175] d) Traverse two by two to calculate the distance between the "CAD" object frame and the "free line segment";

[0176] e) Merge the “free segments” whose distance is less than the “free segment” merging threshold θ into the “CAD” object range. For each “free segment”:

[0177] i. If the distance of a "CAD" object that satisfies the merging threshold θ of the "free line segment" is 0, the "free line segment" is not merged into any "CAD" object;

[0178] ii. If the distance of the "CAD" object is less than the "free line segment" merging threshold θ, the "free line segment" is merged into this unique "CAD" object;

[0179] iii. If the number of "CAD" objects whose distance is less than the "free line segment" merging threshold θ is greater than 1, the "free line segment" is merged into the "CAD" object with the closest distance;

[0180] f) After the merging is completed, the detection box of the "CAD" object is recalculated and Elements[page] is updated;

[0181] g) Traverse the detection boxes of all "CAD" objects in Elements[page], capture blocks from the drawing image according to the target box, and store them for archiving;

[0182] h) Add the "CAD" object element as a special text tag to the "Text Element" Texts[page] of the page;

[0183] The fifth step is to calculate the position of each cell of the "table segment" on each page, locate all "text elements" in specific cells, obtain the final extracted table information, and complete the archiving:

[0184] 1) Initialize the "table" object Tables, the data structure is an array;

[0185] 2) Traverse each page of the drawing and define the current page index as page:

[0186] a) Get the "table line segment" TableLines[page] of the current page;

[0187] b) Divide the table into rows and columns according to the "horizontal line segment elements" hLines[page] and "vertical line segment elements" vLines[page], traverse each row and column, that is, a cell, and the cell is expressed as a bounding box;

[0188] c) Traverse the "text element" Texts[page] of the current page, and determine whether the position of the "text element" is within the cell position based on the position relationship between the "text element" and the cell, and record it as CellTexts[page][position], where position is the top left corner of the cell;

[0189] d) Traverse all CellTexts[page][position] and sort all the "text elements" in it in the order of vertical coordinates from large to small and horizontal coordinates from small to large;

[0190] e) Sort the cell positions recorded in CellTexts[page] in positive order and record all "text elements" in Tables[page];

[0191] 3) The information of the home improvement product drawing PDF document is extracted, the final extracted table information is obtained, and archiving is completed.

[0192] In summary, compared with the existing technology, PDF documents exported from CAD are often difficult to efficiently identify and extract complete information and content, including text objects and CAD blocks, etc. The existing traditional text processing technology and the processing technology based entirely on deep learning each have certain problems, which are unable to be solved by each. Therefore, the embodiment of the present invention is aimed at a specific type of home improvement product drawing PDF document, combines the deep target detection model and the traditional drawing processing technology, extracts the CAD elements and their surrounding elements separately through a specific algorithm logic, and expresses the entire document structure as a complete table structure. The information order in the document is organized through the characteristics of the table, and the extraction of the document content is completed relatively completely, providing support for subsequent document information technology.

[0193] Another aspect of the embodiment of the present invention further provides a home decoration product drawing recognition system combined with a visual model, including:

[0194] The first module is used to obtain the PDF document of the home decoration product drawing, construct the page image, and obtain the line elements and text elements in the page;

[0195] The second module is used to detect the CAD elements and Table elements of the page image through a detection network, and exclude the CAD elements that are misdetected;

[0196] The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments;

[0197] The fourth module is used to merge the free line segments and text elements around the CAD elements to determine the final text elements;

[0198] The fifth module is used to reorganize all text elements and CAD elements in a table format and complete document archiving in the form of text tables.

[0199] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0200] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method for recognizing home improvement product drawings in combination with a visual model when executing the computer program. The electronic device may be any intelligent terminal including a tablet computer, a car computer, etc.

[0201] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0202] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0203] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0204] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the method for recognizing home improvement product drawings combined with a visual model in the embodiment of the present invention;

[0205] Input / output interface 503, used to implement information input and output;

[0206] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0207] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0208] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0209] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for recognizing home improvement product drawings combined with a visual model is implemented.

[0210] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0211] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0212] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0213] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0214] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0215] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0217] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0218] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0219] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0220] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0222] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0223] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for recognizing home decoration product drawings in combination with a visual model, characterized in that: The following steps are involved: Get the PDF document of home decoration product drawings, build page images, and get the line elements and text elements in the page; Detecting the CAD elements and Table elements of the page image through the detection network, and excluding the misdetected CAD elements; Dividing the line segment elements into two parts: line segments belonging to the table and free line segments; Merge the free line segments and text elements around the CAD elements to determine the final text element; Rearrange all text elements and CAD elements in table format and complete document archiving in text table form.

2. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The method of obtaining a PDF document of a home decoration product drawing, constructing a page image, and obtaining line elements and text elements in the page includes the following steps: Get the PDF document of home decoration product drawings; Initialize line elements, text elements, and page images according to the home improvement product drawing PDF document; Traverse each page of the drawing, record all line segment elements into the line segment element collection, record all text elements into the text element collection, and record all page images into the page image collection.

3. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The method of detecting the CAD element and the Table element of the page image by detecting the network includes the following steps: Call the target detection model Grounding DINO; Initialize the CAD element and confirm the data structure of the CAD element as a dictionary type; Traverse each page of the drawing and perform the following processing: Get the current page image; Use the object detection model to identify the "CAD" and "Table" elements respectively, where the detection threshold of the target box is set to 0.35 and the text threshold is set to 0.25; Get all detection results, each of which includes category, target box and confidence level; Traverse all the detection results of the CAD category. If the first target frame of the detection result overlaps with the second target frame of a detection result of the Table category, and the overlapping area accounts for more than 90% of the second target frame, then delete the detection result of the CAD category; retain other detection results; All the detection frame objects of the retained detection results of the CAD category are recorded and stored.

4. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The dividing the line segment elements into two parts, line segments belonging to the table and free line segments, comprises the following steps: Initialize the line segments and free line segments to which the table belongs, and the data structure is all dictionary type; Go through each sheet and perform the following steps: Get the line segment element of the current page; Traverse all line segment elements and divide them into horizontal line segment elements and vertical line segment elements according to the coordinate relationship between the starting and ending points of the line segment; Traverse the combination of horizontal line segment elements and vertical line segment elements in pairs. If the x-axis coordinate of a horizontal line segment element is equal to the x-axis coordinate of a vertical line segment element, they are classified as the line segment to which the table belongs. Traverse all line segment elements and classify all line segment elements that do not belong to the line segment to which the table belongs as free line segments; Get the text element of the current page.

5. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The step of merging free line segments and text elements around the CAD element to determine the final text element includes the following steps: According to the custom element merging threshold, all free line segments and text elements whose distance is less than the element merging threshold are merged into the CAD object range, the CAD object range is recalculated, and the CAD object element is added to the text element set of the page as a special text tag; specifically: Go through each sheet and perform the following steps: Define the threshold for merging free segments; Get the CAD detection results of the current page, where each result is the detection box corresponding to the CAD element; Get the offline segments of the current page; Traverse two by two to calculate the distance between the CAD object frame and the free line segment; Merge the free line segments whose distance is less than the free line segment merging threshold into the CAD object range; After the merging is completed, the detection frame of the CAD object is recalculated and the CAD detection result of the current page is updated; Traverse the detection frames of all CAD objects in the CAD detection results of the current page, capture blocks from the drawing image according to the target frame based on the detection frame, and store them for archiving; Add the CAD object element as a special text tag to the text element of the page.

6. The method for recognizing home decoration product drawings in combination with a visual model according to claim 5, characterized in that: The step of merging the free line segments whose distance is less than the free line segment merging threshold into the CAD object range comprises the following steps: If the CAD object whose distance is less than the threshold of merging free line segments is 0, the free line segment will not be merged into any CAD object; If the CAD object whose distance is less than the free line segment merging threshold is 1, then merge the free line segment into this unique CAD object; If the distance of the CAD objects less than the threshold of merging free segments is greater than 1, the free segment is merged to the closest CAD object.

7. The method for recognizing home decoration product drawings in combination with a visual model according to claim 1, characterized in that: The step of reorganizing all text elements and CAD elements in a table format and completing document filing in a text table format includes the following steps: Calculate the position of each cell for each line segment of the table on each page, locate all text elements in specific cells, obtain the final extracted table information, and complete the archiving; specifically include: Initialize the table object and configure the data structure as an array; Go through each sheet and perform the following steps: Get the line segment to which the table on the current page belongs; Divide the table into rows and columns according to the horizontal line segment elements and the vertical line segment elements, traverse each row and column, and determine a cell and the bounding box corresponding to the cell; Traverse the text elements of the current page, determine whether the position of the text element is within the position of the cell according to the position relationship between the text element and the cell, and record it in the cell text set; Traverse all cell text sets and sort all text elements in them in the order of vertical coordinates from large to small and horizontal coordinates from small to large; Sort the cell positions recorded in the cell text collection in positive order and record all text elements into the table object; The information extraction of the home improvement product drawing PDF document is completed, the final extracted table information is obtained, and the archiving is completed.

8. A home decoration product drawing recognition system combined with a visual model, characterized in that: include: The first module is used to obtain the PDF document of the home decoration product drawing, construct the page image, and obtain the line elements and text elements in the page; The second module is used to detect the CAD elements and Table elements of the page image through a detection network, and exclude the CAD elements that are misdetected; The third module is used to divide the line segment elements into two parts: line segments belonging to the table and free line segments; The fourth module is used to merge the free line segments and text elements around the CAD elements to determine the final text elements; The fifth module is used to reorganize all text elements and CAD elements in a table format and complete document archiving in the form of text tables.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images