Building engineering drawing structured information extraction method, device, equipment and medium

Abstract

Embodiments of the present disclosure disclose a construction drawing structured information extraction method, device, equipment and medium. A specific embodiment of the method comprises: obtaining a construction drawing image; in response to determining that the size information of the construction drawing image does not satisfy a preset condition, performing down-sampling processing on the construction drawing image; performing image quality detection processing on the scaled construction drawing image; in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, performing character recognition processing on the construction drawing image; updating preset prompt word information; inputting the scaled construction drawing image and the updated preset prompt word information into a pre-trained visual language cross-modal inference model; and performing backtracking verification and adaptive error correction processing on the initial construction drawing structured information. The embodiment improves the accuracy of construction drawing structured information recognition and extraction.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and specifically to methods, apparatus, equipment, and media for extracting structured information from architectural drawings. Background Technology

[0002] In the field of digital delivery and intelligent drawing review for architectural engineering projects, it is necessary to extract structured engineering information (such as drawing number, project name, design date, and professional category) from architectural drawings to support subsequent applications such as automated drawing review, BIM model construction, and quantity surveying. Structured information extraction from architectural drawings is a technology for extracting structured information from architectural drawings. Currently, the commonly used method for extracting structured information from architectural drawings is based on Optical Character Recognition (OCR).

[0003] However, when using the above method to extract structured information from architectural drawings, the following technical problems often arise: When using Optical Character Recognition (OCR)-based extraction methods to extract structured information from architectural drawings, various visual interference factors exist, such as differences in font styles between different design institutes (variations in thickness, slant, and character spacing), structural confusion of similar characters (e.g., the uppercase letter "J" and the number "1", the letter "O" and the number "0"), and background graphic overlay interference (table lines crossing text, drawing frames covering text areas). These interferences cause OCR to misidentify correct characters as similar-looking characters, for example, misidentifying the drawing number "J-02" as "1-02". Because the OCR model lacks semantic understanding capabilities, it cannot self-correct misidentifications based on context (e.g., "the architectural code should be J"), resulting in low accuracy in extracting structured information from architectural drawings.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide methods, apparatus, electronic devices, and computer-readable media for extracting structured information from architectural drawings to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for extracting structured information from architectural drawings. The method includes: acquiring an image of an architectural drawing; in response to determining that the size information of the architectural drawing image does not meet a preset condition, performing downsampling processing on the architectural drawing image to obtain a scaled architectural drawing image; performing image quality detection processing on the scaled architectural drawing image to obtain image quality detection information; in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, performing character recognition processing on the architectural drawing image to obtain initial text recognition information; performing shadow context injection processing on preset prompt word information based on the initial text recognition information to update the preset prompt word information; inputting the scaled architectural drawing image and the updated preset prompt word information into a pre-trained visual language cross-modal reasoning model to obtain initial structured information of the architectural drawing; and performing backtracking verification and adaptive error correction processing on the initial structured information of the architectural drawing to obtain the final structured information of the architectural drawing.

[0008] Secondly, some embodiments of this disclosure provide a structured information extraction device for architectural engineering drawings. The device includes: an acquisition unit configured to acquire an image of an architectural engineering drawing; a downsampling processing unit configured to, in response to determining that the size information of the architectural engineering drawing image does not meet a preset condition, perform downsampling processing on the architectural engineering drawing image to obtain a scaled architectural engineering drawing image; an image quality detection unit configured to perform image quality detection processing on the scaled architectural engineering drawing image to obtain image quality detection information; and a character recognition unit configured to, in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, perform character recognition processing on the image quality drawing image. The aforementioned architectural engineering drawing image undergoes character recognition processing to obtain initial text recognition information; the shadow context injection unit is configured to perform shadow context injection processing on preset prompt word information based on the aforementioned initial text recognition information to update the preset prompt word information; the input unit is configured to input the aforementioned scaled architectural engineering drawing image and the updated preset prompt word information into a pre-trained visual language cross-modal reasoning model to obtain the initial architectural engineering drawing structured information; the backtracking verification and adaptive error correction unit is configured to perform backtracking verification and adaptive error correction processing on the aforementioned initial architectural engineering drawing structured information to obtain the architectural engineering drawing structured information.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: the method for extracting structured information from architectural drawings according to some embodiments of this disclosure improves the accuracy of structured information recognition and extraction from architectural drawings. Specifically, the reason for the low accuracy of structured information recognition and extraction from architectural drawings is that when using an extraction method based on Optical Character Recognition (OCR) to extract structured information from architectural drawings, there are various visual interference factors in the drawings, such as differences in font styles (thickness, slant, character spacing) between different design institutes, structural confusion of similar characters (such as the uppercase letter "J" and the number "1", the letter "O" and the number "0"), and background graphic superposition interference (table lines passing through text, drawing frames covering text areas). These interferences cause OCR to misidentify correct characters as similar-looking characters, for example, misidentifying the drawing number "J-02" as "1-02". Because the OCR model lacks semantic understanding ability, it cannot self-correct misidentification based on context (such as "the architectural code should be J"), resulting in low accuracy of structured information recognition and extraction from architectural drawings. Based on this, the method for extracting structured information from architectural drawings according to some embodiments of this disclosure first acquires an image of the architectural drawing. Then, in response to determining that the size information of the architectural drawing image does not meet a preset condition, the architectural drawing image is downsampled to obtain a scaled architectural drawing image. This allows large-size drawings to be adjusted to a size range suitable for subsequent model processing, avoiding low processing efficiency or memory overflow due to excessively large images. Next, image quality detection processing is performed on the scaled architectural drawing image to obtain image quality detection information. This allows for quantitative evaluation of image quality, providing a decision-making basis for different subsequent processing paths. Then, in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, character recognition processing is performed on the architectural drawing image to obtain initial text recognition information. This allows for the extraction of preliminary text information through character recognition processing even when image quality is poor. Afterwards, based on the initial text recognition information, shadow context injection processing is performed on preset prompt word information to update the preset prompt word information. This allows the recognized text, i.e., the initial text recognition information, to be injected into the prompt words as contextual clues, so that the prompt words contain initial text recognition information specific to the current drawing. Next, the scaled architectural drawing image and the updated preset prompt information are input into a pre-trained visual-language cross-modal inference model to obtain the initial structured information of the architectural drawing. Thus, the visual-language model can simultaneously refer to the visual information of the image and the prompts containing the initial text recognition information, utilizing its semantic understanding capabilities to correct errors in the initial text recognition information, such as misidentifying "J" as "1" due to similar-looking characters. Finally, the initial structured information of the architectural drawing is subjected to backtracking verification and adaptive error correction processing to obtain the final structured information of the architectural drawing.Therefore, the model output can be further verified and corrected. Because the initial text recognition information is used as a shadow context injection prompt, and a visual language cross-modal reasoning model is employed to combine images and context for semantic understanding, the model can self-correct misidentification of similar-looking characters in the initial text recognition information based on the contextual semantics of the drawing (e.g., "the architectural code should be J"), thereby improving the accuracy of structured information recognition and extraction from architectural engineering drawings. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 These are flowcharts of some embodiments of the method for extracting structured information from architectural drawings according to the present disclosure; Figure 2 These are schematic diagrams of some embodiments of the structural information extraction device for architectural drawings according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flowchart 100 is shown, illustrating some embodiments of the method for extracting structured information from architectural drawings according to the present disclosure. This method for extracting structured information from architectural drawings includes the following steps: Step 101: Obtain architectural engineering drawings.

[0021] In some embodiments, the entity executing the method for extracting structured information from architectural drawings (e.g., a computing device) can acquire architectural drawing images via wired or wireless connections. These architectural drawing images can be images containing engineering information (such as drawing number, project name, design date, and professional category). Architectural drawing images are typically very large in size.

[0022] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0023] Step 102: In response to the determination that the size information of the architectural engineering drawing image does not meet the preset conditions, the architectural engineering drawing image is downsampled to obtain a scaled architectural engineering drawing image.

[0024] In some embodiments, the execution entity may, in response to determining that the size information of the architectural drawing image does not meet a preset condition, perform downsampling processing on the architectural drawing image to obtain a scaled architectural drawing image. The preset condition may be that the size of the architectural drawing image (such as the pixel width, pixel height, or total number of pixels) is not within a pre-defined range (e.g., the pre-defined range may be that the pixel width is less than or equal to a preset width (e.g., 1024 pixels) and the pixel height is less than or equal to a preset length (e.g., 1024 pixels)).

[0025] In some optional implementations of certain embodiments, the aforementioned execution entity may perform downsampling processing on the aforementioned architectural drawing image through the following steps to obtain a scaled architectural drawing image: The first step is to perform Gaussian blurring on the aforementioned architectural drawings to obtain a blurred preprocessed image. In practice, the executing entity can use Gaussian blurring technology to perform Gaussian blurring on the architectural drawings to obtain a blurred preprocessed image.

[0026] The second step involves downsampling the blurred preprocessed image to reduce it to a preset target size. In practice, the executing entity can employ image downsampling technology to reduce the blurred preprocessed image to a preset target size. For example, the preset target size can be 1024 pixels × 1024 pixels.

[0027] The third step is to identify the reduced and blurred preprocessed image as a scaled architectural drawing image.

[0028] Step 103: Perform image quality detection processing on the scaled architectural engineering drawing image to obtain image quality detection information.

[0029] In some embodiments, the execution entity may perform image quality detection processing on the scaled architectural drawing image to obtain image quality detection information. In practice, the execution entity may employ a no-reference image quality assessment technique to perform image quality detection processing on the scaled architectural drawing image to obtain image quality detection information. This image quality detection information may be a quality score. The quality score may be a score characterizing the degree of image quality.

[0030] Step 104: In response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, character recognition processing is performed on the architectural engineering drawing image to obtain initial text recognition information.

[0031] In some embodiments, the execution entity may perform character recognition processing on the architectural drawing image in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, thereby obtaining initial text recognition information.

[0032] In some optional implementations of certain embodiments, the aforementioned execution entity may perform character recognition processing on the aforementioned architectural engineering drawing image through the following steps to obtain initial text recognition information: The first step is to perform dynamic slicing on the above-mentioned architectural engineering drawings to obtain an image block set.

[0033] The second step involves asynchronously sending the aforementioned image block set to a preset character recognition engine to generate an image block recognition information set corresponding to the aforementioned architectural engineering drawing image. The preset character recognition engine can be an OCR engine used for optical character recognition of graphic blocks. Each image block recognition information set sent is text obtained by recognizing one image block in the aforementioned image block set through OCR recognition.

[0034] The third step is to generate initial text recognition information based on the above image block recognition information set.

[0035] In some optional implementations of certain embodiments, the aforementioned execution entity may perform dynamic slicing processing on the aforementioned architectural drawing image through the following steps to obtain an image block set: The first step is to generate dynamic sliding window information based on the length and width of the aforementioned architectural engineering drawing image. This dynamic sliding window information includes the sliding window length and width. In practice, firstly, the executing entity can determine the non-overlap rate by subtracting a preset overlap rate from 1. Then, the product of the non-overlap rate and a preset number of horizontal slides is determined as a first value. Next, the sum of the first value and 1 is determined as a second value. Then, the executing entity can determine the sliding window width by dividing the width by the first value. Then, the executing entity can determine the non-overlap rate and a preset number of vertical slides as a third value. Then, the sum of the third value and 1 is determined as a fourth value. Finally, the executing entity can determine the sliding window length by dividing the length by the fourth value.

[0036] The second step involves generating sliding step size information based on the aforementioned dynamic sliding window information and the preset overlap rate. In practice, the executing entity can determine the non-overlap rate by subtracting the preset overlap rate from 1. Next, the executing entity can determine the vertical sliding step size by multiplying the sliding window length (included in the dynamic sliding window information) by the non-overlap rate. Then, the horizontal sliding step size is determined by multiplying the sliding window width (included in the dynamic sliding window information) by the non-overlap rate. Finally, the vertical and horizontal sliding step sizes are combined to form the sliding step size information.

[0037] The third step involves segmenting the architectural drawing image using a sliding window based on the aforementioned dynamic sliding window information and sliding step size information, resulting in an image block set. Each image block in the set corresponds to a row and column index, which includes a row index number and a column index number. In practice, the executing entity can, based on the dynamic sliding window information and sliding step size information, start from the top left corner of the architectural drawing image and slide the windows corresponding to the dynamic sliding window information sequentially from left to right and from top to bottom according to the sliding step size information to obtain the image block set. The row index number identifies the vertical arrangement sequence of the image blocks. The column index number identifies the horizontal arrangement sequence of the image blocks.

[0038] In addressing the aforementioned technical problems in the process of adopting technical solutions, the application scenario—digital analysis of architectural engineering drawings—often presents the following technical challenges: After being segmented by a sliding window, architectural engineering drawing images generate hundreds to thousands of image blocks. If a synchronous block-by-block sending method is used, each image block must wait for the previous recognition result before sending the next, resulting in idle CPU, network interface, and OCR server resources during the waiting period, leading to wasted processing resources. If unlimited batch concurrent sending is used, it may instantly exhaust the operating system's network port resources, trigger connection throttling or memory overflow on the OCR server, causing connection failures, recognition task interruptions, and an increased number of service crashes. Given the following requirements for this application scenario: the architectural engineering drawing analysis system needs to achieve high throughput and low latency image block recognition under the dual constraints of client hardware resources (network connection count, memory buffer) and server concurrency limits. Faced with these technical problems, we decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may asynchronously send the image block set to a preset character recognition engine through the following steps to generate an image block recognition information set corresponding to the architectural engineering drawing image: The first step is to store the image patch set into a preset queue to be processed.

[0039] The second step is to remove a preset number of image blocks from the preset queue to be processed and send them to the preset character recognition engine so that the preset character recognition engine can perform character recognition on the preset number of image blocks.

[0040] The third step involves performing the following generation steps on the preset queue to be processed: The first sub-step is to receive image block recognition information corresponding to an image block from the preset character recognition engine and to ensure that the preset processing queue is not empty, and to send an image block from the preset processing queue to the preset character recognition engine for character recognition of the image block.

[0041] The second sub-step involves removing the sent image blocks from the preset processing queue in order to update the preset processing queue.

[0042] The third step is to execute the above generation steps again based on the updated preset queue of pending processes.

[0043] The fourth step involves determining the received image block recognition information (at least one) as an image block recognition information set. Each image block recognition information in this set includes recognized text information and text box position information. The recognized text information can be the text obtained after OCR recognition of the image block by a preset character recognition engine. The text box position information can be the position of the text box containing the aforementioned text within the image block. This position information can be represented by the coordinates of the four vertices of the text box, i.e., the rectangle.

[0044] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "wasteful processing resources and increased service crashes during the recognition of image blocks in architectural engineering drawings." Factors leading to wasted processing resources and increased service crashes often include: After architectural engineering drawing images are cut by a sliding window, hundreds to thousands of image blocks are generated. If a synchronous block-by-block sending method is used, each image block must wait for the previous recognition result to return before the next one can be sent, causing CPU, network interface, and OCR server resources to be idle during the waiting period, resulting in wasted processing resources. If unlimited batch concurrent sending is used, it may instantly exhaust the operating system's network port resources, trigger connection rate limiting or memory overflow on the OCR server, causing connection failures, recognition task interruptions, and increased service crashes. Solving these factors can reduce wasted processing resources and decrease the number of service crashes. To achieve this effect, firstly, the image block set is stored in a preset processing queue. Thus, all image blocks to be recognized can be temporarily stored in the queue, providing a buffer for subsequent asynchronous scheduling. Then, a preset number of image blocks are removed from the preset processing queue and sent to the preset character recognition engine for character recognition. This allows a batch of image blocks to be sent immediately upon startup, enabling the OCR server and network interface to begin operation and avoiding idle waiting during the startup phase. Next, a generation step is performed on the preset processing queue: in response to receiving image block recognition information corresponding to an image block from the preset character recognition engine and ensuring the preset processing queue is not empty, one image block from the preset processing queue is sent to the preset character recognition engine for character recognition. Thus, whenever an image block is recognized and a concurrent slot is released, the next image block is immediately retrieved from the queue and sent, implementing an asynchronous pipeline scheduling mechanism of completing one and replenishing one, ensuring that the number of image blocks being processed always remains at the preset number, and the network connection and OCR server processing unit remain continuously busy. Then, the sent image blocks are removed from the preset processing queue to update the queue. This ensures that each image block is sent only once, avoiding duplicate processing. Based on the updated preset queue of images to be processed, the above generation steps are executed again. This forms a loop processing logic until all image blocks in the queue have been sent and recognized. Finally, the received image block recognition information at least one is determined as an image block recognition information set, where each image block recognition information in the set includes recognition text information and text box position information. Thus, a complete set of recognition results can be obtained, and each recognition result is associated with the spatial position information of the image block.Because it adopts an asynchronous pipeline scheduling mechanism that first sends a preset number of image blocks in parallel, and then sends an additional image block for each recognition result received, the number of image blocks being processed simultaneously remains stable near the preset number. This avoids the idle waiting of CPU, network interface, and OCR server resources caused by synchronous block-by-block sending, reducing the waste of processing resources; and also avoids the exhaustion of network port resources, server connection rate limiting, and memory overflow caused by unlimited batch concurrent sending, thereby reducing the number of connection failures, recognition task interruptions, and service crashes.

[0045] In some optional implementations of certain embodiments, the aforementioned execution entity may generate initial text recognition information based on the aforementioned image patch recognition information set through the following steps: The first step is to perform the following steps for each image patch recognition information in the above image patch recognition information set: The first sub-step involves determining the row and column index information of the image block corresponding to the image block identification information above as the target index information. For example, if an image is cut into a 3-row × 4-column grid, the row and column index information of the image block located in the 2nd row and 3rd column is: row index number 2 and column index number 3.

[0046] The second sub-step generates a horizontal offset based on the row index number included in the target index information and the sliding window width included in the dynamic sliding window information. In practice, the execution entity can determine the horizontal offset by multiplying the row index number by the sliding window width. For example, if the row index number of the current image block is 2 and the sliding window width is 1024 pixels, then the horizontal offset is 2048 pixels.

[0047] The third sub-step involves generating a vertical offset based on the column index number included in the target index information and the sliding window length included in the dynamic sliding window information. In practice, the vertical offset can be determined by multiplying the column index number by the sliding window length.

[0048] The fourth sub-step involves generating global text box position information corresponding to the identified text information included in the image patch recognition information, based on the aforementioned horizontal offset, vertical offset, and text box position information included in the image patch recognition information. In practice, for each of the four vertex coordinates included in the text box position information, the executing entity can add the horizontal offset to the vertex coordinate to determine the global vertex horizontal coordinate, and add the vertical offset to the vertex coordinate to determine the global vertex vertical coordinate. Then, the global vertex horizontal coordinate and global vertex vertical coordinate can be determined as the global vertex coordinates. Finally, the executing entity can determine each of the determined global vertex coordinates as the global text box position information corresponding to the identified text information included in the image patch recognition information.

[0049] The fifth sub-step involves generating the center point position information of the text box corresponding to the identified text information, based on the aforementioned global text box position information. This center point position information includes the ordinate and abscissa of the text box center point. In practice, the average of the abscissas of all global vertices included in the global text box position information is used as the abscissa of the text box center point. The average of the ordinates of all global vertices included in the global text box position information is then used as the ordinate of the text box center point. Finally, the ordinate and abscissa of the text box center point can be combined to form the text box center point position information.

[0050] The second step involves grouping the generated text box center point location information, including the ordinates of each text box's center point, into groups to obtain various text box groups and determine the vertical height of each group. In practice, the execution entity can use a ordinate-based clustering algorithm (e.g., sorting by ordinate and grouping adjacent text boxes with a ordinate difference less than a preset threshold (e.g., 20 pixels) into the same group) to group the text boxes. The vertical height refers to the maximum ordinate of the center points of all text boxes within a text box group.

[0051] The third step involves sorting the identified text information groups according to their vertical heights, resulting in a sequence of identified text information groups. This sequence refers to a list formed by arranging the identified text information groups in ascending order of their vertical height (i.e., from top to bottom). This sequence reflects the natural reading order of the text lines in the original image. In practice, the execution entity can use a standard sorting algorithm (such as quicksort or mergesort) with the vertical height value as the sorting key to sort the identified text information groups in ascending order.

[0052] The fourth step is to generate initial text recognition information based on the above-mentioned sequence of recognized text information groups.

[0053] In some optional implementations of certain embodiments, the aforementioned execution entity may generate initial text recognition information based on the aforementioned sequence of recognized text information groups through the following steps: The first step is to perform a horizontal sorting of the identified text information within each identified text information group in the aforementioned sequence, thereby updating the sequence. This horizontal sorting refers to rearranging multiple identified text information items within the same identified text information group according to the ascending (left-to-right) order of the x-coordinates of their text box center points. In practice, the execution entity can iterate through each identified text information group in the sequence, applying a standard sorting algorithm to each group's identified text information using the x-coordinate of the text box center point as the sorting key to sort it in ascending order. The sorted order within each group represents the left-to-right reading order of the text in that line.

[0054] The second step is to determine the updated recognition text information group sequence as the target recognition text information group sequence.

[0055] The third step involves performing the following deduplication and update steps on the target recognition text information sequence: The first sub-step involves performing the following steps for every three consecutive groups of identified text information in the target identification text information group sequence: In sub-step one, for each of the three consecutive groups of identified text information, a set of candidate overlapping text information corresponding to the identified text information is selected from the three consecutive groups of identified text information. In practice, the executing entity can determine the global text box position information corresponding to the identified text information as the target global text box position information. Then, each identified text information other than the identified text information in the three consecutive groups of identified text information can be determined as a candidate identified text information. Next, the global text box position information corresponding to each candidate identified text information can be determined as a candidate global text box position information. Afterward, for each candidate global text box position information, the executing entity can determine the intersection area of ​​the region represented by the candidate global text box position information and the region represented by the target global text box position information as the first area. Then, the union area of ​​the region represented by the candidate global text box position information and the region represented by the target global text box position information is determined as the second area. Then, the ratio of the first area to the second area can be determined as the target ratio corresponding to the candidate global text box position information. Then, at least one identified text information corresponding to at least one target ratio greater than a preset ratio threshold can be determined as a candidate overlapping text information set corresponding to the aforementioned identified text information.

[0056] Sub-step two: In response to determining that the candidate overlapping text information set is empty, the above-mentioned identified text information is retained in the above-mentioned target identified text information group sequence.

[0057] Sub-step three: In response to determining that the candidate overlapping text information set is not empty, determine the similarity between the identified text information and each candidate overlapping text information in the candidate overlapping text information set, and determine the similarity with the largest similarity among the determined similarities as the target similarity.

[0058] Sub-step four: Identify the overlapping text information in the candidate overlapping text information set that corresponds to the above target similarity as the target overlapping text information.

[0059] Sub-step five involves comparing the target overlapping text information with the aforementioned identified text information. Text information that meets preset conditions from both the target overlapping text information and the identified text information is retained in the target identified text information group sequence, while the other text information is deleted from the target identified text information group sequence to update the target identified text information group sequence. The preset condition can be that the area represented by the global text box position information corresponding to the identified text information is the largest of the two. Optionally, in practice, the executing entity can adopt the following preset conditions: comparing the text lengths of the target overlapping text information and the identified text information, retaining the one with more characters and deleting the one with fewer characters; or comparing the recognition confidence levels (provided by the OCR engine), retaining the one with higher confidence and deleting the one with lower confidence. For example, if the identified text information is "J-02" (confidence level 0.95) and the target overlapping text information is "1-02" (confidence level 0.72), then according to the preset conditions, "J-02" with higher confidence is retained, and "1-02" is deleted from the target identified text information group sequence.

[0060] Sub-step six: Based on the updated target recognition text information group sequence, generate a reference image block recognition information group sequence. Each reference image block recognition information group in the aforementioned reference image block recognition information group sequence corresponds to one target recognition text information in the aforementioned target recognition text information group sequence. Each reference image block recognition information in the aforementioned reference image block recognition information group sequence includes recognition text information and global text box position information. The aforementioned reference image block recognition information group sequence refers to the sequence formed by associating each target recognition text information (i.e., the recognition text information retained after sorting and deduplication) in the target recognition text information group sequence with its corresponding global text box position information, and reorganizing it according to the original group structure. In this sequence, each reference image block recognition information group corresponds to one target recognition text information group in the target recognition text information group sequence, and each reference image block recognition information within each reference image block recognition information group includes one recognition text information and the corresponding global text box position information.

[0061] The fourth step involves converting the reference image block recognition information sequence into a two-dimensional matrix of text, following the reading order, as the initial text recognition information. The reading order refers to the natural order from left to right and top to bottom in document reading habits. The two-dimensional matrix of text refers to organizing the recognized text information into a two-dimensional array structure according to its position on the page. The rows of the matrix correspond to the text lines in the original image, and the columns correspond to the text blocks or characters arranged from left to right in each line of text. In practice, the execution entity can iterate through each group in the reference image block recognition information sequence (in top-to-bottom order). For the reference image block recognition information within each group (in left-to-right order), the recognized text information is concatenated sequentially, with spaces or direct connections between the recognized text information within each group. Then, the text strings from different groups are separated by newline characters, ultimately forming a text string that retains the original page layout. This string is the initial text recognition information.

[0062] Step 105: Based on the initial text recognition information, perform shadow context injection processing on the preset prompt word information to update the preset prompt word information.

[0063] In some embodiments, the executing entity may perform shadow context injection processing on the preset prompt word information based on the initial text recognition information to update the preset prompt word information. The preset prompt word information refers to a predefined natural language instruction template used to guide the visual language cross-modal inference model to perform a specific structured extraction task. This template includes the task objective, output format requirements, and the names of the fields to be extracted. The preset prompt word information includes placeholders for filling the initial text recognition information. In practice, the executing entity may fill the empty spaces represented by the placeholders in the preset prompt word information with the initial text recognition information to update the preset prompt word information. For example, the default prompt message is: "The following is auxiliary text information identified from the drawing (for reference only): [Initial text recognition information to be filled]. Please perform the task in conjunction with the image: Extract the structured information of the architectural engineering drawing in the image. Task rules: 1. Visual semantic priority, that is, when the image information is clear, the visual content of the image is used as the primary basis for judgment; 2. OCR-assisted completion, that is, the above auxiliary text information is only referred to when the image is blurry or details are lost; 3. Conflict arbitration, that is, when the visual content of the image is inconsistent with the auxiliary text information, the visual content of the image shall prevail. The output format is JSON, and the fields include: drawing number, drawing name." Execute shadow context. After injection processing, the updated preset prompt information becomes: "The following is auxiliary text information identified from the drawing (for reference only): [Content of initial text recognition information]. Please perform the task in conjunction with the image: Extract the structured information of the architectural engineering drawing in the image. Task rules: 1. Visual semantic priority, that is, when the image information is clear, priority is given to judging based on the visual content of the image; 2. OCR-assisted completion, that is, the above auxiliary text information is only referred to when the image is blurry or details are lost; 3. Conflict arbitration, that is, when the visual content of the image is inconsistent with the auxiliary text information, the visual content of the image shall prevail. The output format is JSON, and the fields include: drawing number, drawing name."

[0064] Step 106: Input the scaled architectural engineering drawing image and the updated preset prompt information into the pre-trained visual language cross-modal reasoning model to obtain the initial architectural engineering drawing structured information.

[0065] In some embodiments, the aforementioned execution entity can input the scaled architectural drawing image and the updated preset prompt information into a pre-trained visual-language cross-modal reasoning model to obtain initial structured information of the architectural drawing. Here, the aforementioned visual-language cross-modal reasoning model refers to a deep learning model capable of simultaneously processing image visual information and text semantic information. This model can learn and establish cross-modal alignment relationships between image content and text description, and generate structured text output based on the input image and prompts.

[0066] In some optional implementations of certain embodiments, the aforementioned execution entity may input the scaled architectural drawing image and the updated preset prompt information into a pre-trained visual language cross-modal reasoning model through the following steps to obtain the initial structured information of the architectural drawing: The first step involves inputting the scaled architectural drawing image into the visual encoder included in the aforementioned visual-language cross-modal inference model to obtain the visual feature information of the architectural drawing. The aforementioned visual-language cross-modal inference model includes the visual encoder, text encoder, cross-modal fusion layer, and language decoder. The visual encoder can be a convolutional neural network (such as a deep residual network (ResNet)) used to extract the visual features of the scaled architectural drawing image. The aforementioned visual feature information of the architectural drawing refers to the multi-dimensional feature map or feature vector sequence output by the visual encoder after the scaled architectural drawing image is input.

[0067] The second step involves inputting the updated preset prompt word information into the text encoder included in the aforementioned visual-language cross-modal reasoning model to obtain the prompt word text semantic information. The text encoder can be an embedding layer used to extract the text semantic information of the preset prompt word information. This prompt word text semantic information can be a vector sequence output by the text encoder after the updated preset prompt word information is input. Each vector in this sequence corresponds to a token in the input text, representing the semantic role and meaning of that token in the context.

[0068] The third step involves outputting the visual feature information of the architectural drawings and the semantic information of the prompt words to the cross-modal fusion layer to obtain multimodal joint embedding feature information. The cross-modal fusion layer can be a Transformer layer based on a cross-attention mechanism, used to align, interact with, and fuse feature information from different modalities (visual and textual). The multimodal joint embedding feature information refers to the fused feature vector obtained after inputting the visual feature information of the architectural drawings and the semantic information of the prompt words into the cross-modal fusion layer and processing them through the cross-modal attention mechanism.

[0069] The fourth step involves inputting the aforementioned multimodal joint embedding feature information into the language decoder to obtain the initial structured information of the architectural engineering drawings. This initial structured information includes various field information, each with a field name and value, and each field value has a corresponding global text box position. The language decoder can be used to progressively generate initial structured information of the architectural engineering drawings that conforms to the grammatical and semantic rules of natural language based on the multimodal joint embedding feature information. This initial structured information refers to the structured data output by the language decoder, organized according to a preset format. This data describes the key information in the architectural engineering drawings in text form, specifically including various field information, each with a field name and value, and each field value has a corresponding global text box position. The field name refers to the key in the structured data, used to identify the type of information, such as "drawing number," "drawing name," or "design unit." The field value refers to the specific content corresponding to the field name, such as "J-02," "first floor plan," or "a certain architectural design institute."

[0070] Step 107: Perform backtracking verification and adaptive error correction on the initial architectural engineering drawing structured information to obtain the architectural engineering drawing structured information.

[0071] In some embodiments, the aforementioned execution entity may perform retrospective verification and adaptive error correction on the aforementioned initial architectural engineering drawing structured information to obtain architectural engineering drawing structured information.

[0072] In the process of adopting technical solutions to address the aforementioned technical problems, for the application scenarios—such as automated compliance checks based on drawings (e.g., construction departments or review agencies need to conduct compliance reviews of a large number of architectural engineering drawings to check whether various parameters in the drawings meet regulations)—which often require large-scale, high-precision extraction of structured information from drawings, the following technical problems often arise: When extracting structured information from large-format architectural engineering drawings, due to the limitations of the input resolution of the recognition model, it is often necessary to significantly downsample the original high-definition drawings. This results in the model being unable to clearly see some character details in the scaled image (such as the distinction between "J" and "1", "O" and "0", the absence of the hyphen "-", etc.), thus outputting incorrect field values ​​(e.g., misidentifying "J-02" as "102"). Simultaneously, the model's output lacks an effective self-checking mechanism, failing to utilize the complete detailed information in the original high-definition image to verify and correct its own output. If OCR results are directly used for post-processing replacement, new errors may be introduced due to visual interference errors inherent in OCR itself (such as misidentification caused by font style differences or background graphic interference), resulting in low accuracy of the extracted structural information from architectural drawings. This application scenario requires the following characteristics: the error rate of key fields in the extracted structural information from architectural drawings must be controlled at an extremely low level to meet the stringent requirements of project delivery and compliance review; the entire process should eliminate the need for manual field-by-field verification and should automatically complete error detection and correction. Faced with the above technical challenges, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity can obtain the structured information of the architectural drawings by performing retrospective verification and adaptive error correction on the initial architectural drawing structured information through the following steps: The first step is to extract key field information from the initial text recognition information, thus obtaining the information for each key field. Each key field includes a field name and a field value.

[0073] The second step involves performing the following validation steps for each field in the structured information of the initial architectural drawings: The first sub-step is to determine the field names included in the above field information as the target field names.

[0074] The second sub-step involves determining the field values ​​included in the above field information as the field values ​​to be corrected.

[0075] The third sub-step involves identifying the key field information containing the target field name mentioned above as reference field information.

[0076] The fourth sub-step, in response to determining that the field values ​​included in the above reference field information are the same as the field values ​​to be corrected, performs the following steps: Sub-step one involves performing format validation and completion processing on the aforementioned field values ​​to be corrected, resulting in corrected and completed field values. This format validation and completion processing refers to checking the format of field values ​​against preset field format rules (such as date format, drawing number format, and numeric format). For field values ​​that conform to the format but lack necessary components, the missing parts are automatically added according to the rules; for field values ​​with incorrect formats, they are corrected according to the rules. The corrected and completed field values ​​refer to field values ​​that conform to preset format specifications after format validation and completion processing. In practice, the execution entity can perform different format validation and completion operations based on the type of the target field name: If the target field name is "drawing number," the execution entity first checks whether the field value to be corrected conforms to a preset drawing number regular expression, for example, one or more letters followed by a hyphen, and then one or more numbers. If it conforms, it is directly used as the corrected and completed field value. If it does not conform but there is an obvious case that can be completed, for example, if the field value to be corrected is "JS01" (missing a hyphen), then a hyphen is inserted after the second character according to the rules, resulting in "JS-01" as the corrected and completed field value. If the target field is named "Date", the execution entity first attempts to parse the value of the field to be corrected into a date object. If the value is "2024-5-12" (the month is missing a leading zero), it is padded to "2024-05-12" as the correction completion field value. If the value is "2024.05.12" (the separator does not conform to the standard), it is converted to "2024-05-12" as the correction completion field value. If the target field is named "Scale", the execution entity first checks whether the value of the field to be corrected conforms to the scale format, such as "1:100" or "1 / 100". If the value is "1:100" (the colon is full-width), the full-width colon is replaced with a half-width colon to obtain "1:100" as the correction completion field value.

[0077] Sub-step two involves replacing the field values ​​included in the above field information with the above-mentioned complete field values ​​to update the field information.

[0078] The fifth sub-step, in response to determining that the field values ​​included in the above reference field information are different from the above field values ​​to be corrected, performs the following steps: Sub-step one: Determine the global text box position information corresponding to the above field information as the local partition position information.

[0079] Sub-step two involves identifying the local image corresponding to the aforementioned local division location information in the architectural engineering drawing image as the local image to be identified. Here, the aforementioned local image to be identified refers to a sub-image cropped from the original architectural engineering drawing image based on the local division location information.

[0080] Sub-step three involves inputting the aforementioned local image to be identified and the aforementioned field values ​​to be corrected into a pre-trained visual language model to obtain the identification field values.

[0081] Sub-step four involves performing format validation and completion processing on the above-mentioned identification field values ​​to obtain corrected and completed field values.

[0082] Sub-step five involves replacing the values ​​of the fields to be corrected in each of the above fields with the values ​​of the correction and completion fields to update the information in each field.

[0083] The third step is to identify the updated field information as the structured information of the architectural drawings.

[0084] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "low accuracy of extracted structural information from architectural engineering drawings." Factors leading to low accuracy of extracted structural information from architectural engineering drawings often include: when extracting structural information from large-format architectural engineering drawings, the input resolution of the recognition model is usually limited, requiring significant downsampling of the original high-definition drawings. This results in the model being unable to clearly see some character details in the scaled image (such as distinguishing between "J" and "1", "O" and "0", missing hyphens, etc.), thus outputting incorrect field values ​​(e.g., misidentifying "J-02" as "102"). Simultaneously, the model's output lacks an effective self-checking mechanism, failing to utilize the complete detail information in the original high-definition image to verify and correct its own output. If OCR results are directly used for post-processing replacement, new errors may be introduced due to visual interference errors inherent in OCR itself (such as font style differences, misidentification caused by background graphic interference), leading to low accuracy of extracted structural information from architectural engineering drawings. Solving these factors can improve the accuracy of extracted structural information from architectural engineering drawings. To achieve this effect, firstly, key field information is extracted from the initial text recognition information to obtain each key field. Then, for each field in the initial architectural drawing structured information, the field name is determined as the target field name, the field value is determined as the field value to be corrected, and the key field information containing the target field name is determined as the reference field information. This establishes a correspondence between the field information to be verified and the reference baseline information. Subsequently, in response to the determination that the field value in the reference field information is the same as the field value to be corrected, the following steps are performed: format verification and completion processing is performed on the field value to be corrected to obtain the corrected and completed field value. The field values ​​in each of the aforementioned field information are replaced with the completed field value to update each field information. Therefore, when the field value to be corrected has been confirmed as correct by the reference baseline information, there is no need to call a complex visual language model for re-recognition; correction can be completed solely through lightweight format verification and completion, thereby reducing unnecessary computational overhead and improving processing efficiency while ensuring accuracy. Alternatively, in response to the determination that the field values ​​included in the aforementioned reference field information are different from the aforementioned field values ​​to be corrected, the following steps are performed: The global text box position information corresponding to the aforementioned field information is determined as the local segmentation position information. The local image corresponding to the aforementioned local segmentation position information in the architectural drawing image is determined as the local image to be identified. The aforementioned local image to be identified and the aforementioned field values ​​to be corrected are input into a pre-trained visual language model to obtain the identified field values. The aforementioned identified field values ​​are subjected to format validation and completion processing to obtain the corrected and completed field values.The field values ​​included in the aforementioned field information are replaced with the supplementary field values ​​to update each field information. Therefore, when the field value to be corrected does not match the field value included in the reference field information, the corresponding local image area in the original high-resolution drawing can be accurately located. A visual language model is then used to specifically identify this local image, thus avoiding the loss of detail caused by global downsampling, effectively distinguishing easily confused characters (such as "J" and "1", "O" and "0") and restoring missing connectors (such as "-"), while avoiding visual interference errors that may be introduced by directly using OCR results. Furthermore, since this recognition process only targets a small local image rather than the entire large image, it overcomes the input resolution limitations of the recognition model. Finally, the updated field information is identified as the structured information of the architectural engineering drawing. Thus, high-precision structured information that has undergone field-by-field verification, correction, and supplementation can be obtained. Because it employs a method of only formatting correct fields and triggering local high-definition re-recognition for incorrect fields, it fully utilizes the detailed information of the original high-definition image while avoiding the inefficiency of global re-recognition and the unreliability of direct OCR replacement, thus improving the accuracy of the extracted structural information from architectural drawings.

[0085] The above-described embodiments of this disclosure have the following beneficial effects: the method for extracting structured information from architectural drawings according to some embodiments of this disclosure improves the accuracy of structured information recognition and extraction from architectural drawings. Specifically, the reason for the low accuracy of structured information recognition and extraction from architectural drawings is that when using an extraction method based on Optical Character Recognition (OCR) to extract structured information from architectural drawings, there are various visual interference factors in the drawings, such as differences in font styles (thickness, slant, character spacing) between different design institutes, structural confusion of similar characters (such as the uppercase letter "J" and the number "1", the letter "O" and the number "0"), and background graphic superposition interference (table lines passing through text, drawing frames covering text areas). These interferences cause OCR to misidentify correct characters as similar-looking characters, for example, misidentifying the drawing number "J-02" as "1-02". Because the OCR model lacks semantic understanding ability, it cannot self-correct misidentification based on context (such as "the architectural code should be J"), resulting in low accuracy of structured information recognition and extraction from architectural drawings. Based on this, the method for extracting structured information from architectural drawings according to some embodiments of this disclosure first acquires an image of the architectural drawing. Then, in response to determining that the size information of the architectural drawing image does not meet a preset condition, the architectural drawing image is downsampled to obtain a scaled architectural drawing image. This allows large-size drawings to be adjusted to a size range suitable for subsequent model processing, avoiding low processing efficiency or memory overflow due to excessively large images. Next, image quality detection processing is performed on the scaled architectural drawing image to obtain image quality detection information. This allows for quantitative evaluation of image quality, providing a decision-making basis for different subsequent processing paths. Then, in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, character recognition processing is performed on the architectural drawing image to obtain initial text recognition information. This allows for the extraction of preliminary text information through character recognition processing even when image quality is poor. Afterwards, based on the initial text recognition information, shadow context injection processing is performed on preset prompt word information to update the preset prompt word information. This allows the recognized text, i.e., the initial text recognition information, to be injected into the prompt words as contextual clues, so that the prompt words contain initial text recognition information specific to the current drawing. Next, the scaled architectural drawing image and the updated preset prompt information are input into a pre-trained visual-language cross-modal inference model to obtain the initial structured information of the architectural drawing. Thus, the visual-language model can simultaneously refer to the visual information of the image and the prompts containing the initial text recognition information, utilizing its semantic understanding capabilities to correct errors in the initial text recognition information, such as misidentifying "J" as "1" due to similar-looking characters. Finally, the initial structured information of the architectural drawing is subjected to backtracking verification and adaptive error correction processing to obtain the final structured information of the architectural drawing.Therefore, the model output can be further verified and corrected. Because the initial text recognition information is used as a shadow context injection prompt, and a visual language cross-modal reasoning model is employed to combine images and context for semantic understanding, the model can self-correct misidentification of similar-looking characters in the initial text recognition information based on the contextual semantics of the drawing (e.g., "the architectural code should be J"), thereby improving the accuracy of structured information recognition and extraction from architectural engineering drawings.

[0086] Further reference Figure 2 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a structured information extraction device for architectural engineering drawings. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0087] like Figure 2 As shown, a structured information extraction device 200 for architectural drawings in some embodiments includes: an acquisition unit 201, a downsampling processing unit 202, an image quality detection unit 203, a character recognition unit 204, a shadow context injection unit 205, an input unit 206, and a backtracking verification and adaptive error correction unit 207. The acquisition unit 201 is configured to acquire an image of an architectural drawing; the downsampling processing unit 202 is configured to perform downsampling processing on the architectural drawing image to obtain a scaled architectural drawing image in response to determining that the size information of the architectural drawing image does not meet a preset condition; the image quality detection unit 203 is configured to perform image quality detection processing on the scaled architectural drawing image to obtain image quality detection information; and the character recognition unit 204 is configured to perform character recognition processing on the architectural drawing image in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold. The initial text recognition information is obtained; the shadow context injection unit 205 is configured to perform shadow context injection processing on the preset prompt word information based on the initial text recognition information to update the preset prompt word information; the input unit 206 is configured to input the scaled architectural engineering drawing image and the updated preset prompt word information into the pre-trained visual language cross-modal reasoning model to obtain the initial architectural engineering drawing structured information; the backtracking verification and adaptive error correction unit 207 is configured to perform backtracking verification and adaptive error correction processing on the initial architectural engineering drawing structured information to obtain the architectural engineering drawing structured information.

[0088] It is understandable that the units described in the device 200 are related to the reference. Figure 1The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0089] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0090] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0091] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0092] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0093] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0094] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0095] A computer-readable medium may be contained within an electronic device or may exist independently, not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an image of an architectural drawing; in response to determining that the size information of the architectural drawing image does not meet a preset condition, downsample the architectural drawing image to obtain a scaled architectural drawing image; perform image quality detection processing on the scaled architectural drawing image to obtain image quality detection information; in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, perform character recognition processing on the architectural drawing image to obtain initial text recognition information; based on the initial text recognition information, perform shadow context injection processing on preset prompt word information to update the preset prompt word information; input the scaled architectural drawing image and the updated preset prompt word information into a pre-trained visual language cross-modal reasoning model to obtain initial architectural drawing structured information; and perform backtracking verification and adaptive error correction processing on the initial architectural drawing structured information to obtain architectural drawing structured information.

[0096] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a downsampling processing unit, an image quality detection unit, a character recognition unit, a shadow context injection unit, an input unit, and a backtracking verification and adaptive error correction unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as a "unit for acquiring architectural engineering drawing images."

[0099] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0100] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for extracting structured information from architectural engineering drawings, characterized by: Obtain architectural engineering drawings and images; In response to the determination that the size information of the architectural engineering drawing image does not meet the preset conditions, the architectural engineering drawing image is downsampled to obtain a scaled architectural engineering drawing image. The scaled architectural drawing image is subjected to image quality detection processing to obtain image quality detection information; In response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, character recognition processing is performed on the architectural engineering drawing image to obtain initial text recognition information; Based on the initial text recognition information, shadow context injection processing is performed on the preset prompt word information to update the preset prompt word information; The scaled architectural drawing image and the updated preset prompt information are input into a pre-trained visual language cross-modal reasoning model to obtain the initial structured information of the architectural drawing. The initial architectural engineering drawing structured information is backtracked and adaptively corrected to obtain the architectural engineering drawing structured information.

2. The method according to claim 1, characterized in that, The downsampling process of the architectural engineering drawing image to obtain a scaled architectural engineering drawing image includes: The architectural engineering drawing image is subjected to Gaussian blurring to obtain a blurred preprocessed image; The blurred preprocessed image is downsampled to reduce it to a preset target size; The reduced and blurred preprocessed image is identified as a scaled architectural drawing image.

3. The method according to claim 1, characterized in that, The character recognition processing of the architectural engineering drawing image to obtain initial text recognition information includes: The architectural engineering drawing image is dynamically sliced ​​to obtain an image block set; The image block set is asynchronously sent to a preset character recognition engine to generate an image block recognition information set corresponding to the architectural engineering drawing image; Based on the image block recognition information set, initial text recognition information is generated.

4. The method according to claim 3, characterized in that, The dynamic slicing process of the architectural engineering drawing image to obtain an image block set includes: Based on the length and width of the architectural engineering drawing image, dynamic sliding window information is generated, wherein the dynamic sliding window information includes the sliding window length and the sliding window width; Based on the dynamic sliding window information and the preset overlap rate, the sliding step size information is generated; Based on the dynamic sliding window information and sliding step information, the architectural engineering drawing image is segmented by sliding window to obtain an image block set. Each image block in the image block set corresponds to row and column index information, which includes row index number and column index number.

5. The method according to claim 4, characterized in that, The process of generating initial text recognition information based on the image patch recognition information set includes: For each image block recognition information in the image block recognition information set, perform the following steps: The row and column index information of the image block corresponding to the image block recognition information is determined as the target index information; Based on the row index number included in the target index information and the sliding window width included in the dynamic sliding window information, a horizontal offset is generated; Based on the column index number included in the target index information and the sliding window length included in the dynamic sliding window information, a vertical offset is generated; Based on the horizontal offset, the vertical offset, and the text box position information included in the image block recognition information, global text box position information corresponding to the recognized text information included in the image block recognition information is generated; Based on the global text box position information, text box center point position information corresponding to the identified text information is generated, wherein the text box center point position information includes the vertical coordinate of the text box center point and the horizontal coordinate of the text box center point; Based on the generated center point location information of each text box, including the vertical coordinates of the center points of each text box, the recognition text information included in the image block recognition information set is grouped to obtain each recognition text information group, and the vertical height of each recognition text information group is determined. According to the vertical height corresponding to each recognition text information group, sort the recognition text information groups to obtain the recognition text information group sequence; Based on the sequence of identified text information groups, initial text identification information is generated.

6. The method according to claim 5, characterized in that, The step of generating initial text recognition information based on the recognized text information group sequence includes: For each identified text information group in the identified text information group sequence, the identified text information in the identified text information group is horizontally sorted in order to update the identified text information group sequence; The updated sequence of recognized text information groups is determined as the target sequence of recognized text information groups. For the target recognition text information group sequence, perform the following deduplication and update steps: For every three consecutive groups of identified text information in the target identification text information sequence, perform the following steps: For each of the three consecutive groups of identified text information, a set of candidate overlapping text information corresponding to the identified text information is selected from the three consecutive groups of identified text information: In response to determining that the candidate overlapping text information set is empty, the identified text information is retained in the target identified text information group sequence; In response to determining that the candidate overlapping text information set is not empty, the similarity between the identified text information and each candidate overlapping text information in the candidate overlapping text information set is determined, and the similarity with the largest similarity among the determined similarities is determined as the target similarity. The overlapping text information in the candidate overlapping text information set that corresponds to the target similarity is determined as the target overlapping text information; The target overlapping text information is compared with the identified text information to retain text information that meets the preset conditions in the target identified text information group sequence, and the other text information is deleted from the target identified text information group sequence to update the target identified text information group sequence. Based on the updated target recognition text information group sequence, a reference image block recognition information group sequence is generated, wherein each reference image block recognition information group in the reference image block recognition information group sequence corresponds to a target recognition text information in the target recognition text information group sequence, and each reference image block recognition information in the reference image block recognition information group sequence includes recognition text information and global text box position information; Following the reading order, the sequence of reference image block recognition information groups is converted into text in the form of a two-dimensional matrix as the initial text recognition information.

7. The method according to claim 1, characterized in that, The process of inputting the scaled architectural drawing image and the updated preset prompt information into a pre-trained visual language cross-modal reasoning model to obtain the initial structured information of the architectural drawing includes: The scaled architectural drawing image is input into the visual encoder included in the visual language cross-modal reasoning model to obtain the visual feature information of the architectural drawing. The visual language cross-modal reasoning model includes the visual encoder, text encoder, cross-modal fusion layer and language decoder. The updated preset prompt word information is input into the text encoder included in the visual language cross-modal reasoning model to obtain the prompt word text semantic information; The visual feature information of the architectural drawings and the semantic information of the prompt words are output to the cross-modal fusion layer to obtain multimodal joint embedding feature information; The multimodal joint embedding feature information is input into the language decoder to obtain the initial architectural engineering drawing structured information, wherein the initial architectural engineering drawing structured information includes various field information, each field information includes a field name and a field value, and each field value has corresponding global text box position information.

8. A device for extracting structured information from architectural drawings, characterized in that: The acquisition unit is configured to acquire images of architectural drawings. The downsampling processing unit is configured to perform downsampling processing on the architectural drawing image in response to determining that the size information of the architectural drawing image does not meet a preset condition, thereby obtaining a scaled architectural drawing image. An image quality detection unit is configured to perform image quality detection processing on the scaled architectural engineering drawing image to obtain image quality detection information; The character recognition unit is configured to perform character recognition processing on the architectural drawing image in response to determining that the quality score represented by the image quality detection information is less than a preset score threshold, so as to obtain initial text recognition information. The shadow context injection unit is configured to perform shadow context injection processing on the preset prompt word information based on the initial text recognition information, so as to update the preset prompt word information; The input unit is configured to input the scaled architectural drawing image and the updated preset prompt information into a pre-trained visual language cross-modal reasoning model to obtain the initial architectural drawing structured information. The backtracking verification and adaptive error correction unit is configured to perform backtracking verification and adaptive error correction processing on the initial architectural engineering drawing structured information to obtain architectural engineering drawing structured information.

9. An electronic device, characterized in that: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images