Page analysis method, apparatus and electronic device

Abstract

The present disclosure provides a layout analysis method and device and electronic equipment, the method comprising: obtaining an electronic component specification sheet image; performing feature extraction of different scales on the electronic component specification sheet image to obtain a plurality of feature maps; performing one feature fusion processing on all feature maps according to an execution path from high to low of the feature level to obtain a one feature fusion map; performing secondary feature fusion processing on all one feature fusion maps according to an execution path from low to high of the feature fusion level to obtain a secondary feature fusion map; performing attention interaction between the secondary feature fusion map and position information and an object query vector, analyzing a corresponding query feature vector, and predicting a layout element type and a corresponding bounding box coordinate. Through the bidirectional fusion path, the features are bidirectionally enhanced, the obtained feature fusion map has the characteristics of rich details and clear semantics, the feature expression capability of each feature level is greatly improved, and the recognition accuracy for complex layout elements is effectively enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a layout analysis method, apparatus, and electronic device. Background Technology

[0002] In the digital age, most documents exist in electronic form. In order to extract key information from electronic documents, it is necessary to perform layout recognition on electronic documents. Related analysis technologies include image processing technology or deep learning models. By analyzing and processing electronic documents, layout elements in electronic documents can be extracted, thereby determining the key information of the documents.

[0003] In related technologies, the accuracy of recognition results is poor for complex electronic documents, such as electronic component specification sheets, which contain complex layouts. It is difficult to accurately identify and locate elements in electronic component specification sheets. Summary of the Invention

[0004] This disclosure provides a layout analysis method, apparatus, and electronic device to at least solve the above-mentioned technical problems existing in the prior art.

[0005] According to a first aspect of this disclosure, a layout analysis method is provided, the method comprising:

[0006] Obtain images of electronic component specification sheets; Feature extraction at different scales is performed on the electronic component specification sheet image to obtain a multi-layer feature map; Perform feature fusion processing on all feature maps according to the execution path from high to low feature level to obtain a feature fusion map corresponding to each layer of feature maps; All primary feature fusion maps are processed into secondary feature fusion maps according to the execution path from low to high feature fusion level; After fusing the secondary feature fusion image with the preset position information of the electronic component specification image, feature matching is performed with the pre-generated object query vector to determine the corresponding query feature vector. By analyzing the query feature vector, the predicted layout element type and the corresponding bounding box coordinates are obtained.

[0007] In one possible implementation, the feature map has N layers, and accordingly, determining the feature map includes: Convolutional processing is performed on a portion of the k-th layer feature map to obtain the corresponding intermediate feature map; After convolving all channels of the intermediate feature map, it is fused with the k-th layer feature map to obtain the (k+1)-th layer feature map; where 1≤k≤N-1, and the first layer feature map is determined based on the feature extraction of the initial feature map of the electronic component specification sheet image.

[0008] In one possible implementation, the primary feature fusion map has N layers. Correspondingly, the process of performing a primary feature fusion process on all feature maps according to an execution path from high to low feature level to obtain the corresponding primary feature fusion map includes: Upsampling is performed on the feature fusion map of the (k+1)th layer to obtain the upsampling result; The upsampling result and the k-th layer feature map are fused once to obtain the k-th layer first feature fusion map; where 1≤k≤N-1, and the N-th layer first feature fusion map is the N-th layer feature map.

[0009] In one possible implementation, the step of performing a feature fusion process on all feature maps according to the execution path from high to low feature level to obtain a corresponding first-level feature fusion map further includes: Perform a convolution operation on the first feature fusion map.

[0010] In one possible implementation, the secondary feature fusion map has N layers. Correspondingly, the secondary feature fusion process, which involves performing secondary feature fusion on all primary feature fusion maps according to an execution path from low to high feature fusion level to obtain the corresponding secondary feature fusion map, includes: The k-th layer quadratic feature fusion map is downsampled to obtain the downsampling result; After fusing the downsampling result with the first-order feature fusion map of the (k+1)th layer, a second-order feature fusion map of the (k+1)th layer is obtained; where 1≤k≤N-1, and the second-order feature fusion map of the first layer is the first-order feature fusion map of the first layer.

[0011] In one possible implementation, the method further includes: The primary feature fusion map and / or the secondary feature fusion map are processed at different resolution scales.

[0012] In one possible implementation, the method further includes: The corresponding loss function is determined based on the coordinate deviation between the predicted bounding box and the true bounding box, and the model is updated and optimized using the loss function.

[0013] In one possible implementation, determining the corresponding loss function based on the coordinate deviation between the predicted bounding box and the true bounding box includes: The cross-union ratio (CUNR) is determined based on the cross-union ratio between the coordinates of the predicted bounding box and the true bounding box, the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the true bounding box, and the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the true bounding box. The center distance loss is determined based on the deviation between the center point coordinates of the predicted bounding box and the center point coordinates of the actual bounding box. The keypoint loss is determined based on the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the actual bounding box, and the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the actual bounding box. The corresponding loss function is obtained by adding the products of the cross-union ratio loss and its corresponding weight, the center distance loss and its corresponding weight, and the keypoint loss and its corresponding weight.

[0014] According to a second aspect of this disclosure, a layout analysis apparatus is provided, comprising: The image acquisition module is used to acquire images of electronic component specification sheets; The feature extraction module is used to extract features at different scales from the electronic component specification sheet image to obtain a multi-layer feature map; The first feature fusion module is used to perform a first feature fusion process on all feature maps according to the execution path from high to low feature level, so as to obtain the first feature fusion map corresponding to each layer of feature map; The secondary feature fusion module is used to perform secondary feature fusion processing on all primary feature fusion maps according to the execution path from low to high feature fusion level, so as to obtain the corresponding secondary feature fusion map. The prediction module is used to fuse the secondary feature fusion image with the preset position information of the electronic component specification image, perform feature matching with the pre-generated object query vector, determine the corresponding query feature vector, and obtain the predicted layout element type and corresponding bounding box coordinates by analyzing the query feature vector.

[0015] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0016] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0017] This disclosure discloses a layout analysis method that extracts features at different scales from electronic component specification sheet images to obtain multi-layer feature maps. For all feature maps, a first-level feature fusion process is performed according to the feature level, from high to low, passing high-level features to the lower levels and supplementing the lower-level features with semantic information from the high-level features, resulting in a corresponding first-level feature fusion map. Then, for all first-level feature fusion maps, a second-level feature fusion process is performed according to the feature level, from low to high, passing low-level features back to high-level features and supplementing the high-level features with detailed information from the lower-level features, resulting in a second-level fused feature map. This second-level fused feature map is then fused with location information and subjected to multi-layer cross-attention interaction with the object query vector. By analyzing the obtained query feature vector, the type of each potential layout element and its corresponding bounding box coordinates can be predicted. In other words, the features are enhanced bidirectionally through a bidirectional fusion path. The resulting second-level feature fusion map possesses both rich detail and clear semantics, significantly improving the feature representation ability at each feature level. Using this as the basis for analysis can effectively enhance the recognition accuracy for complex layout elements such as text, images, and tables, ensuring the accuracy of the layout analysis results.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0020] Figure 1 A schematic diagram illustrating the implementation flow of the layout analysis method according to an embodiment of this disclosure is shown; Figure 2 This diagram illustrates the structure of the lightweight network layer in the layout analysis method of this embodiment. Figure 3 A schematic diagram of the processing flow of the feature pyramid network and path aggregation module in the layout analysis method of this disclosure embodiment is shown; Figure 4 A schematic diagram of the loss function determination process in the layout analysis method of this disclosure is shown; Figure 5 A schematic diagram of the predicted bounding box determination process in the layout analysis method of this disclosure is shown; Figure 6 The diagram shows the effect of the layout analysis method according to an embodiment of the present disclosure; Figure 7 A schematic diagram of the layout analysis device according to an embodiment of this disclosure is shown; Figure 8 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0022] The core issue of layout analysis is how to automatically identify and locate these different elements in a document. This is not just a simple classification problem, but a complex task involving spatial relationships between elements, logical structure, and understanding of the overall layout.

[0023] Specifically, layout analysis needs to solve the problems of element type identification, element position and boundary determination, layout structure, and information extraction and utilization, and is mainly applied in the fields of papers, contracts, invoices or certificates.

[0024] Existing techniques are typically effective only for specific types of document layout analysis, relying on manually designed features and heuristic rules, resulting in poor generalization ability and difficulty adapting to complex and ever-changing layouts. For documents like electronic component specification sheets with variable layouts and formats, these techniques cannot accurately identify and locate all elements. Furthermore, object detection-based methods are computationally intensive and lack real-time performance. In Transformer-based layout analysis tasks, while Transformer possesses powerful global feature capture and relationship modeling capabilities, its ability to acquire local features is insufficient. Accurate capture of local information is crucial for identifying specific elements (such as text, images, tables, etc.) and their details within a document layout.

[0025] The layout analysis method provided in this application is applied to a layout analysis model, which can be deployed on one or more electronic devices. These electronic devices can be smart terminals or servers; smart terminals can be mobile phones, personal digital assistants (PADs), tablets, laptops, desktop computers, etc. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) services, and big data and artificial intelligence platforms.

[0026] The following description, in conjunction with the accompanying drawings, introduces a layout analysis method, apparatus, and device provided in this application.

[0027] like Figure 1 As shown, this application provides a layout analysis method, applied to a layout analysis model and executed by an electronic device deploying the layout analysis model, including: S101. Obtain images of electronic component specification sheets.

[0028] In this step, an image acquisition device can be used to capture images of the electronic component specification sheet. These images include text blocks, images, and charts, distributed across different areas of the image and representing different document information. For example, Table A at the top of the image represents information such as the type, quantity, and specifications of the products contained in a box.

[0029] S102. Extract features at different scales from the electronic component specification sheet image to obtain a multi-layer feature map.

[0030] In this step, features at different scales are extracted from the electronic component specification sheet image based on the backbone network of the layout analysis model. The layout analysis module can be an RT-DETR model, and the backbone network can be a ResNet-18 backbone network. It should be noted that this step replaces the basic block in the ResNet-18 backbone network with the lightweight and fast network module FasterNet Block. The lightweight and fast network module significantly reduces the model's computational load based on techniques such as depthwise separable convolution and inverse residual blocks, thereby reducing computational resource consumption. Furthermore, it can instantly identify and classify various layout elements, including text blocks, images, and charts, in the document flow corresponding to the electronic component specification sheet, optimizing model processing time.

[0031] The lightweight fast network module in this step includes at least three lightweight fast network layers connected in sequence. After the electronic component specification sheet image is input into the backbone network, it passes through the aforementioned lightweight fast network layers for visual feature extraction. Specifically, the first lightweight fast network layer extracts the visual feature map corresponding to the electronic component specification sheet image, denoted as the first visual feature map, corresponding to the bottom layer features. The second lightweight fast network module further extracts features from this first visual feature map to obtain the corresponding visual feature map, denoted as the second visual feature map, corresponding to the middle layer features, and so on, to obtain the high-level features. That is, each lightweight fast network layer outputs a feature map of a certain scale.

[0032] Since electronic component specification sheet images primarily consist of text blocks, images, and charts, their features not only possess rich local details, such as text edges and table lines, but also exhibit strong global semantic relationships, such as the correspondence between parameter text and tables. This step extracts features from electronic component specification sheet images at different scales, obtaining at least three visual feature maps at different scales, which can cover the full-scale target feature requirements of specification sheets, from small text to large tables.

[0033] S103. Perform feature fusion processing on all feature maps according to the execution path from high to low feature level to obtain a feature fusion map corresponding to each feature map level.

[0034] In this step, the visual feature maps at each scale obtained in step S102 are processed once according to the execution path from high to low feature hierarchy. This allows high-level features to be passed down from top to bottom, thus enhancing the expression of low-level features by passing semantic information from high-level to low-level features. Except for the highest-level visual feature map, each remaining visual feature map at each scale, after undergoing one feature fusion process, yields a corresponding first-level feature fusion map. The first-level feature map of the highest-level visual feature map is itself.

[0035] It should be noted that this step implements the aforementioned feature fusion process based on the neck network in the layout analysis model. Accordingly, the neck network uses visual feature maps of different scales output by the backbone network as a basis, and performs upsampling using the Feature Pyramid Network (FPN) set in the neck network to pass high-level features to the middle and bottom layers. This supplements the middle and bottom layer features with semantic information such as text category and table attributes, thereby improving the shortcomings of multi-level features, where the bottom features have many details but weak semantics. This also prevents the layout analysis model from misclassifying special parameter text as ordinary text, thus providing a more accurate data foundation for the subsequent layout analysis process.

[0036] S104. Perform secondary feature fusion processing on all primary feature fusion maps according to the execution path from low to high feature fusion level to obtain the corresponding secondary feature fusion maps.

[0037] In this step, based on step S103, all primary feature fusion maps are further subjected to secondary feature fusion according to the execution path from low to high feature fusion level. The lower-level primary feature fusion map after primary feature fusion is passed from bottom to top to the higher level, that is, secondary feature fusion is performed to supplement the higher-level features with detailed information such as table corners and text edges, so as to improve the problem of strong semantics but little detail in the high-level features of multi-level features.

[0038] It should be noted that this step utilizes the path aggregation module PANet set in the neck network to perform secondary feature fusion processing on the primary feature fusion map obtained from the feature pyramid network, performing downsampling from the bottom up. Except for the lowest-level primary feature fusion map, the primary feature fusion maps at each remaining scale undergo secondary feature fusion processing to obtain corresponding secondary feature fusion maps. The secondary feature fusion map corresponding to the lowest-level primary feature fusion map is itself. All the resulting secondary feature fusion maps possess both rich detail and clear semantics.

[0039] S105. After fusing the secondary feature fusion image with the preset position information of the electronic component specification image, perform feature matching with the pre-generated object query vector to determine the corresponding query feature vector. By analyzing the query feature vector, obtain the predicted layout element type and the corresponding bounding box coordinates.

[0040] In this step, the highest-level secondary feature fusion map, representing the multi-scale secondary feature fusion map processed by the neck network, is fed into the Transformer decoder in the layout analysis model. The Transformer decoder performs target detection and classification based on the highest-level secondary feature fusion map. In this step, the location information of the electronic component specifications is represented by a location embedding vector, which is used to assign a unique spatial identifier to each spatial location in the secondary feature fusion map, i.e., to provide spatial coordinate information. By adding and fusing the secondary feature fusion map with the location embedding vector, spatial location information is supplemented to the secondary feature fusion map, compensating for the lack of spatial localization correlation in pure visual features. This results in a secondary feature fusion map carrying spatial location identifiers, thus providing a prerequisite for the subsequent attention mechanism to focus on specific regions.

[0041] Furthermore, the secondary feature fusion map carrying spatial location identifiers is matched with the pre-generated object query vectors, and the corresponding query feature vector is obtained through attention interaction. It should be noted that the object query vectors in this step are used to detect potential layout element targets; that is, each object query vector corresponds to a potential detection target, such as tables, parameter text blocks, images, and titles in a specification document. During the attention interaction process, the object query vector is used as the query, and the secondary feature fusion map carrying spatial location identifiers is used as the key and value. For each object query vector, by comparing the features of all spatial locations in the secondary feature fusion map that incorporates spatial location information, the target region that best matches the object query vector is selected. In other words, the attention interaction process causes each object query vector to focus on the region in the feature map that is most likely its target. The semantic information and spatial location identifier information corresponding to the target region are then integrated into the object query vector to obtain a query feature vector containing both layout element semantic information and spatial location information, which serves as the basis for subsequent element type identification and bounding box coordinate prediction.

[0042] Furthermore, the above query feature vector is input into the prediction head, which parses the query feature vector through the prediction head in the Transformer decoder, and outputs the layout element type and the corresponding bounding box coordinates for each query, thus ultimately realizing the prediction of layout element type and bounding box coordinates.

[0043] This disclosure discloses a layout analysis method that extracts features at different scales from an image of electronic component specification sheets to obtain multi-layer feature maps. For all feature maps, a first-level feature fusion process is performed, following a path from high to low feature level. High-level features are passed to lower-level features, supplementing them with semantic information from the high-level features, resulting in a corresponding first-level feature fusion map. Then, for all first-level feature fusion maps, a second-level feature fusion process is performed, following a path from low to high feature level. This reverses the process, passing low-level features to high-level features, supplementing them with detailed information from the low-level features, resulting in a second-level fused feature map. This second-level fused feature map is then fused with location information and subjected to multi-layer cross-attention interaction with an object query vector. By analyzing the obtained query feature vector, the type of each potential layout element and its corresponding bounding box coordinates can be predicted. In other words, the features are enhanced bidirectionally through a bidirectional fusion path. The resulting second-level feature fusion map possesses both rich detail and clear semantics, significantly improving the feature representation capabilities at each feature level. Using this as the basis for analysis can effectively enhance the recognition accuracy for complex layout elements such as text, images, and tables, ensuring the accuracy of the layout analysis results.

[0044] In one possible implementation, the feature map has N layers, and accordingly, determining the feature map includes: Convolutional processing is performed on a portion of the k-th layer feature map to obtain the corresponding intermediate feature map; After convolving all channels of the intermediate feature map, it is fused with the k-th layer feature map to obtain the (k+1)-th layer feature map; where 1≤k≤N-1, and the first layer feature map is determined by feature extraction based on the initial feature map of the electronic component specification sheet image.

[0045] In this embodiment, the backbone network of the layout analysis model includes a preliminary feature extraction module and a lightweight fast network module. The preliminary feature extraction module is used to perform preliminary visual feature extraction on the acquired electronic component specification sheet image to obtain an initial feature map. The lightweight fast network module is used to perform multi-scale feature extraction based on the initial feature map to obtain feature maps at different scales. For example, if N=3, three feature maps at different scales are obtained, each corresponding to a layer. Specifically, by performing preliminary visual feature extraction on the electronic component specification sheet image to obtain the initial feature map, convolving some channels of the initial feature map to obtain the corresponding intermediate feature map, and then convolving all channels of the intermediate feature map and fusing it with the initial feature map to obtain the first layer feature map, i.e., the first layer feature map when k=1.

[0046] It should be noted that the preliminary feature extraction module in this embodiment includes a convolution module and a pooling module. The convolution module is used to capture local features of the electronic component specification sheet image, and the pooling module is used for scale adaptation. Specifically, the convolution module preferably uses three 3×3 convolutional layers to accurately capture pixel-level edges of text and line features of table rows and columns, generating low-dimensional, high-resolution feature maps. The pooling module downsamples the feature map output by the convolution module through max pooling, reducing the feature map size to obtain the corresponding initial feature map as the input basis for subsequent lightweight and fast network modules. For example, a 256×256 feature map can be reduced to 128×128 to expand the receptive field. This reduces subsequent computation and adapts to multi-scale targets in the specification sheet, including large tables and small text.

[0047] It is particularly important to emphasize that the lightweight fast network module in this implementation includes at least three lightweight fast network layers, such as... Figure 2 As shown, each lightweight fast network layer includes a partial convolutional layer PConv and two 1×1 convolutional layers arranged from top to bottom. The initial feature map after pooling is passed through the inverted residual structure formed by the partial convolutional layer PConv and the two 1×1 convolutional layers, which can compress redundant information while fusing multi-channel features.

[0048] Accordingly, partial convolutional layers apply convolutions to a subset of the input channels while keeping the remaining channels unchanged. This reduces unnecessary computation and memory access, allowing for more efficient extraction of spatial features. This is particularly suitable for large-scale datasets where feature map redundancy can lead to significant waste of computational resources. In this embodiment, the partial convolutional layer can select to perform 3×3 convolutions on a subset of the input feature map channels, such as the first or last channel, while keeping the remaining channels unchanged to obtain an intermediate feature map. This improves overall processing speed and achieves more continuous or regular memory access. Partial convolutional layers not only simplify the complexity of convolution operations but also reduce storage resource requirements. For example, performing 3×3 convolutions on 1 / 4 of the channels while keeping the remaining 3 / 4 unchanged results in a computational cost that is represented by floating-point operations (FLOPs). The FLOPs for partial convolution are 1 / 16 of those for ordinary convolution, and the corresponding memory access is 1 / 4 of that for ordinary convolution, effectively reducing redundant computation and ensuring low memory access.

[0049] Furthermore, the intermediate feature map obtained from the partial convolutional layer processing is further processed by two 1×1 convolutional layers. The first 1×1 convolutional layer is used to increase the feature channel dimension and enhance the feature representation capability, while the second 1×1 convolutional layer is used for dimensionality reduction to match the input dimension of subsequent modules. The result of the intermediate feature map after processing by two 1×1 convolutional layers is fused with the initial feature map to obtain a target-scale feature map, namely the first-layer feature map (k=1), corresponding to the bottom-level features. The first feature map is used as the input data for the next lightweight fast network layer, i.e., as the basis for extracting the second-layer feature map. The first-layer feature image is further processed using the inverted residual structure formed by the partial convolutional layer PConv and the two 1×1 convolutional layers in the corresponding lightweight fast network layer to obtain the second-layer feature map (k=2), corresponding to the intermediate features. Similarly, the second-layer feature map is used as the input data for the next lightweight fast network layer to continue processing, resulting in the third-layer feature map (k=N=3), corresponding to the high-level features. This yields three feature maps at different scales, which serve as the input data for the subsequent neck network to perform multi-scale feature fusion.

[0050] Therefore, this embodiment replaces the common basic block with a lightweight and fast network module in the backbone network. The lightweight and fast network module effectively reduces the consumption of computing resources and shortens the overall data processing time of the model, meeting the analysis needs of specification analysis that requires large-scale or real-time processing.

[0051] In one implementation, the feature fusion map has N layers. Accordingly, all feature maps are subjected to feature fusion processing according to the execution path from high to low feature level to obtain the corresponding feature fusion map, including: Upsampling is performed on the feature fusion map of the (k+1)th layer to obtain the upsampling result; The upsampling result and the feature map of the kth layer are fused together to obtain the feature map of the kth layer; where 1≤k≤N-1, and the feature map of the Nth layer is the feature map of the Nth layer.

[0052] In this embodiment, for each scale of feature map output by the backbone network, the feature pyramid network (FPN) of the neck network performs a feature fusion process according to the execution path from high to low feature level. Through top-down upsampling operations, such as bilinear interpolation, high-level features are passed to the bottom layer and fused with the bottom layer features element by element to supplement semantic information for the bottom layer features, such as table category or image category.

[0053] Accordingly, the highest-level feature map in the above feature maps is used as the input data (second input feature map) when generating the first feature fusion map from the second-highest-level feature map. For example, if N=3, the third-level feature fusion map is the highest-level feature map, and the third-level feature map itself is the third-level first feature fusion map. When generating the first feature fusion map corresponding to the second-highest-level feature map, an upsampling operation is performed on the highest-level feature map to obtain the first upsampling result. This upsampling result is then fused with the second-highest-level feature map (the second-level feature map) to obtain the first feature fusion map corresponding to the second-highest-level feature map (the second-level first feature fusion map). Furthermore, when generating the first-level feature fusion map (the first-level first-level feature fusion map) corresponding to the next-level feature map of the second-highest level feature map, the first-level feature fusion map corresponding to the second-highest level feature map is used as input data, and an upsampling operation is performed on it to obtain the corresponding first upsampling result. This result is then fused with the first-level feature map of the next-level feature map of the second-highest level feature map (the first-level feature map) to obtain the first-level feature fusion map corresponding to the next-level feature map of the second-highest level feature map. This process continues until all feature maps except the highest-level feature map have generated corresponding first-level feature fusion maps, with the first-level feature fusion map of the highest-level feature map being the highest-level feature map itself.

[0054] In one scenario, multi-layer feature maps extracted at different scales are represented as [C1, C2, C3, C4, C5], where C1 represents the lowest-level feature and C5 represents the highest-level feature. The Feature Pyramid (FPN) module upsamples the high-level feature (C5) to the same resolution as the mid-level feature (C4) through upsampling (e.g., bilinear interpolation), and then fuses it with the C4 feature to obtain a first-stage feature fusion map corresponding to the mid-level feature C4. This map is then passed sequentially to C3 and C2—supplementing the lowest-level feature (C2) with semantic information such as text category and table attributes, preventing the layout analysis model from misclassifying special parameter text as ordinary text.

[0055] The corresponding formula is: ; ; In the formula This is a feature fusion map, where f represents feature fusion.

[0056] That is, the first-order feature fusion map corresponding to the middle-layer feature C4. And so on. , Among them, the first-level feature fusion map M5 corresponding to the highest-level feature C5 is itself.

[0057] In one possible implementation, all feature maps are subjected to a feature fusion process according to the execution path from high to low feature level to obtain a corresponding first-level feature fusion map, and the process further includes: Perform a convolution operation on the feature fusion map.

[0058] In this embodiment, before performing the secondary feature fusion process, a convolution operation is also performed on the primary feature fusion map of each layer. The corresponding convolution formula is as follows:

[0059] In the formula, fc represents the convolution operation. This represents the feature fusion map after convolution.

[0060] For example, the feature fusion map after convolution corresponding to the middle layer feature C4. The result is obtained by upsampling the highest-level feature C5 to the same resolution as the middle-level feature C4, then fusing it with the middle-level features (i.e., adjusting the dimensions after 1×1 convolution after channel concatenation). By performing a convolution operation on the first-stage feature fusion map to adjust the number of channels and remove redundant information, the output [P2, P3, P4, P5] serves as the input data for subsequent second-stage feature fusion processing, collectively forming a multi-scale feature pyramid covering small, medium, and large targets. In other words, the feature pyramid network achieves top-down upsampling in feature extraction and fusion, passing semantic information from higher-level features to lower-level features, thereby enhancing the expression of lower-level features, improving the model's ability to perceive small and densely packed samples, and effectively improving the model's recognition accuracy.

[0061] In one implementation, the secondary feature fusion map has N layers. Correspondingly, all primary feature fusion maps are subjected to secondary feature fusion processing according to an execution path from low to high feature fusion level to obtain the corresponding secondary feature fusion map, including: The k-th layer quadratic feature fusion map is downsampled to obtain the downsampling result; After fusing the downsampling result with the first-order feature fusion map of the (k+1)th layer, the second-order feature fusion map of the (k+1)th layer is obtained; where 1≤k≤N-1, and the second-order feature fusion map of the first layer is the first-order feature fusion map of the first layer.

[0062] In this embodiment, the neck network adds a path aggregation module PANet on the basis of the feature pyramid network, so that the neck network can address the problem that the low-level features output by the backbone network have many details but weak semantics, and the high-level features have strong semantics but few details. The neck network uses bidirectional path fusion of the feature pyramid network and the path aggregation module to enhance the features in both directions.

[0063] It should be noted that, as Figure 3 As shown, the path aggregation module PANet adds a reverse path to the feature pyramid network. It performs secondary feature fusion from bottom to top on the primary feature fusion map obtained after the primary feature fusion of the feature pyramid network, to pass the high-resolution features from the lower layers to the deeper layers, supplementing the detailed information of the higher-level features. Specifically, the first-layer secondary feature fusion map is the first-layer primary feature fusion map (k=1). Correspondingly, the secondary feature fusion maps (k=1) corresponding to the lowest-level features in all primary feature fusion maps are downsampled and fused with the primary feature fusion maps corresponding to the middle-level features to obtain the corresponding secondary feature fusion maps for the middle layer. This process continues, performing secondary feature fusion on the next level of features from the middle-level features, until the secondary feature fusion maps of all primary feature fusion maps except the lowest-level feature fusion map are obtained. The secondary feature fusion map of the lowest-level feature fusion map (k=1) is the lowest-level feature fusion map itself. The combination of the path aggregation module PANet and the feature pyramid network can refine the feature selection process at different spatial levels, extract richer detailed information, and automatically adjust the weight distribution in the attention mechanism to optimize the network's attention to key features. This not only enhances the perception of small-scale, dense samples, but also improves subsequent localization and recognition capabilities, thereby ensuring overall recognition accuracy.

[0064] In one scenario, after the path aggregation module PANet receives the first-stage feature fusion map [P2, P3, P4, P5] generated by the feature pyramid network, it performs a second-stage feature fusion process. The corresponding formula can be expressed as:

[0065]

[0066] In the formula, This represents the secondary feature fusion map of the i+1 feature level to be generated. Indicates feature fusion, Indicates and A feature fusion map obtained after convolution at the same feature level is used to generate... ; For example, in this embodiment, the secondary feature fusion map N2 after the convolution of the lowest-level features is equivalent to the primary feature fusion map P2, which is the primary feature fusion map P2 itself. When performing secondary feature fusion processing from bottom to top to pass the detailed information of the lower-level features to the higher-level features to enhance the parsing ability of the higher-level features, for the primary feature fusion map P3 after convolution corresponding to the upper-level feature of the lowest-level features, its secondary feature fusion map N3 is obtained by fusing the first downsampling result obtained by downsampling the secondary feature fusion map N2 of the lowest-level features with the primary feature fusion map P3 after convolution, and so on, to obtain the corresponding secondary feature fusion maps [N2, N3, N4, N5].

[0067] The path aggregation module PANet in this embodiment adds a reverse path to the feature pyramid network. It downsamples the primary feature fusion map P2 corresponding to the bottom-level features after the feature pyramid network is fused, and then merges it with the primary feature fusion map P3 corresponding to the middle-level features. This fusion is then passed to P4 and P5 in sequence, thereby supplementing the high-level features with detailed information such as table corners and text edges. Through the cooperation between the path aggregation module PANet and the feature pyramid network, the final output is a 4-level secondary feature fusion map [N2, N3, N4, N5]. Each level has the characteristics of rich details and clear semantics, which greatly improves the feature expression ability of each level of features, thereby enhancing the overall recognition accuracy.

[0068] In one possible implementation, the method further includes: The primary and / or secondary feature fusion maps are processed at different resolution scales.

[0069] In this embodiment, considering the scale differences of specification elements, a variable resolution extraction function is designed for each feature layer. This allows each feature layer to extract information at different resolutions from the input feature map, generating a multi-resolution feature map. Let the input feature be... Accordingly, the features for each resolution are represented as follows:

[0070] In the formula, To achieve resolution Above, the first Feature map of the layer For the first Layers for resolution The feature extraction function.

[0071] This embodiment generates pyramid features with richer expressive power through resolution transformation and cross-resolution feature fusion. It adopts a variable resolution feature pyramid to adapt to the same type of target of different sizes, allowing the model to integrate and unify feature expressions of different scales. This enables it to better handle text, tables and images of different scales contained in electronic component specifications, effectively capture diverse features from fine-grained text to large-scale charts, increase the model's adaptability to elements of different sizes and complex layouts, and especially enhance the accurate recognition capability for small-font text or complex images.

[0072] In one possible implementation, the method further includes: The corresponding loss function is determined based on the coordinate deviation between the predicted bounding box and the true bounding box, and the model is updated and optimized using the loss function.

[0073] In this embodiment, the true bounding box is the actual box position of the target layout element in the electronic component specification document, serving as a reference standard for training the layout analysis model. Accordingly, the target elements in the electronic component specification document image, such as tables or images, are annotated using annotation tools to obtain the true bounding box.

[0074] For example, import the specification document image into the annotation tool. If the specification document image is tilted, such as due to scanning offset, first adjust it to be horizontal using the tool's built-in rotation correction function to ensure that the target element has no angular deviation and reduce errors in subsequent position information calculations. Click on the top-left vertex of the target element using the annotation tool and drag it to the bottom-right vertex to generate a rectangle that fits the text, completing the annotation of the target element.

[0075] In this embodiment, after marking the target element with a rectangular bounding box, manual verification can be performed. For example, verifying the accuracy of the bounding box involves zooming in on the image to check if the bounding box has not cut off the target element or contained excessive background. For example, a text box should only contain text with no extra blank lines. Tag correctness is also checked, i.e., whether the core tag matches the target element.

[0076] Furthermore, the coordinate deviation between the predicted bounding box output by the layout analysis model and the aforementioned true bounding box is determined, providing a gradient direction for adjusting the layout analysis model parameters. Based on the loss function determined by the coordinate deviation, the layout analysis model is updated and adjusted to improve the accuracy of the analysis results of the layout analysis model for target elements in the electronic component specification sheet image.

[0077] In one possible implementation, the loss function is determined based on the coordinate deviation between the predicted bounding box and the true bounding box, including: The cross-union ratio (CUNR) is determined based on the cross-union ratio between the coordinates of the predicted bounding box and the true bounding box, the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the true bounding box, and the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the true bounding box. The center distance loss is determined based on the deviation between the center point coordinates of the predicted bounding box and the center point coordinates of the actual bounding box. The keypoint loss is determined based on the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the actual bounding box, as well as the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the actual bounding box. The corresponding loss functions are obtained by adding the products of crossover ratio loss and corresponding weights, center distance loss and corresponding weights, and keypoint loss and corresponding weights.

[0078] In this embodiment, the coordinate deviation between the manually annotated ground truth bounding boxes and the predicted bounding boxes output by the layout analysis model is used to determine the loss function for optimizing the layout analysis model. Both the ground truth bounding boxes and the predicted bounding boxes contain category label information and location coordinate information. For example... Figure 4 As shown, a bipartite graph is constructed between the real bounding box and the predicted bounding box output by the layout analysis model. Matching samples are selected by Hungarian algorithm, and the loss value of the matching samples is calculated. The model is then optimized in reverse to gradually improve the analysis accuracy. An end-to-end logic of direct prediction and bipartite graph matching is adopted to replace the non-maximum suppression (NMS) step in traditional object detection.

[0079] It should be noted that the coordinates of the true bounding box include the coordinates of its top-left corner (denoted as the first point) and its top-right corner (denoted as the second point), to capture the geometric features of the target elements in the specification image. The coordinates of the predicted bounding box include the coordinates of its top-left corner (denoted as the third point) and its top-right corner (denoted as the fourth point). When determining the loss function, the first distance between the first and third points, and the second distance between the second and fourth points, can focus on the key corners of the bounding box, directly optimizing the edge alignment accuracy of the specification target (such as tables or text blocks). Accordingly, the calculation formula is:

[0080]

[0081] In the formula, and These represent the coordinates of the top-left and bottom-right corners of the prediction box, respectively. This indicates the x-coordinate of the top left corner of the prediction box. This represents the x-coordinate of the bottom right corner of the prediction box. This represents the y-coordinate of the top left corner of the prediction box. This represents the vertical coordinate of the bottom right corner of the prediction box. These represent the coordinates of the top-left and bottom-right corners of the actual bounding box, respectively. Indicates the first distance. Indicates the second distance.

[0082] In addition, this embodiment incorporates the height (H) and width (W) of the specification image when determining the loss function, which can normalize the distance (e.g., converting pixel distance into a ratio relative to the image size) and solve the problem of incomparable loss values ​​for specification images of different sizes. For example, a 10-pixel deviation in A4 specification images will result in different impacts than a 10-pixel deviation in A3 specification images.

[0083] Accordingly, the intersection and union ratio loss The calculation formula is as follows:

[0084] In the formula, h and These represent the height and width of the input image, respectively. This represents the intersection-over-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. If there is no intersection between the predicted bounding box and the ground truth bounding box, the IoU is 0.

[0085] Furthermore, in this embodiment, the center distance loss is determined based on the deviation between the center point coordinates of the predicted bounding box and the center point coordinates of the actual bounding box. The center point coordinates of the actual bounding box can be the coordinates of the intersection of the diagonals of the actual bounding box's rectangle, denoted as the first center point coordinates. The center point coordinates of the predicted bounding box can be the coordinates of the intersection of the diagonals of the predicted bounding box's rectangle, denoted as the second center point coordinates. The center distance loss is determined based on the coordinate deviation between the first and second center point coordinates. Determining the center distance loss avoids situations where corners are locally aligned but the overall shape is offset. It should be noted that this embodiment also incorporates the height (H) and width (W) of the specification image when determining the center distance loss, allowing for distance normalization. Accordingly, the center distance loss... The formula is:

[0086] In the formula, The x-coordinate of the second center point The x-coordinate of the first center point The ordinate of the second center point. The ordinate of the first center point.

[0087] Furthermore, the keypoint loss is determined based on the deviations between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the true bounding box, and the deviations between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the true bounding box. Accordingly, the keypoint loss formula is... as follows:

[0088] This embodiment obtains the corresponding loss function by adding the products of the intersection-union ratio (CIRR) loss and its corresponding weight, the center distance loss and its corresponding weight, and the keypoint loss and its corresponding weight. It should be noted that the weight corresponding to the CIRR loss is the first weight parameter, the weight corresponding to the center distance loss is the second weight parameter, and the weight corresponding to the keypoint loss is the third weight parameter. The objective function aims to minimize the first distance, the second distance, and the center point distance. The first, second, and third weight parameters of the objective function are dynamically adjusted using the backpropagation algorithm. Accordingly, the loss function... The calculation formula is as follows:

[0089] In the formula The first, second, and third weight parameters are dynamically adjusted.

[0090] It should be noted that in related technologies, the model loss function is usually chosen as the GIoU loss function to improve the calculation of bounding box regression loss. GIoU introduces the concept of a minimum closure region, containing the predicted box and the ground truth box within a minimum rectangular box. However, the effectiveness of GIoU is affected when the predicted box is completely covered by the background box (a region box that does not contain any object of interest). Based on this, this embodiment focuses on minimizing the distance between the top-left and bottom-right corners of the predicted box and the ground truth box, that is, directly optimizing the distance between keypoints, and more effectively handling the bounding box localization problem. Furthermore, in order to focus on the Euclidean distance between the center points of the bounding boxes and enhance the edge alignment constraints of the bounding boxes, this embodiment also normalizes the offset, and eliminates the optimization difficulties caused by scale differences by obtaining the width and height of the specification image, so that the edges of the predicted box are closer to the edges of the ground truth box, thereby improving the localization accuracy.

[0091] In one scenario, an image of an electronic component specification sheet is input into an improved layout analysis model, RT-DETR. The improved RT-DETR model extracts image features and outputs the position parameters of the predicted bounding boxes. The IoU value between the predicted and ground truth bounding boxes is calculated to determine if there is any intersection. For example, when the predicted and ground truth bounding boxes do not intersect, the IoU value is 0, and the loss value is 1. The improved MPDIoU loss function provided in this embodiment optimizes the distance between keypoints through dynamic weights, more effectively handling the bounding box localization problem, and calculates the loss based on this to obtain the MPDIoU loss value. Based on the calculated MPDIoU loss value, the weights of the improved RT-DETR model are updated through the backpropagation algorithm. Through multiple iterations of training, the model parameters are optimized, improving the localization accuracy of the predicted bounding boxes.

[0092] It should be noted that after training, the layout analysis model can accurately identify and locate the required layout information from images of electronic component specification sheets. For example... Figure 5 As shown, the image of the electronic component specification sheet to be detected is input into the trained layout analysis model. The model analyzes the data and generates multiple candidate bounding boxes. These boxes are then filtered according to category confidence levels, retaining high-confidence predictions to select effective bounding boxes, thus achieving the layout analysis objective. The layout analysis results obtained using the layout analysis model of this application are shown below. Figure 6 As shown.

[0093] This embodiment conducted experimental comparisons of different loss functions. Specifically, 500 pages of specifications were randomly selected to test the layout analysis models using different loss functions. The results are shown in Table 1.

[0094]

[0095] Table 1. Test results of layout analysis models with different loss functions. The experimental test results prove that the method provided in this embodiment... The loss function has a higher prediction accuracy.

[0096] In summary, this disclosure replaces the basic blocks of the backbone network in the RT-DETR model with FasterNet Blocks, and enhances the neck network in the RT-DETR object detection model using the Feature Pyramid (FPN) module and the Path Aggregation (PANet) module. Using the minimum point distance intersection-union (GIoU) ​​loss as the objective function, an improved RT-DETR model is obtained. The layout analysis model removes overlapping redundant boxes, retaining candidate boxes with the highest class probability and most accurate location, ultimately determining the bounding box coordinates of each target element. The RT-DETR target model uses the minimum point distance intersection-union (GIoU) ​​loss as the objective function, addressing the shortcomings of the original RT-DETR's GIoU loss in the specification document scenario. For example, the loss fails when the predicted box is covered by a background box, and the corner positioning of text / tables is inaccurate, ultimately achieving more accurate bounding box prediction. This enables end-to-end training from the original specification document to element detection and classification, omitting necessary preprocessing steps in related technologies, simplifying the layout analysis process, and resulting in higher accuracy and efficiency for layout analysis of electronic component specification documents.

[0097] like Figure 7 As shown, this application provides a layout analysis device 700, comprising: Image acquisition module 701 is used to acquire images of electronic component specification sheets; Feature extraction module 702 is used to extract features at different scales from the electronic component specification sheet image to obtain multi-layer feature maps; The first feature fusion module 703 is used to perform a first feature fusion process on all feature maps according to the execution path from high to low feature level to obtain the first feature fusion map corresponding to each layer of feature map; The secondary feature fusion module 704 is used to perform secondary feature fusion processing on all primary feature fusion maps according to the execution path from low to high feature fusion level, so as to obtain the corresponding secondary feature fusion map. The prediction module 705 is used to fuse the secondary feature fusion image with the preset position information of the electronic component specification image, and then perform feature matching with the pre-generated object query vector to determine the corresponding query feature vector. By analyzing the query feature vector, the predicted layout element type and the corresponding bounding box coordinates are obtained.

[0098] It should be noted that the layout analysis device in this application embodiment solves the problem in a similar way to the aforementioned layout analysis method. Therefore, the implementation process, implementation principle, and beneficial effects of the layout analysis device can be found in the description of the implementation process, implementation principle, and beneficial effects of the aforementioned method. Repeated descriptions will not be repeated.

[0099] In one embodiment, the feature extraction module 702 is used to perform convolution processing on a portion of the k-th layer feature map to obtain the corresponding intermediate feature map; After convolving all channels of the intermediate feature map, it is fused with the k-th layer feature map to obtain the (k+1)-th layer feature map; where 1≤k≤N-1, and the first layer feature map is determined by feature extraction based on the initial feature map of the electronic component specification sheet image.

[0100] In one embodiment, the primary feature fusion module 703 is used to perform upsampling processing on the (k+1)th layer primary feature fusion map to obtain the upsampling result; The upsampling result and the feature map of the kth layer are fused together to obtain the feature map of the kth layer; where 1≤k≤N-1, and the feature map of the Nth layer is the feature map of the Nth layer.

[0101] In one embodiment, the primary feature fusion module 703 is further configured to perform a convolution operation on the primary feature fusion map.

[0102] In one embodiment, the secondary feature fusion module 704 is used to perform downsampling processing on the k-th layer secondary feature fusion map to obtain a downsampling result; After fusing the downsampling result with the first-order feature fusion map of the (k+1)th layer, the second-order feature fusion map of the (k+1)th layer is obtained; where 1≤k≤N-1, and the second-order feature fusion map of the first layer is the first-order feature fusion map of the first layer.

[0103] In one embodiment, a resolution processing module is also included, which is used to process the secondary feature fusion map at different resolution scales.

[0104] In one possible implementation, the system further includes an optimization module, which determines a corresponding loss function based on the coordinate deviation between the predicted bounding box and the true bounding box, and uses the loss function to update and optimize the layout analysis model.

[0105] In one possible implementation, the optimization module is further configured to determine the cross-union ratio loss based on the cross-union ratio between the coordinates of the predicted bounding box and the true bounding box, the deviation between the coordinates of the upper left corner of the predicted bounding box and the coordinates of the upper left corner of the true bounding box, and the deviation between the coordinates of the lower right corner of the predicted bounding box and the coordinates of the lower right corner of the true bounding box. The center distance loss is determined based on the deviation between the center point coordinates of the predicted bounding box and the center point coordinates of the actual bounding box. The keypoint loss is determined based on the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the actual bounding box, as well as the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the actual bounding box. The corresponding loss functions are obtained by adding the products of crossover ratio loss and corresponding weights, center distance loss and corresponding weights, and keypoint loss and corresponding weights.

[0106] This application provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the layout analysis method described in any of the above embodiments.

[0107] This application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the layout analysis method described in any of the above embodiments.

[0108] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0109] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0111] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as layout analysis methods. For example, in some embodiments, the layout analysis method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the layout analysis method described above may be performed.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0116] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A layout analysis method, characterized in that, The method includes: Obtain images of electronic component specification sheets; Feature extraction at different scales is performed on the electronic component specification sheet image to obtain a multi-layer feature map; Perform feature fusion processing on all feature maps according to the execution path from high to low feature level to obtain a feature fusion map corresponding to each layer of feature maps; All primary feature fusion maps are processed into secondary feature fusion maps according to the execution path from low to high feature fusion level; After fusing the secondary feature fusion image with the preset position information of the electronic component specification image, feature matching is performed with the pre-generated object query vector to determine the corresponding query feature vector. By analyzing the query feature vector, the predicted layout element type and the corresponding bounding box coordinates are obtained.

2. The layout analysis method according to claim 1, characterized in that, The feature map has N layers. Accordingly, determining the feature map includes: Convolutional processing is performed on a portion of the k-th layer feature map to obtain the corresponding intermediate feature map; After convolving all channels of the intermediate feature map, it is fused with the k-th layer feature map to obtain the (k+1)-th layer feature map; where 1≤k≤N-1, and the first layer feature map is determined based on the feature extraction of the initial feature map of the electronic component specification sheet image.

3. The layout analysis method according to claim 2, characterized in that, The single-layer feature fusion map has N layers. Correspondingly, the process of performing a single-layer feature fusion on all feature maps according to the execution path from high to low feature level to obtain a single-layer feature fusion map corresponding to each layer includes: Upsampling is performed on the feature fusion map of the (k+1)th layer to obtain the upsampling result; The upsampling result and the k-th layer feature map are fused once to obtain the k-th layer first feature fusion map; where 1≤k≤N-1, and the N-th layer first feature fusion map is the N-th layer feature map.

4. The layout analysis method according to claim 1, characterized in that, The step of performing a feature fusion process on all feature maps according to the execution path from high to low feature level to obtain the corresponding first-level feature fusion map also includes: Perform a convolution operation on the first feature fusion map.

5. The layout analysis method according to claim 3, characterized in that, The secondary feature fusion map has N layers. Correspondingly, the secondary feature fusion processing of all primary feature fusion maps is performed according to an execution path from low to high feature fusion level to obtain the corresponding secondary feature fusion map, including: The k-th layer quadratic feature fusion map is downsampled to obtain the downsampling result; After fusing the downsampling result with the first-order feature fusion map of the (k+1)th layer, a second-order feature fusion map of the (k+1)th layer is obtained; where 1≤k≤N-1, and the second-order feature fusion map of the first layer is the first-order feature fusion map of the first layer.

6. The layout analysis method according to claim 1, characterized in that, The method further includes: The primary feature fusion map and / or the secondary feature fusion map are processed at different resolution scales.

7. The layout analysis method according to any one of claims 1-6, characterized in that, The method further includes: The corresponding loss function is determined based on the coordinate deviation between the predicted bounding box and the true bounding box, and the model is updated and optimized using the loss function.

8. The layout analysis method according to claim 7, characterized in that, The step of determining the corresponding loss function based on the coordinate deviation between the predicted bounding box and the true bounding box includes: The cross-union ratio (CUNR) is determined based on the cross-union ratio between the coordinates of the predicted bounding box and the true bounding box, the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the true bounding box, and the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the true bounding box. The center distance loss is determined based on the deviation between the center point coordinates of the predicted bounding box and the center point coordinates of the actual bounding box. The keypoint loss is determined based on the deviation between the top-left corner coordinates of the predicted bounding box and the top-left corner coordinates of the actual bounding box, and the deviation between the bottom-right corner coordinates of the predicted bounding box and the bottom-right corner coordinates of the actual bounding box. The corresponding loss function is obtained by adding the products of the cross-union ratio loss and its corresponding weight, the center distance loss and its corresponding weight, and the keypoint loss and its corresponding weight.

9. A layout analysis apparatus for performing the layout analysis method according to any one of claims 1-8, characterized in that, The device includes: The image acquisition module is used to acquire images of electronic component specification sheets; The feature extraction module is used to extract features at different scales from the electronic component specification sheet image to obtain a multi-layer feature map; The first feature fusion module is used to perform a first feature fusion process on all feature maps according to the execution path from high to low feature level, so as to obtain the first feature fusion map corresponding to each layer of feature map; The secondary feature fusion module is used to perform secondary feature fusion processing on all primary feature fusion maps according to the execution path from low to high feature fusion level, so as to obtain the corresponding secondary feature fusion map. The prediction module is used to fuse the secondary feature fusion image with the preset position information of the electronic component specification image, perform feature matching with the pre-generated object query vector, determine the corresponding query feature vector, and obtain the predicted layout element type and corresponding bounding box coordinates by analyzing the query feature vector.

10. An electronic device, characterized in that, include: At least one processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the layout analysis method according to any one of claims 1-8.

Images