Image processing method and device, and storage medium

Abstract

The invention discloses an image processing method and device, and a storage medium, and the method comprises the steps: obtaining a first image, the image content of the first image comprising an architecture diagram; based on a target detection model, detecting position information of each element in a plurality of elements included in the first image; based on the position information of each element in a plurality of elements included in the first image, determining a hierarchical relationship among the plurality of elements; and constructing the architecture graph based on the hierarchical relationship among the plurality of elements.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method and device, and a storage medium. Background Art

[0002] Charts are useful communication tools that can represent data in a visually appealing format that is easy to understand. Chart mining refers to the process of automatically detecting, extracting, and analyzing charts to reproduce the tabular data originally used to create them. By allowing access to data that may not be available in other formats, chart mining facilitates the creation of many downstream applications and is currently being widely studied. However, most research focuses on common chart types, including architecture diagrams, pie charts, line charts, scatter plots, etc., and there are few directly applicable methods for charts in some special application scenarios.

[0003] An architecture diagram is a type of diagram used to present the overall structure of a system, software, or network and the relationships between components, usually including various symbols, icons, and lines. Architecture diagrams are widely used in various fields such as software development, cloud architecture, and the Internet of Things. Diagram mining methods for architecture diagrams are crucial to improving the editability and comprehensibility of architecture diagrams and improving the work efficiency of developers. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and device, and a storage medium, which can accurately extract the hierarchical relationship between elements of an architecture diagram.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image processing method, the method comprising:

[0007] Obtaining a first image, wherein image content of the first image includes an architecture diagram;

[0008] Based on the target detection model, detecting position information of each element among a plurality of elements included in the first image;

[0009] Determining a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image;

[0010] The architecture diagram is constructed based on the hierarchical relationship between the multiple elements.

[0011] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0012] An acquisition module is configured to acquire a first image, wherein the image content of the first image includes an architecture diagram;

[0013] A detection module, configured to detect position information of each element among a plurality of elements included in the first image based on a target detection model;

[0014] a determination module configured to determine a hierarchical relationship between the plurality of elements included in the first image based on position information of each element in the plurality of elements;

[0015] A construction module is configured to construct the architecture diagram based on the hierarchical relationship between the multiple elements.

[0016] An embodiment of the present application provides an electronic device, including a processor, wherein the processor is configured to:

[0017] Obtaining a first image, wherein image content of the first image includes an architecture diagram;

[0018] Detecting position information of each element among a plurality of elements included in the first image based on the target detection model;

[0019] Determining a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image;

[0020] The architecture diagram is constructed based on the hierarchical relationship between the multiple elements.

[0021] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the above-mentioned image processing method are implemented.

[0022] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0023] The chip provided in the embodiment of the present application is used to implement the above-mentioned image processing method, and the chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned image processing method.

[0024] The image processing method, device, and storage medium provided in the embodiments of the present application obtain a first image, wherein the image content of the first image includes an architecture diagram; based on a target detection model, the position information of each of the multiple elements included in the first image is detected, wherein the position information is used to characterize the area occupied by the element in the architecture diagram; based on the position information of each of the multiple elements included in the first image, the hierarchical relationship between the multiple elements is determined, and based on the hierarchical relationship between the multiple elements, the architecture diagram is constructed. The feature extraction capability of the target detection model is fully utilized to locate the elements of the architecture diagram, and the hierarchical relationship between the elements is determined based on the characteristics of the architecture diagram through the position of each element in the architecture diagram, and the architecture diagram is reconstructed based on the extracted hierarchical relationship, so as to extract the hierarchical relationship of the elements of the architecture diagram in the architecture diagram image, thereby realizing the editability and secondary use of the architecture diagram chart. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0026] Figure 2 It is an optional schematic diagram of an element provided in an embodiment of the present application;

[0027] Figure 3A-3B is an optional schematic diagram of position information representation provided by an embodiment of the present application;

[0028] Figure 4 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0029] Figure 5 is an optional schematic diagram of a candidate parent element provided in an embodiment of the present application;

[0030] Figure 6 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0031] Figure 7 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0032] Figure 8 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0033] Fig. 9 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0034] Fig.10 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0035] Fig.11 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0036] Fig.12 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0037] Fig.13 It is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0039] The embodiments of the present application can provide an image processing method, device and storage medium. In practical applications, the image processing method can be implemented by an electronic device, and each functional entity in the electronic device can be collaboratively implemented by hardware resources of a computer device (such as a terminal device, a network device, etc.), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0040] Of course, the embodiments of the present application are not limited to being provided as methods and hardware, and there may be multiple implementation methods, such as being provided as a storage medium (storing instructions for executing the image processing method provided by the embodiments of the present application).

[0041] Below, various embodiments of the image processing method, device, and storage medium provided in the embodiments of the present application are described.

[0042] The image processing method provided in the embodiment of the present application can be as follows Figure 1 As shown, the method may include:

[0043] S101: Obtain a first image, where image content of the first image includes an architecture diagram.

[0044] The electronic device may determine a first image to be subjected to hierarchical relationship extraction, wherein the image content of the first image includes an architecture diagram. The first image is an image file whose image content includes architecture diagram content. The first image may also be referred to as an architecture diagram image. The formats of the first image include: Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), Bitmap (BMP) and other image formats.

[0045] In the embodiment of the present application, the electronic device obtains the first image in a manner including but not limited to the following manners:

[0046] Method 1: determining a first image from a plurality of stored images based on a selection operation of a user;

[0047] Method 2: acquiring or photographing the architecture diagram displayed by other devices through a camera to obtain a first image;

[0048] Method three: determining, based on the screenshot operation of the user, that the image obtained by the screenshot operation of the user is the first image;

[0049] Method 4: export or save the architecture diagram opened in the architecture diagram software as a first image;

[0050] Method 5: receiving a first image sent by another device.

[0051] In the embodiment of the present application, there is no limitation on the manner in which the electronic device obtains the first image.

[0052] After obtaining the first image, the electronic device may perform image preprocessing such as image denoising, binarization, edge detection, etc. on the first image.

[0053] S102: Detect position information of each element among a plurality of elements included in the first image based on a target detection model.

[0054] After the electronic device obtains the first image, it inputs the first image into the target detection model, and the target detection model detects the elements included in the architecture diagram in the first image. In the embodiment of the present application, the elements can be understood as nodes or objects included in the architecture represented in the architecture diagram. In one example, Figure 2 The architecture diagram shown includes the following elements: elements where the business platform is located, elements where the authority center is located, elements where the authentication center is located, elements where the API center is located, elements where the message center is located, elements where the process center is located, and elements where the application center is located. The business platform, authority center, authentication center, API center, message center, process center, and application center are the text contents on the above elements respectively.

[0055] After inputting the first image, the target detection model inputs the position information of each of the multiple elements included in the architecture diagram in the first image. The position information can be understood as the coordinates of the corresponding element in the area where the architecture diagram in the first image is located, wherein the origin of the coordinate system used by the coordinates can be the center position in the first image or the lower right corner position of the first image.

[0056] In one example, for Figure 2As shown in the architecture diagram, the target detection model outputs the location information of each element including the elements where the business platform is located, the elements where the authority center is located, the elements where the authentication center is located, the elements where the API center is located, the elements where the message center is located, the elements where the process center is located, and the elements where the application center is located.

[0057] In the embodiment of the present application, the location information of an element is used to represent the area occupied by the element in the architecture diagram.

[0058] In an embodiment of the present application, the position information of the element output by the target detection model may be the position information of the detection box for the element, wherein the detection box is used to identify the area occupied by the element identified by the target detection model in the architecture diagram.

[0059] The position information output by the object detection model may represent the position and size of the area occupied by the element in the first image, i.e., the architecture diagram. The position information may be represented in a variety of ways.

[0060] In one example, the position information of element 301 may be represented as follows: Figure 3A As shown in (x1, y1, x2, y2), (x1, y1) and (x2, y2) are the coordinates of two points 302 and 303 on the diagonal line of the detection box respectively.

[0061] In one example, the position information of element 301 may be represented as follows: Figure 3B The (x0, y0, w, h) shown here are the coordinates of the center point of the detection frame, and w and h are the width and height of the detection frame respectively.

[0062] In the embodiment of the present application, the target detection model is a network model obtained by deep learning training. The target detection model can be obtained by training the electronic device based on training data, or it can be received from other devices. The network structure used by the target detection model may include structures such as YOLOv8, NanoDet, and RTDETR. In the embodiment of the present application, there is no limitation on the structure type of the target detection model.

[0063] S103: Determine a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image.

[0064] After determining the position information of each element in the elements included in the first image, the hierarchical relationship between the multiple elements is determined according to the position information of each element. The hierarchical relationship can be understood as the relationship between elements in the architecture diagram included in the first image.

[0065] In the embodiment of the present application, according to the position information of each element in the multiple elements, the multiple elements are matched based on the rules to determine the position relationship between one element and other elements, so as to identify the position relationship between the elements by using the position information of the elements according to the rules. The rules here can be understood as the relationship matching rules that characterize the characteristics of the architecture diagram. The content of the relationship matching rules can be set according to actual needs.

[0066] S104: construct the architecture diagram based on the hierarchical relationship between the multiple elements.

[0067] After determining the hierarchical relationship between the elements, the hierarchical relationship may be stored as a structured result based on a set format, where the set format is a format that can be recognized by the electronic device.

[0068] The electronic device may construct a structured result that the electronic device can support based on the position information of the elements and their parent nodes in the hierarchical relationship, so that the electronic device can reconstruct the architecture diagram based on the structured result.

[0069] In one example, the setting format is (position information of child element, position information of parent element);

[0070] In one example, the format is (ID of child element, ID of parent element);

[0071] In one example, the setting format is (ID of child element, location information of child element, ID of parent element);

[0072] In one example, the setting format is (ID of child element, position information of child element, ID of parent element, position information of parent element).

[0073] The ID in the above example may be an identifier assigned to an element by the electronic device after identifying the detected element.

[0074] In the embodiment of the present application, on the basis that the setting format of multiple elements can identify the hierarchical relationship, the content included in the setting format is not limited.

[0075] In the embodiment of the present application, a first image is obtained, and the image content of the first image includes an architecture diagram; based on a target detection model, the position information of each element among the multiple elements included in the first image is detected, and the position information is used to characterize the area occupied by the element in the architecture diagram; based on the position information of each element among the multiple elements included in the first image, the hierarchical relationship between the multiple elements is determined, and based on the hierarchical relationship between the multiple elements, the architecture diagram is constructed. The feature extraction capability of the target detection model is fully utilized to locate the elements of the architecture diagram, and the hierarchical relationship between the elements is determined based on the characteristics of the architecture diagram through the area occupied by each element in the architecture diagram, and the architecture diagram is reconstructed based on the extracted hierarchical relationship, so as to realize the extraction of the hierarchical relationship of the elements of the architecture diagram in the architecture diagram image, thereby realizing the editability and secondary use of the architecture diagram chart.

[0076] In some embodiments, S203, determining the hierarchical relationship between the multiple elements based on the position information of each element in the multiple elements included in the first image, includes:

[0077] Each of the multiple elements is taken as the first element, and the following processing is performed respectively:

[0078] Based on position information of each element in the multiple elements, determine the parent element of the first element from multiple second elements, the multiple second elements include other elements in the multiple elements except the first element, the area occupied by the first element is located within the area occupied by the parent element of the first element, and no other elements are included between the parent element of the first element and the first element.

[0079] Here, the hierarchical relationship is the relationship of inclusion and being included between elements, that is, the relationship between child elements and parent elements.

[0080] For an element, the parent element can be understood as the element that directly includes the element, and the elements directly included by the parent element are the child elements of the parent element.

[0081] In one example, if Figure 2 As shown, the element where the business platform is located is the parent element of the following elements: the element where the authority center is located, the element where the authentication center is located, the element where the API center is located, the element where the message center is located, the element where the process center is located, and the element where the application center is located, that is, the child nodes of the element where the business platform is located include: the element where the authority center is located, the element where the authentication center is located, the element where the API center is located, the element where the message center is located, the element where the process center is located, and the element where the application center is located.

[0082] In an embodiment of the present application, multiple elements detected by the target detection model are traversed, and for an element, namely, a first element, a relationship between the first element and other elements among the detected elements except the first element, namely, multiple second elements, is determined, and the parent element of the first element is determined from the multiple second elements.

[0083] In a possible implementation manner, the plurality of second elements include elements of the plurality of elements except the first element.

[0084] In one example, the detected elements include: element A, element B, element C, element D and element E, and the following processing is performed:

[0085] Take element A as the first element, and determine the parent element of element A from elements B, C, D, and E;

[0086] Take element B as the first element and determine the parent element of element B from elements A, C, D and E;

[0087] Take element C as the first element and determine the parent element of element C from elements A, B, D and E;

[0088] Take element D as the first element and determine the parent element of element D from elements A, B, C, and E;

[0089] Element E is taken as the first element, and the parent element of element E is determined from element A, element B, element C, and element D.

[0090] In a possible implementation, the plurality of second elements include other elements, except the first element, whose areas are larger than the first element. In this case, the plurality of elements may be sorted from small to large in area based on the area occupied by each element in the plurality of elements, and the plurality of second elements corresponding to the first element may be determined based on the sorting result.

[0091] In one example, the detected elements include: element A, element B, element C, element D and element E. The area sizes of element A, element B, element C, element D and element E are sorted from small to large as: element C, element A, element D, element B and element E. Then, the following processing is performed:

[0092] Take element C as the first element and determine the parent element of element C from elements A, D, B, and E;

[0093] Take element A as the first element and determine the parent element of element A from elements D, B, and E;

[0094] Take element D as the first element and determine the parent element of element D from elements B and E;

[0095] Take element B as the first element and determine whether element E is the parent element of element B.

[0096] In an embodiment of the present application, in the architecture diagram, a parent element may include one or more child elements, a child element includes only one parent element, and the area of ​​the parent element is larger than the area of ​​the child element. For an element, determining the parent element of the element from all elements whose areas are larger than the area of ​​the element can reduce the amount of calculation and improve processing efficiency.

[0097] In the embodiment of the present application, the area of ​​an element may be determined based on the position information of the element.

[0098] If the position information of an element is expressed as (x1, y1, x2, y2), the area of ​​the element can be expressed as (x2-x1)*(y2-y1).

[0099] If the position information of an element is expressed as (x0, y0, w, h), the area of ​​the element can be expressed as w*h.

[0100] In an embodiment of the present application, for an element, according to the position information of each element in the multiple elements, multiple elements are matched based on rules to determine whether an element is in a child-to-parent relationship with other elements, thereby using the position information of the elements according to the rules to identify the position relationship between the elements. The rules here can be understood as the relationship matching rules between the child elements and the parent elements in the representation architecture diagram. The content of the relationship matching rules can be set according to actual needs.

[0101] In one example, the relationship matching rule includes at least one of the following:

[0102] Rule 1: The area of ​​the parent node is not smaller than that of the child node;

[0103] Rule 2: The parent node must almost completely contain the child node;

[0104] Rule 3: When a node has multiple nodes containing it: give priority to selecting the node with the smallest area as the parent node, and give priority to the node with a larger aspect ratio as the parent node.

[0105] In an embodiment of the present application, multiple elements are traversed, and for each element, the parent element of the element is found from other elements, so as to clarify the relationship between the child elements and the parent elements between the elements in the architecture diagram, and clearly sort out the hierarchical relationship between the elements.

[0106] In some embodiments, the determining the parent element of the first element from the plurality of second elements based on the position information of each element in the plurality of elements, such as Figure 4 As shown, including:

[0107] S401, based on the position information of the first element and the position information of each second element in the plurality of second elements, respectively determine a matching cost of the first element and the plurality of second elements, the matching cost being used to represent a possibility that an area occupied by the second element includes an area occupied by the first element;

[0108] S402, based on the matching cost between each second element in the plurality of second elements and the first element, selecting at least one candidate parent element of the first element from the plurality of second elements, wherein the area occupied by the first element is within the area occupied by the candidate parent element of the first element;

[0109] S403: Select a parent element of the first element from at least one candidate parent element of the first element.

[0110] In S401, for each second element, based on the area occupied by the first element and the area occupied by the second element, the matching cost of the first element and the second element is determined, and the matching cost represents the possibility that the area occupied by the second element includes the area occupied by the first element, wherein a matching cost relational expression can be set, and the position information of the first element and the position information of the second element are used as parameters of the relational expression for matching elements to determine the matching cost of the first element and the second element. Wherein, based on different matching relational expressions, the value range of the matching cost is different.

[0111] In the embodiment of the present application, the matching cost represents the positional relationship between regions of different elements, reflecting rule one and rule two.

[0112] In S402, for a second element, whether the second element is a candidate parent node of the first element is determined based on the matching cost between the second element and the first element, thereby determining one or more candidate parent elements of the first element. The candidate parent elements can be understood as including the parent elements of the first element.

[0113] In one example, if Figure 5 As shown, element 501 and element 502 are both candidate parent elements of element 503.

[0114] In the embodiment of the present application, a candidate element list, namely a candidate node list, may be set for a first element, and candidate parent elements among the plurality of second elements may be added to the candidate node list.

[0115] In the embodiment of the present application, after the position information of the element is identified, an element identifier may be assigned to each element's position information to distinguish different elements. When adding a candidate parent element to the candidate element list, the element identifier of the candidate parent element may be added to the candidate element list.

[0116] The one or more candidate parent elements include a parent element. In S403, a parent element is determined from the one or more candidate parent elements.

[0117] If there is only one candidate parent element for the first element, then the candidate parent element is the parent element of the first element.

[0118] If there are multiple candidate parent elements for the first element, an element is selected from the multiple candidate parent elements as the parent element.

[0119] In an embodiment of the present application, candidate parent elements including the first element are found from multiple second elements based on the matching cost, and the parent element is found from the found candidate parent elements, and the relationship between elements is hierarchically sorted to improve the search efficiency of the parent element.

[0120] In some embodiments, the determining the parent element of the first element from the plurality of second elements based on the position information of each element in the plurality of elements further includes:

[0121] In the case that there is no candidate parent element for the first element, the parent node of the first element is identified as a set identifier, and the set identifier is used to indicate that there is no parent node for the first element.

[0122] In the embodiment of the present application, when the first element includes a candidate parent element, the first element includes a parent element, and the parent element is included in the determined one or more candidate parent elements. When the first element does not include a candidate parent element, the first element does not include a parent element. In this case, a set identifier is used to replace the parent element of the first element. When the parent element of the first element is the set identifier, it means that the first element does not have a parent element.

[0123] In one example, the flag is set to 0.

[0124] In one example, the flag is set to -1.

[0125] In the embodiment of the present application, there is no limitation on the identification form of the setting identification.

[0126] In an embodiment of the present application, for an element that does not have a parent element, the parent element of the element is represented based on a set identifier. On the basis of the effective identification element that does not have a parent element, the expression method of the child element and the parent element is followed to simplify the implementation logic of the solution.

[0127] In some embodiments, the step of determining the matching cost of the first element and the second element based on the position information of the first element and the position information of the second element in step S401 includes:

[0128] Determine a size of an overlapping area between the first element and the second element based on the position information of the first element and the position information of the second element;

[0129] Based on the size of the overlapping area, a matching cost of the first element and the second element is determined.

[0130] In the embodiment of the present application, for the first element and the second element, their areas can be expressed as S1 and S2, where the area can be determined based on the position information, and the size of the overlapping area of ​​the first element and the second element can be expressed as formula (1):

[0131] S1∩S2 formula (1);

[0132] In one example, the matching cost csot can be expressed as formula (2):

[0133] cost = S1∩S2 Formula (2);

[0134] When the matching cost is expressed as formula (2), if S2 includes S1, that is, the second element includes the first element, the value of cost is the size of S1; if S2 does not include S1, that is, the second element does not include the first element, the value of cost is 0.

[0135] In one example, the matching cost csot can be expressed as formula (3):

[0136] cost = S1 - S1 ∩ S2 (3);

[0137] When the matching cost is expressed as formula (3), if S2 includes S1, that is, the second element includes the first element, the value of cost is 0; if S2 does not include S1, that is, the second element does not include the first element, the value of cost is the size of S1.

[0138] In one example, the matching cost csot can be expressed as formula (4):

[0139]

[0140] When the matching cost is expressed as formula (4), if S2 includes S1, that is, the second element includes the first element, the cost value is 0; if S2 does not include S1, that is, the second element does not include the first element, the cost value is the ratio of S1 to S2.

[0141] In one example, the matching cost csot can be expressed as formula (5):

[0142]

[0143] When the matching cost is expressed as formula (5), if S2 includes S1, that is, the second element includes the first element, the value of cost is 0; if S2 does not include S1, that is, the second element does not include the first element, the value of cost is 1.

[0144] The above formulas (2) to (5) are examples of several ways of expressing the matching cost based on the size of the overlapping area. In the embodiment of the present application, there is no limitation on the way of expressing the matching cost based on the size of the overlapping area.

[0145] In the embodiment of the present application, based on the size of the overlapping area between the first element and the second element, the possibility that the second element includes the first element, that is, the matching cost, is determined, which can simply, effectively and accurately determine the matching cost.

[0146] In some embodiments, S402 selects at least one candidate parent element of the first element from the plurality of second elements based on the matching cost between each second element in the plurality of second elements and the first element, including:

[0147] For each second element in the plurality of second elements, the following processing is performed respectively:

[0148] comparing the matching cost of the second element and the first element with a matching cost threshold;

[0149] Based on the comparison result, it is determined whether the second element is a candidate parent element of the first element.

[0150] In the embodiment of the present application, a matching cost threshold is set for the matching cost, wherein the matching cost threshold may be different for different representation modes of the matching cost, wherein the matching threshold may be a fixed size or may vary based on the size of the first element.

[0151] In one example, the matching cost is expressed as formula (2), and the matching threshold is the product of the size of the first element and the first coefficient, where the value of the first coefficient is 1, 0.95, 0.9, etc.

[0152] If the matching cost of a second element is greater than or equal to the matching threshold, the second element is a candidate parent element of the first element; otherwise, the second element is not a candidate parent element of the first element.

[0153] In one example, the matching cost is expressed as formula (3), and the matching threshold is the product of the size of the first element and the second coefficient, where the value of the second coefficient is 0, 0.05, 0.1, etc.

[0154] If the matching cost of a second element is less than or equal to the matching threshold, the second element is a candidate parent element of the first element; otherwise, the second element is not a candidate parent element of the first element.

[0155] In one example, the matching cost is expressed as formula (4), and the matching threshold is the fourth coefficient, wherein the value of the fourth coefficient is 0, 0.05, 0.1, etc.

[0156] If the matching cost of a second element is less than or equal to the matching threshold, the second element is a candidate parent element of the first element; otherwise, the second element is not a candidate parent element of the first element.

[0157] In one example, the matching cost is expressed as formula (5), and the matching threshold is the fifth coefficient, where the value of the fifth coefficient is 0, 0.05, 0.1, etc.

[0158] If the matching cost of a second element is less than or equal to the matching threshold, the second element is a candidate parent element of the first element; otherwise, the second element is not a candidate parent element of the first element.

[0159] In the embodiment of the present application, for each second element, whether the second element is a candidate parent element of the first element is quickly determined based on the size of the matching cost between the second element and the first element, thereby improving processing efficiency.

[0160] In some embodiments, the at least one candidate parent element includes a plurality of candidate parent elements, and the step of selecting a parent element of the first element from the at least one candidate parent element of the first element in step S403 includes:

[0161] Based on the area of ​​each candidate parent element among the multiple candidate parent elements, select a candidate parent element with the smallest area from the multiple candidate parent elements as the parent element;

[0162] The area of ​​the candidate parent element is determined based on the position information of the candidate parent element.

[0163] When there are multiple candidate parent elements for the first element, the area sizes of the multiple candidate parent elements are traversed, and the candidate parent element with the smallest area is used as the parent element of the first element.

[0164] In some embodiments, the image processing method provided by the embodiment of the present application further includes Figure 6 Steps shown:

[0165] S601, obtaining text information of each element in the multiple elements;

[0166] Accordingly, S104, constructing the architecture diagram based on the hierarchical relationship between the multiple elements, includes:

[0167] S602: Reconstruct the architecture diagram based on text information of each element in the multiple elements and the hierarchical relationship between the multiple elements.

[0168] In the above Figure 6 In the embodiment, S601 is located after S103. In practical applications, the execution of S601 may be located before S103.

[0169] The target detection model can identify the text information of each element in the first image to obtain the text information of each element. At this time, when the target detection model is obtained by the electronic device based on the training data, the label data in the training data also includes the text information of each element.

[0170] The text information of each element in the first image may also be recognized based on optical character recognition (OCR). In this case, the text information of each element may be recognized one by one.

[0171] In the embodiment of the present application, there is no limitation on the method for obtaining the text information of each element in the first image.

[0172] After obtaining the text information of each of the multiple elements, the position information and text information of the elements and their parent nodes in the hierarchical relationship can be organized by setting a format to form a structured result that can be supported by the electronic device, so that the electronic device can reconstruct the architecture diagram based on the structured result.

[0173] In one example, the format is set as (position information of child element, position information of parent element, text information of child element);

[0174] In one example, the format is set as (ID of child element, ID of parent element, text information of child element);

[0175] In one example, the format is set as (ID of child element, location information of child element, ID of parent element, text information of child element);

[0176] In one example, the format is set as (ID of child element, position information of child element, ID of parent element, position information of parent element, text information of child element).

[0177] The ID in the above example may be an identifier assigned to an element by the electronic device after identifying the detected element.

[0178] In the embodiment of the present application, on the basis that the setting format of multiple elements can identify the hierarchical relationship, the content included in the setting format is not limited.

[0179] In the embodiment of the present application, the layout diagram can be reconstructed in an application program in which the architecture diagram can be edited, so that the user can edit the reconstructed layout diagram according to his or her actual needs.

[0180] In a possible implementation, the structural results of each element are stored or output to other devices so that the other devices can reconstruct the layout diagram.

[0181] In the embodiment of the present application, the determined hierarchical relationship and the text information of each element can reconstruct a layout diagram that can be edited twice.

[0182] In some embodiments, the image processing method provided by the embodiment of the present application further includes:

[0183] Obtaining a sample image set, wherein the sample image set includes a plurality of sample images, and contents of the sample images include a sample architecture diagram;

[0184] For the sample image set, respectively labeling elements in each of the sample images to obtain label data, wherein the label data includes labeling information of each sample image in the plurality of sample images;

[0185] Based on the sample image set and the label data, the initial detection model is trained to obtain the target detection model, and the prime number target detection model is the converged initial detection model.

[0186] In the case where the target detection model is obtained by training the electronic device based on the training data, the training data may include a sample image set and label data, the sample image set includes multiple sample images, the sample image is an architecture diagram image including an architecture diagram, the label data includes annotation information of each sample image in the multiple sample images, and the annotation information includes position information of a rectangular box for annotating elements in the sample image. The position information of the rectangular box may be obtained by annotating the sample image with a bounding box (bbox).

[0187] The initial detection model is trained using the training data, and the trained initial detection model is the target detection model.

[0188] In some embodiments, the image processing method provided by the embodiment of the present application further includes:

[0189] Image preprocessing is performed on a plurality of sample images in the sample image set respectively so that sizes of different sample images are the same.

[0190] In an embodiment of the present application, the method of performing image preprocessing on the sample image includes: resizing, and resizing different sample images to the same size. Here, the resized sample image may be a set size, or the size of the smallest sample image among multiple sample images, or the average size of multiple sample images.

[0191] The image preprocessing method for the sample image may also include: image denoising, binarization, edge detection, etc. The embodiment of the present application does not impose any limitation on the image preprocessing method.

[0192] When the sample images have the same size, multiple sample images can be input into the initial detection model in batches to train the initial detection model, thereby improving the model training efficiency.

[0193] The following is a further description of the image processing method provided in the embodiments of the present application.

[0194] In the related art, the commonly used graph relationship extraction methods include the following:

[0195] Method 1: Rule-based method, which extracts information from the graph and the relationship between nodes based on manually defined rules and patterns. This method requires the rules to be defined in advance and uses classic knowledge representation methods for reasoning, which is suitable for specific fields or tasks. The rule-based method includes the following steps: graph recognition, relationship extraction, and data processing and application.

[0196] Method 2: Deep learning-based method. Deep learning-based method uses labeled training data to train machine learning models to identify and extract relationships in text.

[0197] Method 3: Chart subtitle based method

[0198] Figure captioning refers to a method of automatically generating additional information for non-text components.

[0199] Method 4: Chart-based question-answering method

[0200] We approach the task of graph information extraction from the perspective of visual question answering, which focuses on extracting specific information from input images and answering natural language questions about them.

[0201] For method 1, the disadvantages of complex and difficult implementation in the chart element identification stage become increasingly obvious;

[0202] For method 2, since the relational features of the architecture diagram are not obvious, there are targets with very high feature similarity, but their relational levels are not the same, so it cannot be directly used in the architecture diagram relation extraction task;

[0203] For method 3, it is necessary to use a natural language processing algorithm to extract key information, which requires relatively large computing power and slow reasoning speed. Therefore, this type of method is not very practical.

[0204] For method 4, by combining image analysis and text understanding technology, the question-answering method can directly answer more specific user questions. However, on the one hand, it is a large amount of engineering to construct a question-answering dataset suitable for a specific task; on the other hand, due to the complexity of the data, this type of method is difficult to train.

[0205] The embodiment of the present application provides an image processing method, which combines the advantages of deep learning methods and rule-based methods to provide a diagram recognition and relationship extraction method for architecture diagrams, using deep learning methods to extract architecture diagram image features and identify elements and categories in the diagram. Combined with the characteristics of the architecture diagram, a relationship matching rule that can identify the hierarchical relationship of elements is manually designed to extract the hierarchical relationship.

[0206] The image processing method provided in the embodiment of the present application is as follows: Figure 7 As shown, including:

[0207] S701: pre-process the input architecture diagram image.

[0208] The architecture diagram image is an image file (.jpg / .png / .bmp, etc.) containing the architecture diagram content, and the architecture diagram content includes the key node rectangular area of ​​the architecture diagram and its related text information. In one embodiment, the architecture diagram file displayed on the screen can be captured by a screenshot tool to obtain the architecture diagram image; the opened architecture diagram file can also be exported as an architecture diagram image by using architecture diagram software; the architecture diagram image can also be downloaded or captured from the Internet (such as Baidu Gallery, Google Gallery, etc.), or a clear architecture diagram image can be taken from a display device.

[0209] S702: Input the preprocessed architecture diagram image into the target detection model, and detect image elements through the target detection model.

[0210] The object detection model is a model that has been supervised and trained on the architecture diagram dataset. The object detection model detects image elements, which may include identifying and locating image elements.

[0211] The detection of the target detection model is a target detection task, which can be implemented with a customized trained target detection model. The so-called customized trained target detection model is: collect a large number of architecture diagram images, and perform bbox annotation on the key target nodes in each image to obtain training data; use the training data to train a deep learning-based target detector (such as YOLOv8, NanoDet, RTDETR, etc., without limitation), and the trained target detector is the target detection model. When training the target detector, the sample images can be batch spliced. Therefore, before batch splicing, the sample images of different sizes are resized (resized) so that the sizes of different sample images are the same (for example: 640*640).

[0212] By inputting the architecture diagram image obtained in S701 into the target detection model, a series of detection frames can be obtained, including key information required for architecture diagram analysis (such as the location information of the detection frame). After the image element detection, a series of detection frames of the same category can be obtained, and the attribute value of the coordinate bbox of each detection frame is determined.

[0213] S703: Determine the hierarchical relationship included in the architecture diagram image based on the position information of the identified image elements.

[0214] Hierarchical relationship matching is performed based on set rules. The elements detected by the target detection model and their position information are used to find the parent node of the target element through a rule-based relationship matching method to obtain a hierarchical relationship matrix of all relationships in the graph.

[0215] Through the target detection model in S702, all detection boxes of the rectangle category can be obtained, and the coordinates of the detection boxes can be expressed as {x1, y1, x2, y2}. For all the nodes or elements obtained, the relationship matching rules are as follows Figure 8 As shown, including:

[0216] S801: Sort by node area in ascending order;

[0217] S802: For node a, traverse all nodes b that are larger than it, and calculate the matching cost cost(a,b) corresponding to each node b;

[0218] S803: If the matching cost is less than 0.1, b is a candidate parent node of a and is added to the candidate parent node list pl of node a;

[0219] S804: looping step S802 and step S803 to obtain a complete list of candidate parent nodes for each node;

[0220] S805: traverse the candidate list, and select the node with the smallest area in the list as the final parent node for each node. If the candidate parent node is empty, the parent node of the node is recorded as -1.

[0221] S806. Output hierarchical relationship.

[0222] The obtained hierarchical relationship is processed into a form that is easy to store and output.

[0223] The relationship matching results in S803 are integrated to generate structured results, which can be used for subsequent reconstruction of the architecture diagram.

[0224] In the embodiment of the present application, for the target detection algorithm of S802: fully utilize the advantages of deep learning methods to extract image features, identify architecture diagram elements and locate node positions through supervised training. This process can be implemented using target detection algorithms with better performance, such as the yolo algorithm and the rtdetr algorithm.

[0225] The structured result here can be a json file. Each object contains an ID, four coordinates, parent node coordinates, and text, so that this information is saved in a unified form.

[0226] In the embodiment of the present application, for the rule-based relationship matching method of S803: based on the architecture diagram nodes and their position information extracted by the target detection algorithm, rules are designed based on the characteristics of the architecture diagram to match and construct hierarchical relationships.

[0227] In order to implement the design of architectural diagram relationship matching rules, the relationship hierarchy rules of elements in the architectural diagram can be summarized. First, define each element in the architectural diagram as a node, and each node has its own number, where the corresponding node can be found based on the number. Secondly, by default, each node in the architectural diagram has its own parent node, and the parent node number is recorded. Thirdly, when the parent node of a node does not exist, the node is considered to be the bottom-level node of the relationship, and the parent node is defined as -1. Finally, the following rules can be obtained by summarizing the relationship between elements in the diagram:

[0228] 1. The area of ​​the parent node is not less than the area of ​​the child node;

[0229] 2. The parent node must almost completely contain the child node;

[0230] 3. When a node has multiple nodes containing it: give priority to selecting the node with the smallest area as the parent node, and give priority to the node with a larger aspect ratio as the parent node.

[0231] In the embodiment of the present application, the parent-child relationship matching cost is designed to select the most likely parent node as the final result:

[0232] Assume that there are nodes a and b, and define the regions where the two nodes are located as S a , S b , if S b ≥S a , the cost function between two nodes can be defined as the following formula:

[0233]

[0234] Among them, cost(a,b) represents the matching cost between nodes a and b. When node a completely belongs to the region of node b, the cost is the smallest. That is, when there are multiple nodes b, the node with the largest matching inclusion degree is used as the parent node of node a. Design based on architecture diagram feature analysis and relationship matching cost.

[0235] The image processing method provided in the embodiment of the present application is a rule-based relationship matching method, such as Fig. 9 As shown, including:

[0236] S901. Positioning nodes based on a target detection model.

[0237] S902, arrange the nodes in ascending order of area {b1, b2, ..., bn}.

[0238] S903: Calculate the matching cost cost(bi, bj) between node i and node j.

[0239] Among them, 0 <i<n,i<j<n。

[0240] S904. Determine the parent node list pi of node i based on cost (bi, bj).

[0241] The parent node list can be understood as a list of candidate parent elements.

[0242] S905, selecting the node with the smallest area in the parent node list pi as the parent node, and obtaining a hierarchical relationship.

[0243] The image processing method provided in the embodiment of the present application can normally identify elements in the architecture diagram with its own characteristics and relatively accurately extract the hierarchical relationship between elements. By constructing a corresponding supervised learning data set, the target detection algorithm based on deep learning can accurately extract the position of elements in the architecture diagram. Based on the target positioning results of the target detection model, the relationship matching rules in the present technical solution can accurately construct the hierarchical relationship between the identified targets. In the architecture diagram data set, the target detection part and the relationship extraction part have been preliminarily trained and tested. The current target detection accuracy is 97.5%, and the relationship matching accuracy is 88.5%, which proves the feasibility of the image processing method provided in the embodiment of the present application.

[0244] The embodiment of the present application proposes a diagram recognition and relationship extraction method for architecture diagrams. First, the feature extraction capability of deep learning methods is fully utilized, and the target detection algorithm is used to locate the elements of the architecture diagram. Secondly, by analyzing the characteristics of the architecture diagram, a relationship matching rule based on minimizing the regional cost is designed. Finally, combining the diagram element recognition results and the relationship matching rule design, this solution can realize the extraction of the hierarchical relationship of the elements in the architecture diagram, thereby realizing the editability and secondary use of the architecture diagram.

[0245] An electronic device according to an embodiment of the present application, such as Fig.10 As shown, the electronic device 1000 includes:

[0246] An acquisition module 1001 is configured to acquire a first image, wherein the image content of the first image includes an architecture diagram;

[0247] A detection module 1002 is configured to detect position information of each element of a plurality of elements included in the first image based on a target detection model;

[0248] A determination module 1003 is configured to determine a hierarchical relationship between the multiple elements included in the first image based on position information of each element in the multiple elements;

[0249] The construction module 1004 is configured to construct the architecture diagram based on the hierarchical relationship between the multiple elements.

[0250] In some embodiments, the determination module 1003 is further configured to:

[0251] Each of the multiple elements is taken as the first element, and the following processing is performed respectively:

[0252] Based on position information of each element in the multiple elements, determine the parent element of the first element from multiple second elements, the multiple second elements include other elements in the multiple elements except the first element, the area occupied by the first element is located within the area occupied by the parent element of the first element, and no other elements are included between the parent element of the first element and the first element.

[0253] In some embodiments, the determination module 1003 is further configured to:

[0254] Based on the position information of the first element and the position information of each second element in the plurality of second elements, respectively determining matching costs of the first element and the plurality of second elements, the matching costs being used to represent the possibility that an area occupied by the second element includes an area occupied by the first element;

[0255] Based on the matching cost between each second element in the plurality of second elements and the first element, selecting at least one candidate parent element of the first element from the plurality of second elements, wherein the area occupied by the first element is within the area occupied by the candidate parent element of the first element;

[0256] A parent element of the first element is selected from at least one candidate parent element of the first element.

[0257] In some embodiments, the determination module 1003 is further configured to:

[0258] In the case that there is no candidate parent element for the first element, the parent node of the first element is identified as a set identifier, and the set identifier is used to indicate that there is no parent node for the first element.

[0259] In some embodiments, the determination module 1003 is further configured to:

[0260] Determine a size of an overlapping area between the first element and the second element based on the position information of the first element and the position information of the second element;

[0261] Based on the size of the overlapping area, a matching cost of the first element and the second element is determined.

[0262] In some embodiments, the determination module 1003 is further configured to:

[0263] For each second element in the plurality of second elements, the following processing is performed respectively:

[0264] comparing the matching cost of the second element and the first element with a matching cost threshold;

[0265] Based on the comparison result, it is determined whether the second element is a candidate parent element of the first element.

[0266] In some embodiments, the determination module 1003 is further configured to:

[0267] Based on the area of ​​each candidate parent element among the multiple candidate parent elements included in the at least one candidate parent element, selecting a candidate parent element with the smallest area from the multiple candidate parent elements as the parent element;

[0268] The area of ​​the candidate parent element is determined based on the position information of the candidate parent element.

[0269] In some embodiments, the electronic device 1000 further includes:

[0270] A text acquisition module, configured to acquire text information of each element of the plurality of elements;

[0271] The construction module 1004 is configured to reconstruct the architecture diagram based on the text information of each element in the multiple elements and the hierarchical relationship between the multiple elements.

[0272] In some embodiments, the electronic device 1000 further includes:

[0273] A sample acquisition module is configured to obtain a sample image set, wherein the sample image set includes a plurality of sample images, and the contents of the sample images include a sample architecture diagram;

[0274] a labeling module configured to label elements in each of the sample images to obtain label data, wherein the label data includes labeling information of each of the plurality of sample images;

[0275] The training module is configured to train the initial detection model based on the sample image set and the label data to obtain the target detection model, and the prime number target detection model is the converged initial detection model.

[0276] In some embodiments, the electronic device 1000 further includes:

[0277] The preprocessing module is configured to perform image preprocessing on the multiple sample images in the sample image set respectively so that the sizes of different sample images are the same.

[0278] In practical applications, the above-mentioned acquisition module, detection module, determination module, text acquisition module, construction module, sample acquisition module, labeling module, training module, preprocessing module and other modules can be implemented by a processor located on an electronic device, specifically a combination of one or more of a central processing unit (CPU), an image graphics processor (graphics processing unit, GPU), a microprocessor (MPU, Microprocessor Unit), an application processor (AP), a digital signal processor (DSP, Digital Signal Processing), and a field programmable gate array (FPGA, Field Programmable Gate Array).

[0279] Those skilled in the art should understand that the relevant description of the above-mentioned electronic device in the embodiment of the present application can be understood by referring to the relevant description of the image processing method in the embodiment of the present application.

[0280] Fig.11 An optional implementation of the present application is provided as a structural diagram of an electronic device, such as Fig.11As shown, the embodiment of the present application provides an electronic device 1100, including an electronic chip 1101, which can be implemented as an image processing method as described in one or more embodiments above. The electronic chip can be implemented as a processor of an electronic device or as a microprocessor of a fast charging module.

[0281] An embodiment of the present application provides an electronic device, Fig.12 Another optional implementation provided in the embodiment of the present application is a structural diagram of an electronic device, such as Fig.12 As shown, an embodiment of the present application provides an electronic device 1200, including:

[0282] A processor 1201 and a storage medium 1202 storing executable instructions of the processor 1201, wherein the storage medium 1202 relies on the processor 1201 to perform operations via a communication bus 1203, and when the instructions are executed by the processor 1201, the image processing method performed in one or more of the above embodiments is executed.

[0283] It should be noted that, in actual application, the various components in the terminal are coupled together via the communication bus 1203. It is understandable that the communication bus 1203 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 1203 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as communication buses 1203.

[0284] An embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the image processing method described in one or more of the above embodiments.

[0285] An embodiment of the present application provides a schematic structural diagram of an electronic device 1300. Fig.13 The electronic device 1300 shown includes a processor 1310. The processor 1310 is configured to:

[0286] Obtaining a first image, wherein image content of the first image includes an architecture diagram;

[0287] Based on the target detection model, detecting position information of each element among a plurality of elements included in the first image;

[0288] Determining a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image;

[0289] The architecture diagram is constructed based on the hierarchical relationship between the multiple elements.

[0290] In the embodiment of the present application, the processor 1310 can call and run a computer program from the memory to implement the image processing method in the embodiment of the present application.

[0291] Alternatively, if Fig.13 As shown, the electronic device 1300 may further include a memory 1320. The processor 1310 may call and run a computer program from the memory 1320 to implement the image processing method in the embodiment of the present application.

[0292] The memory 1320 may be a separate device independent of the processor 1310 , or may be integrated into the processor 1310 .

[0293] Alternatively, if Fig.13 As shown, the electronic device 1300 may further include a transceiver 1330, and the processor 1310 may control the transceiver 1330 to communicate with other devices, specifically, to receive signals sent by other devices. Here, the transceiver may include at least two antennas.

[0294] It can be understood that the transceiver includes multiple physical paths for receiving signals or transmitting signals, and the physical elements on one or more physical paths for transmitting signals constitute a transmitter.

[0295] Optionally, the electronic device 1300 can implement the corresponding processes implemented by the electronic device in each method of the embodiment of the present application, and for the sake of brevity, it will not be repeated here. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Arra y, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0296] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0297] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct RAM bus random access memory (Direct RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0298] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0299] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0300] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0301] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0302] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0303] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0304] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0305] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0306] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0307] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0308] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a first image, wherein image content of the first image includes an architecture diagram; Based on the target detection model, detecting position information of each element among a plurality of elements included in the first image; Determining a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image; The architecture diagram is constructed based on the hierarchical relationship between the multiple elements.

2. The method according to claim 1, characterized in that: The determining, based on the position information of each element among the multiple elements included in the first image, the hierarchical relationship between the multiple elements includes: Each of the multiple elements is taken as the first element, and the following processing is performed respectively: Based on position information of each element in the multiple elements, determine the parent element of the first element from multiple second elements, the multiple second elements include other elements in the multiple elements except the first element, the area occupied by the first element is located within the area occupied by the parent element of the first element, and no other elements are included between the parent element of the first element and the first element.

3. The method according to claim 2, characterized in that The determining, based on the position information of each element in the plurality of elements, the parent element of the first element from the plurality of second elements comprises: Based on the position information of the first element and the position information of each second element in the plurality of second elements, respectively determining matching costs of the first element and the plurality of second elements, the matching costs being used to represent the possibility that an area occupied by the second element includes an area occupied by the first element; Based on the matching cost between each second element in the plurality of second elements and the first element, selecting at least one candidate parent element of the first element from the plurality of second elements, wherein the area occupied by the first element is within the area occupied by the candidate parent element of the first element; A parent element of the first element is selected from at least one candidate parent element of the first element.

4. The method according to claim 3, characterized in that The determining the parent element of the first element from the plurality of second elements based on the position information of each element in the plurality of elements further includes: In the case that there is no candidate parent element for the first element, the parent node of the first element is identified as a set identifier, and the set identifier is used to indicate that there is no parent node for the first element.

5. The method according to claim 3, characterized in that: The determining, based on the position information of the first element and the position information of the second element, the matching cost of the first element and the second element includes: Determine a size of an overlapping area between the first element and the second element based on the position information of the first element and the position information of the second element; The matching cost of the first element and the second element is determined based on the size of the overlapping area.

6. The method according to claim 3, characterized in that Selecting at least one candidate parent element of the first element from the plurality of second elements based on a matching cost between each second element in the plurality of second elements and the first element includes: For each second element in the plurality of second elements, the following processing is performed respectively: comparing the matching cost of the second element and the first element with a matching cost threshold; Based on the comparison result, it is determined whether the second element is a candidate parent element of the first element.

7. The method according to claim 3, characterized in that The at least one candidate parent element includes a plurality of candidate parent elements, and selecting the parent element of the first element from the at least one candidate parent element of the first element includes: Based on the area of ​​each candidate parent element among the multiple candidate parent elements, select a candidate parent element with the smallest area from the multiple candidate parent elements as the parent element; The area of ​​the candidate parent element is determined based on the position information of the candidate parent element.

8. The method according to claim 1, characterized in that: The method further comprises: Obtaining text information of each element in the multiple elements; Accordingly, based on the hierarchical relationship between the multiple elements, the architecture diagram is constructed, including: The architecture diagram is reconstructed based on the text information of each of the multiple elements and the hierarchical relationship between the multiple elements.

9. The method according to claim 1, characterized in that: The method further comprises: Obtaining a sample image set, wherein the sample image set includes a plurality of sample images, and contents of the sample images include a sample architecture diagram; For the sample image set, respectively labeling elements in each of the sample images to obtain label data, wherein the label data includes labeling information of each sample image in the plurality of sample images; Based on the sample image set and the label data, the initial detection model is trained to obtain the target detection model, and the prime number target detection model is the converged initial detection model.

10. The method according to claim 9, characterized in that The method further comprises: Image preprocessing is performed on a plurality of sample images in the sample image set respectively so that sizes of different sample images are the same.

11. An electronic device, characterized in that: The electronic device comprises: An acquisition module is configured to acquire a first image, wherein the image content of the first image includes an architecture diagram; a detection module, configured to detect position information of each element among a plurality of elements included in the first image based on a target detection model; a determination module configured to determine a hierarchical relationship between the plurality of elements included in the first image based on position information of each element in the plurality of elements; A construction module is configured to construct the architecture diagram based on the hierarchical relationship between the multiple elements.

12. An electronic device, comprising a processor, characterized in that: The processor is configured to: Obtaining a first image, wherein image content of the first image includes an architecture diagram; Based on the target detection model, detecting position information of each element among the multiple elements included in the first image, wherein the position information is used to represent an area occupied by the element in the architecture diagram; Determining a hierarchical relationship between the multiple elements based on position information of each element among the multiple elements included in the first image; The architecture diagram is constructed based on the hierarchical relationship between the multiple elements.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps in the image processing method according to any one of claims 1 to 10 are implemented.

14. A storage medium storing an executable program, characterized in that: When the executable program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

15. A chip, comprising a processor, characterized in that: The processor is configured to execute the image processing method according to any one of claims 1 to 10.

Images