A card template determination method and related device

Abstract

An embodiment of the present application provides a method and related apparatus for determining a card template in the field of artificial intelligence. The method aggregates elements identified in an input image into at least one component, and then pushes a template matching the component to the user. The user only needs to upload a picture of the card layout he or she likes, and the server can automatically match a suitable combination template based on the picture, thereby avoiding repeated design and the difficulty of designing cards, and improving development efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method for determining a card template and related devices. Background Art

[0002] With the development of modern technology, applications (APPs) on smart terminals such as mobile phones and tablets can present rich and colorful content. Among them, presenting information through cards is a common form. Figure 1 A schematic diagram showing information presented through cards. Figure 1 As can be seen, different food shops are presented through cards, each of which can carry information such as the shop's name, logo, rating, and distance. The content and style of these cards need to be designed and provided by the content provider (CP).

[0003] Content providers can offer apps or mini-programs, presenting information in the form of cards. When designing cards, content providers or individual developers can use the developer platform to generate card styles. To facilitate quick card design, the developer platform provides a variety of card templates for content providers and individual developers to choose from.

[0004] However, the current developer platform can only allow content providers or individual developers to select or design cards themselves, and cannot automatically recommend suitable card templates to content providers or individual developers. Summary of the Invention

[0005] The embodiments of the present application provide a method and related device for determining a card template, which can automatically match a suitable template based on the pictures uploaded by the user, thereby improving development efficiency.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a card template, comprising: obtaining an input image; identifying and classifying elements in the input image; aggregating the elements to obtain at least one component, wherein the component is a combination of at least one of the elements; and determining a target template from the candidate template based on the similarity between the component and the candidate template.

[0007] In combination with the first aspect, in an implementation of the embodiment of the present application, after acquiring the input image and before identifying and classifying elements in the input image, the method further includes:

[0008] The input image is grayscaled, Gaussian low-pass filtered, and binarized to obtain the binarized input image.

[0009] In conjunction with the first aspect, in one implementation of the embodiment of the present application, identifying and classifying elements in the input image includes:

[0010] Performing an erosion process on the input image so that the text area in the input image is processed into a connected area;

[0011] Draw the bounding rectangle of each area in the input image using a contour detection algorithm;

[0012] The element in the input image is determined according to the position of the circumscribed rectangle.

[0013] In conjunction with the first aspect, in one implementation of the embodiment of the present application, identifying and classifying elements in the input image further includes:

[0014] If there is text in the circumscribed rectangle and the area of ​​the text is equal to the area of ​​the circumscribed rectangle, determining that the element corresponding to the circumscribed rectangle is a text element;

[0015] If there is no text in the bounding rectangle and the area of ​​the bounding rectangle is smaller than a preset threshold, determining that the element corresponding to the bounding rectangle is a control element;

[0016] If there is text in the circumscribed rectangle and the area of ​​the text is not equal to the area of ​​the circumscribed rectangle, or if there is no text in the circumscribed rectangle and the area of ​​the circumscribed rectangle is not less than a preset threshold, it is determined that the element corresponding to the circumscribed rectangle is a picture element.

[0017] In conjunction with the first aspect, in an implementation of the embodiment of the present application, after determining that the element corresponding to the circumscribed rectangle is a text element, the method further includes:

[0018] Identifying the meaning of text in the text elements by using a natural language understanding semantic recognition algorithm;

[0019] The specific category of the text element is determined according to the meaning of the text.

[0020] In conjunction with the first aspect, in one implementation of the embodiment of the present application, aggregating the elements to obtain at least one component includes:

[0021] Calculating the offset between the elements;

[0022] The elements are clustered according to the offsets between the elements using a clustering algorithm to obtain at least one component.

[0023] In conjunction with the first aspect, in one implementation of the embodiment of the present application, calculating the offset between the elements includes:

[0024] If the first element and the second element intersect in the horizontal direction, the vertical distance between the first element and the second element is calculated as the element offset;

[0025] If the first element and the second element intersect in the vertical direction, the horizontal distance between the first element and the second element is calculated as the element offset;

[0026] If the first element and the second element do not intersect, the vertical spacing and the horizontal spacing between the first element and the second element are calculated respectively, and the minimum value is selected as the element offset.

[0027] In conjunction with the first aspect, in an implementation of the embodiment of the present application, after aggregating the elements to obtain at least one component and before determining the target template from the candidate templates based on the similarity between the component and the candidate templates, the method further includes:

[0028] Obtaining semantic information corresponding to the element through a natural language understanding semantic recognition algorithm or a computer vision recognition algorithm;

[0029] Elements in the component are adjusted according to the semantic information.

[0030] In conjunction with the first aspect, in an implementation of the embodiment of the present application, determining the target template from the candidate templates based on the similarity between the component and the candidate templates includes:

[0031] Generate an undirected graph corresponding to the component according to the elements in the component;

[0032] Calculating the similarity between the component and the candidate template based on the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template;

[0033] The candidate template corresponding to the highest similarity is determined as the target template corresponding to the component.

[0034] In conjunction with the first aspect, in an implementation of the embodiment of the present application, generating an undirected graph corresponding to the component according to the elements in the component includes:

[0035] Taking one of the elements in the component as an initial node, searching for elements in the component in vertical and horizontal directions and connecting the searched elements to the initial node in sequence with edges;

[0036] In the order in which the elements are searched, with the searched elements as subsequent nodes, the elements are searched in the vertical and horizontal directions in the component and the searched elements are sequentially connected with edges.

[0037] In conjunction with the first aspect, in an implementation of the embodiment of the present application, generating an undirected graph corresponding to the component according to the elements in the component includes:

[0038] The nodes in the undirected graph are labeled with labels corresponding to the categories of the elements.

[0039] In conjunction with the first aspect, in an implementation of the embodiment of the present application, calculating the similarity between the component and the candidate template based on the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template includes:

[0040] Calculating similarities between nodes in an undirected graph corresponding to the component and an undirected graph corresponding to the candidate template;

[0041] The similarity between the component and the candidate template is calculated according to the similarity between the nodes.

[0042] In conjunction with the first aspect, in one implementation of the embodiment of the present application, the similarity between the nodes in the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template is calculated using a node similarity calculation formula, and the node similarity calculation formula is:

[0043] sim(v i ,u j ) = sigmoid(h);

[0044] h=w1s t +w2s d +w3s l +w4s c ;

[0045] Among them, v i is the node in the undirected graph corresponding to the component, u j is the node in the undirected graph corresponding to the candidate template, w1, w2, w3, w4 are the preset weight parameters, s t is the node label similarity, s d is the node degree similarity, s l is the node loop similarity, s c is the link node similarity.

[0046] In conjunction with the first aspect, in one implementation of the embodiment of the present application, the similarity between the component and the candidate template is calculated based on the similarity between the nodes using a graph similarity calculation formula, and the graph similarity calculation formula is:

[0047]

[0048] Among them, SIM(G c ,Gt ) is component G c With the candidate template G t The similarity function between i For component G c The node of φ(v i ) is the candidate template G t and node v in i The corresponding nodes do not overlap with other nodes.

[0049] In a second aspect, an embodiment of the present application provides a device for determining a card template, including:

[0050] Acquisition module, used to obtain input images;

[0051] a processing module, configured to identify and classify elements in the input image;

[0052] The processing module is further configured to aggregate the elements to obtain at least one component, wherein the component is a combination of at least one of the elements;

[0053] The processing module is further configured to determine a target template from the candidate templates based on the similarity between the component and the candidate templates.

[0054] In a third aspect, an embodiment of the present application provides a device for determining a card template, comprising: a processor and a transmission interface, wherein the processor is configured to call program instructions stored in a memory to execute the method of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer or a processor, cause the computer or the processor to execute the method of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, wherein the computer program product includes program instructions, and when the program instructions are executed on a computer or a processor, the computer or the processor executes the method as in the first aspect.

[0057] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0058] An embodiment of the present application provides a method and related device for determining a card template in the field of artificial intelligence. The method aggregates elements identified in an input image into at least one component, and then pushes a template matching the component to the user. The user only needs to upload a picture of his or her favorite card layout, and the server can automatically match a suitable combination template based on the picture, avoiding repeated design and the difficulty of designing cards, thereby improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram for presenting information through cards;

[0060] Figure 2 A flowchart of a method for determining a card template provided in an embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of preprocessing an input image in an embodiment of the present application;

[0062] Figure 4 The input image is grayscaled, Gaussian low-pass filtered, and binarized;

[0063] Figure 5 A schematic diagram of drawing a circumscribed rectangle in an embodiment of the present application;

[0064] Figure 6 This is a flow chart of element classification in the embodiments of this application;

[0065] Figure 7 A schematic diagram of element offsets in an embodiment of the present application;

[0066] Figure 8 This is a schematic diagram of the element offset matrix table corresponding to the input image in the embodiment of the present application;

[0067] Figure 9 A schematic diagram of adjacent element search in an embodiment of the present application;

[0068] Figure 10 A schematic diagram of component identification and error correction in an embodiment of the present application;

[0069] Figure 11a A schematic diagram illustrating components in an embodiment of the present application;

[0070] Figure 11b In the embodiment of this application Figure 11a The corresponding undirected graph;

[0071] Figure 12 This is an example matrix for calculating node similarity in an embodiment of the present application;

[0072] Figure 13 Schematic diagram of the relationship matrix between node label similarity and node label in the embodiment of the present application;

[0073] Figure 14 A schematic diagram of the overall solution of an embodiment of the present application;

[0074] Figure 15 A schematic diagram of a device for determining a card template in an embodiment of the present application;

[0075] Figure 16A schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The embodiments of the present application provide a method and related device for determining a card template, which can automatically match a suitable template based on the pictures uploaded by the user, thereby improving development efficiency.

[0077] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and in the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0078] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0079] To make the description of the following embodiments clear and concise, a brief introduction to the related technologies is first given:

[0080] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0081] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0082] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0083] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0084] The solutions provided in the embodiments of the present application involve technologies such as image recognition and semantic recognition by artificial intelligence, which are specifically described through the following embodiments:

[0085] Figure 2 This is a flowchart of a method for determining a card template provided in an embodiment of the present application. This process can be executed by a device capable of executing computer instructions, such as a server. The server can then display the execution result (the determined card template) on the developer platform's website or push it to a terminal (such as a user device). This embodiment of the present application is described using a server as an example, and other devices can refer to the embodiments of the present application for implementation.

[0086] The process includes:

[0087] 201. Get input image;

[0088] In an embodiment of the present application, a user can upload a desired image to a server, so that the server can obtain the input image. Specifically, the user can prepare a design draft in the form of a picture in advance by hand-drawing or simple puzzle-making, and then click to upload the design draft on the developer platform website to obtain the corresponding card template.

[0089] 202. Identify and classify elements in the input image;

[0090] In an embodiment of the present application, after the server obtains the input image, the server may first pre-process the input image. Figure 3 Schematic diagram of preprocessing the input image in an embodiment of the present application. The preprocessing includes grayscale, Gaussian low-pass filtering and binarization of the input image. Grayscale refers to the conversion of a color image into a grayscale image. Gaussian low-pass filtering refers to a linear smoothing filter that is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. Image binarization is the process of setting the grayscale value of the pixel points on the image to 0 or 255, that is, the process of making the entire image appear in obvious black and white effect. Figure 4 The input image is grayscaled, Gaussian low-pass filtered, and binarized. Grayscaled, Gaussian low-pass filtered, and binarized have corresponding mature processing algorithms in image processing, and the present application does not limit them.

[0091] In an embodiment of the present application, the server can use a connected area method to identify elements. First, the input image is corroded to process the text into connected areas. The embodiment of the present application does not limit the specific algorithm used for the corrosion process. The corrosion process can make the text in the input image thicker and more blurred. After multiple corrosion processes (generally four or five times), the text in the input image is bonded into several connected areas, which allows the subsequent contour detection algorithm to more accurately depict the area of ​​the text.

[0092] Therefore, after the server performs corrosion processing on the input image, the bounding rectangle of each area in the input image can be drawn using a contour detection algorithm. Figure 5 Schematic diagram of drawing a circumscribed rectangle in the embodiment of the present application. In the embodiment of the present application, the server can Figure 4 For the text area in the image, the corresponding bounding rectangle is drawn by the contour detection algorithm, and then the position information of these bounding rectangles is recorded to identify the elements of the text part. Figure 5 As shown, the server can conveniently draw the corresponding bounding rectangles through the contour detection algorithm, and then record the position information of these bounding rectangles.

[0093] Finally, the position information of these bounding rectangles is the position information of the elements. In some cases, the server can number these elements for subsequent processing.

[0094] In the embodiment of the present application, after the server identifies the elements in the input image, it can classify these elements and establish an association relationship between the elements and the categories they belong to. The rules for the server to classify the elements are:

[0095] If there is text in the bounding rectangle and the area of ​​the text is equal to the area of ​​the bounding rectangle, then the element corresponding to the bounding rectangle is determined to be a text element;

[0096] If there is no text in the bounding rectangle and the area of ​​the bounding rectangle is smaller than a preset threshold, the element corresponding to the bounding rectangle is determined to be a control element;

[0097] If there is text in the bounding rectangle and the area of ​​the text is not equal to the area of ​​the bounding rectangle, or if there is no text in the bounding rectangle and the area of ​​the bounding rectangle is not less than a preset threshold, the element corresponding to the bounding rectangle is determined to be a picture element.

[0098] According to the above classification rules, the server can use at least one process to classify the elements. The embodiment of the present application provides one of the better processes such as Figure 6 shown.

[0099] Figure 6 This is a flow chart of element classification in the embodiment of this application. The process is as follows:

[0100] ① The server first uses a text recognition algorithm to determine whether there is text within the bounding rectangle. If so, it executes ②; if not, it executes ③;

[0101] ② The server detects whether the area of ​​the region corresponding to the text is equal to the area of ​​the circumscribed rectangle. If so, it executes ④; if not, it executes ⑤;

[0102] ③ The server detects whether the area of ​​the bounding rectangle is greater than a preset threshold. If so, it executes ⑤; if not, it executes ⑥;

[0103] ④The server determines that the element corresponding to the bounding rectangle is a text element and labels the element;

[0104] ⑤The server determines that the element corresponding to the bounding rectangle is a picture element and labels the element (G: Graph);

[0105] ⑥The server determines that the element corresponding to the bounding rectangle is a control element and labels the element (C: Controller).

[0106] In some embodiments, the server can further subdivide the text elements, for example, into ordinary text (T: Text), rating text (R: Rate), date and time text (D: Date), and price text (P: Price). A specific method can be that the server identifies the text meaning in the text element through a natural language understanding (NLU) semantic recognition algorithm; and determines the specific category of the text element based on the text meaning. In actual applications, the server can also set other text categories and identify them through appropriate semantic recognition algorithms, which is not limited in this embodiment of the present application.

[0107] 203. Aggregate the elements to obtain at least one component, where the component is a combination of at least one element;

[0108] In an embodiment of the present application, after the server identifies and classifies elements in an input image, it may cluster these elements to obtain a plurality of components. Specifically, the server may first calculate the offset distances between the elements, and then cluster these elements using a clustering algorithm based on the offset distances to obtain at least one component.

[0109] In the embodiment of the present application, if the first element and the second element intersect in the horizontal direction (it can be understood that the two elements are in a vertical relationship, such as Figure 7 As shown), the vertical distance between the first element and the second element is calculated as the element offset; if the first element and the second element intersect in the vertical direction (which can be understood as the two elements are in a left-right relationship), the horizontal distance between the first element and the second element is calculated as the element offset; if the first element and the second element do not intersect, the vertical distance and the horizontal distance between the first element and the second element are calculated respectively, and the minimum value is selected as the element offset. For example, Figure 7 Schematic diagram of element offset in the embodiment of the present application. Element O1 and element O2 intersect in the horizontal direction, and the offset between element O1 and element O2 is the distance between them in the vertical direction, such as Figure 7 In actual applications, the server can establish a rectangular coordinate system, determine the area corresponding to the element's circumscribed rectangle based on the position information of the element's circumscribed rectangle, and then determine whether two elements intersect and calculate the vertical or horizontal spacing between the elements based on this position information. This embodiment of the application will not be described in detail.

[0110] For example, the server identifies n elements (i.e., C1, C2, C3...C N ), then after the server calculates the offset between each two elements, it can get the following Figure 8 The table shown, Figure 8Schematic diagram of the element offset matrix corresponding to the input image in the embodiment of this application. Among them, there is no offset between the element and itself, which is represented by "×".

[0111] In the embodiment of the present application, after the server calculates the offsets between the elements, it can cluster the elements according to the offsets between the elements through a clustering algorithm to obtain at least one component. Specifically, the server can start from one element and search for adjacent elements according to the offsets between the elements (for example, two elements can be considered as adjacent elements if the offsets are within a preset range), and aggregate the adjacent elements into components, such as Figure 9 shown. Figure 9 Schematic diagram of adjacent element search in an embodiment of the present application. The server may first start with C1 and search for adjacent elements, namely, C3 and C6. The server then continues to search for elements adjacent to C3 (no adjacent elements are found), then searches for elements adjacent to C6 (C7 and C8 are found), and so on. Ultimately, the server may aggregate C1, C3, C6, C7, and C8 into component 1.

[0112] In actual applications, the server can use a suitable clustering algorithm to cluster these elements. The embodiment of the present application does not limit the clustering algorithm used. Ultimately, the server can obtain at least one component.

[0113] The embodiment of the present application calculates the offsets between elements, aggregates the elements according to the offset sizes, and identifies them as components. It is possible to identify components when only image information is available.

[0114] In some embodiments, in order to further ensure and improve the accuracy of component aggregation, after the server obtains at least one component, it can also obtain the semantic information corresponding to the element through the natural language understanding (NLU) semantic recognition algorithm or the computer vision (CV) recognition algorithm; adjust the elements in the component according to the semantic information. Specifically, the server can use NLU to identify the semantics of the text, use CV to identify the semantics of the image, recombine elements with similar semantics and adjacent elements, and correct the element aggregation results. In actual applications, if the semantic information of an element is dissimilar to most other elements in the group, and the semantic information of the element is similar to the semantic information of the elements in the adjacent components, the element can be reclassified into the adjacent components.

[0115] For example, Figure 10This is a schematic diagram of component identification and error correction in an embodiment of the present application. The server recognizes that the semantics of "China Film International Cinema" is "cinema," which is different from the semantic information of "beverage" of other elements in its component. Furthermore, the semantics of "China Film International Cinema" are similar to the semantic information of elements in adjacent components, both belonging to the movie category. Therefore, the server can determine this and reclassify "China Film International Cinema" into this adjacent component.

[0116] 204. Determine a target template from the candidate templates based on the similarity between the component and the candidate templates.

[0117] In an embodiment of the present application, the server can match the grouped components with existing candidate templates, find the target template with the highest similarity, and combine the target templates according to the positional relationship between the components as a combined template recommended to the user.

[0118] First, the server can generate an undirected graph corresponding to the component based on the elements in the component. Specifically, after obtaining the component, the server abstracts the features of the elements within the component, abstracting them with labeled nodes and linking adjacent nodes with edges. Specifically, the server can use one of the components (usually the top-left element) as the initial node, search for elements in the component vertically and horizontally, and connect the found elements to the initial node in sequence with edges. Then, using the searched elements as subsequent nodes in the order in which they were found, the server searches for elements in the component vertically and horizontally, and connects the searched elements to the subsequent nodes in sequence with edges. Specifically, the following rules can be used: ① Use the top-left element as the initial node. ② If there is no initial node, define an empty node in the top-left corner. ③ Label the nodes according to the element classification results, with a total of six categories: T, R, D, P, G, and C. ④ Starting from the initial node, search for elements vertically and horizontally. ⑤ The search for elements prioritizes downward movement.

[0119] Figure 11a This is a schematic diagram illustrating components in an embodiment of the present application. Figure 11b In the embodiment of this application Figure 11a The corresponding undirected graph. One of the components obtained by the server is Figure 11a As shown. Then, the server can locate the initial node with the upper left corner element in the component, add the G (picture) label, and start searching downward. If there are no other elements, search rightward and find three texts, which are connected with edges. The three newly connected nodes continue to search in the order they were searched until all nodes are abstracted into nodes of an undirected graph, as shown. Figure 11b shown.

[0120] The server can then calculate the similarity between the component and the candidate template based on the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template. Specifically, the server can calculate the similarity between the nodes in the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template; and calculate the similarity between the component and the candidate template based on the similarity between the nodes.

[0121] The server can calculate the similarity between the component and the candidate template based on the similarity between nodes using the graph similarity calculation formula. The graph similarity calculation formula is:

[0122]

[0123] Among them, SIM(G c ,G t ) is component G c With the candidate template G t The similarity function between components G is used to calculate c With the candidate template G t The similarity between them, where the component feature undirected graph G c =(V c ,E c ), template feature undirected graph G t =(U t ,E t ), V and U represent nodes, and E represents edges. i For component G c The node of φ(v i ) is the candidate template G t and node v in i The corresponding nodes do not overlap with other nodes. sim is two nodes (v i and φ(v i )) is a similarity function used to calculate the similarity between two nodes. i ) indicates a one-to-one correspondence between the nodes of the two graphs. They may not correspond, but they cannot correspond repeatedly. For example, [(v1,u1),(v2,u2),(v3)] and [(v1,u2),(v2,u1),(v3)] are both legal correspondences; [(v1,u1),(v2,u1),(v3,u2)] is an illegal correspondence because u1 corresponds repeatedly. Figure 12 This is an example matrix for calculating node similarity in the embodiment of the present application, component G c The corresponding undirected graph contains nodes V C1 、V C2 …V Cn , the undirected graph corresponding to the template contains the node U t1 、U t2 …U tm , Snm Represents node V Cn and node U tm The similarity between them.

[0124] Specifically, the server can calculate the similarity between the nodes in the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template using a node similarity calculation formula. The node similarity calculation formula is:

[0125] sim(v i ,u j ) = sigmoid(h);

[0126] h=w1s t +w2s d +w3s l +w4s c ;

[0127] Among them, v i is the node in the undirected graph corresponding to the component, u j is the node in the undirected graph corresponding to the candidate template, w1, w2, w3, w4 are the preset weight parameters, s t is the node label similarity, s d is the node degree similarity, s l is the node loop similarity, s c is the link node similarity.

[0128] ①Node label similarity s t It can be determined by the labels between two nodes according to pre-set rules. For example, Figure 13 This is a schematic diagram of the relationship matrix between node label similarity and node label in the embodiment of this application. The server can calculate the node label similarity based on this matrix, where the value of node label similarity can be specifically set according to actual needs, and this embodiment of the application does not limit this. The example matrix provided in this embodiment of the application is as follows Figure 13 As shown, the node label similarity between nodes with the same label is 1, the four text controls have high similarity, the image and the control have high similarity, and there is no similarity between (TDPR) and (GC) (i.e., 0).

[0129] ②Node degree similarity s d , the minimum value is greater than the maximum value, if the maximum value is 0, then the value is 1.

[0130]

[0131] Among them, degree c ,degree t are the degrees of the two nodes respectively.

[0132] ③Node loop similarity sl , the minimum value is greater than the maximum value, if the maximum value is 0, then the value is 1.

[0133]

[0134] Among them, loop c , loop t They are loops of two nodes respectively.

[0135] ④Link node similarity s c , the degree of nodes with the same label is divided by the maximum degree. If the maximum degree is 0, the value is 1

[0136]

[0137] Here, samenode is the degree of nodes with the same label, that is, the number of nodes with the same label that are connected to each node. For example, if the labels of the nodes connected to node C include 1 text label and 1 control label, and the labels of the nodes connected to node T include 1 control label, then when calculating the node similarity between nodes C and T, samenode is 1.

[0138] The embodiment of the present application proposes a method for abstracting the layout of component elements into an undirected graph as a feature. When the amount of image information is large, there is a lot of invalid information, and the algorithm processes a large amount of data, the image information is simplified, the amount of data processing is reduced, and the processing efficiency is improved.

[0139] In some embodiments, the server may further optimize the above-mentioned preset weight parameters w1, w2, w3, and w4 through an optimization algorithm. In the optimization algorithm:

[0140] 1. Define the input as: component feature undirected graph G c =(V c ,E c );

[0141] 2. Define output:

[0142] Calculate the similarity between the component and each template:

[0143]

[0144] The template number with the highest similarity is the recommended template:

[0145] j=getIndex(max(S c1 ,S c1 ,…S cm ))

[0146] Matching result vector: (assuming there are m templates, and the fifth template has the highest similarity, then the fifth element is 1, and the others are 0)

[0147]

[0148] r is the output.

[0149] 3. Define the loss function:

[0150] Using cross entropy loss function:

[0151]

[0152] Among them, m is the number of template categories, r ij is the element value of the matching result vector of sample i and subscript j, S ij is the similarity between sample i and the jth template.

[0153] 4. Use gradient descent or other methods to optimize parameters.

[0154] After the server calculates the similarity between the component and each candidate template, it can determine the target template from the candidate templates based on the similarity between the component and the candidate template. For example, if the similarity between component 1 and candidate template 1 is 1, and the similarity between component 1 and candidate template 2 is 99, component 1 can select candidate template 2 with the greatest similarity as the candidate template corresponding to component 1. In actual applications, a component can match one or more candidate templates as the target template according to actual needs, and this embodiment of the application does not limit this.

[0155] After the server determines the target template, it can also combine the determined target templates to form a combined template and push it to the user. For example, the server obtains component 1, component 2, and component 3 in the user's input image through the above steps, and determines target template 1 based on component 1, determines target template 2 based on component 2, and determines target template 3 based on component 3. The server can then combine target template 1, target template 2, and target template 3 to form a combined template and push it together to the user.

[0156] The overall solution of the embodiment of the present application adopts the method of uploading pictures to recommend combination templates. Users only need to upload their favorite card layout pictures, and the server can automatically match the appropriate combination template based on the pictures, avoiding repeated designs and difficult card design problems, and improving development efficiency.

[0157] Figure 14This is a schematic diagram of the overall solution of an embodiment of the present application. The user first uploads an image corresponding to the desired rendering effect to the server. The server then preprocesses the image, identifies the elements in the image, aggregates the elements into at least one component, and performs error correction. Finally, the server abstracts the component features to obtain an undirected graph corresponding to the component. This undirected graph is then matched against the undirected graph of the candidate template for similarity, and the matched template is pushed to the user.

[0158] Figure 15 Schematic diagram of a device for determining a card template in an embodiment of the present application. The device 1500 includes:

[0159] Acquisition module 1501, used to execute the above Figure 2 Corresponding to step 201 in each embodiment;

[0160] Processing module 1502 is used to execute the above Figure 2 These correspond to steps 202, 203, and 204 in the respective embodiments.

[0161] Figure 16 A schematic diagram of a server structure provided for an embodiment of the present application, wherein the server 1600 is a device determined by a card template. The server 1600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1601 (for example, one or more processors) and a memory 1602, and one or more storage media 1603 (for example, one or more massive storage devices) for storing application programs 1633 or data 1632. Among them, the memory 1602 and the storage medium 1603 may be temporary storage or permanent storage. The program stored in the storage medium 1603 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1601 may be configured to communicate with the storage medium 1603 to execute a series of instruction operations in the storage medium 1603 on the server 1600.

[0162] The server 1600 may further include one or more power supplies 1604, one or more wired or wireless network interfaces 1605, one or more input and output interfaces 1606, and / or one or more operating systems 1631, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0163] The steps performed by the server in the above embodiment can be based on the Figure 16 Specifically, CPU 1601 can execute the above Figure 2 In some embodiments, the modules in CPU 1601 may include the following: Figure 15 The acquisition module 1501 and the processing module 1502 in the corresponding embodiment will not be described in detail in the embodiment of the present application.

[0164] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A method for determining a card template, characterized in that: include: Obtain an input image, wherein the input image includes a card layout image; Identifying and classifying elements in the input image; Aggregating the elements to obtain at least one component, wherein the component is a combination of at least one of the elements; aggregating the elements to obtain at least one component comprises: calculating offsets between the elements; clustering the elements according to the offsets between the elements using a clustering algorithm to obtain at least one component; the elements in the component are adjacent to each other; A target template is determined from the candidate templates according to the similarity between the component and the candidate templates.

2. The method according to claim 1, characterized in that After acquiring the input image and before identifying and classifying elements in the input image, the method further includes: The input image is grayscaled, Gaussian low-pass filtered, and binarized to obtain the binarized input image.

3. The method according to claim 1, characterized in that The identifying and classifying elements in the input image includes: Performing an erosion process on the input image so that the text area in the input image is processed into a connected area; Draw the bounding rectangle of each area in the input image using a contour detection algorithm; The element in the input image is determined according to the position of the circumscribed rectangle.

4. The method according to claim 3, characterized in that The identifying and classifying elements in the input image further includes: If there is text in the circumscribed rectangle and the area of ​​the text is equal to the area of ​​the circumscribed rectangle, determining that the element corresponding to the circumscribed rectangle is a text element; If there is no text in the bounding rectangle and the area of ​​the bounding rectangle is smaller than a preset threshold, determining that the element corresponding to the bounding rectangle is a control element; If there is text in the circumscribed rectangle and the area of ​​the text is not equal to the area of ​​the circumscribed rectangle, or if there is no text in the circumscribed rectangle and the area of ​​the circumscribed rectangle is not less than a preset threshold, it is determined that the element corresponding to the circumscribed rectangle is a picture element.

5. The method according to claim 4, characterized in that After determining that the element corresponding to the circumscribed rectangle is a text element, the method further includes: Identifying the meaning of text in the text elements by using a natural language understanding semantic recognition algorithm; The specific category of the text element is determined according to the meaning of the text.

6. The method according to claim 1, characterized in that Calculating the offset between the elements includes: If the first element and the second element intersect in the horizontal direction, the vertical distance between the first element and the second element is calculated as the element offset; If the first element and the second element intersect in the vertical direction, the horizontal distance between the first element and the second element is calculated as the element offset; If the first element and the second element do not intersect, the vertical spacing and the horizontal spacing between the first element and the second element are calculated respectively, and the minimum value is selected as the element offset.

7. The method according to claim 1, characterized in that After aggregating the elements to obtain at least one component and before determining a target template from the candidate templates based on similarities between the component and the candidate templates, the method further includes: Obtaining semantic information corresponding to the element through a natural language understanding semantic recognition algorithm or a computer vision recognition algorithm; Elements in the component are adjusted according to the semantic information.

8. The method according to claim 1, characterized in that Determining a target template from the candidate templates according to the similarity between the component and the candidate templates includes: Generate an undirected graph corresponding to the component according to the elements in the component; Calculating the similarity between the component and the candidate template based on the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template; The candidate template corresponding to the highest similarity is determined as the target template corresponding to the component.

9. The method according to claim 8, characterized in that Generating an undirected graph corresponding to the component according to the elements in the component includes: Taking one of the elements in the component as an initial node, searching for elements in the component in vertical and horizontal directions and connecting the searched elements to the initial node in sequence with edges; In the order in which the elements are searched, with the searched elements as subsequent nodes, the elements are searched in the vertical and horizontal directions in the component and the searched elements are sequentially connected with edges.

10. The method according to claim 9, characterized in that Generating an undirected graph corresponding to the component according to the elements in the component includes: The nodes in the undirected graph are labeled with labels corresponding to the categories of the elements.

11. The method according to claim 8, characterized in that Calculating the similarity between the component and the candidate template according to the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template includes: Calculating similarities between nodes in an undirected graph corresponding to the component and an undirected graph corresponding to the candidate template; The similarity between the component and the candidate template is calculated according to the similarity between the nodes.

12. The method according to claim 11, characterized in that The similarity between the nodes in the undirected graph corresponding to the component and the undirected graph corresponding to the candidate template is calculated using a node similarity calculation formula. The node similarity calculation formula is: sim(v i ,u j )=sigmoid(h); h=w1s t +w2s d +w3s l +w4s c ; Among them, v i is the node in the undirected graph corresponding to the component, u j is the node in the undirected graph corresponding to the candidate template, w1, w2, w3, w4 are the preset weight parameters, s t is the node label similarity, s d is the node degree similarity, s l is the node loop similarity, s c is the link node similarity.

13. The method according to claim 12, characterized in that The similarity between the component and the candidate template is calculated based on the similarity between the nodes using a graph similarity calculation formula. The graph similarity calculation formula is: Among them, SIM(G c ,G t ) is component G c With the candidate template G t The similarity function between i For component G c Node, φ(v i ) is the candidate template G t and node v in i The corresponding nodes do not overlap with other nodes.

14. A device for determining a card template, characterized in that: include: An acquisition module, configured to acquire an input image; the input image includes a card layout image; a processing module, configured to identify and classify elements in the input image; The processing module is further configured to aggregate the elements to obtain at least one component, wherein the component is a combination of at least one of the elements; the processing module is specifically configured to: calculate the offsets between the elements; cluster the elements using a clustering algorithm based on the offsets between the elements to obtain at least one component; the elements in the component are adjacent to each other; The processing module is further configured to determine a target template from the candidate templates based on the similarity between the component and the candidate templates.

15. A device for determining a card template, characterized in that: include: A processor and a transmission interface, wherein the processor is configured to call program instructions stored in a memory to execute the method according to any one of claims 1 to 13. 16 . A computer-readable storage medium comprising instructions, which, when executed on a computer or a processor, cause the computer or the processor to perform the method according to claim 1 .

17. A computer program product comprising instructions, the computer program product comprising program instructions, which, when the program instructions are run on a computer or a processor, cause the computer or the processor to perform the method according to any one of claims 1 to 13.

Images