Image element matching method and device, model training method and device, and data processing method

Abstract

The present application discloses a method and device for matching image elements, a model training method and device, and a data processing method, which converts the problem of finding image information that matches a given image element into the problem of predicting from a matching knowledge graph whether there are edges between nodes corresponding to the given image element and other nodes in the matching knowledge graph. This method flexibly realizes the matching of image elements, improves the matching effect, and reduces labor costs.

Description

Technical Field

[0001] The present application relates to but is not limited to industrial intelligent technology, and in particular to an image element matching method and device, a model training method and device, and a data processing method. Background Art

[0002] In related technologies, the problem of matching image elements is solved through pre-defined templates or rules. This approach has poor flexibility and diversity, and is also costly for designers.

[0003] In addition, a pre-designed graph autoencoder is also used to solve the image element matching problem. However, this method is difficult to train for large-scale graphs, and the visual features of the graph nodes are pre-calculated. Therefore, the matching results are not ideal. Summary of the Invention

[0004] The present application provides an image element matching method and device, a model training method and device, and a data processing method, which can be flexibly adapted to large-scale image training and improve matching effects.

[0005] An embodiment of the present invention provides a data processing method, including:

[0006] Receive images to be processed;

[0007] Obtaining at least one original image element of the image to be processed;

[0008] Based on a knowledge graph, obtaining target image elements corresponding to the original image elements, wherein the knowledge graph uses image elements as nodes and relationships between image elements as edges;

[0009] The target image element is fed back.

[0010] The embodiment of the present invention further provides a method for matching image elements, including:

[0011] Add the image elements to be processed as nodes to be processed into the collocation knowledge graph;

[0012] Extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph;

[0013] According to the adjacent node information and feature information, the nodes in the collocation knowledge graph are encoded to obtain the new feature information of the nodes;

[0014] Determine node pairs with edges based on the new feature information of the nodes, where the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph;

[0015] Determine that the image element corresponding to the node having an edge with the node to be processed is matched with the image element to be processed.

[0016] In an exemplary embodiment, the method further comprises:

[0017] The collocation knowledge graph is constructed based on the image elements of the sample images, wherein the image elements of the sample images serve as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

[0018] In an exemplary embodiment, the image elements of the sample image include:

[0019] Using massive banner PSD image information as material, parse each image file PSD file to obtain several layers included in the PSD file, and each layer is regarded as an independent image element;

[0020] The step of constructing a collocation knowledge graph based on image elements of the sample image includes:

[0021] The independent image elements are used as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

[0022] In an exemplary embodiment, the feature information includes an adjacency matrix; and extracting the feature information of nodes in the collocation knowledge graph includes:

[0023] Based on a convolutional neural network, a visual feature matrix of the node is extracted according to the adjacency matrix.

[0024] In an exemplary embodiment, encoding the nodes in the collocation knowledge graph according to the adjacent node information and feature information to obtain new feature information of the nodes includes:

[0025] The adjacency matrix and the feature information are input into a multi-layer graph convolutional neural network, and the nodes in the collocation knowledge graph are respectively encoded to obtain new feature information of the nodes in the collocation knowledge graph.

[0026] In an exemplary embodiment, determining a node pair having an edge includes:

[0027] It is determined whether there is an edge between the node pairs by calculating the edge probability of the node pairs.

[0028] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-mentioned image element matching methods.

[0029] The present application further provides a device for implementing image element matching, comprising a memory and a processor, wherein the memory stores the following instructions executable by the processor: for executing the steps of any of the above-mentioned image element matching methods.

[0030] This application further provides a method for matching image elements, including:

[0031] Add the image elements to be processed as nodes to be processed into the collocation knowledge graph;

[0032] Input the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph into the pre-trained collocation model for processing;

[0033] If there is an edge between the output node pairs of the collocation model, the image element corresponding to the node that has an edge with the node to be processed is determined to be collocated with the image element to be processed.

[0034] In an exemplary embodiment, inputting the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph into a pre-trained collocation model for processing includes:

[0035] Extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing collocation relationships of nodes in the collocation knowledge graph;

[0036] According to the adjacent node information and feature information, the nodes in the collocation knowledge graph are respectively encoded to obtain new feature information of the nodes;

[0037] Determine whether there is an edge between node pairs based on the new feature information of the nodes, wherein the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph.

[0038] In an exemplary embodiment, the method further includes training the collocation model.

[0039] In an exemplary embodiment, the collocation model is a FedNet model.

[0040] The present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-mentioned image element matching methods.

[0041] The present application further provides a device for implementing image element matching, comprising a memory and a processor, wherein the memory stores the following instructions executable by the processor: for executing the steps of any of the above-mentioned image element matching methods.

[0042] This application also provides a model training method, including:

[0043] Construct a collocation knowledge graph based on the image elements of the sample images, where the image elements serve as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination;

[0044] Divide the collocation knowledge graph into collocation combination number subgraphs, where the collocation combination number is the number of collocation combinations included in the collocation knowledge graph;

[0045] Obtain the first adjacency matrix of each subgraph, input the obtained adjacency matrix into a preset collocation model for end-to-end training, and obtain the second adjacency matrix of the subgraph;

[0046] The first adjacency matrix and the second adjacency matrix are compared, and parameters of the collocation model are adjusted based on the comparison results.

[0047] In an exemplary embodiment, the image elements of the sample image include:

[0048] Using massive banner PSD image information as material, parse each image file PSD file to obtain several layers included in the PSD file, and each layer is regarded as an independent image element;

[0049] The step of constructing a collocation knowledge graph based on image elements of the sample image includes:

[0050] The independent image elements are used as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

[0051] In an exemplary embodiment, dividing the collocation knowledge graph into subgraphs equal to the number of collocation combinations includes:

[0052] In the collocation knowledge graph, the subgraph is formed by taking the k-order neighbor nodes of all nodes in each collocation combination, and the collocation knowledge graph is divided into subgraphs as many as the collocation combinations, and each subgraph includes a collocation combination and the k-order neighbor nodes of the collocation combination node.

[0053] In an exemplary embodiment, inputting the obtained adjacency matrix into a preset collocation model for end-to-end training to obtain a second adjacency matrix of the subgraph includes:

[0054] Extracting feature information of the nodes in the subgraph according to the first adjacency matrix representing the collocation relationship of the nodes in the subgraph;

[0055] Encoding the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes;

[0056] Determine whether there is an edge between node pairs according to the new feature information of the nodes, wherein the node pairs are node pairs in a subgraph, and decode a second adjacency matrix of the collocation relationship of the nodes in the subgraph.

[0057] In an exemplary embodiment, obtaining a first adjacency matrix of each subgraph includes:

[0058] Construct a positive and negative edge set for each subgraph, take the edges connected by all the nodes in the subgraph as the positive edge set, and randomly take the edges consisting of the same number of unconnected node pairs as the positive edge set as the negative edge set. i and node v j When there is an edge connection, the first adjacency matrix A i,j =1, otherwise the first adjacency matrix A i,j =0.

[0059] In an exemplary embodiment, the feature information includes visual features; and extracting the feature information of nodes in the subgraph includes:

[0060] Based on a convolutional neural network, a visual feature matrix of the node is extracted according to the first adjacency matrix.

[0061] In an exemplary embodiment, encoding the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes includes:

[0062] The first adjacency matrix and feature information are input into a multi-layer graph convolutional neural network, and the nodes in the subgraph are respectively encoded to obtain new feature information of the nodes in the subgraph.

[0063] In an exemplary embodiment, determining whether an edge exists between a pair of nodes includes:

[0064] Whether there is an edge between the node pairs is determined by calculating the edge probability of the node pairs.

[0065] In an exemplary embodiment, comparing the first adjacency matrix and the second adjacency matrix and adjusting the parameters of the collocation model according to the comparison result includes:

[0066] Use the loss function to calculate the loss for each sample separately;

[0067] The average loss of all samples is used as the target loss;

[0068] Updating the parameters of the collocation model using a back propagation algorithm according to the target loss;

[0069] Among them, the samples include: positive sample p t =p i,j, negative sample p t =1-p i,j , where p i,j For node pairs (v i ,v j ) of the edge probability.

[0070] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the model training methods described above.

[0071] The present application also provides a device for implementing model training, including a memory and a processor, wherein the memory stores the following instructions that can be executed by the processor: used to execute the steps of any of the model training methods described above.

[0072] The image element matching method provided in the embodiment of the present application converts the problem of finding image information that matches a given image element into the problem of predicting from the matching knowledge graph whether there are edges between the nodes corresponding to the given image element and other nodes in the matching knowledge graph. It flexibly realizes the matching of image elements, improves the matching effect, and reduces labor costs.

[0073] The model training method provided in the embodiment of the present application is suitable for large-scale graph training. On the one hand, the image elements in the matching knowledge graph are divided into multiple sub-graphs for subsequent matching model training, which solves the problem of difficulty in large-scale graph training; on the other hand, the feature information of the node, such as the visual features, is not pre-calculated, but is continuously trained during the model training, which ensures that the trained matching model has a more ideal matching effect.

[0074] Furthermore, in the model training method of this application, the matching model adopts the FedNet model, which enables end-to-end training on massive data, so that the obtained matching model achieves better results.

[0075] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0077] Figure 1 This is a flowchart of the image element matching method of this application;

[0078] FIG2( a ) is a schematic diagram of the first embodiment of constructing a collocation knowledge graph in this application;

[0079] FIG2( b ) is a schematic diagram of a second embodiment of constructing a collocation knowledge graph in the present application;

[0080] FIG2( c ) is a schematic diagram of a third embodiment of the present application for constructing a collocation knowledge graph;

[0081] Figure 3 A schematic diagram of a fourth embodiment of constructing a collocation knowledge graph for this application;

[0082] Figure 4 This is a flowchart of another method for matching image elements in this application;

[0083] Figure 5 A schematic diagram of an application example of the combination of image elements of this application;

[0084] Figure 6 This is a schematic diagram of the structure of the image element matching device of the present application;

[0085] Figure 7 A flowchart of the model training method for this application;

[0086] Figure 8 A schematic diagram of the network structure of the network used for this application;

[0087] Figure 9 This is a schematic diagram of the structure of the model training device of this application. DETAILED DESCRIPTION

[0088] To make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any way.

[0089] In a typical configuration of the present application, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0090] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0091] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0092] The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. Also, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0093] Figure 1 This is a flow chart of the image element matching method of this application, such as Figure 1 Shown, including:

[0094] Step 100: Add the image element to be processed as a node to be processed to the collocation knowledge graph.

[0095] In an exemplary embodiment, the collocation knowledge graph is pre-constructed based on image elements of sample images and may include:

[0096] Using the PSD image information of the advertising design (banner) (e.g., including several image files, i.e., PSD files) as the material, each PSD file is parsed to obtain the several layers included in the PSD file, with each layer as an independent image element. It can be assumed that any two image elements in the same PSD file are coordinated with each other in design, that is, the image elements in each PSD file form a coordinated combination.

[0097] Independent image elements are used as nodes in the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination (that is, edges exist between nodes that coordinate with each other). In this way, using banner PSD image information as material, a large collocation knowledge graph can be constructed, which contains the design knowledge used by designers in the process of creating banners.

[0098] In an exemplary embodiment, parsing each PSD file can be achieved through a related PSD parsing tool. The specific implementation is not intended to limit the scope of protection of this application and will not be described in detail here.

[0099] In an exemplary embodiment, each layer has an identity (ID), and layers with the same ID have the same image elements and are the same node in the collocation knowledge graph.

[0100] Banners are rectangular billboards that span a webpage. When users click on them, they are usually linked to the advertiser's website. Banner PSD images are saved as PSD files. Banner ads are the earliest form of online advertising and are still the most common. Banner ads are also called banner ads.

[0101] In an exemplary embodiment, it is assumed that the collocation knowledge graph G has N nodes and M edges, V = {v1, v2, ..., v N} represents a set of N nodes, E={(v1,v2),(v3,v4),...,(v k ,v m )} represents a set of M edges, (v i ,v j ) represents node v i and node v j There is an edge connection. G=(V,E), if the N×N adjacency matrix A is used to represent the collocation knowledge graph G, then when the node v i and node v j When there is an edge connection, A i,j =1, otherwise A i,j = 0. X represents the node feature matrix, each node contains F-dimensional features, node v i Features of X i ={x i,1 ,x i,2 ,...,x i,F}, X is a matrix of N×F. In an exemplary embodiment, the node features include visual features.

[0102] For example, suppose a psd1 file is parsed and the five layers included in the psd1 file are obtained. Each layer is an independent image element, such as Figure 2(a) to Figure 2(c)As shown, they correspond to node 1, node 2, node 3, node 4 and node 5 respectively. Any two image elements in the same psd1 file are designed to coordinate with each other, that is, node 1, node 2, node 3, node 4 and node 5 form a combination in pairs, as shown in Figure 2(a), which is the edge of node 1 constructed; as shown in Figure 2(b), which is the edge of node 1 and node 2 constructed; as shown in Figure 2(c), which is the edge of node 1, node 2, node 3, node 4 and node 5 constructed. To take another example, based on the collocation knowledge graph shown in Figure 2(c), it is assumed that a psd2 file is continued to be parsed to obtain the 4 layers included in the psd2 file, and each layer is an independent image element, such as Figure 3 As shown, they correspond to nodes 2, 6, 7, and 8 respectively. It can be seen that one of the layers included in the psd2 file has the same image element as the layer corresponding to node 2 in the psd1 file, so they are both reflected as nodes 2 with the same ID in the collocation knowledge graph. Any two image elements in the same psd2 file are designed to be coordinated with each other. In other words, nodes 2, 6, 7, and 8 form a collocation combination, and the constructed edges are as follows: Figure 3 shown.

[0103] Step 100 adds the image element to be processed as a node into the pre-built collocation knowledge graph. In this way, the next step is to convert the problem of finding image information that matches the given image element from the element library into the problem of predicting from the collocation knowledge graph whether there is an edge between the node corresponding to the given image element and other nodes in the collocation knowledge graph.

[0104] Step 101: Extract feature information of nodes in the collocation knowledge graph based on adjacent node information that represents the collocation relationship of nodes in the collocation knowledge graph.

[0105] In an exemplary embodiment, the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph can be represented by an adjacency matrix.

[0106] In an exemplary embodiment, it is assumed that the collocation knowledge graph G has N nodes and M edges, V = {v1, v2, ..., v N} represents a set of N nodes, E={(v1,v2),(v3,v4),...,(v k ,v m )} represents a set of M edges, (v i ,v j ) represents node v i and node v jThere is an edge connection. G=(V,E), if the N×N adjacency matrix A is used to represent the collocation knowledge graph G, then when the node v i and node v j When there is an edge connection, A i,j =1, otherwise A i,j = 0. X represents the node feature matrix, each node contains F-dimensional features, node v i Features of X i ={x i,1 ,x i,2 ,...,x i,F}, X is a matrix of N×F. In an exemplary embodiment, the node features include visual features.

[0107] In an exemplary embodiment, extracting feature information of nodes in the collocation knowledge graph in this step may include:

[0108] Extract the visual feature matrix X of the node based on the convolutional neural network. The convolutional neural network may include but is not limited to: widened network structure (VGGNet, Inception Network), residual network (ResNet, Residual Network), dense convolution network (DenseNet), etc. Among them, VGGNet was proposed by the Visual Geometry Group (VGG) of the University of Oxford.

[0109] Preferably, in an exemplary embodiment, ResNet can be used to extract the visual features X of each node i image in the knowledge graph. i , the visual features of all nodes constitute the feature matrix X of the knowledge graph.

[0110] Step 102: Based on the adjacent node information and feature information, encode the nodes in the collocation knowledge graph to obtain new feature information of the nodes.

[0111] In an exemplary embodiment, this step may include:

[0112] The adjacency matrix and feature information are input into a multi-layer graph convolutional neural network, and the nodes in the collocation knowledge graph are encoded separately to obtain new feature information of the nodes in the collocation knowledge graph.

[0113] In this step, the input information of the multi-layer graph convolutional neural network includes any node v i The feature information and adjacency matrix of , therefore, for any node v i , the new feature information z i It not only encodes the information of the node itself, but also encodes the information of other nodes that work with the node.

[0114] In one exemplary embodiment, for node v i and the node v i Neighbor node N i , N i ={v j |A i , j =1}, z i =f encoder (X i ,N i ), where f encoder This is achieved through a multi-layer graph convolutional neural network (GCN). The implementation of a certain layer of GCN can be shown as formula (1):

[0115]

[0116] In formula (1), represents the i-th node in the l-th layer, yes The output value of , ReLU is the activation layer, and the output result after all nodes in the knowledge graph are processed by GCN shown in formula (1) is shown in formula (2):

[0117]

[0118] In formula (2), represents the Laplacian matrix of the adjacency matrix A, Where I represents the identity matrix and D represents the degree matrix of the vertex s represents the s-order neighbor node of the node, W (l) Represents the learning parameters of the lth layer. In the first learning, W (l) Can be initialized using kaiming_uniform.

[0119] Step 103: Determine node pairs with edges based on the new feature information of the node, where the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph.

[0120] In an exemplary embodiment, this step may include: determining whether there is an edge between the node pairs by calculating the edge probability of the node pairs, i.e., the collocation score. In an exemplary embodiment, the edge probability p of the node pair (i.e., node i and node j) may be calculated as shown in formula (3): i,j :

[0121] p i，j =h(z i -z j )w+b (3)

[0122] In formula (3), z i 、z j is node v i , node v j The new feature information is obtained after the encoding (Encoder) in step 102; w and b are learning parameters, and h(.) represents the sigmoid activation function.

[0123] Step 104: Determine whether image elements corresponding to nodes that have edges with the node to be processed are matched with the image element to be processed.

[0124] The image element matching method provided in this application converts the problem of finding image information that matches a given image element from an element library into the problem of predicting from a matching knowledge graph whether there are edges between nodes corresponding to a given image element and other nodes in the matching knowledge graph. This method flexibly realizes the matching of image elements, improves the matching effect, and reduces labor costs.

[0125] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing Figure 1 A method for matching image elements of any of the items shown.

[0126] The present application further provides a device for implementing image element matching, comprising a memory and a processor, wherein the memory stores the following instructions that can be executed by the processor: Figure 1 The steps of any one of the image element matching methods shown.

[0127] Figure 4 A flowchart of another method for matching image elements according to the present application, including:

[0128] Step 400: Add the image element to be processed as a node to be processed to the collocation knowledge graph.

[0129] Step 401: input the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph into a pre-trained collocation model for processing, and the collocation model outputs whether there is an edge between the node pairs.

[0130] In an exemplary embodiment, the processing in this step includes:

[0131] Extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph;

[0132] According to the adjacent node information and feature information, the nodes in the collocation knowledge graph are encoded to obtain the new feature information of the nodes;

[0133] Determine whether there is an edge between node pairs based on the new feature information of the nodes, where the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph.

[0134] Step 402: Determine whether image elements corresponding to nodes that have edges with the node to be processed are matched with the image element to be processed.

[0135] Figure 4 The image elements shown are matched with Figure 1 The differences shown are: Figure 1 The implementation of steps 101 to 103 in Figure 4 The image element matching method shown is implemented using a pre-trained matching model. The training method of the matching model is as follows: Figure 6 shown.

[0136] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing Figure 4 A method for matching image elements of any of the items shown.

[0137] The present application further provides a device for implementing image element matching, comprising a memory and a processor, wherein the memory stores the following instructions that can be executed by the processor: Figure 4 The steps of any one of the image element matching methods shown.

[0138] In an exemplary embodiment, the present application also provides a data processing method, including: receiving an image to be processed; obtaining at least one original image element of the image to be processed; obtaining a target image element corresponding to the original image element based on a knowledge graph, wherein the knowledge graph uses image elements as nodes and the relationships between image elements as edges; and feeding back the target image element. Here, taking a user uploading a product main image as an example, Figure 5 As shown, the image element matching method provided in the embodiment of the present application automatically matches the required image information from the knowledge graph based on the main image of the product, such as one or more images that match the main image of the product uploaded by the user. In this way, the matched image can be recommended to the user for use, or multiple matched images can be displayed to the user for selection.

[0139] In one illustrative example, Figure 5 The image element to be processed can also be the image element itself. Through the data processing method described in this application, a set of image elements will be automatically matched from the knowledge graph based on the image element to be processed. In this way, a set of matched image elements can be recommended as a whole.

[0140] Figure 6This is a schematic diagram of the structure of the image element matching device of this application, such as Figure 6 As shown, it includes a pre-processing module, a processing module and a determination module; wherein,

[0141] The preprocessing module is used to add the image elements to be processed as nodes to be processed into the collocation knowledge graph.

[0142] a processing module for extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing collocation relationships of nodes in the collocation knowledge graph; encoding the nodes in the collocation knowledge graph based on the adjacent node information and the feature information to obtain new feature information of the nodes; and determining node pairs with edges based on the new feature information of the nodes, wherein the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph;

[0143] The determination module is used to determine whether the image element corresponding to the node having an edge with the node to be processed is matched with the image element to be processed.

[0144] In an exemplary embodiment, the processing module can be a pre-trained collocation model, the input of the collocation model is the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph, and the output of the collocation model is whether there is an edge between the node pairs.

[0145] Figure 7 This is a flowchart of the model training method for this application, such as Figure 7 Shown, including:

[0146] Step 700: Construct a collocation knowledge graph based on the image elements of the sample image, wherein the image elements serve as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

[0147] In an exemplary embodiment, the sample image can be based on a large amount of banner PSD image information (e.g., including several image files, i.e., PSD files). Each PSD file is parsed to obtain several layers included in the PSD file, with each layer being an independent image element. It can be assumed that any two image elements in the same PSD file are designed to coordinate with each other, that is, each pair of image elements in each PSD file constitutes a coordinated combination.

[0148] Independent image elements are used as nodes in the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination (that is, edges exist between nodes that coordinate with each other). In this way, using massive amounts of banner PSD image information as material, a massive collocation knowledge graph can be constructed, which contains the design knowledge used by designers in the banner creation process.

[0149] Step 701: Divide the collocation knowledge graph into subgraphs equal to the number of collocation combinations, where the number of collocation combinations is the number of collocation combinations included in the collocation knowledge graph.

[0150] In an exemplary embodiment, assuming that the collocation knowledge graph G includes Q collocation combinations, this step divides the collocation knowledge graph G into subgraphs equal to the number of collocation combinations, including:

[0151] In the collocation knowledge graph G, the k-order neighbor nodes of all nodes in each collocation combination are taken to form a subgraph, and the collocation knowledge graph G is divided into Q subgraphs. Each subgraph G q It includes a collocation combination and its k-order neighboring nodes. K is a hyperparameter in the training process, and different training results can be obtained by setting different k values.

[0152] In this step, the image elements in the collocation knowledge graph G are divided into multiple sub-graphs for subsequent collocation model training, which solves the problem of difficulty in large-scale graph training.

[0153] Step 702: Obtain a first adjacency matrix for each subgraph, input the obtained adjacency matrix into a preset collocation model for end-to-end training, and obtain a second adjacency matrix for the subgraph.

[0154] In an exemplary embodiment, in this step, each subgraph G is obtained q The first adjacency matrix A q ,include:

[0155] For each subgraph G q Construct a set of positive and negative edges and take subgraph G q In the combination, all the edges connecting two nodes are taken as the positive edge set, and the edges consisting of the same number of unconnected node pairs as the positive edge set are randomly selected as the negative edge set. q Node v in i and node v j When there is an edge connection, A i,j =1, otherwise A i,j =0.

[0156] In one exemplary embodiment, for each subgraph, Figure 8 In step 702, inputting the obtained adjacency matrix into a preset collocation model for end-to-end training to obtain a second adjacency matrix of the subgraph may include:

[0157] Extracting feature information of nodes in the subgraph according to a first adjacency matrix representing collocation relationships of nodes in the subgraph;

[0158] According to the first adjacency matrix and the feature information, the nodes in the subgraph are respectively encoded to obtain new feature information of the nodes;

[0159] Determine whether there is an edge between node pairs based on the new feature information of the nodes, where the node pairs are node pairs in the subgraph, and decode (Decoder) the second adjacency matrix of the collocation relationship of the nodes in the subgraph. As shown in formula (3), p i,j It is the value of the i, j position in the second adjacency matrix. In other words, the second adjacency matrix is composed of the predicted collocation scores of all node pairs.

[0160] In this application, the feature information of the nodes, such as visual features, is not pre-calculated, but is continuously trained during model training, which ensures that the trained matching model has a more ideal matching effect.

[0161] In an exemplary embodiment, extracting feature information of nodes in the subgraph in this step may include:

[0162] Extract the visual feature matrix X of the node based on the convolutional neural network. The convolutional neural network may include but is not limited to: widened network structure (VGGNet, Inception Network), residual network (ResNet, Residual Network), dense convolution network (DenseNet), etc. Among them, VGGNet was proposed by the Visual Geometry Group (VGG) of the University of Oxford.

[0163] Preferably, in an exemplary embodiment, ResNet can be used to extract the visual features X of each node i image in the knowledge graph. i , the visual features of all nodes constitute the feature matrix X of the knowledge graph.

[0164] In an exemplary embodiment, encoding the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes in this step may include:

[0165] The first adjacency matrix and feature information are input into a multi-layer graph convolutional neural network, and the nodes in the subgraph are encoded to obtain new feature information of the nodes in the subgraph. The specific implementation can be found in the description of formula (1) and formula (2) above, and will not be repeated here.

[0166] In this step, the input information of the multi-layer graph convolutional neural network includes any node v i The feature information and the first adjacency matrix, therefore, for any node v i , the new feature information z iIt not only encodes the information of the node itself, but also encodes the information of other nodes that work with the node.

[0167] In one exemplary embodiment, determining the node pairs that have edges in this step may include determining whether an edge exists between the node pairs by calculating the edge probability, i.e., the collocation score, of the node pairs. The specific implementation can be found in the description of formula (3) and step 103 above, and will not be repeated here.

[0168] In an exemplary embodiment, the collocation model includes a feature extraction part (Featurer), an encoding part (Encoder) and a decoding part (Decoder), which can be referred to as a FedNet model in this article.

[0169] The FedNet model can be trained end-to-end on massive amounts of data, and the resulting matching model can achieve better results.

[0170] Step 703: Compare the first adjacency matrix and the second adjacency matrix, and adjust the parameters of the collocation model according to the comparison result.

[0171] In an exemplary embodiment, this step may include:

[0172] Using Focal Loss function FL(p t )=-α t (1-p t ) β log(p t ), where α t and β are the hyperparameters of the new joint model, and can be taken as follows in the experiment: t =0.5, β=1, calculate the loss for each sample i =FL(p t );

[0173] The average loss of all samples is taken as the target loss;

[0174] According to the target loss, the back propagation algorithm is used to update the parameters of the matching model.

[0175] Among them, the samples include positive samples and negative samples. Take all A in the first adjacency matrix i,j = 1 for the node pair (vi,v j ) constitutes the edge e i,j As a positive sample, from A i,j =0 randomly samples a certain proportion as negative samples; the second adjacency matrix is composed of all node pair prediction collocation scores, node pair (v i ,v j) constitutes the edge e i,j The predicted score p i,j =ā i,j , so the true value for all positive samples is 1, and the true value for all negative samples is 0. Assuming the mathematical symbol p t , then, the positive sample p t =p i,j , negative sample p t =1-p i,j .

[0176] The model training method provided in this application is suitable for large-scale graph training. On the one hand, the image elements in the collocation knowledge graph are divided into multiple subgraphs for subsequent collocation model training, which solves the problem of large-scale graph training difficulties. On the other hand, the feature information of the nodes, such as visual features, is not pre-calculated, but is continuously trained during model training, which ensures that the trained collocation model has a more ideal collocation effect. Furthermore, in the model training method of this application, the collocation model adopts the FedNet model, which realizes end-to-end training on massive data, so that the resulting collocation model achieves better results.

[0177] It should be noted that Figure 4 The image element matching method shown is itself also a training process for the matching model. That is to say, in addition to the preliminary training using application samples, the matching model in this application is also dynamically trained in real time during the use of the matching model, further ensuring the practicality of the matching model.

[0178] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-mentioned model training methods.

[0179] The present application further provides a device for implementing matching model training, including a memory and a processor, wherein the memory stores the following instructions that can be executed by the processor: used to execute the steps of the model training method described in any of the above items.

[0180] Figure 9 This is a schematic diagram of the structure of the model training device of this application, such as Figure 9 As shown, it includes: a construction unit, a division unit, a training unit, and an adjustment unit; wherein,

[0181] A construction unit, configured to construct a collocation knowledge graph based on the image elements of the sample image, wherein the image elements serve as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination;

[0182] A division unit, configured to divide the collocation knowledge graph into subgraphs having the number of collocation combinations, where the number of collocation combinations is the number of collocation combinations included in the collocation knowledge graph;

[0183] A training unit is configured to obtain a first adjacency matrix of each subgraph, input the obtained adjacency matrix into a preset collocation model for end-to-end training, and obtain a second adjacency matrix of the subgraph;

[0184] The adjustment unit is used to compare the first adjacency matrix and the second adjacency matrix, and adjust the parameters of the collocation model according to the comparison result.

[0185] In an exemplary embodiment, the training unit may include: an extraction module, an encoding module and a decoding module, wherein:

[0186] An extraction module, configured to extract feature information of nodes in the subgraph according to a first adjacency matrix representing collocation relationships of nodes in the subgraph;

[0187] An encoding module, configured to encode the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes;

[0188] A decoding module is used to determine whether there is an edge between node pairs according to the new feature information of the nodes, wherein the node pairs are in the node pair subgraph, and decode a second adjacency matrix of the collocation relationship of the nodes in the subgraph.

[0189] In an exemplary embodiment, the extraction module may be ResNet.

[0190] In an exemplary embodiment, the training unit may be a FedNet model.

[0191] The FedNet model can be trained end-to-end on massive amounts of data, and the resulting matching model can achieve better results.

[0192] Although the embodiments disclosed in this application are as described above, the contents described are merely embodiments adopted to facilitate understanding of this application and are not intended to limit this application. Any person skilled in the art to which this application belongs may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application. However, the scope of patent protection of this application shall still be based on the scope defined by the attached claims.

Claims

1. A data processing method, comprising: Receive images to be processed; Obtaining at least one original image element of the image to be processed; Based on a knowledge graph, a target image element corresponding to the original image element is obtained, wherein the knowledge graph uses image elements as nodes and the relationship between any two image elements in a collocation combination as edges, the image elements are used to represent layers obtained by parsing an image file (.psd) file, and the collocation combination is constructed based on multiple layers in the .psd file in pairs; The target image element is fed back.

2. A method for matching image elements, comprising: Add the image elements to be processed as nodes to be processed into the collocation knowledge graph; Extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph; According to the adjacent node information and feature information, the nodes in the collocation knowledge graph are encoded to obtain the new feature information of the nodes; Determine node pairs with edges based on the new feature information of the nodes, where the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph; Determine the image elements corresponding to the nodes that have edges with the nodes to be processed and match them with the image elements to be processed, wherein the image elements are used to represent the layers obtained by parsing the image file psd file, and the edges are used to represent the relationship between any two image elements in the combination, and the combination is constructed based on multiple layers in the psd file.

3. The image element matching method according to claim 2, further comprising: The collocation knowledge graph is constructed according to the image elements of the sample image, wherein the image elements of the sample image serve as nodes of the collocation knowledge graph.

4. The image element matching method according to claim 3, wherein: The image elements of the sample image include: Using massive banner PSD image information as material, each image file PSD file is parsed to obtain several layers included in the PSD file, and each layer is regarded as an independent image element; The step of constructing a collocation knowledge graph based on image elements of the sample image includes: The independent image elements are used as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

5. The image element matching method according to claim 2, wherein: The feature information includes a visual feature matrix, and the adjacent node information is represented by an adjacency matrix; The step of extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing collocation relationships of nodes in the collocation knowledge graph includes: Based on a convolutional neural network, the visual feature matrix of the node is extracted according to the adjacency matrix.

6. The image element matching method according to claim 5, wherein: The nodes in the collocation knowledge graph are encoded according to the adjacent node information and feature information to obtain new feature information of the nodes, including: The adjacency matrix and the feature information are input into a multi-layer graph convolutional neural network, and the nodes in the collocation knowledge graph are respectively encoded to obtain new feature information of the nodes in the collocation knowledge graph.

7. The image element matching method according to claim 2, wherein: The determining of node pairs having edges includes: It is determined whether there is an edge between the node pairs by calculating the edge probability of the node pairs.

8. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image element matching method according to any one of claims 2 to 7.

9. A device for implementing image element matching, comprising a memory and a processor, wherein: The memory stores the following instructions that can be executed by the processor: used to execute the steps of the image element matching method described in any one of claims 2 to claim 7.

10. A method for matching image elements, comprising: Add the image element to be processed as a node to be processed to the collocation knowledge graph, wherein the image element is used to represent the layer obtained by parsing the image file PSD file; Input the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph into the pre-trained collocation model for processing; If there is an edge between the output node pairs of the collocation model, the image element corresponding to the node having the edge with the node to be processed is determined to be collocated with the image element to be processed, wherein the edge is used to characterize the relationship between any two image elements in the collocation combination, and the collocation combination is constructed based on the multiple layers in the psd file in pairs.

11. The image element matching method according to claim 10, wherein: The step of inputting the adjacent node information representing the collocation relationship of nodes in the collocation knowledge graph into a pre-trained collocation model for processing includes: Extracting feature information of nodes in the collocation knowledge graph based on adjacent node information representing collocation relationships of nodes in the collocation knowledge graph; According to the adjacent node information and feature information, the nodes in the collocation knowledge graph are respectively encoded to obtain new feature information of the nodes; Determine whether there is an edge between node pairs based on the new feature information of the nodes, wherein the node pairs include node pairs consisting of the node to be processed and any other node in the collocation knowledge graph. 12 . The image element matching method according to claim 10 , further comprising training the matching model.

13. The image element matching method according to claim 12, wherein: The collocation model is a FedNet model and includes at least one of the following: a feature extraction part, an encoding part, and a decoding part.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image element matching method according to any one of claims 10 to 13.

15. A device for implementing image element matching, comprising a memory and a processor, wherein: The memory stores the following instructions that can be executed by the processor: used to execute the steps of the image element matching method described in any one of claims 10 to 13.

16. A model training method comprising: Constructing a collocation knowledge graph based on the image elements of the sample image, wherein the image elements serve as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the collocation combination. The image elements are used to represent layers obtained by parsing the image file PSD file, and the collocation combination is constructed based on multiple layers in the PSD file. Divide the collocation knowledge graph into collocation combination number subgraphs, where the collocation combination number is the number of collocation combinations included in the collocation knowledge graph; Based on the positive and negative edge sets of each subgraph, a first adjacency matrix of each subgraph is obtained, and the obtained first adjacency matrix is input into a preset collocation model to obtain a second adjacency matrix of the subgraph; Comparing the losses between the first adjacency matrix and the second adjacency matrix to obtain a comparison result, and adjusting parameters of the collocation model according to the comparison result.

17. The model training method according to claim 16, wherein: The image elements of the sample image include: Using massive banner PSD image information as material, each image file PSD file is parsed to obtain several layers included in the PSD file, and each layer is regarded as an independent image element; The step of constructing a collocation knowledge graph based on image elements of the sample image includes: The independent image elements are used as nodes of the collocation knowledge graph, and edges are constructed between any two image elements in the same collocation combination.

18. The model training method according to claim 16, wherein: The collocation knowledge graph is divided into subgraphs of the number of collocation combinations, including: In the collocation knowledge graph, the subgraph is formed by taking the k-order neighbor nodes of all nodes in each collocation combination, and the collocation knowledge graph is divided into subgraphs as many as the collocation combinations, and each subgraph includes a collocation combination and the k-order neighbor nodes of the collocation combination node.

19. The model training method according to claim 16, wherein: The step of inputting the obtained first adjacency matrix into a preset collocation model to obtain a second adjacency matrix of the subgraph includes: Extracting feature information of the nodes in the subgraph according to the first adjacency matrix representing the collocation relationship of the nodes in the subgraph; Encoding the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes; Determine whether there is an edge between node pairs according to the new feature information of the nodes, wherein the node pairs are node pairs in a subgraph, and decode a second adjacency matrix of the collocation relationship of the nodes in the subgraph.

20. The model training method according to claim 19, wherein: The step of obtaining a first adjacency matrix of each subgraph based on the positive and negative edge sets of each subgraph includes: Construct a positive and negative edge set for each subgraph, take the edges connected by all the nodes in the subgraph as the positive edge set, and randomly take the edges consisting of the same number of unconnected node pairs as the positive edge set as the negative edge set. v i and nodes v j When there is an edge connection, the first adjacency matrix A i,j =1, otherwise the first adjacency matrix A i,j =0.

21. The model training method according to claim 19, wherein: The feature information includes visual features; and extracting the feature information of the nodes in the subgraph according to the first adjacency matrix representing the collocation relationship of the nodes in the subgraph includes: Based on a convolutional neural network, a visual feature matrix of the node is extracted according to the first adjacency matrix.

22. The model training method according to claim 19, wherein: The step of encoding the nodes in the subgraph according to the first adjacency matrix and the feature information to obtain new feature information of the nodes includes: The first adjacency matrix and feature information are input into a multi-layer graph convolutional neural network, and the nodes in the subgraph are respectively encoded to obtain new feature information of the nodes in the subgraph.

23. The model training method according to claim 19, wherein: Determining whether an edge exists between a pair of nodes includes: Whether there is an edge between the node pairs is determined by calculating the edge probability of the node pairs.

24. The model training method according to claim 23, wherein: The comparing the loss between the first adjacency matrix and the second adjacency matrix to obtain a comparison result, and adjusting the parameters of the collocation model according to the comparison result, includes: Use the loss function to calculate the loss for each sample separately; The average loss of all samples is used as the target loss; Updating the parameters of the collocation model using a back propagation algorithm according to the target loss; Among them, the samples include: positive samples = p i, j , negative samples = 1 - p i, j ,in, p i, j For node pairs ( v i , v j ) of the edge probability.

25. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the model training method described in any one of claims 16 to claim 24.

26. A device for implementing model training, comprising a memory and a processor, wherein: The memory stores the following instructions that can be executed by the processor: used to execute the steps of the model training method described in any one of claims 16 to claim 24.

Images