A depth map matching network and a fashion matching recommendation method based on the network

Abstract

This invention discloses a deep graph matching network and a fashion outfit recommendation method based on this network. The deep graph matching network includes an encoding module, a propagation and matching module, a residual aggregation module, and a synchronization aggregation module. The encoding module takes the in-degree, out-degree, and node type of graph nodes as input and outputs a fixed-size feature vector. The propagation and matching module consists of a propagation layer, forming a multi-layer structure. Each layer includes a neighbor node aggregation propagation unit and a cross-graph node matching fusion unit. The residual aggregation module includes a two-layer fully connected network and a four-layer fully connected network, outputting a similarity score. The synchronization aggregation module is divided into an aggregation part and a synchronization consistency part, with the aggregation part including a GRU gated recurrent unit. The fashion outfit recommendation method based on this network first constructs a fashion outfit rule knowledge base, calculates the fashion outfit migration cost between two graphs, and then, after processing the user's input fashion outfit rules, provides recommendation results and reasons.

Description

Technical Field

[0001] This invention belongs to the field of deep graph similarity learning models and fashion matching recommendation, and involves deep graph matching networks and fashion matching recommendation methods based on these networks. The fashion matching recommendation method uses tree-shaped matching rules extracted from visual images and user-input supplementary rules as the most basic inputs. Background Technology

[0002] As people's quality of life gradually improves, their pursuit of beauty also increases. In traditional brick-and-mortar clothing stores, people typically rely on experience and spend a significant amount of time selecting and trying on clothes to create a satisfactory outfit. However, with the development of e-commerce, online shopping has permeated people's daily lives. When shopping online, the inability to try on clothes makes people rely even more on experience, making it harder to find a satisfactory outfit and resulting in higher return rates and time costs. Furthermore, as people's aesthetic demands continue to rise, various fashion styling tutorials, fashion bloggers, and styling apps have emerged online, and the effectiveness of fashion styling is receiving increasing attention. All of the above examples raise the same question: how to coordinate clothing, and how to combine body attributes, occasions, and desired effects to achieve the expected or even unexpected results. The ability to match any two pieces of clothing, accessories, etc., to suitable individuals and occasions is a stylist's professional skill. An excellent stylist usually needs to have sufficient artistic attainments, aesthetic literacy, painting skills, historical knowledge, and the ability to observe and analyze life. They need to master knowledge related to facial makeup, hair design, clothing matching, nail art, etc., and are rare and professional.

[0003] With the development of artificial intelligence, deep learning has made interpretable intelligent outfit recommendations feasible. Current research on outfit recommendations focuses on image matching, suggesting corresponding bottoms based on images of tops, which has achieved good results. However, these recommendation methods lack aesthetic interpretation and do not consider people's accumulated fashion experience in daily life. In addition, how to evaluate the recommendation results is also a significant problem. Summary of the Invention

[0004] This invention proposes a deep graph matching network and a fashion outfit recommendation method based on fashion matching rules. Several fashion matching tree-shaped rules are extracted from relevant books, materials, and the internet, and converted into fashion matching rule graphs to form a fashion matching rule knowledge base. A fashion matching migration cost calculation method is designed, enabling the retrieval of the most experienced and explained matching rules with the lowest migration cost from the user-input fashion matching data within the knowledge base. Then, based on the difference between the input rules and the retrieved recommendation rules, and the expected effects of the recommendation rules, reasons for recommendations are provided to the user. The deep graph matching network is used to learn and calculate the fashion matching migration cost, accelerating the retrieval speed in the fashion rule dataset. Based on this approach, this invention proposes a fashion outfit recommendation method based on a deep graph matching network.

[0005] This invention proposes a deep graph matching network, comprising: an encoding module that encodes all nodes in the local graph and neighboring graphs, taking the in-degree, out-degree, and node type of the graph nodes as inputs and outputting a fixed-size feature vector; a propagation and matching module, consisting of a multi-layered propagation layer, each layer containing two parts: a neighbor node aggregation propagation unit and a cross-graph node matching fusion unit. The neighbor node aggregation propagation unit propagates node features along edges to neighboring nodes and outputs message vectors. All message vectors are summed with the terminal node to obtain the first propagation message for each node. The cross-graph node matching fusion unit calculates the similarity between any two nodes in the two graphs and outputs a matching retention vector. The difference between the matching retention vector and the node feature vector is calculated to obtain a residual vector; and a residual aggregation module, comprising a two-layer fully connected network and a four-layer fully connected network, which concatenates the residual vectors generated by all propagation and matching modules and feeds them into the two-layer fully connected network for processing. Feature transformation, after passing through the saliency block aggregation module, yields two graph-level residual vectors. The difference between the two graph-level residual vectors is squared and then fed into the four-layer fully connected network to output a similarity score. The synchronization aggregation module is divided into an aggregation part and a synchronization consistency part. The aggregation part includes a GRU gated recurrent unit, which concatenates the message vector and matching retention vector of each layer in the propagation and matching module as input and feeds it into the GRU gated recurrent unit to obtain a new feature vector, which is then used as the input for the next propagation layer. The synchronization consistency part uses synchronization consistency loss as additional supervision information. The node feature vector of each propagation layer is input into the saliency block aggregation module to obtain a graph-level feature vector. The synchronization consistency loss is used to calculate the L2 distance between the input graph-level feature vectors of the local graph and the neighboring graphs. The sum of the L2 distances obtained from each layer is calculated, and the sum is minimized to ensure that the L2 distance gradually decreases from the first layer to the last layer of the propagation module, i.e., the two graphs gradually become consistent.

[0006] The saliency block aggregation module divides the input vector into three equal parts according to the number of nodes, sums each part to obtain three block-level vectors, and concatenates the block-level vectors to form a graph-level vector.

[0007] Furthermore, the encoding module uses a two-layer fully connected network and sorts the feature vectors of each node in the image from largest to smallest by the sum of their in-degree and out-degree, and sorts the nodes with the same feature vector by node type.

[0008] Furthermore, the neighbor node aggregation and propagation unit adopts a two-layer fully connected network, the input of which is a node, an edge, and a node triple; the cross-graph node matching and fusion unit determines similar and dissimilar nodes based on the similarity, and inputs the proportion of similar nodes and dissimilar nodes in the graph into the decision unit of the cross-graph node matching and fusion unit to obtain the neighbor node addition ratio and the local node retention ratio. The neighbor node addition ratio is multiplied by the neighbor node feature vector to obtain the neighbor node matching retention vector, and the local node retention ratio is multiplied by the local node feature vector to obtain the local node matching retention vector; the neighbor node matching retention vector and the local node matching retention vector are added to obtain the matching retention vector.

[0009] Furthermore, nodes with a similarity greater than 75% are defined as similar nodes, and nodes with a similarity less than 25% are defined as dissimilar nodes.

[0010] In addition, a fashion outfit recommendation method based on a depth graph matching network is proposed, including:

[0011] S1. Extract several fashion matching rules from relevant books, materials and the Internet, standardize them and convert them into fashion matching rule trees, and build a fashion matching rule knowledge base;

[0012] S2, For the fashion matching rule tree in the knowledge base, calculate the fashion matching migration cost between pairs; the cost depends on the matching degree of the elements in the two images, and the higher the matching degree, the lower the cost.

[0013] S3, convert the fashion matching rule tree in the knowledge base into a fashion matching rule graph, and construct a fashion matching rule dataset based on the fashion matching rule graph and the calculated pairwise fashion matching migration costs;

[0014] S4, Using a portion of the data in the fashion matching rules dataset as the training set, the deep graph matching network is trained using an end-to-end supervised training strategy, and the parameters of the deep graph matching network are determined.

[0015] S5, based on the fashion matching rules input by the user, calculates recommended fashion combinations using a depth map matching network with defined parameters.

[0016] Furthermore, the fashion matching rule knowledge base contains six matching rules: clothing, inherent human attributes, variable external human attributes, occasion, weather, and effect. Each matching rule is described using a tree structure or forest structure, and the combinations of the six matching rules constitute the fashion matching rule knowledge base.

[0017] Further, S1 includes:

[0018] S11, Construct sub-knowledge bases of matching rules with different focuses, including sub-knowledge bases of color matching rules, type matching principles, attribute matching principles, classic matching examples, and incorrect matching examples. Each rule contains rule entries of clothing, inherent human attributes, variable external human attributes, occasion, weather, and effect. Construct each rule domain according to the tree-shaped modification relationship.

[0019] S12, integrate the various sub-knowledge bases, classify types and attributes, perform deduplication and merging after preprocessing based on edit distance, and delete ambiguous rules and complex rules that are difficult to process;

[0020] S13, construct a tree-like hierarchy among fashion element types to facilitate the calculation of subsequent fashion matching rule migration costs.

[0021] Furthermore, the matching degree in S2 is determined by the tree distance between the two fashion element types in their respective tree hierarchy. The distance between classes is 1, while the distance between classes within a class decreases exponentially according to the level of the nearest common ancestor. The distance of the coverage relationship is 0, and the greater the distance, the lower the matching degree. The method for calculating the fashion matching migration cost from input 1 to input 2 is to search from each fashion element in input 2 to input 1. Elements that are not found or not fully matched will be included in the cost. The amount of cost depends on the matching degree of the subtree rooted at that element. The higher the matching degree, the lower the cost. If there are multiple matches, the highest matching degree is used to calculate the cost. The method for calculating the matching degree of the subtree is as follows: the root node calculates the highest score of the subtree matching based on the matching degree of the fashion element type, and directly obtains half of the highest score as the basic score. The child nodes share the remaining score equally according to whether they match. The specific score of the child node is determined by whether each child node's child nodes match. If there are no child nodes, the remaining score is directly obtained.

[0022] Furthermore, when training the deep graph matching network, the input is a pair of fashion matching rules graphs. In one iteration, the nodes of the graph pair are encoded and sorted, and then passed through a 5-layer propagation and matching module with shared weights. The feature vectors of the nodes are updated through neighboring nodes and neighboring graph nodes. During the 5-layer propagation, the distance between the feature vectors of the two graphs is gradually reduced through synchronous consistency loss. The residual module takes the multi-layer residuals generated during propagation and matching as input, outputs the final similarity score, compares it with the true value, and uses the stochastic optimization strategy of Adam adaptive momentum to update the deep graph matching network.

[0023] Further, step S4 includes:

[0024] S41, divide the fashion matching rule dataset into training set, validation set and test set according to the proportion, and calculate the transfer cost between each pair of graphs in the training set, the transfer cost between the training set and the validation set, and the transfer cost between the training set and the test set.

[0025] S42, During the training iteration, the training graph pairs are used as inputs, and the mean squared error between the transfer cost and the output of the depth graph matching network is used as the main supervision information, and the cycle consistency loss is used as the additional supervision information. The stochastic optimization strategy of Adam adaptive momentum is used to update the parameters of the depth graph matching network.

[0026] S43. During the training iteration, after training a certain number of graph pairs, the deep graph matching network is validated once using the pairwise transfer cost between the training set and the validation set, and the mean square error of the deep graph matching network is recorded. After training, the parameters of the deep graph matching network with the best validation results are used as the final parameters of the deep graph matching network, and the deep graph matching network is tested once using the pairwise transfer cost between the training set and the test set to evaluate the performance of the final deep graph matching network.

[0027] The beneficial effects of this invention are as follows: This invention proposes a fashion matching recommendation method based on a deep graph matching network, aiming to help users make more professional and reasonable fashion combinations and provide reasons for the recommendations. Specifically, to structure fashion matching rules, this invention proposes a matching rule structure, where the rule domain includes clothing, human attributes, occasion, weather, and effect, and specifies a tree-like representation structure for each rule domain. To better suit fashion matching recommendation tasks, a fashion matching migration cost method is proposed, which calculates the cost of adding and replacing matching components only, without calculating the cost of deleting matching components, when migrating between fashion combinations. To improve the interpretability of cross-graph matching, a decision unit is introduced to guide the addition and retention of nodes in the current node and neighboring graphs. To better fit the computation process of the original algorithm, a synchronization consistency loss is introduced to gradually converge the propagation process. To reduce information loss when generating graph-level vectors, a saliency block aggregation module is proposed to retain the saliency difference information between blocks. This invention adopts an end-to-end training strategy to ensure clear and concise training steps. This framework has great potential for practical applications such as fashion matching recommendation. Attached Figure Description

[0028] Figure 1 This is a model framework diagram of the depth map matching network of the present invention.

[0029] Figure 2 This is a model framework diagram of the fashion matching recommendation method based on depth graph matching network of the present invention.

[0030] Figure 3 This is a flowchart of the fashion matching recommendation method based on a depth graph matching network according to the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] Figure 1This is a model framework diagram of the deep graph matching network of the present invention. The deep graph matching network includes four parts: an encoding module, a propagation and matching module, a residual aggregation module, and a synchronization aggregation module. The encoding module takes the in-degree, out-degree, and node type of a graph node as input and outputs a feature vector of a fixed size. The propagation and matching module consists of a propagation layer, which is a multi-layered structure. Each layer contains two parts: a neighbor node aggregation propagation unit and a cross-graph node matching fusion unit. The neighbor node aggregation propagation unit propagates node features along the edges to neighbor nodes and outputs message vectors. All message vectors are summed at the terminal node to obtain the first propagation message for each node. The cross-graph node matching fusion unit calculates the similarity between the feature vectors of any two nodes in two graphs and outputs a matching retention vector. Vector similarity can be cosine similarity or dot product similarity, etc. The difference between the matching retention vector and the node feature vector is used to obtain the residual vector. The residual aggregation module includes a two-layer fully connected network and a four-layer fully connected network. The residual aggregation module concatenates the residual vectors generated by all propagation and matching modules and sends them to the two-layer fully connected network for feature transformation. After passing through the saliency block aggregation module, two graph-level residual vectors are obtained. The difference between the two graph-level residual vectors is calculated, squared, and then input into the four-layer fully connected network to output the similarity score. The synchronization aggregation module is divided into an aggregation part and a synchronization consistency part. The aggregation part includes a GRU-gated recurrent unit, which concatenates the message vector and matching retention vector of each layer in the propagation and matching module as input, and feeds them into the GRU-gated recurrent unit to obtain a new feature vector, which serves as the input for the next propagation layer. The synchronization consistency part uses synchronization consistency loss as additional supervision information. It inputs the node feature vector of each propagation layer into the saliency block aggregation module to obtain a graph-level feature vector. The synchronization consistency loss is used to calculate the L2 distance between the input local graph and the neighboring graphs' graph-level feature vectors. The sum of the L2 distances obtained from each layer is calculated, and the sum is minimized to ensure that the L2 distance gradually decreases from the first layer to the last layer of the propagation module, i.e., the two graphs gradually become consistent. Specifically, the saliency block aggregation module divides the input vector into three equal parts according to the number of nodes, sums each part separately to obtain three block-level vectors, and concatenates these block-level vectors to obtain a graph-level vector.

[0033] Figure 2 This is a model framework diagram of the fashion matching recommendation method based on a depth graph matching network according to the present invention. The model takes tree-shaped matching rules extracted from visual images and user-input supplementary rules as inputs, such as "(beige, long, wool) knitwear, (blue, loose) jeans, (medium shade) skin tone, (170cm) height". It retrieves several standard rules with the lowest fashion matching migration cost from the fashion matching rule knowledge base, presents the above standard matching to the user, and shows the effect described in the rules to complete the recommendation.

[0034] Figure 3 The flowchart of the fashion matching recommendation method based on depth graph matching network provided by the present invention is shown, and its details are as follows:

[0035] Step S1: Construct a fashion matching rule knowledge base. The training data used in this invention comes from books such as "Change Your Clothes, Change Your Life," "Choose the Right Colors and Wear the Right Clothes (Collector's Edition)," and "Identify Your Body Shape and Wear the Right Clothes (Collector's Edition)." Each book has its own focus, such as detailed and specific matching rules, color matching techniques, and general matching principles. This invention mainly constructs matching combinations among six matching items: clothing, inherent human attributes, variable external human attributes, occasion, weather, and effect. Each matching item is described using a tree structure or forest structure, such as "(yellow, (cartoon) pattern) T-shirt, (khaki) cargo pants," where cartoon modifies pattern, yellow and pattern modify T-shirt, and khaki modifies cargo pants, constructing a hierarchical relationship in the tree structure based on the modification relationship. Clothing refers to the combination of clothing and accessories such as tops, bottoms, outfits, and necklaces; inherent human attributes refer to human attributes that are not easily changed, such as skin color, body shape, and face shape; variable external human attributes refer to human attributes such as makeup, hair color, and hairstyle; occasion refers to the appropriate occasion, style, and purpose; weather refers to temperature, season, and weather conditions; and effect refers to the overall effect of the combination, which can be good or bad.

[0036] Step S2: Design a method for calculating the migration cost of fashion matching. To construct a fashion matching rule dataset, this invention requires a calculation method to calculate the cost of migrating between pairs of rules. Since migration is best performed between two similar matching rules, and the cost should include unmet requirements (e.g., clothing items in the new matching need to be purchased, while items already met in the original matching will not have their costs calculated), this is an asymmetric calculation method. That is, the migration cost from matching A to matching B is different from the migration cost from matching B to matching A.

[0037] First, extract all non-modifier terms from the knowledge base and construct a tree structure relationship of fashion elements. For example, clothing includes tops and bottoms; tops include trench coats, sportswear, hoodies, shirts, etc.; shirts include tank tops, sweaters, etc. Then, define the distance between fashion elements in the tree structure. Define the depth of the node containing the nearest common ancestor of two elements in the tree as h, starting from 0. If the ancestor is exactly one of the two elements, the distance between the two elements is 0; otherwise, the distance is 1 - (1 / 2). h .

[0038] When calculating the migration cost of fashion combinations, such as the migration cost from combination A to combination B, all elements k in combination B are searched in combination tree A. If no match is found, the score is recorded as 0. The initial remaining score is 0. When a match is found, the score is half of the current remaining score. The child nodes of the two matched nodes are then matched at the same level in the subtree from B to A, and the remaining half of the score is divided equally. If there are multiple levels, half the score is obtained after a successful match, and the remaining score is redistributed to the subtree. For example, if there is an element k that is successfully searched from tree B to tree A, 0.5 points are awarded. Let's assume that element k in tree B has 4 children. When the first child node is successfully found, has the same content, and has no child nodes, 0.125 points are awarded; when the second and third children cannot find a matching node, 0 points are awarded; when the fourth child is successfully found, has the same content, but has two children, one of which failed to find a match and the other succeeded, 0.125 / 2 + 0.125 / 4 = 0.09375 points are awarded. The total score is 0.71875 points, so the cost is 1 - 0.71875 = 0.28125.

[0039] Step S3: Construct a fashion matching rule dataset. This invention constructs a fashion matching rule dataset based on the aforementioned fashion matching rule knowledge base and the fashion matching migration cost calculation method. The dataset is divided into training, validation, and test sets in a 6:2:2 ratio. The fashion matching migration cost calculation method is used to calculate the pairwise migration cost between each graph in the training set, as well as the pairwise migration costs between the training and validation sets and between the training and test sets. Since the migration cost calculation method is order-dependent, each pair of matching rules in the fashion matching rule knowledge base can generate two data pairs in the fashion matching rule dataset.

[0040] Step S4: Design a depth map matching network. This invention proposes a multi-scale saliency block aggregation depth map matching network model, such as... Figure 1 As shown, the network node embedding and edge embedding use the following fully connected networks (MLPs):

[0041]

[0042]

[0043] Where V represents the set of nodes in the graph; E represents the set of edges in the graph; MLP node A node encoder representing a fully connected network; MLP edge An encoder representing the edges of a fully connected network; x i The initial vector representing node i; x ijThis represents the initial vector of the edge connected to node i and node j. The initial vectors of nodes and edges are generally one-hot encoded according to their type. If there is no type, a uniform random vector is used for encoding. and These represent the initial node feature vector and the initial edge feature vector, respectively, which serve as the input to the first propagation layer.

[0044] The propagation and matching module will set up the node set. The vector representation is mapped to the new node representation. The calculation process of the neighbor node aggregation propagation unit is represented as follows:

[0045]

[0046] Where f message It is a fully connected network that outputs message vectors.

[0047] This patent uses the dissimilarity of nodes across graphs for calculation, and adds the proportion of similar and dissimilar nodes as auxiliary input. Referring to the definition of graph edit distance, the distance is related to the addition, modification, and deletion of nodes. The proportion of each node with similar nodes and dissimilar nodes in the matched graph are the decision conditions for retaining, modifying, or deleting that node. Using dissimilarity, a vector representation of dissimilar nodes can be obtained, so the model can learn which nodes to modify the node features to when making modification decisions. The graph node edit decision table is shown in Table 1.

[0048] Table 1. Node Editing Decision Table (t = 0.25)

[0049]

[0050] The calculation method for the proportion of similar nodes and the proportion of dissimilar nodes in the cross-graph node matching and fusion unit is as follows:

[0051]

[0052]

[0053]

[0054] Where, 0≤a j→i ≤1 represents the normalized vector similarity between node i and node j, s h Similarity calculation metrics include normalized cosine similarity; the hyperparameter t < 0.5 is the dissimilarity threshold; I{z} = 1 when z is true, otherwise I{z} = 0. Therefore, the cross-graph node matching fusion unit is represented as follows:

[0055]

[0056]

[0057] Where σ is the Sigmoid function; It is a fully connected network, and the Sigmoid function outputs the retention ratio of the current node; it is also a fully connected network, and the Sigmoid function outputs the addition ratio of neighboring nodes. This is the residual vector.

[0058] Therefore, the aggregation part of the synchronization aggregation module is represented as follows, where f node Indicates a GRU gated cyclic unit:

[0059]

[0060] Indicators such as graph edit distance are calculated by continuously adding, modifying, and deleting nodes and edges in one graph during the calculation process, ultimately affecting the other graph. Figure 1 Therefore, the model proposes a synchronous consistency loss, which makes the difference between the node feature vectors of the two graphs smaller as the number of propagation layers increases. To facilitate the calculation of graph-level differences, a saliency-based block aggregation module is used in the synchronous aggregation module to convert the node feature vectors of the two graphs into graph-level representations. The saliency-based block aggregation module divides the nodes into a fixed number of blocks, sums the values ​​in each block, and then concatenates them to obtain a fixed-size input vector. Since the nodes are sorted by node degree, the nodes represented by each block have different saliencies, thus preserving the node differences between blocks. Then, the synchronous aggregation module calculates the graph-level difference change between layers, ensuring that the difference between the two graphs in the later layer is smaller than the difference between the two graphs in the previous layer. The cycle-consistent part of the aggregation module is represented as follows:

[0061]

[0062]

[0063]

[0064] Where Ц represents vector concatenation; m represents the number of blocks in the saliency block aggregation module, i.e. the size of the output. The saliency block aggregation module performs vector summation within each block. Represents graph G i The number of nodes, i.e. D represents the feature length of the graph-level feature vector; T represents the total number of propagation layers. Represents graph G i The graph-level representation of the t-th propagation layer; d (t) This indicates the difference in the graph-level representation of the two graphs at the t-th propagation level; L sync This represents the loss due to synchronization issues.

[0065] To avoid the loss of low-order features caused by using only the result of the last convolutional layer as input, and to address the issue that due to the smoothing nature of graph convolutional neural networks, the features of each node tend to be consistent after multiple convolutions, making it impossible to reflect the differences between nodes, this patent performs multi-scale fusion of the residual vectors from multiple propagation layers, using them as input to the residual aggregation module for graph similarity prediction. The residual aggregation module uses a saliency block aggregation module to fuse them into a graph-level representation, and then performs similarity prediction. The process is as follows:

[0066]

[0067]

[0068]

[0069] in, Represents the residual vector generated in the cross-graph node matching fusion unit; MLP fuse This represents a two-layer fully connected network that performs feature fusion on the multi-scale residual vectors before block aggregation; α fuse,i Represents the residual vector after multi-scale fusion; α G Represents graph-level residual vectors; MLP score This represents a four-layer fully connected network used to calculate graph similarity scores; s prec The graph similarity predicted by a deep graph matching network model that aggregates multi-scale saliency blocks.

[0070] To ensure that the two graphs eventually converge during propagation, we need to make s sync →0. The training objective of the model is to minimize the loss function. The model's loss function is described as follows:

[0071] loss = L sync 2 +|s prec -y| 2 ,

[0072] Where y is the actual value calculated using the fashion matching migration cost calculation method.

[0073] Step S5: Design an end-to-end training strategy. In one iteration, the nodes of the fashion matching rule graph are encoded and sorted, and then passed through a 5-layer propagation module with shared weights. The feature vectors of each node are updated through neighboring nodes and neighboring graph nodes. During the 5-layer propagation, the distance between the feature vectors of the two graphs is gradually reduced through synchronous consistency loss. The residual aggregation module takes the multi-layer residuals generated during propagation matching as input and outputs the final similarity score. This score is compared with the true value, and the model is updated using a stochastic optimization strategy with Adam adaptive momentum.

[0074] The main contributions of this invention are as follows: (1) A tree-structured fashion matching rule knowledge base containing 6 items was established, and a fashion matching rule dataset was constructed based on the fashion migration cost calculation method proposed in this invention. (2) A fashion matching recommendation method based on a deep graph matching network was proposed, aiming to help users make more professional and reasonable fashion matching and to provide reasons for the recommendation. A cross-graph node matching fusion unit based on the decision unit and a synchronous consistency loss were proposed. The difference vector prediction results generated by the cross-graph node matching fusion unit were used to enhance the interpretability of the network model inference and the model performance. A saliency block aggregation module was proposed to reduce the loss of information in the aggregation layer and enhance the model's matching performance for different graph sizes. This opens up a huge research space for interpretable fashion matching recommendations in the future.

Claims

1. A fashion outfit recommendation method based on a depth graph matching network, characterized in that, include: S1. Extract several fashion matching rules from relevant books, materials and the Internet, standardize them and convert them into fashion matching rule trees, and build a fashion matching rule knowledge base; S2, For the fashion matching rule tree in the knowledge base, calculate the fashion matching migration cost between pairs; the cost depends on the matching degree of the elements in the two images, and the higher the matching degree, the lower the cost. S3, convert the fashion matching rule tree in the knowledge base into a fashion matching rule graph, and construct a fashion matching rule dataset based on the fashion matching rule graph and the calculated pairwise fashion matching migration costs; S4, Using a portion of the data in the fashion matching rules dataset as the training set, a deep graph matching network is trained using an end-to-end supervised training strategy, and the parameters of the deep graph matching network are determined. S5, based on the fashion matching rules input by the user, calculates recommended fashion combinations using a depth map matching network with defined parameters; The depth map matching network includes: The encoding module encodes all nodes in the current graph and its neighboring graphs. The inputs are the in-degree, out-degree, and type of the graph nodes, and the output is a feature vector of a fixed size. The propagation and matching module consists of a multi-layered propagation layer. Each layer contains two parts: a neighbor node aggregation propagation unit and a cross-graph node matching and fusion unit. The neighbor node aggregation propagation unit propagates node features along the edges to neighbor nodes and outputs message vectors. All message vectors are summed with the final node to obtain the first propagation message for each node. The cross-graph node matching and fusion unit calculates the similarity between any two nodes in the two graphs and outputs a matching retention vector. The difference between the matching retention vector and the node feature vector is calculated to obtain the residual vector. The residual aggregation module includes a two-layer fully connected network and a four-layer fully connected network. The residual aggregation module concatenates the residual vectors generated by all propagation and matching modules and sends them to the two-layer fully connected network for feature transformation. After passing through the saliency block aggregation module, two graph-level residual vectors are obtained. The difference between the two graph-level residual vectors is calculated, the square is obtained, and then the similarity score is sent to the four-layer fully connected network to output the similarity score. The synchronization aggregation module is divided into an aggregation part and a synchronization consistency part. The aggregation part includes a GRU gated recurrent unit, which concatenates the message vector and matching retention vector of each layer in the propagation and matching module as input and sends them to the GRU gated recurrent unit to obtain a new feature vector, which is then used as the input for the next propagation layer. The synchronization consistency part uses synchronization consistency loss as additional supervision information. It inputs the node feature vector of each propagation layer into the saliency block aggregation module to obtain graph-level feature vectors. It uses synchronization consistency loss to calculate the L2 distance between the graph-level feature vectors of the input local graph and neighboring graphs. It sums the L2 distances obtained from each layer and minimizes the sum to ensure that the L2 distance from the first layer to the last layer of the propagation module gradually decreases, that is, the two graphs gradually become consistent. The saliency block aggregation module divides the input vector into three equal parts according to the number of nodes, sums each part to obtain three block-level vectors, and concatenates the block-level vectors to form a graph-level vector.

2. The fashion matching recommendation method as described in claim 1, characterized in that, The encoding module uses a two-layer fully connected network and sorts the feature vectors of each node in the image from largest to smallest by the sum of their in-degree and out-degree. If they are equal, they are sorted by node type.

3. The fashion matching recommendation method as described in claim 1, characterized in that, The neighbor node aggregation and propagation unit adopts a two-layer fully connected network. The input of the two-layer fully connected network is a node, an edge, and a node triple. The cross-graph node matching and fusion unit determines similar and dissimilar nodes based on the similarity. The proportion of similar nodes and dissimilar nodes in the graph is input into the decision unit of the cross-graph node matching and fusion unit to obtain the neighbor node addition ratio and the local node retention ratio. The neighbor node addition ratio is multiplied by the neighbor node feature vector to obtain the neighbor node matching retention vector. The local node retention ratio is multiplied by the local node feature vector to obtain the local node matching retention vector. The neighbor node matching retention vector and the local node matching retention vector are added to obtain the matching retention vector.

4. The fashion matching recommendation method as described in claim 3, characterized in that, Nodes with a similarity greater than 75% are defined as similar nodes, and nodes with a similarity less than 25% are defined as dissimilar nodes.

5. The fashion matching recommendation method as described in claim 1, characterized in that, The fashion matching rules knowledge base contains six matching items: clothing, inherent human attributes, variable external human attributes, occasion, weather, and effect. Each matching item is described using a tree structure or forest structure. The combinations of the six matching items constitute the fashion matching rules knowledge base.

6. The fashion matching recommendation method as described in claim 1, characterized in that, S1 includes: S11, Construct sub-knowledge bases of matching rules with different focuses, including sub-knowledge bases of color matching rules, type matching principles, attribute matching principles, classic matching examples, and incorrect matching examples. Each rule contains rule entries of clothing, inherent human attributes, variable external human attributes, occasion, weather, and effect, and constructs each rule domain according to tree-shaped modification relationship; S12, integrate the various sub-knowledge bases, classify types and attributes, perform deduplication and merging after preprocessing based on edit distance, and delete ambiguous rules and complex rules that are difficult to process; S13, Construct a tree-like hierarchy among fashion element types to facilitate the calculation of subsequent fashion matching rule migration costs.

7. The fashion matching recommendation method as described in claim 1, characterized in that, The degree of matching in S2 is determined by the distance between the two fashion element types in their respective tree-like affiliation relationships. The distance between classes is 1, while the distance between classes within a class decreases exponentially according to the level of the nearest common ancestor. The distance of the coverage relationship is 0, and the greater the distance, the lower the degree of matching. The method for calculating the fashion matching migration cost from input 1 to input 2 is as follows: search from each fashion element in input 2 to input 1. Elements that are not found or do not fully match will be included in the cost. The amount of cost depends on the matching degree of the subtree rooted at that element. The higher the matching degree, the lower the cost. If there are multiple matches, the highest matching degree is used to calculate the cost. The method for calculating the matching degree of the subtree is as follows: the root node calculates the highest score of the subtree matching based on the matching degree of the fashion element type, and directly obtains half of the highest score as the base score. The child nodes share the remaining score equally according to whether they match. The specific score of the child node is determined by whether each child node's child nodes match. If there are no child nodes, the remaining score is directly obtained.

8. The fashion matching recommendation method as described in claim 1, characterized in that, When training the deep graph matching network, the input is a pair of fashion matching rules graphs. In one iteration, the nodes of the graph pair are encoded and sorted, and then passed through a 5-layer propagation and matching module with shared weights. The feature vectors of the nodes are updated through neighboring nodes and neighboring graph nodes. During the 5-layer propagation, the distance between the feature vectors of the two graphs is gradually reduced by the synchronization consistency loss. The residual module takes the multi-layer residuals generated during propagation and matching as input, outputs the final similarity score, compares it with the true value, and uses the Adam adaptive momentum stochastic optimization strategy to update the deep graph matching network.

9. The fashion matching recommendation method as described in claim 1, characterized in that, Step S4 includes: S41, Divide the fashion matching rule dataset into training set, validation set and test set according to the proportion, and calculate the transfer cost between each pair of graphs in the training set, the transfer cost between the training set and the validation set, and the transfer cost between the training set and the test set. S42, During the training iteration, the training graph pairs are used as inputs, and the mean squared error between the transfer cost and the output of the depth map matching network is used as the main supervision information, and the cycle consistency loss is used as the additional supervision information. The stochastic optimization strategy of Adam adaptive momentum is used to update the parameters of the depth map matching network. S43. During the training iteration, after training a certain number of graph pairs, the deep graph matching network is validated once using the pairwise transfer cost between the training set and the validation set, and the mean square error of the deep graph matching network is recorded. After training, the parameters of the deep graph matching network with the best validation results are used as the final parameters of the deep graph matching network, and the deep graph matching network is tested once using the pairwise transfer cost between the training set and the test set to evaluate the performance of the final deep graph matching network.

Images