Fabric emotion labeling method based on multi-view stack type depth TSK fuzzy system

Through multi-view stack-type depth TSK fuzzy system and manifold constraints, combined with convolutional neural network, the distortion and overfitting problems caused by feature splicing in home textile design are solved, the model interpretability is improved, and the fabric emotional annotation is achieved is achieved. Sexual and personalized design support.

CN118982701APending Publication Date: 2024-11-19NANTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410973976.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the design of home textiles, the existing technology is difficult to easily integrate the objective characteristics of textile patterns such as textures, patterns, and colors and the emotional experience of emotions into artificial intelligence systems for image aesthetic evaluation, emotion classification and retrieval, and the deep learning model lacks explanation. It is prone to overfitting.

Method used

Using a multi-view stack-deep TSK fuzzy system, a single TSK fuzzy system based on manifold constraints is constructed by extracting multi-view feature of the initial fabric image, forming a stack-deep TSK fuzzy system, combining a convolutional neural network and emotional labeling unit. Perform fabric sentiment annotation to ensure that samples are approached during feature space transformation, avoid overfitting, and improve model interpretability through fuzzy rules and Shannon entropy adaptive weighting.

Benefits of technology

Solve the distortion of aesthetic or emotional expression caused by feature splicing, avoid overfitting, improve the interpretability of the model and the accuracy of emotional annotation, and provide a more personalized and artistic design resource management system for home textile design. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982701A_ABST
    Figure CN118982701A_ABST
Patent Text Reader

Abstract

The invention discloses a fabric emotion labeling method based on a multi-view stack type depth TSK fuzzy system. The method comprises the following steps: performing multi-view feature map extraction on an initial fabric image; inputting the multi-view feature maps into a convolutional neural network to respectively obtain deep multi-view feature maps; constructing an emotion labeling system; inputting the deep multi-view feature map into an emotion labeling system to carry out emotion labeling on the fabric image; wherein the step of constructing the emotion labeling system specifically comprises the following steps: constructing a single TSK fuzzy system based on manifold constraint; based on a single TSK fuzzy system, constructing a stack type depth TSK fuzzy system as an emotion labeling unit; and constructing an emotion labeling system based on an emotion labeling unit. Based on home textile design resources, a novel intelligent resource management system integrating multi-view aesthetic feature extraction, multi-view deep aesthetic feature extraction and emotion labeling is established.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a fabric emotion labeling method based on a multi-view stacked deep TSK fuzzy system. Background Art

[0002] Emotion is the spiritual essence of product artistry and individuality. Consumers will have different emotions or feelings when facing different home textile products. Patterns are important design elements and carriers of aesthetic transmission of home textile products, and contain rich emotional information, such as consumers' identity, taste, aesthetic preferences and values. Therefore, modern home textile design has transitioned from the traditional rational indicator satisfaction stage oriented towards physiological factors and technological innovation to the perceptual indicator that meets consumers' emotional needs. As a result, pattern design with rich connotation and emotion is increasingly respected by designers. With the continuous development of informatization and the real-time sharing of global design information, the accumulated patterns and pattern materials in home textile design and production are increasing day by day, and popular information is increasing in a blowout manner. However, a large number of surveys have found that it is difficult for designers to conveniently and rationally analyze and utilize them in home textile design. Therefore, how to integrate the objective characteristics of textile patterns such as texture, pattern, and color with the sensory experience of emotions into an artificial intelligence-based intelligent management system for image aesthetic evaluation, emotion classification and retrieval, emotion labeling, etc. is very important for home textile design practitioners. It is also an important issue that needs to be urgently solved in the process of home textile companies focusing on the personality, fashion, and artistry that satisfy consumers' emotional satisfaction, shifting from functionality and usability.

[0003] From the current research situation, both traditional machine learning assisted sentiment labeling and deep learning assisted sentiment labeling have achieved remarkable results. However, there are still urgent problems to be solved in the current research in terms of feature utilization and model interpretability: In the research of traditional machine learning assisted sentiment labeling, most of them adopt the method of feature splicing to form a new feature input space with different types of features (salient areas, colors, textures, etc.), and then modeling. The new feature input space formed by this feature splicing method makes the original feature space with certain aesthetics or emotional expressions "distorted", and cannot fully utilize the pattern information carried by all the original features; whether it is traditional machine learning assisted sentiment labeling or deep learning assisted sentiment labeling, when the training samples contain noise or the number is insufficient, it is easy to produce overfitting. In the training process, that is, when converting from feature space to emotion label space, the present invention expects that samples belonging to the same emotion in feature space can be closer in emotion label space, thereby changing its loose class structure and avoiding overfitting; although deep learning assisted sentiment labeling has good feature learning ability, it is almost a black box model and has no interpretability. Therefore, interpretable models can help home textile designers better understand the causes and reasoning process of the results. Summary of the invention

[0004] To solve the above problems, the present invention provides a fabric emotion labeling method based on a multi-view stacked deep TSK fuzzy system, comprising the following steps:

[0005] Extracting a multi-view feature map from the initial fabric image, wherein the multi-view feature map includes: an edge feature map, a texture feature map and a significant area feature map;

[0006] Input the multi-view feature map into the convolutional neural network to obtain deep multi-view feature maps, wherein the deep multi-view feature map includes: a deep edge feature map, a deep texture feature map and a deep salient area feature map;

[0007] Build a sentiment annotation system;

[0008] The deep multi-view feature map is input into the sentiment annotation system for sentiment annotation of fabric images.

[0009] Furthermore, constructing a sentiment annotation system specifically includes the following steps:

[0010] Construct a single TSK fuzzy system based on manifold constraints;

[0011] Based on a single TSK fuzzy system, a stacked deep TSK fuzzy system is constructed as a sentiment labeling unit;

[0012] Based on the emotion annotation unit, a emotion annotation system is constructed.

[0013] Furthermore, a single TSK fuzzy system based on manifold constraints is constructed, which specifically includes the following steps:

[0014] If the training samples in the same class are regarded as nodes in an undirected graph, each pair of nodes is connected by an undirected edge, and each edge corresponds to a weight coefficient. The specific formula is:

[0015]

[0016] in, and denote the i-th and j-th samples in the v-th visual feature space, σ denotes the heat kernel parameter, and E ij Represents the weight coefficient between the i-th sample and the j-th sample;

[0017] The proposed manifold assumption is satisfied by minimizing the objective expression. The specific formula is:

[0018]

[0019] in, and Respectively represent the results of sentiment labeling of the i-th and j-th samples;

[0020] For the TSK fuzzy system, after the antecedent parameters are determined through unsupervised learning, the solution process of the consequent parameters is solved by optimizing the objective function. The specific formula is:

[0021]

[0022] Among them, J TSK-FS represents the objective function of the TSK fuzzy system, p (v) represents the consequent parameters that the fuzzy system needs to learn at the view v, y i The labels of the training samples are used to label the sentiment of fabric images for model training;

[0023] By embedding formula (2) into the objective function of the TSK fuzzy system, a single TSK fuzzy system based on manifold constraints is obtained. The specific formula is:

[0024]

[0025] Among them, J Manifold-TSK-FS represents the objective function of a single TSK fuzzy system based on manifold constraints, λ 1 and λ 2 represents the regularization parameter; the objective function J Manifold-TSK-FS The optimization solution is obtained by Lagrange multiplier method.

[0026] Furthermore, the stacked deep TSK fuzzy system includes L single TSK fuzzy systems based on manifold constraints, namely: FS 1 , FS 2 , ..., FS L ; Among them, the input of each single TSK fuzzy system contains the original input features The outputs of each single TSK fuzzy system are: i,1 ,y i,2 , ..., y i,L , and introduce the outputs of the first L-1 single TSK fuzzy systems into the antecedents of the next single TSK fuzzy system;

[0027] The fuzzy rule is based on the stack as the dynamic adaptation factor of the rule. The specific formula of the fuzzy rule is:

[0028]

[0029] in, represents the dynamic adaptation factor, represents the first-order partial parameter of the consequent of the kth rule, represents the 0th order part parameter, represents the fuzzy set function used by the kth rule; Represents the fuzzy set used by the i-th feature in the k-th rule.

[0030] A single TSK fuzzy system is fuzzified, inferred, and defuzzified to obtain the result output. The specific formula is:

[0031]

[0032] Among them, y i,l represents the lth single TSK fuzzy system.

[0033] Further, the emotion annotation system includes three emotion annotation units, which respectively input the deep edge feature map, the deep texture feature map, and the deep salient region feature map;

[0034] Among them, each sentiment labeling unit minimizes the same objective function, the specific formula is:

[0035]

[0036] in, Indicates the mean of the annotation results of the i-th sample from other perspectives except the current perspective. It indicates the result of the i-th sample being labeled by the emotion labeling unit in the feature space of the current perspective;

[0037] The three emotion annotation units are subjected to multi-view collaborative learning, including the inter-view consistency constraint part and the Shannon entropy adaptive weighting part. The specific formula is:

[0038]

[0039] Among them, w r represents the weight of the rth sentiment annotation unit,

[0040] is the inter-view consistency constraint part, It is the adaptive weighted part of Shannon entropy.

[0041] According to one aspect of the present invention, a storage medium is provided, in which instructions are stored. When a computer reads the instructions, the computer executes any one of the above-mentioned fabric emotion labeling methods based on a multi-view stacked deep TSK fuzzy system.

[0042] According to another aspect of the present invention, there is provided an electronic device, comprising a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Aiming at the problem of "distortion" of aesthetic or emotional expression caused by the new feature input space formed by feature splicing, the present invention constructs a multi-perspective feature representation method for home textile design resources, and uses multi-perspective learning to mine the complementary aesthetic and emotional expressions carried by the original features from different perspectives.

[0045] 2. The present invention aims at the fitting phenomenon caused when the training samples contain noise or are insufficient in number. It uses the "compactness graph" in manifold learning to ensure that the training samples in the same category should be as close as possible in the conversion space when they are converted from the original feature space to the label space, so as to fit the labels more appropriately and avoid overfitting.

[0046] 3. To address the problem of poor interpretability of deep learning models, the present invention uses the TSK fuzzy system as the basic learning unit, constructs a fuzzy system with a special hierarchical structure, and combines it with a feature random strategy to ensure the interpretability of the constructed fuzzy system while improving its approximation capability.

[0047] 4. Based on home textile design resources, the present invention establishes a new intelligent resource management system that integrates multi-perspective aesthetic feature extraction, multi-perspective deep aesthetic feature extraction, and emotional labeling, providing auxiliary support for home textile designers to make full use of existing design resources and stimulate creative inspiration. It also provides solutions for consumers to screen out designs and products that better meet their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0050] Figure 2 It is a structural diagram of a stacked deep TSK fuzzy system according to an embodiment of the present invention;

[0051] Figure 3 It is a fuzzy rule transformation diagram of the stacked deep TSK fuzzy system according to an embodiment of the present invention;

[0052] Figure 4 A diagram of a multi-perspective collaborative learning solution according to an embodiment of the present invention;

[0053] Figure 5 It is an overall process framework diagram of an embodiment of the present invention;

[0054] Figure 6 This is a diagram of the deep feature extraction network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] like Figure 1 As shown, the fabric emotion labeling method based on the multi-view stacked deep TSK fuzzy system of the present invention comprises the following steps:

[0057] Extracting a multi-view feature map from the initial fabric image, wherein the multi-view feature map includes: an edge feature map, a texture feature map and a significant area feature map;

[0058] Input the multi-view feature map into the convolutional neural network to obtain deep multi-view feature maps, wherein the deep multi-view feature map includes: a deep edge feature map, a deep texture feature map and a deep salient area feature map;

[0059] Build a sentiment annotation system;

[0060] The deep multi-view feature map is input into the sentiment annotation system for sentiment annotation of fabric images.

[0061] Building a sentiment annotation system includes the following steps:

[0062] Construct a single TSK fuzzy system based on manifold constraints;

[0063] Based on a single TSK fuzzy system, a stacked deep TSK fuzzy system is constructed as a sentiment labeling unit;

[0064] Based on the emotion annotation unit, a emotion annotation system is constructed.

[0065] In order to suppress the occurrence of "overfitting", based on the idea of ​​manifold learning, it is assumed that "when training samples in the same class are converted from the original feature space to the label space, they should be as close as possible in the converted space". Therefore, if the training samples in the same class are regarded as nodes in an undirected graph, each pair of nodes is connected by an undirected edge, and each edge corresponds to a weight coefficient, the relationship between the training samples can be described by the following mathematical expression:

[0066]

[0067] in, and represents the i-th and j-th samples in the v-th visual feature space, and σ is the heat kernel parameter. Obviously, any two converted samples (i.e., sentiment labeling results) in the same category and The undirected edges between them should have large weights. Therefore, the proposed manifold assumption can be satisfied in the best way by minimizing the following objective expression:

[0068]

[0069] E ij Represents the weight coefficient between the i-th sample and the j-th sample.

[0070] For the TSK fuzzy system, after its antecedent parameters are determined by unsupervised learning (such as fuzzy C-means clustering), the solution process of the consequent parameters can be seen as the following linear regression problem, which can be solved by optimizing the following objective function:

[0071]

[0072] By embedding the compactness graph in equation (2) into the objective function of the TSK fuzzy system, we can obtain a single TSK fuzzy system based on manifold constraints.

[0073]

[0074] Among them, λ 1 and λ 2 is the regularization parameter. The objective function shown in formula (4) can be optimized and solved by the Lagrange multiplier method.

[0075] The feature expression of home textile design resources within a certain perspective (such as texture) is often located in a high-dimensional space. When using a single TSK fuzzy system for modeling, the phenomenon of "rule explosion" is prone to occur, which significantly reduces its interpretability. The generalized stack principle believes that adding predictive information to the sample input feature space can help open the manifold structure of the input feature space and improve training accuracy. The present invention uses the TSK fuzzy system embedded in the compactness graph in formula (4) as the basic component, and designs a stacked deep TSK fuzzy system as the sentiment labeling unit of this project.

[0076] The fuzzy rule is based on the stack as the dynamic adaptation factor of the rule. The specific formula of the fuzzy rule is:

[0077]

[0078] in, represents the dynamic adaptation factor, represents the first-order partial parameter of the consequent of the kth rule, represents the 0th order part parameter, represents the fuzzy set function used by the kth rule; Represents the fuzzy set used by the i-th feature in the k-th rule.

[0079] A single TSK fuzzy system is fuzzified, inferred, and defuzzified to obtain the result output. The specific formula is:

[0080]

[0081] Among them, y i,l represents the lth single TSK fuzzy system.

[0082] The reason why the stacked deep TSK fuzzy system can solve the "rule explosion" and maintain interpretability without reducing accuracy is mainly due to two aspects:

[0083] from Figure 2 As shown, the input of all single TSK fuzzy systems contains the original input feature space. Therefore, for FS1, FS2, etc., the present invention can be used in its input feature space Some features (much smaller than d) are randomly selected from the training set, thereby reducing the number of fuzzy rules required. In addition, due to the use of a stack structure, the manifold structure of the FSl feature space will be continuously opened by the first l-1 outputs, and the training accuracy will be improved;

[0084] This stack structure will contain y in the input feature space i,1, y i,2 ...etc., that is, these outputs are introduced in the antecedents of fuzzy rules. However, these outputs are not as interpretable as features, such as Figure 3 As shown. On the surface, this stack structure will destroy the interpretability of fuzzy rules. However, the present invention can make some equivalent mathematical changes while ensuring that the output results remain unchanged, hide these output results in the rule consequences, and use them as the dynamic adaptation factor (Certainly Factor, CF) of the rule, so that the fuzzy rules with a stack structure are still interpretable.

[0085] The constructed stacked deep TSK fuzzy system (i.e., emotion annotation unit) can complete the emotion annotation of home textile design resources in a single feature space. After the original home textile design resources are preprocessed, the present invention can obtain multiple feature spaces that depict emotions. Next, we will design how to use the emotion annotation unit to perform multi-perspective collaborative learning in multiple feature spaces and mine the complementary emotion pattern information contained in different feature spaces.

[0086] like Figure 4 As shown in the figure, there are two collaborative learning strategies: First, for the same home textile design resource, the sentiment labeling results in different feature spaces should be as consistent as possible. For example, if a home textile design pattern is very pleasing to the eye and belongs to the category of high aesthetics, then its texture features, edge features, and significant area features should all belong to the category of high aesthetics. Second, the weights of different perspectives are adaptively learned using Shannon entropy, and the perspectives with weights less than a certain threshold are reduced. represents the expression of the feature space of the i-th sample at the v-th perspective, It represents the result of the i-th sample being labeled by the emotion labeling unit in the feature space of the current perspective. It represents the average value of the annotation results of the i-th sample from other perspectives except the current perspective. l Represents the weight coefficient of the lth view.

[0087] The emotion annotation system includes three emotion annotation units, which input the deep edge feature map, deep texture feature map and deep salient region feature map respectively;

[0088] Among them, each sentiment labeling unit minimizes the same objective function, the specific formula is:

[0089]

[0090] in, Indicates the mean of the annotation results of the i-th sample from all other perspectives except the current perspective;

[0091] The three emotion annotation units are subjected to multi-view collaborative learning, including the inter-view consistency constraint part and the Shannon entropy adaptive weighting part. The specific formula is:

[0092]

[0093] Among them, w r represents the weight of the rth sentiment annotation unit, is the inter-view consistency constraint part, It is the adaptive weighted part of Shannon entropy.

[0094] like Figure 5 As shown, in the initial multi-view feature extraction stage, the Sobel operator is used to extract edge shape features, the Gabor filter is used to extract texture features, and the LC (Luminance Contrast) algorithm is used to extract significant area features.

[0095] Based on the initial multi-view feature map, the present invention designs a deep convolutional neural network to extract deep features (the features of the penultimate layer are the deep features finally learned). The designed network structure is as follows: Figure 6 shown.

[0096] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.

Claims

1. Fabric sentiment labeling method based on multi-view stacked deep TSK fuzzy system, It is characterized in that The steps include: Extracting a multi-view feature map from the initial fabric image, wherein the multi-view feature map includes: an edge feature map, a texture feature map and a significant area feature map; Input the multi-view feature map into the convolutional neural network to obtain deep multi-view feature maps, wherein the deep multi-view feature map includes: a deep edge feature map, a deep texture feature map and a deep salient area feature map; Build a sentiment annotation system; The deep multi-view feature map is input into the sentiment annotation system for sentiment annotation of fabric images.

2. The method according to claim 1, It is characterized in that Building a sentiment annotation system includes the following steps: Construct a single TSK fuzzy system based on manifold constraints; Based on a single TSK fuzzy system, a stacked deep TSK fuzzy system is constructed as a sentiment labeling unit; Based on the emotion annotation unit, a emotion annotation system is constructed.

3. The method according to claim 2, It is characterized in that Constructing a single TSK fuzzy system based on manifold constraints includes the following steps: If the training samples in the same class are regarded as nodes in an undirected graph, each pair of nodes is connected by an undirected edge, and each edge corresponds to a weight coefficient. The specific formula is: in, and denote the i-th and j-th samples in the v-th visual feature space, σ denotes the heat kernel parameter, and E ij Represents the weight coefficient between the i-th sample and the j-th sample; The proposed manifold assumption is satisfied by minimizing the objective expression. The specific formula is: in, and Respectively represent the results of sentiment labeling of the i-th and j-th samples; For the TSK fuzzy system, after the antecedent parameters are determined through unsupervised learning, the solution process of the consequent parameters is solved by optimizing the objective function. The specific formula is: Among them, J TSK-FS represents the objective function of the TSK fuzzy system, p (v) represents the consequent parameters that the fuzzy system needs to learn at the view v, y i The labels of the training samples are used to label the sentiment of fabric images for model training; By embedding formula (2) into the objective function of the TSK fuzzy system, a single TSK fuzzy system based on manifold constraints is obtained. The specific formula is: Among them, J Manifold-TSK-FS represents the objective function of a single TSK fuzzy system based on manifold constraints, λ 1 and λ 2 represents the regularization parameter; the objective function J Manifold-TSK-FS The optimization solution is obtained by Lagrange multiplier method.

4. The method according to claim 3, It is characterized in that The stacked deep TSK fuzzy system consists of L single TSK fuzzy systems based on manifold constraints, namely: FS 1 , FS 2 , ..., FS L ; Among them, the input of each single TSK fuzzy system contains the original input features The outputs of each single TSK fuzzy system are: i,1 ,y i,2 , ..., y i,L , and introduce the outputs of the first L-1 single TSK fuzzy systems into the antecedents of the next single TSK fuzzy system; The fuzzy rule is based on the stack as the dynamic adaptation factor of the rule. The specific formula of the fuzzy rule is: in, represents the dynamic adaptation factor, represents the first-order partial parameter of the consequent of the kth rule, represents the 0th order part parameter, represents the fuzzy set function used by the kth rule; Represents the fuzzy set used by the i-th feature in the k-th rule. A single TSK fuzzy system is fuzzified, inferred, and defuzzified to obtain the result output. The specific formula is: Among them, y i,l Indicates l A single TSK fuzzy system.

5. The method according to claim 4, It is characterized in that The emotion annotation system includes three emotion annotation units, which input the deep edge feature map, deep texture feature map and deep salient region feature map respectively; Among them, each sentiment labeling unit minimizes the same objective function, the specific formula is: in, Indicates the mean of the annotation results of the i-th sample from other perspectives except the current perspective. It indicates the result of the i-th sample being labeled by the emotion labeling unit in the feature space of the current perspective; The three emotion annotation units are subjected to multi-view collaborative learning, including the inter-view consistency constraint part and the Shannon entropy adaptive weighting part. The specific formula is: Among them, w r represents the weight of the rth sentiment annotation unit, is the inter-view consistency constraint part, It is the adaptive weighted part of Shannon entropy.

6. A storage medium, It is characterized in that The storage medium stores instructions, and when a computer reads the instructions, the computer executes the fabric emotion labeling method based on a multi-view stacked deep TSK fuzzy system as described in any one of claims 1 to 5.

7. An electronic device, It is characterized in that The invention comprises a processor and the storage medium as claimed in claim 6, wherein the processor executes instructions in the storage medium.