Class-level relationship constraint and structured graph enhancement method, system, medium and terminal

Abstract

The present invention belongs to the field of image classification technology, and specifically relates to a class-level relationship constraint and structured graph enhancement method, system, medium and terminal. Clustering is used to construct the visual representation distribution of the data set image, and a hierarchical structured relationship is constructed based on the distribution information; the typical batch construction process is extended under the guidance of the relationship, and the associated knowledge is hierarchically aggregated through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner; a sampling strategy based on visual distribution is designed to construct intra-class and inter-class constraints, and at the same time, the course construction and class-level distribution regularization operations are combined to further explicitly enhance the visual representation. Finally, the enhanced visual representation is used for classification tasks, thereby improving the visual classification effect. This innovative method provides an effective solution to the visual classification problem by deeply mining and utilizing the distribution information of the visual representation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a class-level relationship constraint and structured graph enhancement method, system, medium and terminal. Background Art

[0002] Visual classification has received increasing attention in academia and has been widely used in large-scale models. However, effectively encoding the diversity of visual data and extracting discriminative features remains a challenging task. Current research tends to use methods such as visual cues, cross-modal enhancement, and complex learning theory guidance to address this problem. However, these methods often rely on data-driven techniques and are susceptible to feature shifts caused by data heterogeneity and data bias. Therefore, how to extract discriminative features in a more data-independent way to improve the accuracy of visual classification has become an urgent problem that needs to be solved.

[0003] Existing methods usually only use instance-level information to enhance representation or use complex theories to improve deep models. They do not deeply explore the distribution of representations in the visual space and the structured relationship between visual representations, and are unable to construct reasonable and effective constraints. They have certain defects and shortcomings in dealing with the problems that need to be solved in this field. Summary of the invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention provides a class-level relationship constraint and structured graph enhancement method, system, medium and terminal, which significantly improve the image classification effect of the existing method when facing complex image data.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] In a first aspect, the present invention provides a class-level relationship constraint and structured graph enhancement method, comprising:

[0007] Step 1: Mining visual distribution information based on the distribution of image visual representation to form hierarchical structured association knowledge of instance-VP-CP;

[0008] Step 2: Based on the output associated knowledge information, the typical batch construction process is extended, and the associated knowledge is hierarchically fused through graph smoothing operations to enhance the model's representation learning ability in a guided knowledge manner;

[0009] Step 3: Design a sampling strategy based on visual distribution, construct intra-class and inter-class constraints, and combine class-level regularization operations to further explicitly enhance visual representation.

[0010] In a typical implementation, the specific process of step 1 is: mapping the image into a visual representation, using a clustering algorithm to construct the distribution of the visual representation of the image in the data set, obtaining the visual pattern (VP) and class pattern (CP) by mining the visual distribution information, and orderly forming the hierarchical structured association knowledge of instance-VP-CP.

[0011] Specifically, the Kmeans clustering algorithm is used to construct the feature space according to the clustering relationship of the samples and output the visual distribution structure , constructing the visual distribution pattern of the image, the distribution structure It is expressed as:

[0012] ,

[0013] in, Represents visual mapping, that is, characterizing the extraction process and distribution structure , Indicates the number of clusters.

[0014] Furthermore, for cluster ,use Indicates that the cluster belongs to The number of images of the class, represents the number of all images in the cluster, so the cluster About Class The purity of Indicates that the cluster The maximum value of the purity relative to each category is the cluster According to the representativeness of the cluster, it can be divided into the dominance of the cluster. For the training data, the corresponding VP is the prototype representation of the cluster to which it belongs, and the corresponding CP is the dominant cluster of all the categories in which it belongs. Purity It can be expressed as:

[0015] ,

[0016] in, Representation Cluster The middle category is The number of images, Representation Cluster The sum of the number of all images in , represents the dominant cluster, represents a mixed cluster, Indicates the set threshold.

[0017] for , selected from 0.5 to 0.55 on the CIFAR100 and Vireo172 datasets, and from 0.55 to 0.65 for the CIFAR10 dataset.

[0018] For each dominant cluster , the cluster belongs to the category The image representation constitutes a set ,in , calculate the central mode of the images in the collection As The corresponding visual pattern. , all by The dominant in-cluster category is The image representation constitutes a set , calculate the central mode of all image representations in the collection As a class pattern, the expression is:

[0019] ,

[0020] in, represents the function for computing the central mode, Represents a visual mapping.

[0021] In a typical implementation, the specific process of step 2 is: based on the instance-VP-CP structured association knowledge constructed in step 1, guide the input image The corresponding VP 、 Information fusion between CPs to generate enhanced feature representation; set input image representation Related to it adjacent, and Related to it Adjacency, so the adjacency matrix can be constructed , then the information fusion process can be expressed as:

[0022] ,

[0023] in, represents the splicing process of representation, represents the hierarchical graph enhancement process, represents the concatenated vector, Represents a visual mapping, represents the process of computing visual patterns, Represents the input layer-hidden layer (with feature maps), Represents the weight matrix between the hidden layer and the output layer. represents the vector obtained by graph smoothing, softmax represents the normalized exponential function, which is a specific activation function. represents the adjacency matrix and ReLU represents the activation function.

[0024] Furthermore, in step 2, for cluster The training set images in , if its cluster If it is dominant, Corresponding Cluster The corresponding visual pattern is: If cluster Not dominant, calculation and Represents the closest visual pattern As Corresponding , expressed as:

[0025] ,

[0026] in, represents the function for calculating distance, Indicates selection , Indicates ascending order. express The category to which it belongs, express The cluster to which it belongs. represents the dominant cluster, represents a mixed cluster; represents the process of computing class patterns, Represents the process of computing visual patterns.

[0027] Furthermore, all the visual patterns and class patterns calculated in the training set form a knowledge base, and the calculation and The representation distance is closest to the three visual patterns and three class patterns:

[0028] ,

[0029] Get the image Corresponding and After that, a structured association can be constructed , used in subsequent characterization enhancement modules.

[0030] In a typical implementation, in step 3, a representative weighted dominant pattern positive sampling strategy and a visually similar pattern negative sampling strategy are used to construct intra-class and inter-class constraints, and curriculum construction and class-level regularization constraints are combined to enhance visual representation.

[0031] Furthermore, the representative weighted dominant mode positive sampling strategy is specifically as follows: first, calculate each category The purity of the dominant cluster, constituting the set At the same time, the distance between each dominant cluster and other clusters is calculated, which represents the distance between the cluster and the edge of the class. The positive sample sampling process can be expressed as:

[0032] ,

[0033] in, Indicates descending order. represents the function for calculating distance, Indicates bitwise comparison. represents the sampling process of positive samples, Represents the positive sample pool.

[0034] Select distance center mode Recent samples as positive sample set , this process can be expressed as:

[0035] ,

[0036] in, Represents a visual mapping, Indicates the index. Indicates selection .

[0037] The visually similar negative sampling strategy is specifically as follows: the negative sampling pool selection process is expressed as:

[0038] ,

[0039] in, represents the negative sample pool, express The cluster where express The category to which it belongs.

[0040] Input image at distance The nearest cluster Sampling and The nearest This process is expressed as:

[0041] ,

[0042] in, Representative cluster The The representation of an image, Represents descending order.

[0043] In a typical implementation, in step 3, the course construction method measures the difficulty of sample representation learning by quantitatively estimating the representativeness of the clusters. Specifically, for clusters ,use Indicates that the cluster belongs to The number of images of the class, represents the number of all images in the cluster, so the cluster About Class Representative use Then, for the The input image , whose difficulty level is mapped as:

[0044] ,

[0045] in, Indicates a threshold value that is set. Indicates easy samples, Represents hard samples.

[0046] Further, for , selected from 0.75 to 0.85 on the CIFAR100 and Vireo172 datasets, and from 0.85 to 0.95 for the CIFAR10 dataset.

[0047] After that, the “attenuation method” is used to adjust the representation learning of the sample. Specifically, different penalty coefficients are set for each sample, and the difficulty level is easy The image penalty coefficient is , the difficulty is hard The image penalty coefficient is , and use the coefficient To adjust the penalty coefficient:

[0048] ,

[0049] Based on experience The initial value is close to 1. When the loss converges to less than 0.01 during training and the average difference in loss between two consecutive iterations is less than 0.0001 (that is, when the training converges), it starts to decrease gradually. .

[0050] In a typical implementation, in step 3, the constraint method includes intra-class constraints, inter-class constraints, and class-level regularization constraints.

[0051] Furthermore, the intra-class constraint is specifically as follows: first, sample from the positive sample pool p samples, and then in the input image Construct loss between its corresponding positive sample feature representation , explicitly constraining the distance between feature representations:

[0052] ,

[0053] in, Represents a visual mapping, P and Q represent images respectively Visual representation, image Its corresponding visual representation of the positive sample, represents the selection of p positive samples, Represents the positive sample pool.

[0054] The inter-class constraint is specifically as follows: first, sample from the negative sample pool samples, and then in the input image Construct loss between its corresponding negative sample feature representation , the distance between explicit constraint feature representations is expressed as:

[0055] ,

[0056] in, Represents a visual mapping, represents a fixed coefficient, and Represent the number of images and the number of negative samples respectively, Indicates the selection of n negative samples, Represents the negative sample pool.

[0057] The class-level regularization constraint is specifically as follows: First, the prototype representation of each class in the batch is enhanced. , and then construct constraints between the prototype representations of each different analogy. The process is as follows:

[0058] ,

[0059] in, represents the process of computing prototype representation, Indicates category After the representation of class-level relationship constraints, is a fixed parameter.

[0060] In a second aspect, the present invention provides a class-level relationship constraint and structured graph enhancement system, comprising:

[0061] A visual distribution mining module is configured to mine visual distribution information based on the distribution of visual representations of the image to form hierarchical structured association knowledge of instance-VP-CP;

[0062] The structured graph enhancement module is configured to extend the typical batch construction process based on the output contextual knowledge information and fuse the contextual knowledge through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner;

[0063] The distribution-guided relation constraint module is configured to design a sampling strategy based on visual distribution, construct intra-class and inter-class constraints, and combine class-level regularization operations to further explicitly enhance the visual representation.

[0064] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned class-level relationship constraint and structured graph enhancement method.

[0065] In a fourth aspect, the present invention provides a terminal comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the above-mentioned class-level relationship constraint and structured graph enhancement method.

[0066] In a typical implementation, the terminal includes a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, and a smart watch.

[0067] The present invention achieves the following technical effects:

[0068] The present invention proposes a distribution-guided class-level relationship constraint and structured graph enhancement method, called Drc-Hge, which aims to build class-level constraints to explicitly enhance visual representation through curriculum learning strategies. This method is based on the distribution of visual representation, mines hierarchical association knowledge, and uses graph enhancement to fuse knowledge to improve the representation learning ability of the model, thereby improving the visual classification effect. In previous studies, the inventors have made certain progress in class-level knowledge representation, mainly by building a variety of sampling strategies and curriculum learning strategies to establish class-level constraints. Under the guidance of class-level relationships, feature smoothing, class prototype alignment and class-level constraint explicit regularization are adopted to significantly alleviate the complexity of visual representation distribution. In the invention, the guidance of visual distribution is further introduced to deeply mine the distribution of visual representation, and a more reasonable and effective sampling strategy is designed to construct class-level constraints. Based on the visual distribution, structured hierarchical relationships are mined, and multi-level pattern alignment is completed using graph smoothing operations, effectively integrating hierarchical relationships, thereby further enhancing visual representation. Specifically, the present invention includes three modules: Visual Distribution Mining: In the offline stage, this module uses a clustering algorithm to construct the distribution of visual representations of images in the data set, and calculates the visual pattern and class pattern based on the visual distribution information to form a hierarchical structured association knowledge of instance-visual pattern-class pattern. Hierarchical Graph Enhancement: In the training stage, this module receives the association knowledge information output by the visual distribution mining module, expands the typical batch construction process, and hierarchically fuses the association knowledge through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner. Distribution-guided Relation Constraints: Based on the hierarchical graph enhancement module, a sampling strategy based on visual distribution is designed to construct intra-class and inter-class constraints, and at the same time, combined with class-level distribution regularization operations, the visual representation is further explicitly enhanced. Finally, the enhanced visual representation is used for classification tasks, thereby improving the visual classification effect. This innovative method provides an effective solution to the visual classification problem by deeply mining and utilizing the distribution information of visual representations.

[0069] It is worth emphasizing that the distribution-guided class-level relationship constraint and structured graph enhancement method proposed in the present invention is highly versatile and can be easily integrated into various algorithmic frameworks. Experiments were conducted on three datasets, including performance comparisons, ablation experiments of key parts, in-depth analysis of sampling strategies, and case studies to comprehensively evaluate the method. The experimental results show that Drc-Hge can better construct class-level representation constraints under the guidance of knowledge and enhance visual representation, thereby continuously improving classification accuracy. The present invention mines and models class-level relationships based on the visual distribution of image representation to enhance the representation learning effect, thereby extracting discriminative features and alleviating the difficulty of mapping diverse features to the same category. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the class-level relationship constraint and structured graph enhancement method in Embodiment 1 of the present invention;

[0071] Figure 2 Schematic diagram of a strategy for alleviating intra-class diversity and inter-class similarity of visual patterns in Embodiment 1 of the present invention;

[0072] Figure 3 This is a schematic diagram of the class-level structural relationship in Example 1 of the present invention;

[0073] Figure 4 This is a schematic diagram of the visualization of features extracted by ResNet50 and Drc-Hge in Example 1 of the present invention;

[0074] Figure 5 A schematic diagram of correcting errors in the class-level relationship constraint and structured graph enhancement method in Example 1 of the present invention. DETAILED DESCRIPTION

[0075] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0076] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0077] The present invention is further described below in conjunction with embodiments.

[0078] Example 1

[0079] This embodiment provides a class-level relationship constraint and structured graph enhancement method. This embodiment uses the method applied to a server as an example. It is understandable that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0080] like Figure 1-3 As shown in the figure, the framework of this method includes three main modules: Visual Distribution Mining Module: In the offline stage, this module uses clustering algorithm to construct the distribution of visual representation of images in the dataset, and calculates visual pattern (VP) and class pattern (CP) according to the visual distribution information, and orderly forms the hierarchical structured association knowledge of instance-VP-CP. Hierarchical Graph Enhancement Module: In the training stage, this module receives the association knowledge information output by the Visual Distribution Mining Module, expands the typical batch construction process, and hierarchically fuses the association knowledge through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner. Distribution-guided Relation Constraints Module: Based on the Hierarchical Graph Enhancement Module, a sampling strategy based on visual distribution is designed to construct intra-class and inter-class constraints, and at the same time, combined with class-level regularization operations, the visual representation is further explicitly enhanced. Finally, the enhanced visual representation is used for classification tasks, thereby improving the visual classification effect.

[0081] Specifically, the distribution-guided class-level relational constraints and structured graph enhancement method (Drc-Hge) consists of three steps:

[0082] Step 1: Use vision algorithms to train images As input, mapped to visual representation ,in is the number of images in the training set. Construct feature space based on the clustering relationship of samples, use clustering algorithm to output visual distribution . Calculate the relational knowledge based on the distribution information and output the visual pattern ( ) and class pattern ( );

[0083] Step 2: Based on visual distribution C and relational knowledge , , calculate the structured association relationship and output: . Using graph smoothing algorithms under the guidance of structured relationships and The level of integration;

[0084] Step 3: Guided by the visual distribution, positive and negative samples are sampled through a diversified strategy to construct intra-class and inter-class constraints, and the visual representation is explicitly enhanced by combining curriculum construction and class-level regularization.

[0085] The three modules are explained in detail below.

[0086] 1. Distribution Mining Module

[0087] In order to construct the structured relationship between the image representations of the dataset and guide the design of the sampling strategy and the construction of the class-level constraints, Drc-Hge uses a clustering algorithm in the distribution mining module to identify the clustering relationship of the image representation in the visual space, and then obtain the distribution pattern of the image visual representation. Then, the visual pattern and class pattern are calculated based on the distribution of the visual representation.

[0088] 1. Visual distribution model construction

[0089] Given a size of N Dataset , the K-means clustering algorithm is used to construct the visual distribution pattern of the feature space based on the visual representation of the image. Specifically, firstly, the images in the dataset are Input the visual mapping model to obtain its visual representation , after performing this operation on all images in the dataset, we can get a set of visual representations of the images The visual representation of each image is actually a vector in a high-dimensional space, so the set In fact, a feature space has been formed, but the distribution of this space is messy and unknowable. The clustering algorithm is used to construct its distribution pattern for the next step of mining related knowledge and guiding the design of sampling strategies. This process can be expressed as:

[0090] ,

[0091] in, represents the image collection in the dataset, Represents a visual mapping, , Indicates the number of clusters. In the experiment, a pre-trained visual model is used as the visual mapping , extracting the visual representation of the image to ensure the quality of the prior knowledge. Afterwards, the obtained distribution structure , visual representation collection and tag collection It will be used as input for associative knowledge mining.

[0092] 2. Distribution-guided knowledge mining

[0093] For each image A corresponding relationship can be obtained by matching with cluster and class information , the distribution structure has been obtained before , visual representation collection and tag collection Based on this information, we will mine and build structured related knowledge. , the sum of all images in the cluster is recorded as , the clusters belonging to the category The number of images is expressed as Similarly, we can get the sequence of the number of images belonging to all categories in the cluster , is the number of categories. Purity Expressed as The maximum value in and The ratio of Representative cluster Belongs to category of images), The size of the cluster can be used as a criterion to determine Is the dominant cluster or mixed cluster:

[0094] ,

[0095] in, Representation Cluster The middle category is The number of images, Representation Cluster The sum of the number of all images in , represents the dominant cluster, represents a mixed cluster, Indicates the set threshold.

[0096] For each dominant cluster , the cluster belongs to the category The image representation constitutes a set ,in , calculate the central mode of the images in the collection As The corresponding visual pattern. , all by The dominant in-cluster category is The image representation constitutes a set , calculate the central mode of all image representations in the collection As a class pattern, the expression is:

[0097] ,

[0098] in, represents the function for computing the central mode, Represents a visual mapping.

[0099] 2. Structured Graph Enhancement

[0100] The structured image enhancement module takes relational knowledge and visual representation as input, builds structured associative knowledge, and uses graph smoothing methods to hierarchically fuse them to enhance the visual representation.

[0101] 1.Structured relationship building

[0102] For the images in the training set , its corresponding The calculation is simple, Category classpattern: That is Corresponding .and The corresponding visual pattern is relatively complex. The training set images in , first determine its cluster Is it dominant? If so, Corresponding Cluster The corresponding visual pattern is: ; If the cluster Not dominant, calculation and Characterize the closest visual pattern As Corresponding , expressed as:

[0103] ,

[0104] in, represents the function for calculating distance, Indicates selection , Indicates ascending order. express The category to which it belongs, express The cluster to which it belongs. represents the dominant cluster, represents a mixed cluster; represents the process of computing class patterns, Represents the process of computing visual patterns.

[0105] For the images in the test set Since no prior information can be obtained, the way to construct a structured relationship is to calculate the distance, and all the visual patterns and class patterns calculated in the training set form a knowledge base, and calculate the distance between them. The representation of the three visual patterns and three class patterns closest to each other is expressed as:

[0106] ,

[0107] in, represents the function for calculating distance, Indicates selection ; Get the image Corresponding and After that, a structured association can be constructed , used in subsequent characterization enhancement modules.

[0108] 2. Relationship-guided graph smoothing

[0109] The image has been obtained above Corresponding and , HGE builds hierarchical relationships and uses the characteristics of graph convolution to perform hierarchical fusion and enhance visual representation. Specifically, let Representing images The visual representation of express The corresponding visual pattern, express The corresponding classpattern, represents the concatenation of three visual information. To fuse the representation information in a hierarchical form, it is necessary to set and Adjacent, and Adjacent, from this we can get the adjacency matrix:

[0110] , then the process of hierarchical graph enhancement can be expressed as:

[0111] ,

[0112] in, represents the splicing process of representation, represents the hierarchical graph enhancement process, represents the concatenated vector, Represents a visual mapping, represents the process of computing visual patterns, Represents the input layer-hidden layer (with feature maps), Represents the weight matrix between the hidden layer and the output layer. represents the vector obtained by graph smoothing, softmax represents the normalized exponential function, which is a specific activation function. represents the adjacency matrix and ReLU represents the activation function.

[0113] 3. Distribution-guided relationship constraint module

[0114] To explicitly alleviate the complexity of visual patterns, including intra-class diversity and inter-class complexity, this module constructs class-level constraints through diversified sampling strategies, effectively enhancing the distribution of visual representations and regularizing representations.

[0115] 1. In-class sample construction

[0116] In order to alleviate the intra-class diversity of representation distribution, a variety of positive sample sampling strategies are designed based on the visual distribution pattern, and intra-class relationship constraints are constructed to enhance visual representation. The sampling process of positive samples is divided into two key steps: 1. Select a positive sample pool from the entire dataset; 2. Sampling positive samples in the positive sample pool.

[0117] (1) Positive sample selection method

[0118] Based on the distribution of samples, the most representative cluster in the class is selected as the positive sample pool. For each cluster, the purity of the cluster and the distribution of the cluster are used as the criterion to determine its representativeness.

[0119] (a) Cluster purity: Clusters with high purity are more representative within the class. Purity The calculation method for the category , the purity of all clusters dominated by this class can form a set ;

[0120] (b) Cluster distribution: The closer the cluster is to the “edge of the class”, the more representative it is within the class. Dominant Cluster , the distance between its center mode and all clusters dominated by other categories is of interest, which represents the cluster The inter-class similarity between the representations in and the representations of other categories. Dominant Cluster , we can get a two-dimensional distance sequence .pass and By comparing the purity of the clusters and the distribution close to the "class edge", we can filter out the clusters with high purity and close to the "class edge" and determine such clusters as categories. Representative clusters of The positive sample pool is sampled from the image during training. This process can be expressed as:

[0121] ,

[0122] in, Indicates descending order. represents the function for calculating distance, Indicates bitwise comparison. represents the sampling process of positive samples, Represents the positive sample pool.

[0123] (2) Sampling in the positive sample pool

[0124] After selecting the positive sample pool, based on the distribution of samples, samples close to the center of the sample pool are sampled as positive samples. Specifically, for the input image , follow the process described in (1) to obtain the corresponding positive sample pool For the positive sample pool The central mode can be calculated , and then get each sample in the sample pool and the central mode Next, select the distance center mode The most recent 100 samples are used as positive sample sets , this process can be expressed as:

[0125] ,

[0126] in, represents the function for computing the central mode, Represents a visual mapping, Indicates the index. Indicates selection .

[0127] 2. Construction of inter-class samples

[0128] Based on the visual distribution, a variety of negative sample sampling strategies are designed, and inter-class relationship constraints are constructed to enhance visual representation, further alleviating the inter-class similarity of visual pattern distribution. Similar to the sampling process of positive samples, the sampling process of negative samples is also divided into two steps: 1. Negative sample pool selection process; 2. Sampling within the negative sample pool.

[0129] (1) Negative sample pool selection method

[0130] Compared with intra-class samples, the construction of inter-class samples is relatively simple. Based on the visual distribution, images with high similarity between classes are selected as negative sample pools. Specifically, the distribution pattern obtained above In the example, images belonging to the same cluster have similar visual representations. Therefore, the cluster where the distribution is selected is similar to the input image. The same images but belonging to different categories are used as negative sample pools. For the input image In general, this process can be expressed as:

[0131] ,

[0132] in, represents the negative sample pool, express The cluster where express The category to which it belongs.

[0133] (2) Sampling in the negative sample pool

[0134] After selecting the negative sample pool, based on the distribution of samples, samples closest to the input image are sampled as the negative sample set. Specifically, for the input image , the corresponding negative sample pool can be obtained by the process in (1) , and then we can find the visual representation and Next, select The samples with the closest visual representation distance are taken as the negative sample set. , this process can be expressed as:

[0135] ,

[0136] in, Representation Cluster The The representation of an image, Indicates descending order. Represents the input image visual representation of .

[0137] 3. Class-level relationship constraints

[0138] After obtaining the positive sample set and the negative sample set, Drc-Hge constructs intra-class constraints and inter-class constraints to alleviate the intra-class diversity and inter-class similarity, explicitly regularizes the visual distribution, and thus enhances the visual representation.

[0139] (1) Intra-class constraints

[0140] In the previous article, we have obtained the input image The corresponding positive sample set , in the training phase, for the input image , randomly sample from the corresponding positive sample set samples as positive samples and construct intra-class relationship constraints. Specifically:

[0141] ,

[0142] in, Represents a visual mapping, represents the selection of P positive samples, Represents intra-class relationship constraints.

[0143] (2) Inter-class constraints

[0144] In the previous article, we have obtained a negative sample set. , in the training phase, for the input image , sampled from the corresponding negative sample set samples as negative samples to construct inter-class relationship constraints. Specifically:

[0145] ,

[0146] in, Represents a visual mapping, Represents the relationship constraints between classes.

[0147] (3) Class-level distribution regularization

[0148] let The visual representation of the enhanced image is further regularized using the Loss constraint after class-level relationship constraints are performed through the above operations. Specifically, during the training process, the prototype representation of the enhanced representation of each category image at the batch-level is calculated. , and then build constraints between prototype representations of different categories. The process can be expressed as:

[0149] ,

[0150] in, represents the process of calculating the prototype, Indicates category After the representation of class-level relationship constraints, is a fixed parameter.

[0151] (4) Curriculum Construction

[0152] Since the representations belonging to the same cluster are visually similar to each other during the clustering process, it can be considered that when the number of representations of a specific category in a cluster accounts for a very high proportion, the remaining representations in the cluster are hard samples relative to the category to which they belong. For example: suppose the cluster The number of visual representations belonging to category 0 accounts for a large proportion of the total number of images in the cluster, and the remaining visual representations in the cluster all belong to category 1. Representation Cluster The set of representations belonging to category 0, Representation Cluster The set of representations belonging to category 1. Internal representation and The representations within are visually highly similar, and when classifying The representation in is more likely to be classified as category 0 than category 1. That is to say There are extremely high inter-class similarities and intra-class differences in the representations within. The image corresponding to the representation in is a hard sample relative to category 1. Specifically, through the function To determine the hard sample. Belongs to category Image , the expression is:

[0153] ,

[0154] in, Indicates a threshold value that is set. Indicates easy samples, represents hardsamples, Representation Cluster Belong to The number of images, Representation Cluster The number of all images in the .

[0155] After obtaining the hard sample judgment results, Drc-Hge uses a "Decay Method" to adjust the representation learning process. Specifically, different penalty coefficients are set for images during training: For hard samples, For easysample, Used to adjust the penalty coefficient, the adjustment method is defined as:

[0156] ,

[0157] According to experience, during initial training, The initial value is close to 1. When the loss converges to less than 0.01 during training and the average difference in loss between two consecutive iterations is less than 0.0001 (that is, when the training converges), it starts to decrease gradually. .

[0158] In order to verify the effectiveness of the algorithm, the present invention uses three data sets in the experiment, including CIFAR10, CIFAR100 and the challenging data set VireoFood172. The statistical information is shown in Table 1.

[0159] Table 1 Statistics of the datasets used in the experiment

[0160] .

[0161] When evaluating classification performance, Top-1 and Top-5 accuracy were selected as the main indicators. These two indicators are widely used to evaluate the classification accuracy of models on different data sets. When comparing model performance on these data sets, the present invention aims to fully understand and analyze the differences in model performance under different data backgrounds. The accuracy expression is:

[0162] .

[0163] In addition to the quantitative evaluation of classification performance, the qualitative analysis of representation enhancement is also emphasized. The inventors chose the T-SNE method as a tool because it can effectively map data points from a high-dimensional feature space to a two-dimensional or three-dimensional space, thereby visualizing the distribution of data in the feature space. Through this method, the impact of the Drc-Hge method on the distribution of visual representations can be intuitively observed, further verifying its superiority in constraining and enhancing the distribution of representations. This qualitative analysis not only helps to gain a deeper understanding of the model's performance in data representation, but also provides a more comprehensive evaluation, thereby better guiding subsequent model optimization and the selection of application scenarios.

[0164] The visual mapping encoder used by Drc-Hge in the process of obtaining visual representation can use all common visual models as backbone. In the present invention, four popular visual backbones are implemented: LeNet-5, ResNet18, ResNet50 and ViT, expressed as Drc-Hge(LeNet-5), Drc-Hge(ResNet18), Drc-Hge(ResNet50) and Drc-Hge(ViT). During training, SGD is fixedly used as the optimizer. For the hyperparameters used in the training process, specifically, batch-size is fixed to 64, the learning rate is selected from 5e-3 to 1e-1, the interval of lr-decay is selected from [30, 40, 60], lr-decay-rate is set to 0.1 or 0.5, and the training epoch is set to 100, 150 or 200.

[0165] The distance between the image and the positive and negative samples mentioned in the previous article can be measured using cosine similarity or Euclidean distance. The inventors conducted multiple experiments on the threshold used in the previous article. , CIFAR100 and Vireo172 datasets are selected from 0.5 to 0.55, and CIFAR10 dataset is selected from 0.55 to 0.65; , CIFAR100 and Vireo172 datasets are selected from 0.75 to 0.85, and CIFAR10 dataset is selected from 0.85 to 0.95. For the number of positive samples sampled And the number of negative samples sampled Choose from 5 to 25, as demonstrated below with an in-depth analysis of the experimental data.

[0166] 1. Performance comparison

[0167] The indicator is Top-1 / Top-5 accuracy (Acc).

[0168] This section introduces the experimental performance of Drc-Hge and various baseline algorithms in image classification. These algorithms are evaluated using widely used visual backbones, including LeNet5, ResNet18, ResNet50, ViT, and state-of-the-art algorithms such as TransNeXt and FASTERVIT based on ViT, as well as LGCOAMix, TST, and SGN combined with ResNet50. The hyperparameters of all algorithms are carefully tuned as described above to achieve the best performance. The parameter details of Drc-Hge have been described in detail above. For SGN, EMA is set to 0.9999, and for the CIFAR10 dataset Weight Parameters It is set to 0.999, and is set to 0.997 for CIFAR100 and Vireo172 datasets. For LGCOAMix, the number of superpixels mentioned in the paper is randomly selected from a uniform distribution, and the superpixel area is selected according to the Bernoulli probability with a probability of 0.5. When selecting the superpixel with the highest attention weight, the Top percentage is set to 70%. For FasterVit, given that the algorithm chooses ViT-S as the backbone, the inventors choose FasterVit-2, which has the same number of parameters as ViT-S, and the window size is set to 7*7. At the same time, the LAMB optimizer is used during training. This is shown in Table 2 below.

[0169] Table 2 Algorithm performance comparison

[0170] .

[0171] According to Table 2, the following can be observed:

[0172] 1. The proposed Drc-Hge method can continuously improve the generalization ability of visual skeletons. On datasets of different domains and sizes, Drc-Hge can combine various types of backbones, including convolution-based models and transform-based models, to improve classification performance;

[0173] 2. Drc-Hge has achieved more significant improvements when the category complexity is high. Most methods achieved better performance on the CIFAR-10 dataset with lower classification complexity, while on the virefood-172 dataset with a higher number of categories, Drc-Hge alleviated the intra-class diversity and inter-class similarity problems, and better improved the classification effect;

[0174] 3. Visual backbones with more complex structures show significant advantages on larger datasets. In the case of models with relatively basic architectures (such as LeNet5), the possibility of overfitting is higher when training datasets with more categories (such as CIFAR100) or datasets with more samples (such as Virefood-172 and NUS-WIDE). The performance gap is more obvious when compared with ResNet and ViT models.

[0175] 2. Ablation experiment

[0176] With ResNet50 and ViT as the backbone, an ablation study is conducted on Drc-Hge, and the results are shown in Table 3 below.

[0177] Table 3 Ablation experiment results

[0178] .

[0179] From the results in Table 3, we can conclude that:

[0180] 1. The intra-class and inter-class relationship constraints constructed by Drc-Hge can explicitly alleviate visual complexity and improve classification results: positive sampling based on base and constructing intra-class constraints ("+Sp"), and negative sampling based on the intra-class constraints and constructing inter-class constraints ("+Sn") can significantly improve classification accuracy. This is true for both datasets with two visual models as backbones;

[0181] 2. The course construction and the “attenuation strategy” used can further reduce the difficulty of representation learning and improve classification results: adding the “Cu” module can make the model converge faster and reduce the risk of overfitting by explicitly constraining difficult examples, thereby improving the model’s visual representation learning effect and achieving higher classification accuracy;

[0182] 3. Hierarchical fusion of visual pattern and class pattern can further alleviate the difficulty of mapping diverse representations to the same category and improve classification results: "+VP" and "+CP" can significantly improve the results of the model on all datasets. This is because the fusion pattern can effectively achieve the alignment of representations of the same category and alleviate the diversity of representations of the same category;

[0183] 4. Regularization constraints on inter-class representations can explicitly reduce the similarity of inter-class representations and improve classification results: The introduction of "CDR" further improves the classification results of the model. This is because by explicitly regularizing the representation distribution, the difference between representations of different categories can be explicitly increased, the representation learning effect can be improved, and thus better classification accuracy can be obtained.

[0184] 3. Analysis of the impact of different strategies on classification accuracy

[0185] Table 4 Ablation analysis results of different sampling strategies

[0186] .

[0187] The present invention evaluates the impact of different sampling strategies on classification results. Representative weighted dominant mode positive sampling can better alleviate intra-class differences: OC-Posi involves sampling from clusters outside the cluster of the input image itself, although the improvement in classification effect is slightly better than random sampling. SC-Posi refers to a stratified sampling strategy that divides clusters into three levels: high representativeness, medium representativeness, and mixed clusters, and then samples several samples from clusters at each level for constraint construction. This sampling strategy can obtain better classification results to a certain extent through the guidance of visual distribution information. RC-Posi refers to a dominant mode sampling strategy that determines the representativeness of each cluster according to the method described above, and samples from the most representative cluster to construct constraints. In RC-Posi-a, only positive samples are sampled without adding additional constraints; while in RC-Posi-b1, the loss function is weighted according to the representativeness of the cluster where the positive sample is located, and the higher the representativeness, the lower the weight; on the contrary, RC-Posi-b2 has a higher weight when the representativeness is higher when weighted. The results in Table 4 above show that RC-Posi-b2 can achieve the highest classification accuracy. This is because dominant mode sampling can better unify the representation learning of samples under the guidance of visual distribution information, reduce intra-class differences, and alleviate the diversity of similar representations. At the same time, by weighting the loss function in a targeted manner, it can effectively guide the representation learning process, thereby improving the classification accuracy.

[0188] Negative sampling guided by visual similarity patterns can better alleviate inter-class similarity: OC-Nega mentioned in the previous article involves sampling from clusters outside the cluster of the input image itself, which is slightly better than random sampling to improve classification results. NC-Nega refers to sampling from clusters of different categories closest to the input image in pure category clustering mode, and can achieve better classification results under the guidance of distribution information. SC-Nega is a sampling strategy based on visual similarity, that is, in mixed category clustering mode, sampling is performed from the cluster where the input image is located, and by sampling images of different categories that are most similar to the image, it can better alleviate inter-class similarity and achieve better classification results. Specifically, SC-Nega-1 refers to random sampling from the cluster where the image is located, while SC-Nega-2 refers to sampling the negative samples closest to the input image in the cluster where the image is located according to the distance. According to the table results, SC-Nega-2 can achieve the highest classification results when the distribution information is further guided.

[0189] 4. Analysis of the effect of relational modeling

[0190] This section aims to evaluate the impact of relational modeling on representation learning. Confusion matrix analysis shows that in the CIFAR-10 dataset, cats, dogs, and horses have high levels of misclassification. To further investigate this issue, the present invention uses 2D-t-SNE to visualize the image representations encoded by ResNet-50 and Drc-Hge, as shown in Figure 2. Figure 4 As shown. Figure 4 In (a), ResNet-50 shows good discrimination between the three classes. However, the representation distributions of different classes are mixed, while the representation distributions of the same class are widely scattered. In contrast, Figure 4 (b) shows that Drc-Hge effectively solves this problem by more tightly aggregating representations of the same class while separating representations of different classes. This provides strong evidence that the introduction of relational modeling significantly enhances the effect of representation learning.

[0191] 5. Analysis on the impact of different representative calculation methods on classification accuracy

[0192] P-DM refers to a purity-based determination method, where "purity" is defined as the ratio of the number of images of a specific category within a cluster to the total number of images. The higher the ratio, the higher the purity of the image type within the cluster, and therefore the cluster is more representative of the category. PaD-DM is the abbreviation of the purity plus distribution determination method. In PaD-DM, "purity" still refers to the proportion of images of a specific category within a cluster, while "distribution" refers to the distribution position of the cluster described above, that is, the distance from the boundary of the category domain. The closer the distance, the more representative the cluster is.

[0193] Under the guidance of distribution information, PaD-DM can achieve better classification results. PaD-DM comprehensively considers both purity and distribution information to determine the representativeness of clusters relative to categories. The images in clusters close to the class domain boundary are more different from other clusters, so sampling from these clusters can more effectively avoid the problem of inter-class similarity, thereby improving the representation learning effect. This is shown in Table 5.

[0194] Table 5 Comparison of Drc-Hge performance using different representative calculation methods

[0195] .

[0196] (VI) Analysis on the impact of graph enhancement fusion elements on classification accuracy

[0197] Represents that for the test set image, select the k closest to it According to the data in Table 6 below, when selecting the three most recent The highest classification result can be achieved when , which is also the basis for the inventor to choose when conducting other experiments. During the training process, the graph enhancement module also integrates and positive samples ( ) and fusion only Compared with the classification accuracy, the accuracy is lower. The possible reason is that the fused information is too much and inconsistent, resulting in a decrease in the representation learning effect. At the same time, using Loss to constrain positive samples can achieve better classification results.

[0198] Table 6 The impact of different image enhancement fusion elements on classification accuracy

[0199] .

[0200] 7. Analysis on the impact of clustering parameters on classification accuracy

[0201] Table 7 Comparison of image classification performance of Drc-Hge under different clustering parameters

[0202] .

[0203] From Table 7 above, we can see that different clustering results have little effect on the final classification accuracy. Although different clustering results will affect the accuracy of image recognition, the impact is limited because adaptive clustering relies on visual commonalities and incorrect clustering accounts for only a small part. As the number of iterations in the training phase increases, these inaccuracies gradually decrease because the model learns more accurate feature representations and corrects incorrect clusters. Therefore, there is no need to pay too much attention to examples with poor clustering quality; instead, we should focus on improving the accuracy and generalization ability of feature expression to obtain better recognition results.

[0204] 8. Error analysis of Drc-Hge

[0205] This section details the 2Dt-SNE visualization and error analysis to verify the effectiveness of Drc-Hge in dealing with both intra-class diversity and inter-class similarity. Figure 5 Comparison of visual representations of four images generated by Drc-Hge and ResNet-50, along with their predicted components and confidence scores. Figure 5 In (a), the objects in the image are clearly visible and both models provide accurate predictions. However, at the predicted location of “bird”, Drc-Hge shows a higher confidence score. In addition, in the feature space, Drc-Hge Figure 5 The representation in (a) is placed closer to the center of the feature set for “bird”. Figure 5 In (b), the objects in the image are not clear enough and ResNet-50 often produces incorrect predictions. In contrast, Drc-Hge places the features closer to the center of the “airplane” feature set, resulting in more accurate predictions. Figure 5 In (c), although Drc-Hge produces incorrect predictions, the results generated by Drc-Hge are very similar to those of ResNet-50. In addition, the feature space generated by ResNet-50 is Figure 5 There is an unusually chaotic pattern around (c), so this could have happened by chance. Figure 5 In (d), none of the models can accurately predict the image because the objects in the image are too blurred and the pixel range is limited. Nevertheless, Drc-Hge places the features closer to the “airplane” representation set, resulting in a higher confidence score. These findings strongly support the effectiveness of Drc-Hge in improving representation learning by alleviating intra-class diversity and inter-class similarity.

[0206] Example 2

[0207] The present invention provides a class-level relationship constraint and structured graph enhancement system, comprising:

[0208] A visual distribution mining module is configured to mine visual distribution information based on the distribution of visual representations of the image to form hierarchical structured association knowledge of instance-VP-CP;

[0209] The structured graph enhancement module is configured to extend the typical batch construction process based on the output contextual knowledge information and fuse the contextual knowledge through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner;

[0210] The distribution-guided relation constraint module is configured to design a sampling strategy based on visual distribution, construct intra-class and inter-class constraints, and combine class-level regularization operations to further explicitly enhance the visual representation.

[0211] Example 3

[0212] The present invention provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned class-level relationship constraint and structured graph enhancement method.

[0213] Example 4

[0214] The present invention provides a terminal, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the above-mentioned class-level relationship constraint and structured graph enhancement method. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0215] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A class-level relationship constraint and structured graph enhancement method, characterized in that: include: Step 1: Mining visual distribution information based on the distribution of image visual representation to form hierarchical structured association knowledge of instance-VP-CP; Step 2: Based on the output related knowledge information, the typical batch construction process is extended, and the related knowledge is hierarchically fused through graph smoothing operations to enhance the model's representation learning ability in a guided knowledge manner; Step 3: Design a sampling strategy based on visual distribution, construct intra-class and inter-class constraints, and combine class-level regularization operations to further explicitly enhance visual representation; The specific process of step 1 is: map the image into a visual representation and use the clustering algorithm to output the visual distribution structure , visual pattern and class pattern are obtained by mining visual distribution information, and hierarchical structured association knowledge of instance-VP-CP is orderly formed. The distribution structure It is expressed as: ， in, Represents visual mapping, that is, characterizing the extraction process and distribution structure , Indicates the number of clusters; The specific process of step 2 is: based on the instance-VP-CP structured association knowledge constructed in step 1, guide the input image The corresponding VP 、 Information fusion between CPs to generate enhanced feature representation; set input image representation Related to it adjacent, and Related to it Adjacency, so the adjacency matrix can be constructed , then the information fusion process can be expressed as: ， in, represents the splicing process of representation, represents the hierarchical graph enhancement process, represents the concatenated vector, Represents a visual mapping, represents the process of computing visual patterns, represents the weight matrix between the input layer and the hidden layer, represents the weight matrix between the hidden layer and the output layer, represents the vector obtained by graph smoothing, softmax represents the normalized exponential function, which is a specific activation function. represents the adjacency matrix, ReLU represents the activation function; In step 3, a representative weighted dominant mode positive sampling strategy and a visual similarity mode negative sampling strategy are used to construct intra-class and inter-class constraints, and the visual representation is enhanced by combining curriculum construction and class-level regularization constraints; the positive sample sampling process in the representative weighted dominant mode positive sampling strategy is expressed as: ， in, Indicates descending order. represents the function for calculating distance, Indicates bitwise comparison. represents the sampling process of positive samples, represents the positive sample pool; The visually similar negative sampling strategy is specifically as follows: the negative sampling pool selection process is expressed as: ， in, represents the negative sample pool, express The cluster where express The category to which it belongs.

2. A class-level relationship constraint and structured graph enhancement method according to claim 1, characterized in that: cluster The maximum value of the purity relative to each category is the cluster The representativeness of the cluster can be divided into the dominance of the cluster according to its representativeness. For the training data, the corresponding VP is the prototype representation of the cluster to which it belongs, and the corresponding CP is the dominant cluster of all the categories in which it belongs. It can be expressed as: ， in, Representation Cluster The middle category is The number of images, Representation Cluster The sum of the number of all images in , represents the dominant cluster, represents a mixed cluster, Indicates the set threshold.

3. A class-level relationship constraint and structured graph enhancement method according to claim 2, characterized in that: For each dominant cluster , the cluster belongs to the category The image representation constitutes a set ,in , calculate the central mode of the images in the collection As Corresponding visual pattern; for category , all by The dominant in-cluster category is The image representation constitutes a set , calculate the central mode of all image representations in the collection As a class pattern, the expression is: ， in, represents the function for computing the central mode, Represents a visual mapping.

4. A class-level relationship constraint and structured graph enhancement method according to claim 1, characterized in that: The curriculum construction method measures the difficulty of learning sample representations by quantitatively estimating the representativeness of clusters. Specifically, for clusters ,use Indicates that the cluster belongs to The number of images of the class, represents the number of all images in the cluster, so the cluster About Class Representative use express; Then, for the class The input image , whose difficulty level is mapped as: ， in, Indicates a threshold value that is set. Indicates easy samples, represents hard samples; after that, the "attenuation method" is used to adjust the representation learning of the samples. Specifically, different penalty coefficients are set for each sample. The difficulty level is easy The image penalty coefficient is , the difficulty is hard The image penalty coefficient is , and use the coefficient To adjust the penalty coefficient: ， Based on experience The initial value is close to 1. When the loss converges to less than 0.01 during training and the average difference in loss between two consecutive iterations is less than 0.0001, it starts to decrease gradually. .

5. A class-level relationship constraint and structured graph enhancement system, characterized in that: The method for class-level relationship constraint and structured graph enhancement according to any one of claims 1 to 4 comprises: A visual distribution mining module is configured to mine visual distribution information based on the distribution of visual representations of the image to form hierarchical structured association knowledge of instance-VP-CP; The structured graph enhancement module is configured to extend the typical batch construction process based on the output associated knowledge information and fuse the associated knowledge through graph smoothing operations to enhance the representation learning ability of the model in a guided knowledge manner; The distribution-guided relation constraint module is configured to design a sampling strategy based on visual distribution, construct intra-class and inter-class constraints, and combine class-level regularization operations to further explicitly enhance the visual representation.

6. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the class-level relationship constraint and structured graph enhancement method described in any one of claims 1-4.

7. A terminal comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the class-level relationship constraint and structured graph enhancement method described in any one of claims 1-4.

Images