Image recognition method, system and device based on hierarchical prototype multi-view learning

Abstract

The application relates to an image recognition method, system and device based on hierarchical prototype multi-view learning. The method comprises the following steps: constructing a multi-layer prototype model; initializing the multi-layer prototype model; inputting a multi-view image sample set into the initialized multi-layer prototype model; constructing a target function according to a sample-prototype association relationship between original data in the multi-view image sample set and a bottom layer prototype matrix, a mapping relationship of adjacent layer prototype matrices and a structure constraint of a cross-layer consensus prototype graph; solving the target function; outputting an optimized sample-prototype similarity graph and a cross-layer consensus prototype graph; and fusing the optimized sample-prototype similarity graph and the cross-layer consensus prototype graph in the multi-layer prototype model and then obtaining a clustering result of the multi-view image sample set through spectral clustering processing. The method can effectively improve the accuracy and stability of image clustering and enhance the interpretability and reliability of the image recognition result.

Description

Technical Field

[0001] This application relates to the field of multi-view clustering technology, and in particular to an image recognition method, system and device based on hierarchical prototype multi-view learning. Background Technology

[0002] With the development of technologies in the fields of artificial intelligence perception and data mining, processing technologies for multi-view image data have gradually emerged. Among them, multi-view clustering technology has become one of the core technologies. Its core feature is that it can preserve the local differences of each view in multi-view image data (image information under different modalities, angles, and feature domains, such as color, texture, structure, etc.) while integrating the potential shared structural information of the data, thereby achieving a more accurate clustering representation. It has shown strong applicability potential in fields such as image recognition that have high requirements for multi-modal structural modeling.

[0003] In the development of multi-view clustering technology, the multi-view subspace clustering method has gained widespread attention. This method completes the clustering task by jointly learning the subspace representation across views and mining the correlation between views. It performs well in the structural representation of multi-view image data and has become the mainstream traditional method for processing multi-view image clustering tasks.

[0004] However, traditional multi-view subspace clustering methods suffer from significant inefficiencies and excessive resource consumption. When processing large-scale image data, this method faces a severe computational bottleneck: in the graph construction phase, it is necessary to calculate the pairwise similarity between all samples, resulting in both high space and time complexity. (M is the number of views, n is the number of samples); in the subsequent spectral clustering step, the time complexity of eigenvalue decomposition of the constructed graph is also O(n). This high demand for computing resources severely restricts the processing of large-scale, high-resolution image data in imaging scenarios, greatly limiting its practicality and scalability in real-world applications.

[0005] To address the aforementioned scalability challenges, prototype-based multi-view clustering methods have been proposed and gradually applied. These methods select a small number of "prototypes" (representative data points) to construct a bipartite graph structure between the prototypes and the original samples, thereby reducing the computational complexity of subspace learning from... Downgraded to ( , c Indicates the number of prototypes. n (Indicates the number of samples), which reduces computational overhead while preserving data structure information, thus improving image data processing efficiency to some extent.

[0006] However, prototype-based multi-view clustering methods still have significant shortcomings: On the one hand, their performance is highly dependent on the prototype selection strategy. Early heuristic strategies (such as random sampling and k-means selection) are prone to producing unstable or suboptimal prototype representations, making it difficult to effectively cover the semantic structure of image data. On the other hand, existing prototype-based methods mostly adopt a single-layer prototype structure, which has two major drawbacks: First, it is easy to cause uneven distribution or insufficient coverage of prototypes, resulting in weak discrimination ability for sparse and marginal classes, weakening the overall clustering performance, especially reducing the ability to express small samples and fine-grained categories. Second, the expressive ability is limited, and it is unable to capture the local details and global structure of image data at the same time. It is difficult to adapt to the complex fusion relationship of multi-scale structure and heterogeneous features in real images, resulting in the problems of accuracy bottleneck and insufficient generalization ability in real-world multi-view image recognition. The comprehensive requirements of multi-view image clustering in terms of efficiency, performance and generalization ability have not yet been completely solved. Summary of the Invention

[0007] Therefore, it is necessary to provide an image recognition method, system, and device based on hierarchical prototype multi-view learning to address the aforementioned technical problems.

[0008] An image recognition method based on hierarchical prototype multi-view learning, the method comprising:

[0009] Construct a multi-layered prototype model; the multi-layered prototype model includes a multi-layered prototype structure corresponding to each view, the multi-layered prototype structure including layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram;

[0010] Initialize the multi-layer prototype model by inputting the multi-view image sample set into the initialized multi-layer prototype model. Construct the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer.

[0011] Solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph;

[0012] The optimized sample-prototype similarity graph and cross-layer consensus prototype graph in the multi-layer prototype model are fused and then subjected to spectral clustering to obtain the clustering result of the multi-view image sample set.

[0013] An image recognition system based on hierarchical prototype multi-view learning, the system comprising:

[0014] The model building module is used to construct a multi-layer prototype model; the multi-layer prototype model includes a multi-layer prototype structure corresponding to each view, and the multi-layer prototype structure includes layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram;

[0015] The objective function construction module is used to initialize the multi-layer prototype model. It inputs the multi-view image sample set into the initialized multi-layer prototype model and constructs the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer.

[0016] The optimization and solution module is used to solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph.

[0017] The result output module is used to fuse the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model, and then obtain the clustering result of the multi-view image sample set through spectral clustering processing.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0019] Construct a multi-layered prototype model; the multi-layered prototype model includes a multi-layered prototype structure corresponding to each view, the multi-layered prototype structure including layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram;

[0020] Initialize the multi-layer prototype model by inputting the multi-view image sample set into the initialized multi-layer prototype model. Construct the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer.

[0021] Solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph;

[0022] The optimized sample-prototype similarity graph and cross-layer consensus prototype graph in the multi-layer prototype model are fused and then subjected to spectral clustering to obtain the clustering result of the multi-view image sample set.

[0023] The beneficial effects of this invention are as follows:

[0024] The aforementioned image recognition methods, systems, and devices based on hierarchical prototype multi-view learning, by constructing a multi-layer prototype structure with progressively abstracting layers from the bottom to the top and decreasing prototype numbers, can accurately and comprehensively represent high-dimensional image data from fine-grained features such as local textures and edges to semantic-level global patterns, greatly enhancing the structural modeling capability of image data in multi-view space. Simultaneously, by constructing an objective function based on the sample-prototype association between the original data and the bottom-level prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of cross-layer consensus prototype diagrams, the model's associations and constraints can be accurately quantified, providing a clear direction for subsequent optimization and further improving the model's adaptability and integration capability for multi-modal image features. Utilizing a learnable cross-layer guidance mechanism, the model automatically adjusts the prototype consistency between views at each layer to address the alignment challenges of different modal features in images, effectively improving the accuracy and stability of image clustering. Furthermore, the method of this invention can be flexibly adapted to image datasets of different sizes and dimensions, and has good robustness to conditions such as density imbalance and class distribution imbalance. It is suitable for a variety of image application scenarios, and can also provide layer-by-layer visualization and structural interpretation of clustering results through hierarchical prototype structure and cross-layer consistency propagation mechanism, thereby enhancing the interpretability and credibility of image recognition results. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating an image recognition method based on hierarchical prototype multi-view learning in one embodiment.

[0026] Figure 2 This is a schematic diagram of a hierarchical prototype multi-view learning framework in one embodiment;

[0027] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] To address the aforementioned problems, this invention proposes an image recognition method based on hierarchical prototype multi-view learning. This method aims to establish a coarse-to-fine prototype representation from top to bottom by introducing a multi-level prototype structure, thereby achieving layer-by-layer modeling of multi-scale features in image data; based on multi-view... Figure 1 The coherence learning mechanism jointly models the shared structure and differential representations among prototypes, improving the alignment capability for multimodal data. It balances structural expressiveness and computational efficiency, enhancing model discriminability while avoiding the resource overhead of high-order fully connected computations. This invention not only improves the fusion representation capability between views but also enhances the interpretability of the semantic structure of image data and clustering performance, making it particularly suitable for multi-view image recognition applications and possessing broad engineering application potential.

[0030] In one embodiment, such as Figure 1 As shown, an image recognition method based on hierarchical prototype multi-view learning is provided, including the following steps:

[0031] Step 102: Construct a multi-layer prototype model.

[0032] The multi-layer prototype model includes a multi-layer prototype structure corresponding to each view. The multi-layer prototype structure includes layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding bottom-level prototype matrix, and constructing a cross-view shared sample-prototype similarity map based on the association between samples and prototypes. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First A prototype diagram of cross-layer consensus. (e.g.) Figure 2 As shown, a schematic diagram of a hierarchical prototype multi-view learning framework is provided, in which the view-specific prototype and its consistency prototype graph are jointly learned in each layer. To maintain semantic consistency across layers, this invention introduces a learnable cross-layer guidance mechanism through prototype-to-prototype consistency graphs. This design not only improves the consistency between different views but also enhances the interpretability of the hierarchical representation.

[0033] It is understandable that by processing data in layers, the features of multi-view imagery data are gradually abstracted and fused, thereby improving the ability to represent data across views, the interpretability of semantic structure, and clustering performance. In the first layer, across all views (e.g., View 1 to View...), ...M A shared sample-prototype similarity graph is constructed on top of the prototype. This step captures fine-grained local relationships between samples and prototypes, laying the foundation for subsequent layers of processing. It allows each view's raw data (Data Points) to initially establish a connection with the underlying prototype, preserving local details of the data. In subsequent layers (l≥2 layers), first, a view-specific prototype set is learned. Because different views capture information about the same target object in different modalities, angles, etc., learning view-specific prototypes better preserves the local differences between views. Then, a consensus graph is established between prototypes to explore the first... Layer Prototype and the First The relationships between layer prototypes are explored. Cross-layer connections explicitly encode higher-level structural dependencies, enabling information to propagate from bottom to top while maintaining semantic consistency at lower levels. Finally, learnable inter-layer mappings are introduced to progressively optimize prototype embeddings between adjacent layers and fuse multi-scale structural information. This helps integrate information from different levels and views, allowing prototypes to represent not only local details but also global structures. Through this hierarchical processing, on the one hand, the number of prototypes decreases layer by layer from the bottom to the top, achieving a hierarchical abstraction from fine-grained local features to coarse-grained global structures, improving the interpretability of the semantic structure of image data; on the other hand, the information interaction and fusion between layers and between views enhance the fusion representation capability between views, ultimately improving clustering performance.

[0034] Step 104: Initialize the multi-layer prototype model. Input the multi-view image sample set into the initialized multi-layer prototype model. Construct the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer.

[0035] By constructing an objective function that integrates multi-dimensional associations, the representational association between the original data and the underlying prototype, the semantic mapping between inter-layer prototypes, and the structural constraints of the prototype diagram are incorporated into a unified optimization framework. This ensures that the underlying prototype accurately captures the fine-grained features of the image, while the structural constraints guarantee the rationality of cross-layer prototype associations. This provides the model with an optimization direction that balances feature fidelity and semantic coherence, effectively improving its adaptability to multimodal heterogeneous features of images.

[0036] Step 106: Solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph.

[0037] By solving the objective function, iterative optimization of the prototype and graph structure is achieved, making the sample-prototype similarity graph more accurately reflect the relationship between the original image and the underlying prototype. The cross-layer consensus prototype graph more reliably transmits the semantics between layers. The final optimized output provides a high-quality feature association foundation for subsequent fusion, significantly enhancing the model's ability to characterize complex image feature distributions.

[0038] Step 108: The optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model are fused and then the clustering result of the multi-view image sample set is obtained by spectral clustering.

[0039] By fusing and optimizing the association graph and processing it through spectral clustering, the multi-level and multi-view feature associations are transformed into low-dimensional features that can be directly used for clustering. This not only preserves the complete semantic information from fine-grained to coarse-grained, but also achieves accurate classification of image samples through clustering algorithms, ultimately improving the accuracy and robustness of multi-view image recognition, while providing a traceable hierarchical interpretation basis for the results.

[0040] The image recognition method based on hierarchical prototype multi-view learning described above constructs a multi-layer prototype structure with progressively abstracting layers from the bottom to the top, and a decreasing number of prototypes. This enables precise and comprehensive representation of high-dimensional image data, from fine-grained features such as local textures and edges to semantic-level global patterns, greatly enhancing the structural modeling capability of image data in multi-view space. Simultaneously, by constructing an objective function based on the sample-prototype association between the original data and the bottom-level prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype graph, the method accurately quantifies the associations and constraints of each part of the model, providing a clear direction for subsequent optimization and further improving the model's adaptability and integration capability for multi-modal image features. Furthermore, by leveraging a learnable cross-layer guidance mechanism, the method automatically adjusts the prototype consistency between views at each layer to address the alignment challenges of different modal features in images, effectively improving the accuracy and stability of image clustering. Furthermore, the method of this invention can be flexibly adapted to image datasets of different sizes and dimensions, and has good robustness to conditions such as density imbalance and class distribution imbalance. It is suitable for a variety of image application scenarios, and can also provide layer-by-layer visualization and structural interpretation of clustering results through hierarchical prototype structure and cross-layer consistency propagation mechanism, thereby enhancing the interpretability and credibility of image recognition results.

[0041] In one embodiment, initializing a multi-layer prototype model includes: initializing the multi-layer prototype model according to the number of layers in a pre-set multi-layer prototype structure and the number of prototypes at each layer decreasing progressively.

[0042] In one embodiment, constructing a cross-view shared sample-prototype similarity graph includes: extracting the corresponding initial underlying prototypes for each view using a clustering method; constructing a similarity relationship between samples and initial underlying prototypes for each view, and forming a bipartite graph connection matrix from samples to prototypes using a non-negative normalization constraint; fusing the bipartite graph connection matrices of each view to obtain an initial sample-prototype similarity graph; iteratively optimizing the shared connection relationships of the initial sample-prototype similarity graph until a pre-set iteration stopping condition is met, at which point the iteration stops, and a cross-view shared sample-prototype similarity graph is obtained.

[0043] Specifically, for the input raw multi-view image data, a clustering method (such as k-means) is applied to each view to obtain an initial prototype set. For each view, a similarity relationship is constructed between the sample and the corresponding first-layer prototype, and a bipartite graph connection matrix from sample to prototype is formed using a non-negative normalization constraint. The connection matrices of all views are then fused into a shared initial sample-prototype similarity graph to capture fine-grained local semantic structures. This is to improve the discriminative power and cross-view similarity of the similarity graph. Figure 1 To further improve consistency, an optimization learning mechanism is introduced to iteratively update the shared connection relationships based on the prototype features of each view, ensuring semantic alignment and structural consensus of the prototype structures in different views.

[0044] In one embodiment, through the first Layer to the first The prototypes of each layer are extracted in a progressively decreasing manner to obtain the corresponding high-level prototype matrix, including: constructing a joint minimization objective function; the joint minimization objective function includes the first... Layer Prototype and the First Reconstruction loss term of the layer prototype, the first The loss term for fitting the high-level prototype to the sample features and the regularization term constraining the sparsity of the high-level prototype; the 1st Layer prototype matrix, sample feature matrix of multi-view image sample set and the first The layer prototype indicator matrix is ​​used as input to jointly minimize the objective function with respect to the th layer. Differentiating the prototype matrix of each layer and setting the derivative to zero yields a closed-form analytical solution. Based on the predetermined decreasing number of prototypes per layer, the solution for the 1st layer is determined. High-level prototype matrix.

[0045] Specifically, the first The prototype matrix of the layer, the sample feature matrix, and the prototype indicator matrix of the current layer are used as input variables. A joint minimization objective function is constructed, and a closed-form analytical solution is obtained by differentiating the objective function with respect to the prototype matrix of the current layer. The high-level prototype matrix is ​​the joint mapping result of multiple low-level information sources, which includes the semantic structure between views and the guidance relationship between layers. The number of prototypes decreases layer by layer, forming a hierarchical semantic space from fine-grained to abstract levels.

[0046] In one embodiment, construct the first Layer and First The cross-layer consensus prototype diagram includes: constructing a fusion objective function; the fusion objective function includes a same-layer structure preservation term, a cross-layer structure preservation term, and a regularization term, wherein the same-layer structure preservation term is calculated by... Euclidean distance quantization between layer prototypes, and cross-layer structure preservation terms are calculated by the first layer prototype. Layer prototype to the first The reconstruction error of the layer prototype is quantified, and the regularization term is an L2 regularization term that constrains the sparsity of elements in the cross-layer consensus prototype graph. Under the constraints of nonnegativity and row normalization, a quadratic programming method is used to optimize the fusion objective function to obtain the th... Layer and First Prototype diagram of cross-layer consensus.

[0047] Specifically, while maintaining the compactness of high-level semantics, semantic guidance from low-level prototypes is introduced to guide the generation of the current-level prototype structure; for the first... Each prototype node in the layer is minimized by... The reconstruction error of the layer prototype is investigated to establish inter-layer structural consistency. A fusion objective function is constructed, which includes: same-layer structure preservation term, cross-layer structure preservation term, and regularization term. Under the condition of ensuring the non-negativity of the cross-layer consensus prototype graph and row normalization constraint, the optimal cross-layer consensus graph structure is obtained by quadratic programming optimization.

[0048] This hierarchical organization offers two significant advantages. First, by progressively optimizing the prototype's position within the learned structural dependencies, it enhances the prototype's stability and mitigates the instability issues arising from fixed or heuristically sampled prototypes. Second, by jointly modeling local sample-level details with global prototype-level abstractions, it improves the prototype's representation quality and ensures alignment between low-level geometry and high-level semantic information.

[0049] In mathematics, data is organized hierarchically as L A set of layer prototypes, with the number of prototypes decreasing layer by layer. In the first layer, each view contains a data matrix. Depend on The prototype approximation. In each subsequent layer, the... The prototypes of the layers are clustered as This process involves creating higher-level prototypes, building cross-level prototype diagrams, and implementing structural abstractions. This process continues until the final level, resulting in a compact and high-level representation of the data structure.

[0050] In one embodiment, the objective function is:

[0051] (1)

[0052] in, Indicates the first t Data matrix of views, M Total number of views L This indicates the total number of layers in the multi-layered prototype structure. t For the index of the sample, Indicates the first t The prototype matrix of the first layer of views This represents a bipartite graph from the sample to the first-level prototype, for the... Layer and , Indicates the first t The prototype matrix of each view. Indicates the first Layer and First Cross-layer consensus prototype diagram It is a regularization parameter. For the first Number of prototype layers This indicates that all elements are 1. 3D column vector, It is the square of the norm. Specifically, , , , , ,in , Used to control the Layer Prototype Diagram Smoothness, n For the sample size, For feature dimensions.

[0053] In this embodiment, the first term in the optimization objective function is used to reconstruct the original input data from the first-level prototype; the second term is used to maintain consistency between adjacent prototype layers to promote hierarchical abstraction; and the third term is used to regularize the prototype graph to prevent overfitting.

[0054] In one embodiment, solving the objective function and outputting the optimized sample-prototype similarity graph and cross-layer consensus prototype graph includes: optimizing the objective function using an alternating optimization strategy, iteratively updating the cross-layer consensus prototype graph and view-specific prototype matrix of the multi-layer prototype model until the iteration stops when the iteration stopping condition is met, and outputting the optimized sample-prototype similarity graph and cross-layer consensus prototype graph.

[0055] In this embodiment, to facilitate efficient optimization, the objective function in formula (1) is rewritten as a trace matrix expression. This equivalent form makes the optimization of the objective function more concise, as follows:

[0056] (2)

[0057] Then, the present invention updates the prototype diagram and prototype matrix iteratively until convergence. Finally, the present invention constructs the final prototype diagram, which is represented as follows: The prototype diagram The left singular vector is equivalent to the complete similarity graph. The feature vector. Based on this, the present invention... Perform singular value decomposition (SVD) to obtain a low-dimensional embedding matrix representing the clustering structure. and in the matrix Upload Application k The -means algorithm is used to obtain the final clustering results.

[0058] For the objective function (2), an alternating optimization strategy is used to solve it. Since this objective function applies to all variables... and Since the function is not a joint convex function, optimization is performed by fixing some variables and iteratively updating the remaining variables. The complete solution process is summarized in the algorithm in Table 1.

[0059]

[0060] (1) Update With other variables fixed, the objective function (2) can be rewritten as follows:

[0061] (3)

[0062] Obviously, in equation (3) regarding The optimization process can be represented as the following quadratic programming (QP) problem:

[0063] (4)

[0064] in, and ;

[0065] (2) Update With other variables fixed and irrelevant terms ignored, the objective function (2) can be rewritten as follows:

[0066] (5)

[0067] Because each The optimization is independent of other views, therefore the above formula is applied to... Taking the derivative and setting it to zero, we get:

[0068] (6)

[0069] (3) Update With other variables fixed, the objective function (2) can be rewritten as follows:

[0070] (7)

[0071] Similarly, regarding The optimization process can be represented as the following quadratic programming problem:

[0072] (8)

[0073] in, and ;

[0074] (4) Update With other variables fixed and irrelevant terms ignored, the solution can be calculated as follows:

[0075] (9)

[0076] (5) Update With other variables fixed and irrelevant terms ignored, the solution can be calculated as follows:

[0077] (10)

[0078] The overall optimization objective function shown in formula (2) is in the variable set and The above variables do not possess joint convexity. To address this issue, an alternating optimization strategy is proposed, whereby in each iteration, other variables are fixed while only one variable is updated. Each subproblem has a closed-form solution or can be efficiently solved using quadratic programming. Let the... i The objective function value of the next iteration is Then the update process satisfies This indicates that the objective function value decreases monotonically in each iteration. Since the objective function is bounded by 0, the proposed algorithm can guarantee convergence to a local optimum.

[0079] In one embodiment, fusing the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model and then performing spectral clustering to obtain the clustering result of the multi-view image sample set includes: fusing the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model to obtain a fused prototype map; performing singular value decomposition on the fused prototype map to obtain a low-dimensional embedding matrix; inputting the low-dimensional embedding matrix into a clustering algorithm to obtain the clustering result of the multi-view image sample set; the number of clusters in the clustering algorithm is consistent with the number of true categories.

[0080] This invention addresses the problems of inappropriate prototype selection, insufficient structural expressiveness, and poor robustness in image data processing inherent in multi-view clustering tasks. It proposes an image recognition method and system based on hierarchical prototype multi-view learning. This method effectively enhances the expressive power of complex image feature distributions by introducing a progressively abstract prototype graph structure. Furthermore, it provides a more generalizable and adaptive clustering solution to address the modal differences and structural heterogeneity exhibited by image data in multi-view scenarios. The core advantages of this invention are reflected in the following aspects:

[0081] (1) It has strong structural modeling capabilities for high-dimensional image data. By constructing multi-level, progressively decreasing prototype structures, it can achieve progressive abstract modeling from fine-grained features such as local texture and edges of images to semantic-level global patterns, which greatly enhances the expressive power of image data in multi-view space.

[0082] (2) The guidance mechanism enhances modal consistency. In response to the feature alignment problem of different modalities (such as color distribution, texture features, edge gradients, etc.) in image data, a learnable cross-layer guidance mechanism is proposed. In each layer, the prototype consistency between different views is automatically adjusted, thereby improving the accuracy and stability of image clustering.

[0083] (3) It has good scalability and adaptability. This method can be flexibly adapted to image datasets of different sizes and dimensions, and is robust to challenges such as density imbalance and class distribution imbalance in image clustering tasks. It is suitable for typical application scenarios such as medical image analysis, face clustering, and remote sensing image interpretation.

[0084] (4) The clustering results are more interpretable. Through the hierarchical prototype structure and cross-layer consistency propagation mechanism, the system can provide layer-by-layer visualization and structural explanation of the clustering results, thereby enhancing the usability and credibility of the image recognition results.

[0085] This invention evaluates the computational efficiency and scalability of the proposed hierarchical prototype learning method by analyzing its time and space complexity:

[0086] (1) Time complexity

[0087] The time consumption of the proposed hierarchical prototype learning framework mainly stems from the similarity matrix. With prototype matrix The iterative update. The specific analysis is as follows: for the bipartite graph from sample to prototype... Its complexity mainly includes matrix multiplication and n Solving the quadratic programming subproblems has an overall complexity of O(n log n). ,in This represents the total feature dimension across all views. For cross-layer prototype diagrams... Its computational complexity is For the first-level prototype matrix Its time complexity is For the intermediate layer prototype matrix Its time complexity is In summary, based on prototype numbers Much smaller than the sample size n The proposed method demonstrates good computational efficiency and scalability, with overall time complexity decreasing with respect to the number of samples. n Linear growth, i.e. This makes it applicable to clustering tasks on large-scale datasets.

[0088] (2) Space complexity

[0089] The overall space complexity of this method mainly comes from the storage overhead of various matrices in the hierarchical prototype learning framework, specifically including the prototype representation matrix. Cross-layer prototype diagram And the bipartite diagram from sample to prototype In consideration M Each view and L Given a hierarchical structure, the total space overhead is ,in This represents the total feature dimension of all views. Because... Therefore, the space complexity increases with the number of samples. n Linear growth ensures the scalability of the proposed model on large-scale multi-view datasets.

[0090] This invention was experimented on different benchmark datasets and compared with several state-of-the-art multi-view clustering methods under consistent settings to evaluate the effectiveness and scalability of the proposed framework. The code for each benchmark method was directly obtained from GitHub without modification. The hyperparameters were fine-tuned within the range suggested in the original paper using a grid search method to determine the optimal settings for each baseline. For methods containing... k -The baseline method of -means, this invention selects the best result from 50 runs to mitigate the impact of randomness. Number of clusters k It is pre-configured and conforms to common experimental setups. For this method, the initialization strategy is also based on... k -means. In specific implementations, this invention traverses the number of levels of the hierarchical structure. L The range is 1 to 4, used to explore the impact of the number of layers (i.e., model depth) on performance. Subsequent experiments will further verify the rationale for choosing a maximum of 4 layers. For each layer... l The number of its corresponding prototypes is from the predefined set. Selected from, among which kThis represents the actual number of clusters. This setting ensures that deeper levels correspond to coarser-grained prototype representations, thus constructing a hierarchical structure of progressively abstracting data. The regularization parameter for each level... Unified setting as This means assigning the same weights to all layers. Apart from the prototype configuration, the proposed model does not depend on any additional adjustable hyperparameters, simplifying the overall hyperparameter tuning process.

[0091] To evaluate the effectiveness of the proposed algorithm, this invention was compared with the following eleven prototype-based multi-view clustering baseline methods: 1) LMVSC: employs prototype-based graph learning to achieve linear-time multi-view subspace clustering; 2) SMVSC: integrates prototype learning and similarity construction into a unified optimization framework; 3) FPMVS-CAG: jointly optimizes prototype selection and subspace graph construction without hyperparameter involvement; 4) OMSC: fuses consensus prototype representation, consensus graph, and final clustering results into a unified framework; 5) FMVACC: solves the problem through a flexible prototype graph fusion mechanism. 6) IPMVSC: Learns view-specific prototypes to capture the distribution characteristics of heterogeneous data; 7) FastMICE: Improves the robustness of clustering through random view grouping and hybrid fusion strategies; 8) UDBGL: Learns a unified discrete bipartite graph representation under Laplace rank constraints; 9) MVSC-HFD: Projects data to a shared subspace through a hierarchical projection matrix to achieve clustering; 10) 3AMVC: Automatically identifies and aligns high-quality prototypes in different views; 11) HBGMVSC: Constructs a hierarchical bipartite graph and achieves clustering through an adaptive weighted fusion mechanism.

[0092] Table 2 shows a comparison of the clustering performance of the method of this invention with eleven state-of-the-art methods in terms of ACC, NMI, Purity, and F-score. The best and second-best results among all methods are indicated by bold and underline, respectively. Based on these results, the present invention makes the following observations:

[0093] (1) The proposed algorithm has significant advantages over other multi-view clustering algorithms. The proposed algorithm outperforms other algorithms on all datasets. For example, in ACC, the proposed method improves performance by 2.78%, 7.86%, and 3.33% compared to the suboptimal method, respectively. It also achieves the best results in other evaluation metrics such as NMI, Purity, and F-score. Finally, experiments verify the effectiveness and performance superiority of the proposed method.

[0094] (2) Compared with several classic large-scale multi-view clustering methods (UDBGL, FMVACC, and SMVSC) that employ bipartite graph construction, prototype matching correspondence, or projection strategies, the proposed algorithm achieves the best performance. As an important baseline algorithm for prototype-based multi-view clustering, LMVSC shows significant performance improvement from the algorithm of this invention, which verifies the effectiveness of the hierarchical prototype learning strategy guided by the multi-level prototype graph construction of this invention.

[0095] (3) The algorithm of the present invention has made significant improvements over several dynamic prototype learning methods such as FPMVSCAG, IPMVSC, and MVSC-HFD. This is because the method of the present invention introduces structural constraints, which makes the prototype allocation process more regular, thereby obtaining a more robust and semantically consistent similar graph.

[0096] Table 2. Comparison of clustering performance of multi-view datasets

[0097]

[0098] Experimental results on multiple public image datasets (such as Dermatology, HandWritten, and MFeat) demonstrate that the proposed method significantly outperforms existing methods in metrics such as ACC, NMI, Purity, and F-score, while maintaining fast convergence speed and excellent time efficiency. These results fully validate the application value and engineering feasibility of this invention in image recognition tasks, showcasing its broad potential in tasks such as image clustering, image preprocessing for classification, and image representation learning.

[0099] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0100] In one embodiment, an image recognition system based on hierarchical prototype multi-view learning is provided, comprising:

[0101] The model building module is used to construct a multi-layer prototype model; the multi-layer prototype model includes a multi-layer prototype structure corresponding to each view, and the multi-layer prototype structure includes layers from the first layer to the second layer.L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram;

[0102] The objective function construction module is used to initialize the multi-layer prototype model. It inputs the multi-view image sample set into the initialized multi-layer prototype model and constructs the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer.

[0103] The optimization and solution module is used to solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph.

[0104] The result output module is used to fuse the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model, and then obtain the clustering result of the multi-view image sample set through spectral clustering processing.

[0105] Specific limitations regarding the image recognition system based on hierarchical prototype multi-view learning can be found in the limitations of the image recognition method based on hierarchical prototype multi-view learning described above, and will not be repeated here. Each module in the aforementioned image recognition system based on hierarchical prototype multi-view learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0106] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, display screen, and input system connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements an image recognition method based on hierarchical prototype multi-view learning. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input system can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0107] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0108] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image recognition method based on hierarchical prototype multi-view learning, characterized in that, The method includes: Construct a multi-layered prototype model; the multi-layered prototype model includes a multi-layered prototype structure corresponding to each view, the multi-layered prototype structure including layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram; Initialize the multi-layer prototype model by inputting the multi-view image sample set into the initialized multi-layer prototype model. Construct the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer. Solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph; The optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model are fused and then subjected to spectral clustering to obtain the clustering result of the multi-view image sample set. The objective function is: in, Indicates the first t Data matrix of views, M Total number of views L This indicates the total number of layers in the multi-layered prototype structure. t For the index of the sample, Indicates the first t The prototype matrix of the first layer of views This represents a bipartite graph from the sample to the first-level prototype, for the... Layer and , Indicates the first t The prototype matrix of each view. Indicates the first Layer and First Cross-layer consensus prototype diagram It is a regularization parameter. For the first Number of prototype layers This indicates that all elements are 1. 3D column vector, It is the square of the norm.

2. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, Solving the objective function and outputting the optimized sample-prototype similarity graph and cross-layer consensus prototype graph includes: The objective function is optimized using an alternating optimization strategy. The cross-layer consensus prototype graph and view-specific prototype matrix of the multi-layer prototype model are iteratively updated until the iteration stops when the stopping condition is met. The optimized sample-prototype similarity graph and cross-layer consensus prototype graph are then output.

3. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, The process of fusing the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model and then performing spectral clustering to obtain the clustering results of the multi-view image sample set includes: The optimized sample-prototype similarity graph and cross-layer consensus prototype graph in the multi-layer prototype model are fused to obtain a fused prototype graph. Singular value decomposition is performed on the fused prototype image to obtain a low-dimensional embedding matrix. The low-dimensional embedding matrix is ​​then input into a clustering algorithm to obtain the clustering results of the multi-view image sample set. The number of clusters in the clustering algorithm is consistent with the number of true categories.

4. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, The construction of the cross-view shared sample-prototype similarity graph includes: For each view, a clustering method is used to extract the corresponding initial underlying prototype; For each view, a similarity relationship between the sample and the initial underlying prototype is constructed, and a bipartite graph connection matrix from the sample to the prototype is formed in a non-negative normalization constraint manner. The bipartite graph connection matrices of each view are fused to obtain the initial sample-prototype similarity graph. The shared connectivity of the initial sample-prototype similarity graph is iteratively optimized until the pre-set iteration stop condition is met, at which point the iteration stops and a sample-prototype similarity graph shared across views is obtained.

5. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, The passage through the first Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, which includes: Construct a joint minimization objective function; the joint minimization objective function includes the first... Layer Prototype and the First Reconstruction loss term of the layer prototype, the first The loss term for fitting the layer prototype to the sample features and the regularization term for constraining the sparsity of the high-level prototype; The first Layer prototype matrix, sample feature matrix of multi-view image sample set and the first The layer prototype indicator matrix is ​​used as input to the joint minimization objective function with respect to the first layer. Differentiating the prototype matrix of each layer and setting the derivative to zero yields a closed-form analytical solution. Based on the predetermined decreasing number of prototypes per layer, the [number of prototypes] is determined. High-level prototype matrix.

6. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, Construct the first Layer and First The cross-layer consensus prototype diagram includes: Construct a fusion objective function; the fusion objective function includes a same-layer structure preservation term, a cross-layer structure preservation term, and a regularization term, wherein the same-layer structure preservation term is calculated by... Euclidean distance quantization between layer prototypes, and cross-layer structure preservation terms are calculated by the first layer prototype. Layer prototype to the first The reconstruction error of the layer prototype is quantified, and the regularization term is an L2 regularization term that constrains the sparsity of elements in the cross-layer consensus prototype graph. Under the constraints of nonnegativity and row normalization, the quadratic programming method is used to optimize the fusion objective function, resulting in the... Layer and First Prototype diagram of cross-layer consensus.

7. The image recognition method based on hierarchical prototype multi-view learning according to claim 1, characterized in that, The initialization of the multi-layer prototype model includes: The multi-layer prototype model is initialized based on the pre-set number of layers in the multi-layer prototype structure and the decreasing number of prototypes at each layer.

8. An image recognition system based on hierarchical prototype multi-view learning applied to the method described in any one of claims 1-7, characterized in that, The system includes: The model building module is used to construct a multi-layer prototype model; the multi-layer prototype model includes a multi-layer prototype structure corresponding to each view, and the multi-layer prototype structure includes layers from the first layer to the second layer. L The prototype structure of the layers involves extracting prototypes from the corresponding view data in the first layer to obtain the corresponding underlying prototype matrix, and constructing a sample-prototype similarity map shared across views. Layer to the first The prototypes of each layer are extracted in descending order to obtain the corresponding high-level prototype matrix, and the 1st prototype matrix is ​​constructed. Layer and First Cross-layer consensus prototype diagram; The objective function construction module is used to initialize the multi-layer prototype model. It inputs the multi-view image sample set into the initialized multi-layer prototype model and constructs the objective function based on the sample-prototype association relationship between the original data in the multi-view image sample set and the bottom prototype matrix, the mapping relationship between adjacent layer prototype matrices, and the structural constraints of the cross-layer consensus prototype diagram of each layer. The optimization and solution module is used to solve the objective function and output the optimized sample-prototype similarity graph and cross-layer consensus prototype graph. The result output module is used to fuse the optimized sample-prototype similarity map and cross-layer consensus prototype map in the multi-layer prototype model, and then obtain the clustering result of the multi-view image sample set through spectral clustering processing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images