Visual classification processing method and device based on large model and multi-modal data fusion

Abstract

The invention relates to the field of visual processing, and provides a visual classification processing method and device based on large model and multi-modal data fusion. The method comprises the following steps: inputting a to-be-classified input image and a corresponding category text description into a text encoder for multi-level feature extraction to obtain global text features and local text features; performing fine-grained cross-modal alignment on the local visual features and the local text features, calculating association weights between the image regions and the text phrases through a bidirectional cross attention mechanism, and generating aligned intermediate features; splicing and fusing the aligned middle features and the global visual features, and inhibiting background noise in a fusion result and reinforcing discriminative features in the fusion result through a feature mask algorithm in combination with the global text features to obtain multi-modal fusion features; and synchronously inputting the multi-modal fusion features into a multi-space classifier to generate respective classification results, and adaptively outputting an image classification result according to a confidence threshold in combination with a dynamic routing mechanism.

Description

Technical Field

[0001] This application relates to the field of visual processing, and more specifically, to a visual classification processing method and apparatus based on the fusion of large models and multimodal data. Background Technology

[0002] In the context of the rapid development of artificial intelligence technology, multimodal data fusion visual classification methods based on large models have become core technologies in fields such as image understanding, autonomous driving, and medical image analysis. These technologies aim to enhance machines' ability to recognize complex scenes by combining visual information with multimodal data such as textual descriptions.

[0003] However, while existing mainstream methods such as CLIP (Contrastive Language-Image Pre-training) achieve cross-modal alignment through global contrastive learning, they are essentially single-scale global alignment mechanisms. These methods calculate similarity between the entire image and the entire text description, ignoring the correlation between local regions (such as key components of objects in an image) and specific phrases in the text description (such as "the short nose feature of a Pomeranian"). For example, when distinguishing between Pomeranians and similar species like Japanese Spitz, large models often struggle to spontaneously focus on subtle differences in features such as nasal bone length and fur texture, reducing the classification accuracy of large models. Although existing technologies have introduced attention mechanisms (such as cross-attention modules) to address these technical problems, the unidirectional or shallow attention interactions employed by most existing solutions still fail to learn the correlation between subtle regions in the image and lexical semantics, significantly impacting image classification accuracy. Therefore, there is an urgent need to design a technical solution to address the fine-grained image classification problem existing in current technologies. Summary of the Invention

[0004] In this context, the embodiments of this application aim to provide a visual classification processing method and apparatus based on the fusion of large models and multimodal data, which can effectively improve the accuracy and robustness of image classification in complex scenes, enhance the model's ability to extract discriminative features, and improve the generalization performance and stability of classification results under different data distributions.

[0005] In a first aspect of the embodiments of this application, a visual classification processing method based on the fusion of large models and multimodal data is provided, the method comprising: The input image to be classified is obtained, and the input image is input into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features. Obtain the category text description corresponding to the input image, and input the category text description into the text encoder through a large model to perform multi-level feature extraction to obtain global text features and local text features. The local text features are obtained by encoding key text phrases in the category text description. Fine-grained cross-modal alignment of local visual features and local text features is performed, and the association weight between image regions and text phrases is calculated through a bidirectional cross-attention mechanism to generate aligned intermediate features. The bidirectional cross-attention mechanism includes: image-to-text attention flow with local image features as queries and local text features as keys, and text-to-image attention flow with local text features as queries and local image features as keys. The aligned intermediate features are concatenated and fused with global visual features, and combined with global text features, a feature masking algorithm is used to suppress background noise in the fusion result and enhance the discriminative features in the fusion result to obtain multimodal fusion features; The multimodal fusion features are synchronously input into a multi-space classifier, which includes an Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

[0006] In a second aspect of the embodiments of this application, a visual classification processing apparatus based on the fusion of large models and multimodal data is provided. The apparatus includes the following modules: An extraction module is used to acquire an input image to be classified, input the input image into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features; acquire a category text description corresponding to the input image, input the category text description into a text encoder for multi-level feature extraction to obtain global text features and local text features, wherein the local text features are obtained by encoding key text phrases in the category text description; The alignment module is used to perform fine-grained cross-modal alignment of local visual features and local text features. It calculates the association weight between image regions and text phrases through a bidirectional cross-attention mechanism to generate aligned intermediate features. The bidirectional cross-attention mechanism includes: image-to-text attention flow with local image features as queries and local text features as keys, and text-to-image attention flow with local text features as queries and local image features as keys. The fusion module is used to concatenate and fuse the aligned intermediate features with the global visual features, and combine the global text features to suppress background noise in the fusion result and enhance the discriminative features in the fusion result through a feature masking algorithm to obtain multimodal fusion features. The classification module is used to synchronously input the multimodal fusion features into a multi-space classifier, which includes a Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

[0007] This application discloses a visual classification processing method and apparatus based on the fusion of large models and multimodal data. This embodiment, through multi-level feature extraction and fine-grained cross-modal alignment, combined with a multi-space classifier and dynamic routing mechanism, effectively improves the accuracy and robustness of image classification in complex scenes. It enhances the model's ability to extract discriminative features and improves the generalization performance and stability of classification results under different data distributions. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a visual classification processing method based on the fusion of large models and multimodal data, as shown in this application. Figure 2 This is a schematic diagram of the structure of a visual classification processing device based on the fusion of large models and multimodal data, as shown in this application. Detailed Implementation

[0009] The following is for reference. Figure 1 , Figure 1 This is a flowchart illustrating a visual classification processing method based on the fusion of large models and multimodal data, provided as an embodiment of this application.

[0010] To address at least one of the aforementioned technical problems, this application provides a visual classification processing method and apparatus based on large-scale model and multimodal data fusion. Specifically, this application extracts global and local visual features of the input image through a visual encoder, and uses a large-scale model to perform multi-level semantic encoding on the category text description, obtaining global text features and local text features based on key text phrases. This method overcomes the limitations of traditional single-modal visual classification and achieves deep fusion of visual and linguistic information. Furthermore, a bidirectional cross-attention mechanism is used to achieve fine-grained alignment between image regions and text phrases, effectively capturing cross-modal semantic associations, thereby enhancing the model's ability to identify and discriminate key targets in the image. In the local feature interaction stage, this application employs bidirectional attention flows from image to text and from text to image, using local visual features and local text features as queries to interactively capture soft alignment relationships between modalities. This mechanism can adaptively calculate the association weights between different image regions and text phrases, suppress irrelevant background interference, strengthen discriminative visual-semantic correspondences, and provide high-quality aligned feature representations for subsequent classification. This application's implementation method simultaneously inputs multimodal fusion features into three geometric space classifiers: Euclidean, Hilbert, and hyperbolic. The model can then perform discriminative analysis of features from different geometric assumptions. Combined with a dynamic routing mechanism, the optimal classification result is adaptively selected based on the confidence scores of each classifier, effectively addressing data distribution differences and domain bias issues, and significantly improving the model's robustness and generalization ability in complex open environments. This application's implementation method introduces a feature masking algorithm in the feature fusion stage. Aligned intermediate features are concatenated and fused with global visual and textual features, and the fusion result is subjected to noise suppression and discriminative feature enhancement based on attention weights. This strategy helps filter redundant or misleading information introduced during multimodal fusion, highlighting key classification features and further improving the clarity of classification boundaries and the overall performance of the model.

[0011] Figure 1 The flowchart of a visual classification processing method based on the fusion of large models and multimodal data, as shown in one embodiment of this application, includes: Step S101: Obtain the input image to be classified, and input the input image into the visual encoder for multi-level feature extraction to obtain global visual features and local visual features; Step S102: Obtain the category text description corresponding to the input image, and input the category text description into the text encoder through the large model to perform multi-level feature extraction to obtain global text features and local text features; Step S103: Perform fine-grained cross-modal alignment of local visual features and local text features, calculate the association weight between image regions and text phrases through a bidirectional cross-attention mechanism, and generate aligned intermediate features; Step S104: The aligned intermediate features are spliced ​​and fused with the global visual features, and the background noise in the fusion result is suppressed and the discriminative features in the fusion result are enhanced by combining the global text features with the feature masking algorithm to obtain the multimodal fusion features. Step S105: The multimodal fusion features are synchronously input into a multi-space classifier, which includes a Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

[0012] In this embodiment, the input image to be classified can include digital images from any source, such as natural scene images, medical images, remote sensing images, industrial inspection images, security monitoring images, traffic scene images, and product appearance inspection images. The input images involved in this application are primarily visual models input in common image formats (such as RGB, grayscale, infrared, and multispectral images). Furthermore, the size of the input image can be a fixed resolution or dynamically scaled while maintaining aspect ratio to meet the structural requirements of the visual encoder.

[0013] Step S101: Obtain the input image to be classified, and input the input image into the visual encoder for multi-level feature extraction to obtain global visual features and local visual features.

[0014] Understandably, the visual encoder in the above steps can employ a Convolutional Neural Network (CNN), Visual Transformer (ViT), Swin Transformer, or a hybrid CNN-Transformer architecture. After inputting the image, the visual encoder first performs preprocessing such as standardization, zero-padding, and random cropping to ensure the stability of the input features. Subsequently, the visual encoder models the image content at multiple scales through stacked convolutional or self-attention modules. Shallow layers focus on extracting local image features such as edges and textures, while deeper layers gradually acquire global semantic information such as target shape and scene layout. During this process, the input image is divided into multiple regions or patches, each of which, after independent modeling by the encoder, forms fine-grained local visual features. Simultaneously, the encoder generates global visual features that represent the semantic content of the entire image, either at a deeper level or through specific global pooling operations (such as global average pooling or token-like aggregation).

[0015] Through the aforementioned multi-level feature extraction mechanism, the embodiments of this application can simultaneously capture the overall shape information and key local details of the target object. This not only enhances the model's expressive power when processing images with high structural complexity but also lays a high-quality visual representation foundation for subsequent cross-modal alignment and multi-spatial classification. By leveraging the synergistic effect of global and local visual features, the algorithm's ability to recognize fine-grained categories, appearance variations, and local discriminative regions can be improved, thereby effectively enhancing the generalization performance and accuracy of visual classification.

[0016] As an optional embodiment, in step S101, multi-level visual feature extraction of the input image can be performed based on the Visual Transformer (ViT) architecture. Taking a 224×224 resolution RGB natural image as an example, the system first performs normalization processing on the input image, mapping pixel values ​​to a preset distribution range, and performs necessary random cropping and edge padding to enhance the robustness of the model under different lighting, scale, and viewpoint conditions. Subsequently, the image is divided into fixed-size patches, for example, each patch is 16×16, resulting in 196 independent image patches. Each patch is unfolded into a vector and input into the encoder layer of ViT after adding position encoding. The encoder consists of multiple layers of self-attention modules. In the shallow layers, the attention mechanism models the texture changes, edge contrast, and small local structures between neighboring patches, thereby generating preliminary local visual features. As the network propagates deeper layer by layer, the model gradually focuses on cross-regional correlations, and the self-attention weights can dynamically capture the overall contour, semantic structure, and scene context of key targets globally. To address this, the network sets up a dedicated token class at the highest layer to perform aggregation operations on semantic information from all patches and integrates it into global visual features for the entire image through adaptive weights. Ultimately, the visual encoder generates both fine-grained local visual feature sequences covering all patches and global visual features that can express the semantic core of the entire image, providing sufficient representational basis for subsequent cross-modal alignment and multi-spatial classification.

[0017] In another optional embodiment, in step S101, the input image is input into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features. This includes: performing multi-scale salient region detection on the input image; generating dual candidate salient maps using the SLIC superpixel segmentation algorithm and the Frequency-Tuned (FT) algorithm; merging the dual candidate salient maps into a unified visual saliency map using an adaptive weight fusion strategy; locating candidate key regions in the input image based on the visual saliency map using a region proposal network; encoding local features for each candidate key region using a mid-level Transformer block in the visual encoder to obtain the local visual features of each candidate key region; and extracting global context information of the input image using a high-level Transformer block in the visual encoder to obtain the global visual features of the input image.

[0018] In this embodiment, to further enhance the discriminative power and structural awareness of visual features, step S101 combines multi-scale salient region detection and region-level feature encoding techniques with traditional visual encoders to achieve more refined and semantically focused global and local visual representations. Specifically, firstly, multi-scale salient region detection is performed on the input image, and a scale-sensitive mechanism is used to initially screen potentially important regions in the image. For this purpose, the SLIC (SimpleLinear Iterative Clustering) superpixel segmentation algorithm is used to structurally divide the image. SLIC automatically segments the image into several superpixel regions with high boundary fit and strong texture consistency based on a joint distance metric of pixel color and spatial location, allowing subsequent saliency analysis to be conducted at a more natural regional scale. Simultaneously, the FT algorithm is used to capture visual anomalies of salient regions from a frequency domain perspective. The FT algorithm performs frequency domain smoothing and inverse transformation on the image color distribution to highlight regions that deviate from the mean in the overall color statistics. The combination of these two methods generates candidate saliency maps from the two dimensions of spatial structure consistency and color frequency anomalies, respectively.

[0019] In the embodiments of this application, the SLIC superpixel segmentation algorithm is used to perform structured region division of the input image, so that salient region detection no longer relies on pixel-by-pixel analysis, but is based on more stable and visually structurally consistent region-level units, thereby improving the accuracy and robustness of subsequent saliency fusion and region proposal. The SLIC algorithm achieves adaptive block division of image content by clustering the image on a joint distance metric in color space and spatial coordinate space, making it very suitable as a pre-step for saliency detection.

[0020] In the above embodiment, the input image is first mapped to a color space suitable for color perception differences (such as Lab space). Then, several cluster centers are initialized according to a preset number of superpixels, and these cluster centers are regularly distributed in the image space. Subsequently, the SLIC algorithm iteratively assigns each pixel to the cluster containing the nearest cluster center based on the weighted combination distance between color distribution and spatial location (i.e., the SLIC distance metric). This distance metric ensures that adjacent pixels maintain spatial consistency and makes pixels with similar colors more likely to aggregate into the same superpixel region. As the iteration continues, each cluster center continuously updates its position based on the mean of the pixels in its cluster, ultimately forming superpixel segments with high boundary fit and strong texture consistency.

[0021] In particular, during the saliency detection process in this embodiment, the SLIC algorithm can elevate saliency calculation from isolated pixels to the structural unit level through superpixel regions with consistent spatial structure, thereby effectively reducing the impact of noise and enhancing the semantic representation of regions. This provides a structured reference for subsequent saliency map fusion and region proposal stages, making the propagation of saliency weights smoother and the boundaries clearer. Compared to the color anomaly saliency map generated by the frequency-tuned FT algorithm, the structural saliency map generated by SLIC emphasizes the geometric boundaries, texture direction, and regional uniformity of the image. This structural information is crucial for identifying key regions with clear shapes and prominent edges.

[0022] In the subsequent saliency fusion process, the regionalized saliency map obtained by SLIC can improve the boundary accuracy of the saliency map, and can still maintain stable saliency localization ability when dealing with scenes with complex textures, local noise, or low local contrast. Finally, when the region proposal network generates candidate key regions based on the visual saliency map, the addition of SLIC can ensure that the proposed regions fit the target boundary more accurately, reduce the interference of redundant background regions, and thus improve the effectiveness of local visual feature encoding. This makes the entire multi-level feature extraction process have both local fine perception capability and global structural stability.

[0023] In this embodiment, the Fourier Transform (FT) algorithm is used to perform saliency detection on the input image to extract regions that differ significantly from the surrounding background from the global color distribution, thereby assisting in the formation of a highly reliable visual saliency map. The FT algorithm is essentially a saliency detection method based on frequency domain analysis. Its core idea is to identify pixel regions with significant contrast that deviate from the global mean in the color space by modeling the overall color statistical characteristics of the image and smoothing the frequency components. In accordance with the requirements of this embodiment, after basic preprocessing, the input image is first converted to a suitable color space, such as the Lab color space, to more accurately reflect the perceptual differences between pixels. The FT algorithm constructs a global color reference model of the input image by calculating the global mean vector of the entire input image in this color space. Subsequently, Gaussian smoothing is applied to the input image to remove local noise and enhance large-scale color distribution features. For any point in the input image, its saliency is calculated by the Euclidean distance between the point's color and the global mean color. The larger the distance, the more its color deviates from the overall distribution, and therefore, the more likely it is to belong to a visually salient region. By using this global measurement method based on color deviation, the FT algorithm can quickly generate a saliency map that reflects the overall saliency structure of an image.

[0024] In this embodiment, the FT algorithm and the structural saliency map generated by SLIC superpixels together constitute a dual candidate saliency map. The saliency map generated by the FT algorithm has a natural advantage for regions with significant color differences, especially in natural scenes, industrial defect detection, or medical images, where color anomalies often correspond to potential key content. Unlike the structured salient regions obtained by SLIC, the FT saliency map emphasizes color distribution anomalies, thus supplementing structural saliency information driven by texture or boundaries. The combination of the two provides rich and complementary visual cues for the subsequent adaptive weight fusion strategy. During the fusion stage, the FT saliency map enables the system to more accurately identify candidate key regions with color anomalies but indistinct structures, avoiding misclassification of key regions as insignificant against complex texture backgrounds. Ultimately, the introduction of the FT algorithm significantly improves the globality and stability of salient region detection, making the generated unified saliency map more robust to noise interference, illumination changes, and color mixing scenes, thereby providing a more reliable region guidance signal for the region proposal network and further improving the quality and accuracy of local visual feature extraction.

[0025] After obtaining the aforementioned dual candidate saliency maps, an adaptive weight fusion strategy is introduced in step S101. This strategy dynamically adjusts the fusion ratio between the two types of saliency maps based on the texture complexity of the local region, the stability of the color distribution, and the global contrast, thereby generating a unified and more robust visual saliency map. This ensures that the saliency map can accurately focus on structural regions with clear texture boundaries, while also remaining sensitive to key patches with significant color deviations.

[0026] Subsequently, in step S101, a Region Proposal Network (RPN) is used to guide spatial attention in the visual saliency map, and several candidate key regions are generated based on the response intensity of salient regions. The RPN uses a sliding window and anchor box mechanism to locate potential key patches based on local peak positions in the saliency map, effectively eliminating irrelevant background areas and enhancing the priority of key targets in the region proposal stage. When extracting local features, the intermediate Transformer block in the visual encoder specifically encodes local features for the aforementioned candidate key regions. Based on a self-attention structure, the intermediate Transformer can model local texture, structural details, and semantic relationships at the region scale, enabling each key region to obtain high-resolution local visual features.

[0027] After encoding local regions, the visual encoder further integrates the contextual semantics of the entire image through its high-level Transformer block. The high-level Transformer possesses a larger receptive field and stronger cross-regional association modeling capabilities, enabling it to capture spatial relationships between targets, scene composition structure, and category-related semantic patterns globally, and generate global visual features representing the semantic core of the entire input image. Through this multi-level feature extraction method combining saliency-guided local enhancement strategies with global context modeling, the system can significantly improve its perception capabilities for complex scenes, fine-grained categories, and key local regions. This provides a richer local feature base and a more robust global feature base for multimodal task input, which is beneficial for improving the accuracy and robustness of subsequent cross-modal alignment and classification decisions.

[0028] Further optionally, in the above steps, multi-scale salient region detection is performed on the input image, and dual candidate salient maps are generated using the SLIC superpixel segmentation algorithm and the frequency-tuned FT algorithm, including: The input image is subjected to Gaussian pyramid multi-scale layering processing to generate at least three image layers of different scales; SLIC superpixel segmentation is performed on each image layer to divide the image in each image layer into superpixel units with similar visual features; based on spatial region contrast calculation, background prior, and position weight allocation algorithms, sub-saliency maps at different scales are generated, and weighted fusion is performed to obtain the first candidate map; the frequency-tuned Fourier transform algorithm is used to perform global frequency analysis on the input image through Gaussian smoothing and color space transformation, and the saliency value of each pixel in the input image is calculated to obtain the second candidate map.

[0029] In the above embodiments, to improve the adaptability of salient region detection to complex scenes, scale changes, and background interference, a multi-scale saliency modeling mechanism is introduced. This mechanism combines the SLIC superpixel segmentation algorithm and the frequency-tuned Fourier transform (FT) algorithm to construct a dual candidate saliency map, thereby achieving collaborative perception of structural and color saliency. Specifically, the input image is first processed by a Gaussian pyramid to generate multiple scale layers, each representing the image structure at different resolutions. The Gaussian pyramid, through layer-by-layer Gaussian blurring and downsampling, causes a gradual change in the spatial frequency of the image. Small-scale layers highlight the global contour, while large-scale layers retain more local details, enabling saliency analysis to remain sensitive to targets of different sizes.

[0030] At each scale level, SLIC superpixel segmentation is performed, dividing the image into superpixel units with color consistency and spatial continuity. SLIC clusters pixels based on a joint color-spatial distance metric, ensuring that each superpixel region naturally conforms to image boundaries and maintains consistent texture structure. Based on this, the system calculates region-level saliency values ​​for each scale level. Spatial region contrast measures the difference in color and texture between the superpixel region and its neighboring regions. Background priors utilize the statistical regularity of background appearance in image edge regions to suppress low-saliency regions near the edges, while the positional weighting algorithm assigns higher saliency probabilities to regions near the geometric center based on geometric center offset. Through the combination of these three evaluation factors, each scale level generates an independent sub-saliency map. A cross-scale weighted fusion strategy then integrates the advantages of each scale to form a first candidate saliency map with stronger structural sensitivity. This map demonstrates excellent characterization of target boundaries and local structures in complex texture images.

[0031] Meanwhile, a frequency-tuned Fourier transform (FT) algorithm is employed to construct a second candidate saliency map from a global color statistics perspective. The FT algorithm first transforms the input image to a perceptually uniform color space such as Lab, and then performs Gaussian smoothing to eliminate local noise interference. Subsequently, it calculates the global color mean of the entire image, and represents its saliency by measuring the distance between each pixel's color vector and the global mean vector. A larger distance indicates that the pixel deviates more from the average pattern in the overall color distribution, thus possessing stronger visual saliency. Therefore, the FT algorithm is naturally adept at identifying regions that are abnormally prominent in color or brightness, and can compensate for the shortcomings of the SLIC structural saliency in scenarios where color changes are not obvious.

[0032] By jointly modeling multi-scale SLIC structural saliency analysis and FT global color saliency analysis, the dual candidate saliency map obtained in this embodiment can comprehensively evaluate image content from multiple dimensions such as color, structure, location, and scale, making the results of salient region detection more comprehensive, accurate, and robust. In the subsequent saliency fusion, region proposal, and local feature encoding stages, this dual candidate saliency map provides a more reliable attention guidance signal for the visual encoder, helping the model to more accurately locate key regions, reduce background interference, and further improve the discriminative ability of global and local visual feature extraction.

[0033] Optionally, in the above steps, an adaptive weight fusion strategy is used to synthesize the dual candidate saliency maps into a unified visual saliency map, including: evaluating the image content complexity based on the global information entropy and edge density of the input image; assigning dynamic fusion weights to the first candidate map and the second candidate map based on the evaluated image content complexity; performing multi-scale fusion calculations on the first candidate map and the second candidate map in combination with the dynamic fusion weights, and upsampling the fusion results of each scale to the original image size through bilinear interpolation to obtain the final unified visual saliency map.

[0034] In the above embodiments, to further improve the adaptability of salient region detection to different scenes and imaging conditions, an adaptive weight fusion strategy is introduced to dynamically integrate the dual candidate saliency maps generated by SLIC structural saliency and FT color saliency respectively, ultimately obtaining a unified visual saliency map that more realistically reflects the key regions of the image. Specifically, the image content complexity is first estimated based on the global information entropy and edge density of the input image. Information entropy reflects the randomness of the grayscale or color distribution of the image; a higher entropy value indicates richer image texture details and more complex visual content. Edge density, by detecting gradient changes in the image to count the number of high-frequency edges, can be used to measure the structural complexity of the image. The combination of the two constitutes a global evaluation index of image content complexity, enabling the system to adaptively determine whether the current image relies more on structural saliency or color saliency for region identification.

[0035] Based on the aforementioned content complexity assessment results, fusion weights are dynamically assigned to the first candidate map (a structural saliency map based on multi-scale SLIC) and the second candidate map (a global color saliency map based on the FT algorithm). When the image has rich texture structure, high edge density, or complex geometric contours, the system increases the weight of the first candidate map, allowing structural saliency to dominate in the final map. Conversely, when the image has significant color contrast, simple texture, or a stable lighting background, the weight of the second candidate map is increased, allowing color saliency to play a greater role. Through dynamic adjustment, the system can avoid the mismatch problem caused by fixed weights, making the fusion strategy more scene-adaptive.

[0036] After obtaining the dynamic fusion weights, multi-scale fusion calculations are performed on the two candidate maps. During the fusion process, the structural saliency maps and color saliency maps at different scales are linearly or non-linearly combined according to the dynamic weights, and the cross-scale consistency of salient regions is preserved during the fusion process, so that the saliency response can be balanced across small targets, high-frequency textures, and large-area targets. Subsequently, the saliency results at each scale obtained by fusion are upsampled to the original values ​​using bilinear interpolation. Figure 1 The image is sized to ensure that the spatial resolution of the final saliency map is consistent with that of the input image, while maintaining smooth transitions in edge regions and continuity of saliency values. The resulting unified visual saliency map exhibits excellent performance in terms of structural accuracy, color sensitivity, and scale adaptability.

[0037] The adaptive weight fusion strategy in this embodiment can automatically adjust the saliency construction method for different image types, content structures and imaging conditions, so that the final saliency map can more accurately highlight the truly important regions in the image, reduce background noise interference, and provide more stable and reliable input for subsequent region proposal and local visual feature extraction, thereby comprehensively improving the discriminative ability and overall classification performance of the visual encoder.

[0038] Step S102: Obtain the category text description corresponding to the input image. Input the category text description into a text encoder using a large model for multi-level feature extraction to obtain global text features and local text features. In this embodiment, the local text features are obtained by encoding key text phrases in the category text description.

[0039] In the above steps, the first step is to obtain the category text description corresponding to the input image. This category text description can consist of key information such as category name, semantic attributes, appearance features, and typical components, for example, "red long-sleeved sports jacket" or "broad-leaved plant with curved leaf edges." Subsequently, this category text description is input into a large-model-driven text encoder. The text encoder can employ a pre-trained language model architecture, such as BERT, RoBERTa, Transformer Encoder, or a customized pruned version of a large-scale language model (LLM). Its core uses a self-attention mechanism to model the semantic relationships of the text sequence. The text encoder first segments the text description, decomposing the text into independent words or sub-word units, and then maps each word unit to a high-dimensional word vector through a word embedding layer, forming a word-level vector sequence. Next, the word vector sequence is input into a multi-layer Transformer module. Each Transformer block models the contextual relationships between words through iterative computation using self-attention and a feedforward network, enabling the text encoder to capture both the semantic combination logic between words and the long-range dependency information within the text. Building upon this foundation, the text encoder generates global text features through global pooling or special aggregation operations to characterize the semantic information of the entire category text description. Simultaneously, key phrases in the text are identified through dependency parsing or a neural network parser, and attention pooling or weighted summation is applied to the word vector subsequences corresponding to these phrases to obtain local text features. These local features highlight key semantic units in the category description, enabling the encoder to more effectively match visual features during subsequent cross-modal alignment, thus capturing fine-grained category features.

[0040] On the one hand, in constructing local text features, step S102 mainly focuses on key text phrases with discriminative significance in the categorical text description, such as adjective-noun combinations, component phrases, or attribute phrases. The system automatically identifies these key phrases through the attention matrix of the pre-trained language model or dependency parsing algorithm and encodes them locally. Local phrases rely on the representational capabilities of the encoder's intermediate layers and output local text features that can represent fine-grained semantic concepts through a multi-head attention mechanism. For example, for text describing "blue striped short-sleeved T-shirt," local text features can capture semantic fragments such as "blue," "stripes," "short-sleeved," and "T-shirt," enabling the model to perform accurate matching for specific local regions in the image during subsequent cross-modal alignment.

[0041] In generating global text features, the text encoder utilizes high-level Transformer blocks to fuse the semantics of the entire text sequence and generates global text vectors representing the main semantics of the entire text through special class tokens (such as [CLS]) or global pooling strategies. These global text features contain the overall semantic framework in the category description, which can correspond to the global visual features of the image, enabling the model to understand the semantic space of the category to which the input image belongs at a macro level.

[0042] Through the aforementioned multi-level text feature extraction mechanism, this embodiment achieves the collaborative expression of local and global semantics, enabling text representation to possess both a global category semantic framework and fine-grained semantic cues precisely aligned with local visual regions. This allows the system to more efficiently bind visual details to text phrases during cross-modal alignment and multi-space classification, thereby improving the model's discriminative ability and robustness in complex category recognition, fine-grained classification, and multimodal reasoning tasks.

[0043] As an optional embodiment, in step S102, the category text description is input into a text encoder using a large model for multi-level feature extraction to obtain global text features and local text features, including: The built-in word segmenter of a pre-trained language model based on the Transformer architecture segments the category text description into independent lexical units. The word embedding layer of the pre-trained language model then transforms each lexical unit into a high-dimensional numerical vector representation, forming a word vector sequence. Dependency parsing is performed on the category text description to identify and extract a set of key text phrases, where each key text phrase corresponds to a key semantic unit in the category text description. The word vector sequence is then input into the deep transformer module of the text encoder, where global pooling aggregates the overall semantic information of the word vector sequence to obtain the global contextual meaning of the category text description, resulting in global text features representing the overall semantics. For each extracted key text phrase, its corresponding position is located in the word vector sequence, and a corresponding word vector sub-sequence is extracted. An attention pooling layer is used to weight and sum the word vector sub-sequences by calculating lexical importance weights, generating local text features representing the semantics of each key text phrase.

[0044] In the above embodiments, step S102 can perform multi-level semantic feature modeling on the category text description. Its core lies in utilizing the text encoding capabilities of a large model to transform the original natural language description into a high-quality feature representation that reflects both the overall semantics and key local semantics. Specifically, this embodiment introduces a pre-trained language model based on the Transformer architecture. Its built-in word segmenter performs sub-word-level segmentation on the input category text description, dividing the continuous natural language into independent lexical units that cover semantic and grammatical information. Then, the word embedding layer of the pre-trained language model maps each lexical unit to a high-dimensional word vector, enabling the text to participate in subsequent deep semantic calculations in numerical form. To enhance the model's understanding of key semantic segments in the category description, this embodiment further performs dependency parsing on the text, identifying subject-predicate structures, modification structures, and other dependency relationships within the text. This automatically extracts a set of key phrases reflecting the core semantics of the category, allowing the encoder to no longer rely solely on the overall vector representation but to explicitly focus on the semantic units in the text that play a decisive role in category differentiation.

[0045] For example, a dependency parser based on deep biaffine attention can be used, or neural network parsing models provided by natural language processing toolkits such as Stanford NLP, spaCy, and HanLP can be employed. These parsers are typically based on deep neural network architectures, such as bidirectional LSTM, Transformer, or self-attention network models. They first encode the input text sequence to obtain hidden representations that reflect the semantic context of the word sequence. Then, a biaffine attention mechanism or graph-based prediction network is used to calculate dependency scores between words, generating a complete dependency syntax tree. The dependency syntax tree clearly indicates the modification, dominance, or logical subordination relationships between words within the text, such as subject-verb, verb-object, attributive-head, and adverbial-head relationships. This embodiment, based on the parsed dependency structure, identifies text segments with highly dense categorical semantic information, including core noun phrases, attribute-modifying phrases, action or state phrases, and extracts them into a set of key text phrases. In this way, the system can automatically locate the key phrases that best reflect semantic discriminativeness in the category description without the need for manual rules, providing phrase-level input with accurate semantic structure for subsequent local feature generation, thereby significantly improving the interpretability and discriminativeness of local semantic features.

[0046] Furthermore, during the construction of global features, the word vector sequence is input into the deep Transformer module of the text encoder. This module captures the long-distance dependency between any two words in the sequence through a multi-head self-attention mechanism, thereby establishing global semantic associations. After completing multi-layer semantic modeling, the semantic information of the entire word vector sequence is aggregated through global pooling operations (such as average pooling or [CLS] tag pooling), enabling the encoder to output global text features that represent the overall contextual semantics of the category text description. This global feature has advantages such as strong semantic integrity and sufficient expression of the macro-level meaning of the category, and can provide a stable overall semantic benchmark for subsequent tasks.

[0047] For key text phrases obtained from dependency parsing, this embodiment locates their start and end positions in the original word vector sequence and extracts related word vector subsequences. To obtain the contribution of different words within each key phrase, this embodiment introduces an attention pooling mechanism on the subsequences. By calculating the attention weight corresponding to each word vector to reflect its semantic importance, the word vectors are weighted and summed to generate local text features that highlight the semantic center of the key phrase. The introduction of local features enables the encoder to retain those local semantic structures with outstanding recognition value in the category description, such as keyword groups that can distinguish categories, such as "deep texture," "slender structure," and "metal shell," thereby improving the discriminative power of subsequent feature alignment or semantic matching tasks.

[0048] For example, to obtain discriminative phrase-level semantic representations from categorical text descriptions, the start and end positions of each key text phrase are first precisely located in the original word vector sequence. This location process, based on phrase boundary information obtained from dependency parsing, maps the index positions of each word in the phrase back to the word vector sequence, thereby enabling the extraction of word vector subsequences that are completely consistent with the semantic region of the phrase from the global embedding sequence. To further extract elements with stronger semantic contributions within the phrase from these subsequences, this embodiment introduces an attention pooling mechanism at the subsequence level. The attention pooling layer first calculates the importance weight for each vector in the word vector subsequence. This weight can be generated by a learnable attention scoring function, for example, by inputting each word vector into a single-layer or multi-layer feedforward network and combining it with a softmax normalization operation, so that the weight can reflect the contribution of the word to the phrase semantics. Finally, attention pooling forms a phrase-level semantic representation by weighted summation of the word vectors. This weighted aggregation method not only preserves the overall semantic information of the phrase but also strengthens the influence of the semantic core words within the phrase, enabling the generated local text features to more centrally express the phrase's category discrimination ability. The local text features introduced in this way can highlight the most identifiable key phrases in the category description, such as "deep texture," "slender structure," and "metallic shell," thereby significantly improving the model's ability to capture fine-grained semantic differences and its overall discrimination performance in subsequent cross-modal feature alignment, similarity discrimination, or semantic association calculation.

[0049] Through the aforementioned multi-layered text feature generation mechanism, this embodiment enhances the model's sensitivity to key semantic units within the text while maintaining global semantic integrity, achieving the collaborative expressive ability of global and local features. This not only improves the richness and accuracy of category text representations but also further enhances the discriminative performance of subsequent cross-modal matching, semantic retrieval, or model inference tasks, enabling the system to exhibit higher robustness and discriminative power when faced with detailed differences in category descriptions.

[0050] As an optional embodiment, in step S102, after inputting the category text description into the text encoder for multi-level feature extraction to obtain global text features and local text features, a hierarchical encoding strategy can be adopted. This strategy utilizes network modules at different levels in the text encoder to randomly capture lexical-level local semantics and long-distance contextual semantics of different granularities. The lexical-level local semantics and long-distance contextual semantics are then fused to construct random semantic features of different granularities. The generated global text features and local text features are combined into hierarchical feature pairs. The hierarchical feature pairs are then randomly perturbed by the random semantic features to drive the text encoder to learn more robust feature representations as optimized global text features and local text features.

[0051] In this embodiment, after inputting the category text description into the text encoder and obtaining global and local text features, a hierarchical encoding and random semantic perturbation mechanism are further introduced to improve the robustness and generalization ability of the text feature representation.

[0052] Specifically, text encoders typically consist of stacked Transformer modules. Shallow layers focus on lexical-level local semantics, such as adjacency relationships or short-range dependency structures between words, while deep layers capture long-range contextual information across sentences or even phrases. This embodiment leverages this structural characteristic, employing a hierarchical encoding strategy. It randomly selects intermediate features from several layers as semantic candidates. By fusing the local semantics of the shallow layers with the contextual semantics of the deep layers, it constructs random semantic features with different granularities. For example, for the text fragment describing "a metal shell with deep textures," shallow features can reflect the lexical features of "deep textures," while higher-level features can reflect the implicit association between it and the metal shell, thus forming a hierarchical and semantically complementary multi-granularity representation. After generating random semantic features, this embodiment combines global and local text features into hierarchical feature pairs. These feature pairs are then perturbed using random semantic features. Perturbation methods can include lightweight transformations such as random weighted fusion, random permutation of feature dimensions, or random noise injection to simulate the semantic diversity and expressive uncertainty that may occur in real-world text descriptions. By continuously applying random perturbations to hierarchical features, the text encoder is driven to learn more stable feature representations that are less susceptible to changes in local expression during training. This enables the optimized global text features to more accurately represent the overall semantic structure, and the optimized local text features to more stably characterize the semantic center of keywords and phrases. As a result, the robustness and generalization performance of the features are significantly improved in subsequent tasks such as cross-modal alignment, semantic matching, or category discrimination.

[0053] Step S103: Perform fine-grained cross-modal alignment of local visual features and local text features, calculate the association weight between image regions and text phrases through a bidirectional cross-attention mechanism, and generate aligned intermediate features.

[0054] In this embodiment of the application, the bidirectional cross-attention mechanism includes: an image-to-text attention flow with local image features as queries and local text features as keys, and a text-to-image attention flow with local text features as queries and local image features as keys.

[0055] In step S103, to achieve fine-grained cross-modal semantic alignment between local visual features and local text features, a bidirectional cross-attention mechanism is introduced, enabling image regions and text phrases to establish a correspondence in a shared semantic space. Specifically, local visual features are typically obtained by a visual encoder through mid-level feature encoding of salient regions or candidate key regions, and their representation can characterize the texture, shape, or regional attributes of the local image. Local text features, on the other hand, are semantically aggregated by a text encoder from key text phrases, used to express the core meaning of local semantics in the category description.

[0056] To achieve cross-modal matching, this embodiment constructs two attention paths: an image-to-text attention stream and a text-to-image attention stream. The image-to-text attention stream uses local image features as the query vector and text phrase features as the key-value vector. Attention weights are generated by calculating the similarity between the query and the key, and these weights are then used to weightedly fuse the value vectors, enabling the image region to select the most matching semantic information from multiple text phrases. Conversely, the text-to-image attention stream uses text phrase features as the query and local image features as the key-value vector. Through similar attention weight calculations, the text phrases can aggregate visual cues from the most relevant image regions. For example, for the phrase "slender structure" in the category description, the text-to-image attention stream will prioritize regions with a slender shape in the image, while the image-to-text attention stream will activate phrase features corresponding to the semantic meaning of "slender structure" in these slender regions, thus forming a bidirectional reinforced alignment relationship.

[0057] Through the bidirectional cross-attention computation described above, the intermediate features generated in this embodiment not only contain collaborative information between visual and textual data, but also explicitly encode cross-modal correspondences within the features, making the representation of the same semantics more consistent in the embedding space across the visual and textual modalities. Because the attention mechanism itself has weighted selection characteristics, it can automatically suppress the influence of noisy regions or irrelevant phrases, strengthening only semantically highly relevant regions and phrase alignment items, thereby effectively improving fine-grained cross-modal understanding capabilities. With the help of this bidirectional cross-attention mechanism, the aligned intermediate features generated in this embodiment possess higher interpretability and stronger discriminative power, providing a structurally clear and semantically consistent cross-modal foundational representation for subsequent fusion operations, semantic matching, or classification decisions.

[0058] As an optional embodiment, in step S103, local visual features are constructed into an image region feature sequence, where each region feature corresponds to a key discriminative region in the input image; local text features are constructed into a text phrase feature sequence, where each phrase feature corresponds to a key semantic unit in the category text description; layer normalization is performed on both the image region feature sequence and the text phrase feature sequence. In the image-to-text attention flow, the image region feature sequence is used as the query vector, and the text phrase feature sequence is used as the key and value vectors to calculate the cross-attention weights from the image region to the text phrase; a multi-head attention mechanism is used to calculate the region-phrase association in parallel in multiple subspaces to capture the correspondence at different semantic levels; the attention weights are weighted and fused to generate an alignment feature representation from the image perspective, which serves as the image flow alignment feature. In the text-to-image attention flow, the text phrase feature sequence is used as the query vector, and the image region feature sequence is used as the key and value vectors to calculate the cross-attention weights from the text phrase to the image region; a multi-head attention mechanism is also used to explore the guiding role of text on the image region; an alignment feature representation from the text perspective is generated, which serves as the text flow alignment feature. Image stream alignment features and text stream alignment features are concatenated. The concatenated features are then subjected to nonlinear transformation and dimensionality adjustment through a feedforward neural network to generate preliminary aligned intermediate features. The preliminary aligned intermediate features are then further processed to generate the final optimized intermediate features used for feature fusion.

[0059] In the above optional embodiments, to achieve accurate alignment between local image regions and key semantic units of text, the local visual features generated by the visual encoder are first organized into an image region feature sequence, with each vector corresponding to a key discriminative region in the input image. Simultaneously, local text features extracted from the category text description are constructed into a text phrase feature sequence, so that each vector corresponds to a key semantic phrase. To alleviate the differences in numerical distribution between different modal features, this embodiment performs layer normalization on both feature sequences to maintain uniform mean and variance at the layer level, thereby ensuring the stability of attention calculation and reducing gradient shift during cross-modal mapping. Subsequently, in the image-to-text attention path, image region features are used as query vectors, and text phrase features are used as keys and values. The relevance weight of each region to each phrase is calculated using a dot product attention mechanism. Then, a multi-head attention structure is introduced to project the query, key, and value into multiple subspaces for parallel computation, enabling the model to capture region-phrase correspondences from different semantic dimensions such as material, color, shape, and action. For example, when an image contains a region labeled "slender metal structure," different attention heads may focus on textual components such as "slender," "metal," and "structure," and ultimately generate a cross-modal alignment feature representation from the image perspective through weighted fusion. Conversely, in the text-to-image attention path, text phrase features are used as queries, and image region features are used as keys and values. The same multi-head attention mechanism calculates the degree of attention from text phrases to each image region, thus reflecting the reverse guidance ability of text semantics on visual regions. For example, the phrase "curved edge" will cause the model to highlight the response to curved regions in the image during multi-head attention, generating an alignment feature representation from the text perspective. Next, this embodiment concatenates the alignment features from the image stream and text stream, and performs nonlinear transformations and feature dimension adjustments through a feedforward neural network, reorganizing and compressing the concatenated features in a higher-dimensional space to form preliminary alignment intermediate features. To further enhance cross-modal consistency, this embodiment applies deepening processing to the preliminary alignment features, including residual connections, layer normalization, and nonlinear activation, to improve the semantic robustness and cross-modal distribution compatibility of the features, ultimately obtaining optimized intermediate features for subsequent fusion stages. Through the above processing flow, this embodiment not only achieves high-precision fine-grained alignment between local visual regions and text phrases, but also significantly improves the expressive power of cross-modal features, making subsequent visual-semantic fusion more effective and reliable, thereby laying a solid foundation for the model to generate more accurate classification decisions.

[0060] Optionally, in the above steps, the initially aligned intermediate features are further processed to generate optimized intermediate features for final feature fusion. To further improve the semantic consistency and discriminative ability of the initially aligned intermediate features, they are further processed to generate optimized intermediate features, thereby providing high-quality input for subsequent multimodal fusion. For example, firstly, a second cross-attention operation is performed on the initially aligned intermediate features through multi-level deep alignment calculation. This process uses multi-layer stacked attention modules, which allows the semantic correspondence between visual regions and text phrases to be remodeled at a deeper level, enhancing the ability to capture fine-grained semantic differences. After each attention layer calculation, Dropout technology is introduced to randomly deactivate some neurons, preventing the model from overfitting the training data in the deep network, thereby improving feature generalization ability. For example, when similar slender structural regions exist in the image, multi-layer cross-attention can more finely distinguish the correspondence with the text phrase "slender metal parts," while Dropout avoids the network relying on only a single feature dimension. To ensure effective gradient propagation and enhance feature stability during deep alignment, this embodiment employs a residual connection mechanism. This mechanism adds the features from the previous layer to the output features of the current layer, mitigating the vanishing gradient problem that may occur in deep networks. Simultaneously, layer normalization standardizes the output features, stabilizing feature distribution and accelerating training convergence. By combining residual and layer normalization operations, the system can improve network training stability while maintaining feature expressiveness, ensuring consistency of optimized features across different samples and batches. To enhance alignment consistency, a fine-grained alignment loss function is calculated during deep alignment, and a hard negative sample mining mechanism is introduced to automatically identify easily confused sample pairs within similar categories. In the feature space, the relative distance between each image region and the most relevant text phrase is dynamically adjusted, shortening the distance between correctly matched image-text pairs and lengthening the distance between incorrectly matched pairs. For example, when identifying "blue short-sleeved T-shirt" and "blue long-sleeved T-shirt," this mechanism strengthens the alignment between "short-sleeved" and the corresponding image region while weakening the association with the "long-sleeved" region, thereby improving cross-modal discrimination accuracy. Subsequently, the system verifies the consistency of the alignment results through a maximum similarity matching mechanism, ensuring that each image region can find the most semantically relevant text phrase. During iterative optimization, the attention weight distribution is continuously adjusted to fully strengthen the semantic connection between image regions and text phrases. Finally, the optimized intermediate features are evaluated for quality. High-quality feature fragments are selected based on feature activation intensity and distribution uniformity, while low-quality or weakly responding regions undergo local enhancement processing, such as through local attention enhancement or weighted amplification of their feature representation, thereby improving the overall expressive power, discriminative power, and robustness of the intermediate features.Through the above-described in-depth processing flow, this embodiment can generate optimized intermediate features that fully integrate image and text semantics. It not only preserves key local semantic information but also takes into account global consistency, providing a high-precision and high-reliability feature foundation for subsequent multimodal fusion and classification decisions.

[0061] Step S104: The aligned intermediate features are spliced ​​and fused with the global visual features, and the background noise in the fusion result is suppressed and the discriminative features in the fusion result are enhanced by combining the global text features with the feature masking algorithm to obtain the multimodal fusion features.

[0062] In this embodiment, to effectively integrate local cross-modal alignment information with global visual semantics and further enhance discriminative features, the optimized intermediate features are concatenated and fused with global visual features in step S104. Simultaneously, the fusion result is optimized using a feature masking algorithm in conjunction with global text features. Specifically, the optimized intermediate features already contain fine-grained alignment information between local image regions and key text phrases, while the global visual features reflect the overall semantic structure and scene layout of the input image. Through feature concatenation, the system combines local alignment information and global visual semantics along the feature dimension to form a preliminary fusion representation. This allows the model to simultaneously acquire local discriminative information and global contextual information, thereby improving its ability to understand complex image structures or subtle features. For example, for an image containing a red sports jacket and a blue background, the local intermediate features can accurately represent the alignment information between the red clothing region and the text phrase "red sports jacket," while the global visual features provide overall scene and color distribution information, ensuring that the fused features can take into account both local and global semantics.

[0063] Building upon this foundation, this embodiment introduces a feature masking algorithm to suppress background noise and enhance discriminative features. Specifically, firstly, mask weights are generated using global text features. These weights measure the importance of each fused feature dimension or spatial location; that is, regions with strong discriminative power correspond to higher weights, while background or irrelevant regions correspond to lower weights. The feature masking algorithm applies these weights to the fused features through element-wise multiplication, thereby reducing the interference of background information on the final representation and enhancing the response of image regions highly correlated with textual semantics. For example, in the aforementioned image of a red sports jacket, the blue background region is suppressed by the feature mask, while the features of the red clothing region are enhanced, making the fused features more prominent in category discrimination. Through this mechanism, the generated multimodal fused features not only achieve a close integration of visual and textual information at the semantic level but also effectively improve the discriminative power and robustness of the features, providing high-quality input for subsequent multi-spatial classifiers and improving the accuracy and stability of cross-modal visual classification.

[0064] Step S105: The multimodal fusion features are synchronously input into a multi-space classifier, which includes a Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

[0065] In step S105, in order to achieve efficient classification and discrimination of multimodal fusion features, a multi-space classifier architecture is introduced. The fusion features are simultaneously input into the Euclidean space classifier, the Hilbert space classifier, and the hyperbolic space classifier to obtain image category information from different geometric space perspectives.

[0066] Specifically, in step S105, the Euclidean space classifier, based on traditional vector space distance metrics, generates classification probabilities by calculating the Euclidean distance between the fused features and the center vectors of each category, making it suitable for handling categories with obvious local continuous feature distributions. The Hilbert space classifier maps the fused features to a high-dimensional Hilbert space, using kernel functions or inner product operations to capture the nonlinear relationships between features, thus better handling complex nonlinear boundary problems between categories. The hyperbolic space classifier embeds features into a hyperbolic geometric space, using an exponential distance function to characterize the relationship between hierarchical structures and long-tailed categories, enabling the classifier to perform better in scenarios with highly unbalanced category distributions or hierarchical relationships. For example, for an image containing multiple sportswear categories, the Euclidean classifier can quickly distinguish categories with significant differences in color and shape, the Hilbert classifier can distinguish categories with similar textures and styles, while the hyperbolic classifier can effectively handle subtle differences between "short-sleeved sweatshirts" and "long-sleeved sweatshirts" within the category hierarchy.

[0067] To achieve adaptive decision-making among the three spatial classification results, this embodiment introduces a dynamic routing mechanism. This mechanism dynamically adjusts the contribution of each classifier's output to the final classification result based on its confidence value. Classification results with high confidence are given greater weight, while the influence of low-confidence results is suppressed, thus achieving a balance between feature space diversity and discriminative stability. For example, if the probability distribution of the fused features in Euclidean space shows a high confidence level for the "red short-sleeved T-shirt" category, while the classification results in Hilbert and hyperbolic spaces have relatively low confidence levels, the dynamic routing mechanism will favor the Euclidean space output in the final classification result, ensuring the reliability and consistency of the output. Through this multi-space fusion and dynamic routing strategy, this embodiment can fully utilize the sensitivity of different geometric spaces to feature structures, improve the discriminative ability for complex and diverse image categories, and enhance the robustness of the model in long-tailed categories and scenarios with subtle semantic differences, ultimately generating accurate and stable image classification results.

[0068] As an optional embodiment, in step S105, the multimodal fusion features are synchronously input into a multi-space classifier to generate Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results, respectively. This includes: inputting the multimodal fusion features into the Euclidean space classifier; performing a linear transformation through the fully connected layer of the Euclidean space classifier to map the feature dimension to the number of target categories; normalizing the transformed feature vector using the Softmax activation function to generate the probability distribution of each category, thus obtaining the Euclidean space classification result; mapping the multimodal fusion features to a high-dimensional Hilbert space using a kernel function; calculating the inner product of the feature vector in the regenerating kernel Hilbert space using a radial basis function kernel; and performing feature decomposition based on the kernel matrix to obtain the category decision in the Hilbert space. The policy boundary is defined to generate Hilbert space classification results; the multimodal fusion features are projected onto hyperbolic space through exponential mapping; in hyperbolic space, the projected feature vectors are geometrically transformed using a Poincaré sphere model, and the hyperbolic distance between the projected feature vectors and the preset prototype points of each category is calculated, utilizing the tree-like structure of hyperbolic geometry to amplify the difference in detailed features under each category; based on the hyperbolic distance metric and the difference in detailed features under each category, the hyperbolic Softmax function is used to calculate the classification probability of each category, generating hyperbolic space classification results; the curvature parameter of hyperbolic space is adaptively adjusted through learnable parameters; the Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results are used as parallel outputs of the multi-space classifier to form a three-dimensional spatial classification result matrix.

[0069] In step S105, to fully utilize the rich semantic information of the multimodal fusion features and improve the discriminative ability of image classification, a multi-space classifier architecture is further introduced. Classification results are generated in Euclidean space, Hilbert space, and hyperbolic space through parallel processing. Specifically, in step S105, the multimodal fusion features are first input into the Euclidean space classifier. This classifier performs a linear transformation on the input features through a fully connected layer, mapping the high-dimensional features to a vector dimension consistent with the number of target categories. Then, the Softmax activation function is used to normalize the mapped vector, obtaining the probability distribution of each category, thereby generating the Euclidean space classification result. The Euclidean space classifier can efficiently handle categories with significant local feature differences. For example, for "red sports jacket" and "blue sports jacket," this classifier can quickly distinguish them using the Euclidean distance between color and shape features.

[0070] Meanwhile, multimodal fusion features are also mapped to a high-dimensional Hilbert space via kernel functions to capture the complex relationships between features in a non-linear manner. The inner product of feature vectors in the regenerated kernel Hilbert space is calculated using a radial basis function kernel, and feature decomposition is performed based on the kernel matrix to construct the class decision boundary, thereby generating a Hilbert space classification result. This method enhances the model's sensitivity to non-linear boundaries between classes. For example, when distinguishing between "striped short-sleeved T-shirts" and "solid-color short-sleeved T-shirts," kernel mapping can better separate texture information, thus improving fine-grained class discrimination capabilities.

[0071] Simultaneously, multimodal fusion features are projected onto a hyperbolic space via an exponential mapping. In this hyperbolic space, a Poincaré sphere model is used to geometrically transform the projected feature vectors, and the hyperbolic distance between each vector and a predefined category prototype point is calculated. Leveraging the tree-like structure of hyperbolic geometry, the Poincaré sphere model amplifies the differences in detailed features across categories. For example, when identifying different levels of clothing categories, "short-sleeved sweatshirts" and "long-sleeved sweatshirts" show a clear distance distinction in hyperbolic space. Subsequently, combining the hyperbolic distance metric and the detail difference, the hyperbolic Softmax function is used to calculate the classification probability of each category, generating hyperbolic space classification results. The curvature parameter of the hyperbolic space is adaptively adjusted through a learnable method to adapt to the distribution characteristics of different category hierarchies.

[0072] For example, in the above embodiments, the specific implementation of projecting multimodal fusion features onto hyperbolic space and performing geometric transformations using a Poincaré sphere model can be achieved through the following steps: Assuming the input image is a "red short-sleeved sweatshirt", after visual and text encoders and cross-modal alignment, a high-dimensional fusion feature vector is obtained. First, an exponential mapping is performed on the high-dimensional fused feature vector. Projecting the high-dimensional fused feature vector from the Euclidean feature space onto the vicinity of the origin of the hyperbolic space ensures that the embedding of the high-dimensional fused feature vector satisfies the negative curvature property in hyperbolic geometry.

[0073] In hyperbolic space, a Poincaré sphere model is used to represent feature vectors. The position of a vector within the sphere is determined by its magnitude and direction. Each category has a predefined prototype point in hyperbolic space; for example, the prototypes of "short-sleeved sweatshirt" and "long-sleeved sweatshirt" are located on branches at different levels within the sphere. The projected vectors are then calculated. With prototype points of each category hyperbolic distance This approach leverages the tree-like structure of hyperbolic space to naturally amplify hierarchical differences, making the distance between "short-sleeved sweatshirt" and "long-sleeved sweatshirt" in the embedding space significantly greater than the Euclidean distance, thus facilitating the differentiation of fine-grained categories. Subsequently, the distance values ​​are normalized using the hyperbolic Softmax function to calculate the category probabilities. .

[0074] in, This is a temperature regulation coefficient used to control the sharpness of the probability distribution. A larger value indicates a lower probability distribution. This will make the probability distribution more acute, meaning that high-probability classes will dominate and low-probability classes will be compressed; smaller This makes the probability distribution smoother and increases the model's tolerance to uncertainty. This indicates that in hyperbolic space, given an input feature vector... In this case, the image belongs to a category The predicted probability. It is the output of the hyperbolic Softmax function, used for representing multi-class probability distributions. It is through exponential mapping Multimodal fusion feature vector after projection from Euclidean space to hyperbolic space. It incorporates fused features of visual and textual information and represents their semantic location in hyperbolic space. Indicates category Prototype points or center points in hyperbolic space. Each category is pre-assigned a prototype location in hyperbolic space, typically obtained through statistical analysis or optimization learning of training data. Prototype points are used to measure the similarity between input features and each category. It represents any one of all categories. Representing input features With category prototype Hyperbolic distance in hyperbolic space. Hyperbolic distance is calculated using the distance formula under the Poincaré sphere model. Its characteristic is that it can amplify the differences between categories in the hierarchical structure, making it easier to distinguish categories that are hierarchically similar but semantically different in hyperbolic space. The normalization term ensures that the sum of probabilities for all categories is 1. It performs an exponentially weighted summation of the hyperbolic distances for all categories, effectively acting as a standard Softmax normalization. Furthermore, the formula implicitly includes a curvature parameter K in the hyperbolic space that can be adaptively adjusted during training to accommodate different density distributions within the category hierarchy. For example, during training, if the distinction between long-sleeved and short-sleeved categories is insufficient, K can be adjusted to amplify the hyperbolic distance at that level, thereby improving discriminative power. In this way, the hyperbolic space classifier can accurately distinguish fine-grained features even when the category hierarchy is clear or long-tailed categories exist, while maintaining a stable embedding into the overall category space, achieving efficient discrimination and probability output for multimodal fusion features.

[0075] Finally, the Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results are used as parallel outputs of the multi-space classifier to form a three-dimensional spatial classification result matrix. This enables the model to simultaneously discriminate categories from three perspectives: linear distance, nonlinear high-dimensional relationships, and hierarchical geometric structure. Through the multi-space parallel classification mechanism, this embodiment not only fully utilizes the multi-layer semantic information of multimodal fusion features but also improves the ability to distinguish complex categories and the recognition accuracy of long-tailed or slightly differential categories. This provides a reliable and rich decision-making basis for the subsequent dynamic routing mechanism, thereby significantly enhancing the accuracy and robustness of image classification.

[0076] Further, in the above embodiments, after using the Euclidean space classification results and hyperbolic space classification results as parallel outputs of the multi-space classifier, the method further includes: calculating the correlation coefficient between any two sets of spatial classification results (Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results) based on the three-dimensional spatial classification result matrix, and generating a spatial consistency matrix; when the correlation coefficient between any two sets of spatial classification results is lower than a preset threshold, triggering a spatial consistency verification mechanism and initiating a reclassification process; based on the verification result of the spatial consistency matrix, assigning dynamic fusion weights to each spatial classification result using the entropy weight method; linearly combining the weighted three sets of spatial classification results to obtain a preliminary fused multi-space classification result, and introducing a temperature coefficient adjustment during the preliminary fusion process to optimize the smoothness of the probability distribution of the multi-space classification result; calibrating the confidence level of the preliminary fused multi-space classification result, and adjusting the output probability distribution using the Platt scaling method; when the maximum probability value is lower than the confidence threshold, initiating a multi-space voting mechanism to determine the final image classification result according to the majority voting principle.

[0077] In this embodiment, to further improve the reliability and consistency of the multi-space classifier output results, after generating a three-dimensional spatial classification result matrix of Euclidean, Hilbert, and hyperbolic space classification results, the system introduces a spatial consistency check and dynamic fusion mechanism. Specifically, firstly, based on the three-dimensional spatial classification result matrix, the correlation coefficient of any two sets of results in Euclidean, Hilbert, and hyperbolic spaces is calculated to form a spatial consistency matrix, which is used to measure the consistency and mutual support between the spatial classification results. When the correlation coefficient of any two sets of spatial classification results is lower than a preset threshold, it indicates that there is a significant difference in the classification output of different spaces, triggering the spatial consistency check mechanism and initiating a reclassification process to improve the robustness of the classification. For example, if an image outputs the category "blue short-sleeved T-shirt" in Euclidean space and "blue long-sleeved T-shirt" in Hilbert space, this low consistency will be identified and the cross-modal fusion features will be re-evaluated to reduce the potential risk of misclassification.

[0078] For example, the probability vectors output by Euclidean, Hilbert, and hyperbolic space classifiers for the same input image are collected to construct a three-dimensional spatial classification result matrix. Each dimension corresponds to the output of a classifier, and each element represents the predicted probability of that category. Then, for any two sets of classification result vectors in the matrix, the Pearson correlation coefficient is calculated along the category dimension to measure the similarity in probability distribution between the two sets of classification results. The calculation method includes first calculating the mean of the two sets of vectors, then calculating the covariance, and finally dividing the covariance by the product of the standard deviations of the two sets of vectors to obtain the correlation coefficient. By performing pairwise calculations on the three sets of classification results, a symmetric spatial consistency matrix can be obtained, with each element representing the consistency level between the corresponding two sets of classification results. This spatial consistency matrix can be used to determine the degree of mutual support between the spatial classification results. When the correlation coefficient between any two sets of classification results is lower than a preset threshold, reclassification or dynamic weighted fusion can be triggered to improve the robustness and reliability of the overall image classification.

[0079] Based on spatial consistency verification, this embodiment employs the entropy weighting method to assign dynamic fusion weights to each spatial classification result. The confidence level is determined by the entropy of the classification information output from each space; a lower entropy value indicates a more certain classification result and a higher contribution to the final fusion result. Subsequently, the weighted three sets of spatial classification results are linearly combined to obtain a preliminary fused multi-spatial classification result. A temperature coefficient is introduced during the fusion process to smooth the probability distribution, thereby preventing excessive amplification or compression of some category probabilities and making the final probability distribution more reasonable and stable. For example, a higher temperature coefficient can alleviate the overly confident high-probability output of the Euclidean space, making the fusion result more balanced. To further improve classification reliability, this embodiment performs confidence calibration on the preliminary fusion result, using the Platt scaling method to adjust the output probability distribution, making the model's probability predictions more consistent with the confidence distribution of the true classification.

[0080] Finally, when the maximum probability value of the initial fusion result is lower than the preset confidence threshold, the system activates a multi-space voting mechanism to determine the final classification result based on the majority voting principle. This means comparing the voting results of the three spatial classifiers and selecting the category most supported by the majority space as the final output. For example, if both Euclidean and hyperbolic spaces vote for "red sports jacket," while Hilbert space votes for "red jacket," the final classification will be "red sports jacket." Through this process, this embodiment not only improves the robustness and accuracy of the multi-space classification results but also effectively handles minor differences between categories and long-tailed categories, ensuring high consistency and reliability of the final image classification results in terms of multimodal fusion semantics and multi-space discriminative information.

[0081] As an optional embodiment, in step S105, the image classification result is adaptively output based on the confidence threshold using a dynamic routing mechanism, including: inputting the input image into a lightweight pre-screening network based on the MobileNetV3 architecture, performing feature extraction and fast inference through depthwise separable convolution to obtain a coarse-grained classification result and the corresponding confidence score; inputting the coarse-grained classification result into a pre-trained dedicated confidence network CLCNet, and analyzing the numerical distribution pattern of the coarse-grained classification result through a Restricted Self-Attention module; if the confidence score is higher than the coarse classification confidence threshold, the coarse-grained classification result of the lightweight pre-screening network is directly output as the image classification result; if the confidence score is not higher than the coarse classification confidence threshold, a multi-space classification result is output through Euclidean space, Hilbert space, and hyperbolic space.

[0082] In this embodiment, to achieve efficient and reliable image classification, step S105 incorporates a dynamic routing mechanism to adaptively select the final output classification result based on a confidence threshold. Specifically, the input image is first fed into a lightweight pre-screening network based on the MobileNetV3 architecture. Image features are efficiently extracted using depthwise separable convolutions, thereby quickly generating coarse-grained classification results and their corresponding confidence scores while ensuring computational efficiency. This lightweight network can provide preliminary judgments in most easily distinguishable classification scenarios, such as differentiating between "red T-shirt" and "blue T-shirt," while significantly reducing computational overhead. After obtaining the coarse-grained classification result, it is input into a pre-trained dedicated confidence network, CLCNet. The Restricted Self-Attention module is used to analyze the numerical distribution pattern of the classification result to evaluate the reliability of the coarse-grained judgment. Restricted Self-Attention focuses on the relative distribution of key categories in the coarse classification probability vector through a locally constrained attention mechanism. It can accurately capture potential uncertainties or category conflicts. For example, when the probabilities of "red T-shirt" and "orange T-shirt" are close, the module can determine that the confidence of the current coarse classification is insufficient.

[0083] Based on the confidence assessment results, the dynamic routing mechanism determines the final output path. If the confidence level is higher than the set coarse classification confidence threshold, the coarse-grained classification result generated by the lightweight pre-screening network is directly output, quickly completing the classification task and saving computational resources. If the confidence level is lower than the coarse classification confidence threshold, it indicates that the input image has fine-grained features or high similarity between categories. In this case, a multi-space classification path is activated, utilizing the previously generated Euclidean, Hilbert, and hyperbolic space classification results for more refined discrimination. Through this dynamic routing mechanism, this embodiment can ensure both high efficiency and classification accuracy. It achieves rapid output on easily classified samples, while automatically calling the multi-space classifier for deep inference on complex or potentially confusing samples, thereby significantly improving the overall accuracy, robustness, and computational efficiency of image classification.

[0084] Optionally, in this embodiment, the setting of the coarse classification confidence threshold directly affects whether the output of the lightweight pre-screening network is directly used as the final classification result, or whether it triggers a multi-space classifier for fine discrimination. This threshold is usually determined based on statistical analysis of the training and validation sets. Specifically, by analyzing the confidence distribution of the lightweight pre-screening network on the validation set, the score range with high confidence and significantly improved classification accuracy can be used as a candidate threshold. For example, when coarsely classifying categories such as "red short-sleeved T-shirt" and "blue sweatpants," if the accuracy exceeds 95% when the output probability of the lightweight network in the validation set is greater than 0.85, then 0.85 can be considered as the initial confidence threshold. Secondly, to balance classification accuracy and inference efficiency, the threshold can be optimized within a certain range (e.g., 0.7~0.95) during the training phase using methods such as grid search or Bayesian optimization to find the optimal point that retains the advantage of fast output while avoiding misclassification due to low confidence. During the runtime phase, the coarse classification confidence threshold can also be dynamically adjusted to adapt to different scenarios. For example, in scenarios with a balanced distribution of image categories and a large number of easily classifiable samples, the threshold can be appropriately increased to make the lightweight network output more stringent. Conversely, in scenarios with significant differences in fine-grained category differences or the presence of long-tailed categories, the threshold can be decreased to allow more samples to enter the multi-space classifier for processing, thereby improving overall accuracy. Furthermore, the distribution entropy of the coarse classification confidence score can be used for auxiliary adjustments. A lower entropy value indicates high determinism in the network output, allowing for a lower threshold, while a higher entropy value indicates greater uncertainty in the output, requiring a higher threshold to trigger fine-grained classification. This method, combining statistical analysis with dynamic adjustment, ensures a balance between efficiency and accuracy in the coarse classification confidence score threshold.

[0085] In this embodiment, by using multi-level feature extraction and fine-grained cross-modal alignment, combined with a multi-space classifier and dynamic routing mechanism, the accuracy and robustness of image classification in complex scenarios can be effectively improved. This not only enhances the model's ability to extract discriminative features, but also improves the generalization performance and stability of the classification results under different data distributions.

[0086] After introducing the methods of exemplary embodiments of this application, the following references are made. Figure 2 This application describes an exemplary embodiment of a visual classification processing apparatus based on large model and multimodal data fusion. The apparatus includes an extraction module for acquiring an input image to be classified, inputting the input image into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features; acquiring a category text description corresponding to the input image, inputting the category text description into a text encoder for multi-level feature extraction to obtain global text features and local text features, wherein the local text features are obtained by encoding key text phrases in the category text description. The alignment module is used to perform fine-grained cross-modal alignment of local visual features and local text features. It calculates the association weight between image regions and text phrases through a bidirectional cross-attention mechanism to generate aligned intermediate features. The bidirectional cross-attention mechanism includes: image-to-text attention flow with local image features as queries and local text features as keys, and text-to-image attention flow with local text features as queries and local image features as keys. The fusion module is used to concatenate and fuse the aligned intermediate features with the global visual features, and combine the global text features to suppress background noise in the fusion result and enhance the discriminative features in the fusion result through a feature masking algorithm to obtain multimodal fusion features. The classification module is used to synchronously input the multimodal fusion features into a multi-space classifier, which includes a Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

[0087] The above-described apparatus can implement each step described in the above-described method embodiments, and the specific implementation methods of each step will not be repeated here. After introducing the methods and apparatus of the exemplary embodiments of this application, a terminal device of the exemplary embodiments of this application will be described next. This terminal device can implement each step described in the above-described method embodiments, and the specific implementation methods of each step will not be repeated here.

[0088] It should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Therefore, the protection scope of this application should be determined by the scope of the claims.

Claims

1. A visual classification processing method based on the fusion of large models and multimodal data, characterized in that, The method includes: The input image to be classified is obtained, and the input image is input into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features. Obtain the category text description corresponding to the input image, and input the category text description into the text encoder through a large model to perform multi-level feature extraction to obtain global text features and local text features. The local text features are obtained by encoding key text phrases in the category text description. Fine-grained cross-modal alignment of local visual features and local text features is performed, and the association weight between image regions and text phrases is calculated through a bidirectional cross-attention mechanism to generate aligned intermediate features. The bidirectional cross-attention mechanism includes: image-to-text attention flow with local image features as queries and local text features as keys, and text-to-image attention flow with local text features as queries and local image features as keys. The aligned intermediate features are concatenated and fused with global visual features, and combined with global text features, a feature masking algorithm is used to suppress background noise in the fusion result and enhance the discriminative features in the fusion result to obtain multimodal fusion features; The multimodal fusion features are synchronously input into a multi-space classifier, which includes an Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

2. The method according to claim 1, characterized in that, The step of inputting the input image into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features includes: The input image is subjected to multi-scale salient region detection, and dual candidate salient maps are generated by the SLIC superpixel segmentation algorithm and the frequency-tuned FT algorithm. An adaptive weight fusion strategy is used to synthesize the dual candidate saliency maps into a unified visual saliency map; Based on the visual saliency map, candidate key regions in the input image are located using a region proposal network; By using the intermediate Transformer block in the visual encoder, local feature encoding is performed on each candidate key region to obtain the local visual features of each candidate key region. The global context information of the input image is extracted by using the high-level Transformer block in the visual encoder to obtain the global visual features of the input image.

3. The method according to claim 2, characterized in that, The step of performing multi-scale salient region detection on the input image, and generating dual candidate salient maps using the SLIC superpixel segmentation algorithm and the frequency-tuned FT algorithm, includes: The input image is subjected to Gaussian pyramid multi-scale layering processing to generate at least three image layers of different scales; SLIC superpixel segmentation is performed on each image layer, dividing the image in each image layer into superpixel units with similar visual features; Based on spatial region contrast calculation, background prior and location weight allocation algorithm, sub-saliency maps at different scales are generated, and weighted fusion is used to obtain the first candidate map. The frequency-tuned Fourier transform algorithm is used to perform global frequency analysis on the input image and calculate the saliency value of each pixel in the input image through Gaussian smoothing and color space transformation, so as to obtain the second candidate spectrum.

4. The method according to claim 3, characterized in that, The step of employing an adaptive weight fusion strategy to synthesize the dual candidate saliency maps into a unified visual saliency map includes: The image content complexity is evaluated based on the global information entropy and edge density of the input image; Based on the evaluation value of image content complexity, dynamic fusion weights are assigned to the first candidate map and the second candidate map; By combining dynamic fusion weights, multi-scale fusion calculations are performed on the first candidate map and the second candidate map, and the fusion results of each scale are upsampled to the original image size through bilinear interpolation to obtain the final unified visual saliency map.

5. The method according to claim 1, characterized in that, The process involves inputting the category text description into a text encoder using a large model for multi-level feature extraction, resulting in global and local text features, including: The built-in word segmenter of the pre-trained language model based on the Transformer architecture performs word segmentation on the category text description, decomposing it into independent lexical units; and uses the word embedding layer of the pre-trained language model to transform each lexical unit into a high-dimensional numerical vector representation, forming a word vector sequence. Dependency parsing is performed on the category text description to identify and extract a set of key text phrases, wherein the key text phrases correspond to key semantic units in the category text description; The word vector sequence is input into the deep transformer module of the text encoder. The overall semantic information of the word vector sequence is aggregated through global pooling operation to obtain the global contextual meaning of the category text description and obtain global text features used to represent the overall semantics. For each extracted key text phrase, locate the corresponding position of each key text phrase in the word vector sequence and extract the corresponding word vector subsequence; An attention pooling layer is used to perform a weighted summation of the word vector subsequences by calculating the lexical importance weights, generating local text features to represent the semantics of each key text phrase.

6. The method according to claim 1, characterized in that, The process of fine-grained cross-modal alignment of local visual features and local text features, and the calculation of association weights between image regions and text phrases through a bidirectional cross-attention mechanism to generate aligned intermediate features, includes: Local visual features are constructed into an image region feature sequence, where each region feature corresponds to a key discriminative region in the input image; local text features are constructed into a text phrase feature sequence, where each phrase feature corresponds to a key semantic unit in the category text description; layer normalization is performed on both the image region feature sequence and the text phrase feature sequence. In the image-to-text attention flow, the image region feature sequence is used as the query vector, and the text phrase feature sequence is used as the key vector and value vector to calculate the cross-attention weight from the image region to the text phrase. A multi-head attention mechanism is adopted to compute the region-phrase association in parallel in multiple subspaces to capture the correspondence at different semantic levels. The attention weights are weighted and fused to generate an alignment feature representation from the image perspective, which serves as the image flow alignment feature. In the text-to-image attention flow, the text phrase feature sequence is used as the query vector, and the image region feature sequence is used as the key vector and value vector to calculate the cross-attention weight from the text phrase to the image region. Similarly, a multi-head attention mechanism is used to explore the guiding role of the text on the image region. Alignment feature representation from the text perspective is generated as the text flow alignment feature. Image stream alignment features and text stream alignment features are concatenated. The concatenated features are then subjected to nonlinear transformation and dimensionality adjustment through a feedforward neural network to generate preliminary aligned intermediate features. The preliminary aligned intermediate features are then further processed to generate the final optimized intermediate features used for feature fusion.

7. The method according to claim 1, characterized in that, The step of synchronously inputting the multimodal fusion features into a multi-space classifier to generate Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results includes: The multimodal fusion features are input into a Euclidean space classifier, and a linear transformation is performed through the fully connected layer of the Euclidean space classifier to map the feature dimension to the number of target categories. The transformed feature vector is normalized using the Softmax activation function to generate the probability distribution of each category, thus obtaining the Euclidean space classification result. The multimodal fusion features are mapped to a high-dimensional Hilbert space through a kernel function. The inner product of the feature vectors in the regenerated kernel Hilbert space is calculated using a radial basis kernel function. Feature decomposition is performed based on the kernel matrix to obtain the class decision boundary in the Hilbert space, and a Hilbert space classification result is generated. The multimodal fusion features are projected onto hyperbolic space using an exponential mapping. In hyperbolic space, a Poincaré sphere model is used to perform a geometric transformation on the projected feature vectors. The hyperbolic distance between the projected feature vectors and the preset prototype points of each category is calculated. The tree-like structure of hyperbolic geometry is used to amplify the difference in detail features under each category. Based on the hyperbolic distance metric and the difference in detail features under each category, the hyperbolic Softmax function is used to calculate the classification probability of each category, generating hyperbolic space classification results. The curvature parameter of the hyperbolic space is adaptively adjusted through a learnable parameter. The classification results of Euclidean space, Hilbert space, and hyperbolic space are used as parallel outputs of a multi-space classifier to form a three-dimensional spatial classification result matrix.

8. The method according to claim 7, characterized in that, After using the Euclidean space classification results and hyperbolic space classification results as parallel outputs of a multi-space classifier, the method further includes: Based on the 3D spatial classification result matrix, the correlation coefficient of any two sets of spatial classification results among Euclidean spatial classification results, Hilbert spatial classification results, and hyperbolic spatial classification results is calculated to generate a spatial consistency matrix; when the correlation coefficient of any two sets of spatial classification results is lower than a preset threshold, the spatial consistency verification mechanism is triggered and the reclassification process is started. Based on the verification results of the spatial consistency matrix, the entropy weight method is used to assign dynamic fusion weights to each spatial classification result; The three weighted spatial classification results are linearly combined to obtain a preliminary fused multi-spatial classification result. Temperature coefficient adjustment is introduced during the preliminary fusion process to optimize the smoothness of the probability distribution of the multi-spatial classification result. The confidence level of the initial fused multi-space classification results is calibrated, and the output probability distribution is adjusted using the Platt scaling method. When the maximum probability value is lower than the confidence threshold, the multi-space voting mechanism is activated, and the final image classification result is determined according to the majority voting principle.

9. The method according to claim 1, characterized in that, The dynamic routing mechanism adaptively outputs image classification results based on a confidence threshold, including: The input image is fed into a lightweight pre-screening network based on the MobileNetV3 architecture. Feature extraction and fast inference are performed through depthwise separable convolution to obtain coarse-grained classification results and corresponding confidence scores. The coarse-grained classification results are input into the pre-trained dedicated confidence network CLCNet, and the numerical distribution pattern of the coarse-grained classification results is analyzed through the Restricted Self-Attention module. If the confidence level is higher than the coarse classification confidence threshold, the coarse classification result of the lightweight pre-screening network is directly output as the image classification result. If the confidence level is not higher than the coarse classification confidence threshold, the multi-space classification result via Euclidean space, Hilbert space, and hyperbolic space will be output.

10. A visual classification processing device based on the fusion of large models and multimodal data, characterized in that, The device includes the following modules: An extraction module is used to acquire an input image to be classified, input the input image into a visual encoder for multi-level feature extraction to obtain global visual features and local visual features; acquire a category text description corresponding to the input image, input the category text description into a text encoder for multi-level feature extraction to obtain global text features and local text features, wherein the local text features are obtained by encoding key text phrases in the category text description; The alignment module is used to perform fine-grained cross-modal alignment of local visual features and local text features. It calculates the association weight between image regions and text phrases through a bidirectional cross-attention mechanism to generate aligned intermediate features. The bidirectional cross-attention mechanism includes: image-to-text attention flow with local image features as queries and local text features as keys, and text-to-image attention flow with local text features as queries and local image features as keys; The fusion module is used to concatenate and fuse the aligned intermediate features with the global visual features, and combine the global text features to suppress background noise in the fusion result and enhance the discriminative features in the fusion result through a feature masking algorithm to obtain multimodal fusion features. The classification module is used to synchronously input the multimodal fusion features into a multi-space classifier, which includes a Euclidean space classifier, a Hilbert space classifier, and a hyperbolic space classifier, respectively generating Euclidean space classification results, Hilbert space classification results, and hyperbolic space classification results. Combined with a dynamic routing mechanism, the image classification results are adaptively output according to the confidence threshold.

Images