Method, apparatus and medium for generating visual token based on shared index

Abstract

The application relates to the technical field of artificial intelligence, can be applied to business scenes such as financial technology and medical health, and discloses a visual token generation method and device based on a shared index, equipment and a medium, which comprises the following steps: acquiring an input image and extracting semantic features and pixel features through a semantic encoder and a pixel encoder; calculating the distance between each feature and its codebook and determining a shared index by weighted summation; retrieving quantized features from the codebook by using the shared index; generating a reconstructed image and reconstructed semantic features by using a pixel decoder and a semantic decoder respectively; jointly optimizing the encoder, the codebook and the decoder based on the reconstruction results; and generating a unified visual token sequence for a target task image by using the optimized components. Through double-flow feature extraction, shared mapping quantization and joint loss optimization, the application can simultaneously retain global semantic information and local pixel details in the visual token, so that the model has both accurate understanding ability and can generate high-fidelity images, and the performance of the understanding and generation tasks is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a visual token generation method and device based on shared index, equipment and medium. BACKGROUND

[0002] In the field of artificial intelligence, the development of multi-modal technology has become a key direction to improve the comprehensive understanding and generation ability of intelligent systems. Especially in application scenarios that require processing visual and language information simultaneously, how to enable the model to share a unified processing mechanism between different tasks still faces great challenges. Traditional technology often uses independent models to process visual understanding and image generation. This separate design not only leads to complex systems, high maintenance costs, but also low training and inference efficiency, making it difficult to meet the dual requirements of flexibility and efficiency in business.

[0003] In the field of financial technology business, tasks such as risk assessment, intelligent investment, and customer profiling need to process multi-modal data from users, such as combining user-uploaded identity pictures and text information for understanding, and generating reports accordingly. In existing technology, the generation of visual tokens usually relies on single pixel feature encoding or semantic feature encoding, which cannot simultaneously consider image details and overall semantics, resulting in insufficient accuracy and generalization of visual-related tasks in financial risk control systems. This limitation makes it difficult to ensure both fine-grained features and sufficient expression of image semantics in tasks such as customer identity verification that require high-precision visual expression.

[0004] In the field of medical health business, scenarios such as image diagnosis and auxiliary report generation also have strict requirements for multi-modal visual representation. For example, when processing medical images (such as MRI and CT) and inputting them as tokens into a unified model, existing methods often focus on capturing pixel-level details to ensure reconstruction quality, ignoring high-level semantic features such as global anatomical structure, which affects the accuracy of diagnosis and model interpretability in subsequent analysis. SUMMARY

[0005] The main purpose of the present application is to provide a visual token generation method, device, equipment and storage medium based on shared index, aiming to solve the technical problem that the existing single codebook cannot simultaneously meet the high-level semantics required by understanding tasks and the low-level details required by generation tasks when generating visual tokens, making it difficult for the model to balance understanding and generation capabilities.

[0006] To achieve the above purpose, the present application provides a visual token generation method based on shared index, comprising:

[0007] acquire input image data, and process the input image data through a semantic encoder and a pixel encoder to obtain semantic features and pixel features respectively;

[0008] determine distances of the semantic features and the pixel features to a semantic codebook and a pixel codebook respectively to obtain semantic distances and pixel distances respectively;

[0009] determine a shared index by minimizing a weighted sum of the semantic distances and the pixel distances;

[0010] retrieve quantized semantic features and quantized pixel features from the semantic codebook and the pixel codebook respectively using the shared index;

[0011] generate a reconstructed image by processing the quantized pixel features through a pixel decoder, and generate a reconstructed semantic feature by processing the quantized semantic features through a semantic decoder;

[0012] jointly update the semantic encoder, the pixel encoder, the semantic codebook, the pixel codebook, the semantic decoder and the pixel decoder based on a loss function of the reconstructed image and the reconstructed semantic feature;

[0013] process target task image data based on the updated semantic encoder, the pixel encoder, the semantic codebook and the pixel codebook, determine a target task shared index, and output the target task shared index as a unified visual token sequence.

[0014] Further, to achieve the above object, the present application provides a visual token generation device based on a shared index, comprising:

[0015] a feature encoding module configured to acquire input image data, and process the input image data through a semantic encoder and a pixel encoder to obtain semantic features and pixel features respectively;

[0016] a distance calculation module configured to determine distances of the semantic features and the pixel features to a semantic codebook and a pixel codebook respectively to obtain semantic distances and pixel distances respectively;

[0017] a shared index determination module configured to determine a shared index by minimizing a weighted sum of the semantic distances and the pixel distances;

[0018] a quantized feature retrieval module configured to retrieve quantized semantic features and quantized pixel features from the semantic codebook and the pixel codebook respectively using the shared index;

[0019] a feature decoding module configured to generate a reconstructed image by processing the quantized pixel features through a pixel decoder, and generate a reconstructed semantic feature by processing the quantized semantic features through a semantic decoder;

[0020] The joint optimization module is used to jointly update the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder based on the loss function of the reconstructed image and the reconstructed semantic features.

[0021] The visual token generation module is used to process the target task image data based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, determine the target task shared index, and output the target task shared index as a unified visual token sequence.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a shared index-based visual token generation program stored in the memory and executable on the processor, wherein when the shared index-based visual token generation program is executed by the processor, it implements the steps of the shared index-based visual token generation method as described above.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a visual token generation program based on a shared index, wherein when the visual token generation program based on the shared index is executed by a processor, it implements the steps of the visual token generation method based on a shared index as described above.

[0024] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for generating visual tokens based on a shared index, comprising: acquiring an input image and extracting semantic and pixel features using a semantic encoder and a pixel encoder; calculating the distance between each feature and its codebook and performing a weighted sum to determine a shared index; retrieving quantized features from the codebook using the shared index; generating a reconstructed image and reconstructed semantic features using a pixel decoder and a semantic decoder respectively; jointly optimizing the encoder, codebook, and decoder based on the reconstruction results; and generating a unified visual token sequence for the target task image using the optimized components. This invention, through a dual-stream encoder and dual-codebook structure combined with a shared index mechanism, optimizes high-level semantics and low-level details using joint loss during training, thereby achieving the generation of unified visual tokens. It can balance semantic extraction in visual understanding tasks and detail restoration in visual generation tasks, effectively solving the problem that existing models cannot simultaneously satisfy understanding and generation capabilities. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0026] Figure 1 This is a schematic diagram of an application environment for a visual token generation method based on a shared index, according to an embodiment of the present invention.

[0027] Figure 2 This is a flowchart illustrating an embodiment of the visual token generation method based on a shared index according to the present invention.

[0028] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the visual token generation device based on a shared index of the present invention.

[0029] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] The visual token generation method based on shared index provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can acquire the input image from the client and extract semantic and pixel features through a semantic encoder and a pixel encoder; calculate the distance between each feature and its codebook and perform a weighted sum to determine the shared index; retrieve quantized features from the codebook using the shared index; generate reconstructed images and reconstructed semantic features using a pixel decoder and a semantic decoder respectively; jointly optimize the encoder, codebook, and decoder based on the reconstruction results; and generate a unified visual token sequence for the target task image using the optimized components. This invention, through a dual-stream encoder and dual-codebook structure combined with a shared index mechanism, optimizes high-level semantics and low-level details with joint loss during training, thereby achieving the generation of unified visual tokens. It can balance semantic extraction in visual understanding tasks and detail restoration in visual generation tasks, effectively solving the problem that existing models cannot simultaneously satisfy understanding and generation capabilities. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0033] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the visual token generation method based on a shared index provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2As shown, the visual token generation method based on shared index proposed in this invention includes the following steps:

[0035] S10, acquire input image data, and process the input image data through a semantic encoder and a pixel encoder to obtain semantic features and pixel features respectively;

[0036] In this embodiment, acquiring input image data is the initial operation of the system entering the workflow, aiming to collect raw data input of the target visual content. The source of the input image data can include local storage, network transmission, or real-time acquisition via an image acquisition device such as a camera. Different image formats and resolutions can be supported during acquisition, and the data loading interface can be selected according to application requirements to ensure the integrity and availability of the input image data.

[0037] The operation of processing input image data through a semantic encoder involves feeding the acquired input image data as an input tensor into a deep neural network. Semantic encoders are typically built upon visual Transformer networks and possess the ability to semantically abstract the global content of the input image. Specifically, this can be achieved through block-based input preprocessing combined with a self-attention mechanism to establish global relationships between different regions of the input image, outputting feature representations that express high-level semantics. Visual Transformer networks primarily originate from recent research in computer vision, particularly for tasks such as image classification and visual understanding, and possess strong semantic context capture capabilities. These networks are widely used in multimodal applications as semantic abstraction modules at the image level.

[0038] The operations performed by a pixel encoder on input image data include extracting local detail features. Pixel encoders typically employ a convolutional neural network architecture, where the input tensor first passes through multiple convolutional layers, with each kernel extracting local texture information at different scales. The network may include residual connections to mitigate the vanishing gradient problem during deep network training and preserve low-level details. The use of convolutional neural networks stems from deep learning's strong representational power for image detail and has been widely validated in image reconstruction and style transfer tasks.

[0039] After processing the input image data, semantic features and pixel features are obtained. Semantic features, output by a semantic encoder, represent the global semantic abstraction of the input image. Pixel features, output by a pixel encoder, represent the local texture details of the input image. The output tensors of both can be used in parallel in subsequent processing, forming a two-stream representation path that serves different task requirements.

[0040] The acquisition of input image data can be achieved by directly calling image data from a remote storage service API, loading pre-annotated training sets through a local data reading interface, or acquiring data in real time through edge computing devices. The semantic encoder can be implemented using a standard visual Transformer model and supports adjusting the window size and embedding channel number of the patch segmentation module according to different image input sizes. The pixel encoder can be implemented using a multi-layer convolutional network with variable kernel sizes, and the network depth or number of channels can be adjusted to adapt to the requirements of local detail description capabilities in different tasks.

[0041] If the application scenario has specific requirements for the input image resolution, the input image data can be uniformly processed through interpolation scaling or image normalization preprocessing to ensure the consistency of the input tensor format. To adapt to scenarios with higher computational efficiency requirements, the semantic encoder can be replaced with a lightweight Transformer variant such as the Swing Transformer, and the pixel encoder can be replaced with a MobileNet series network to reduce memory consumption and computational latency.

[0042] This embodiment employs a dual-stream approach, simultaneously introducing a semantic encoder and a pixel encoder to process the input image data. This enables the system to fully acquire both global semantic information and local texture detail descriptions from the input image, laying the foundation for accurate shared index selection in subsequent processing. The dual-stream representation path ensures that global semantic abstraction and local detail capture of the same input image proceed in parallel without interference, enhancing the balance between semantic consistency and reconstruction fidelity during subsequent multimodal token generation.

[0043] S20, determine the distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook, and obtain the semantic distance and pixel distance respectively;

[0044] In this embodiment, the distance between semantic features and the semantic codebook is determined by comparing the semantic features obtained from the semantic encoder with each semantic codebook feature vector one by one. Semantic features represent the global semantic description tensor obtained after processing the input image data by the semantic encoder. Semantic codebook feature vectors are a predefined set of standard semantic vectors used to provide a unified semantic representation reference. Distance calculation typically uses the squared Euclidean distance as a metric. For each semantic codebook feature vector, the squared Euclidean distance between the semantic feature and that vector is calculated, ultimately forming a semantic distance set. The calculation of the squared Euclidean distance originates from vector space metric theory and can describe the similarity between different vectors in the semantic space in a concise mathematical form.

[0045] Determining the distance between pixel features and the pixel codebook is also done through a one-to-one comparison. The squared Euclidean distance between the pixel features output by the pixel encoder and each pixel codebook feature vector is calculated. Pixel features represent local texture details of the input image data, while pixel codebook feature vectors are a predefined set of standard detail descriptions, also using the squared Euclidean distance as the comparison benchmark. For each pixel codebook feature vector, the squared Euclidean distance between the pixel feature and that vector is calculated, forming a set of pixel distances.

[0046] Ultimately, the semantic distance set and the pixel distance set are used as the output results of semantic distance and pixel distance, respectively, for the subsequent shared index selection process. The entire distance calculation process requires traversing all codebook feature vectors to ensure that the calculation results cover the entire reference space of the codebook. The determination of semantic distance and pixel distance is carried out in parallel with two streams, with different sets calculated independently without interfering with each other, ensuring the feature alignment capability between the semantic space and the detail space.

[0047] The distance calculation between semantic features and the semantic codebook can be efficiently implemented in a GPU environment using matrix broadcasting. After tensor expansion of the semantic feature vectors and the semantic codebook feature vector matrices, batch calculations are performed directly, avoiding iterative traversal and improving computational efficiency. The distance calculation between pixel features and the pixel codebook can employ the same tensor parallel approach, ensuring consistency with the semantic distance calculation process in terms of resource usage and computational complexity.

[0048] In scenarios requiring large-scale codebooks, approximate nearest neighbor search algorithms can be introduced to reduce the number of comparisons. For example, using the FAISS library or Locality Sensitive Hashing (LSH) can reduce computational complexity, making it particularly suitable for real-time multimodal processing systems with stringent response time requirements. Regarding the organization and storage of codebook feature vectors, contiguous memory arrays can be used, or GPU memory optimization can be employed to improve storage access bandwidth.

[0049] This embodiment achieves accurate measurement of global semantic features and local detail features of the input image in a standard reference space by separately determining the squared Euclidean distance between semantic features and semantic codebook feature vectors, and simultaneously determining the squared Euclidean distance between pixel features and pixel codebook feature vectors. This provides a high-resolution distance comparison basis for subsequent shared index selection. The dual-stream distance calculation process ensures the independence and parallelism of the semantic space and detail space, improving the robustness and applicability of the index selection stage.

[0050] S30, determine the shared index by minimizing the weighted sum of the semantic distance and the pixel distance;

[0051] In this embodiment, determining the shared index based on minimizing the weighted sum of semantic distance and pixel distance is a process of combining and measuring the semantic distance set and pixel distance set obtained in the previous stage. Semantic distance represents the similarity between the semantic features of the input image and the standard vectors of the semantic codebook, while pixel distance represents the similarity between the pixel features of the input image and the standard vectors of the pixel codebook. To establish a balance between the two, a weight parameter is introduced, which adjusts the relative importance of semantic distance and pixel distance in the weighted summation. The weight parameter is derived from the loss weight adjustment principle commonly used in multi-task learning; by adjusting this parameter, a flexible trade-off can be made between global semantic consistency and local detail fidelity.

[0052] The weighted sum calculation is performed separately for each index position shared by the semantic codebook and the pixel codebook. For each index position, the semantic distance value and pixel distance value corresponding to that position are first obtained. The pixel distance value is then multiplied by the weight parameter to obtain the pixel weighted result. Finally, the semantic distance value and the pixel weighted result are added to form the weighted sum result for the current index position. After the weighted sum results for all index positions are calculated, these weighted sum results are compared, and the index position corresponding to the minimum value is selected as the shared index. The shared index is used to indicate that the position has the highest comprehensive matching degree in the semantic space and the detail space, and can serve as a unified reference for both.

[0053] This weighted summation process can be efficiently completed in a parallel computing environment through vectorized operations. The weight parameters participate in the broadcast operation as a scalar, ensuring consistency and efficiency in the calculation of all index positions. This step is closely linked to the semantic distance and pixel distance calculation process in the previous stage. The input data comes directly from the semantic distance set and the pixel distance set, and the output results are directly used as the shared index input for subsequent codebook query operations.

[0054] In practical implementations, weight parameters can be configured and adjusted according to different datasets or application requirements. For example, smaller weight parameter values ​​can be set in tasks requiring stronger global semantic understanding to reduce the impact of pixel distance, while larger weight parameter values ​​can be set in tasks requiring higher fidelity of local details to increase the contribution of pixel distance. The comparison of weighted sum results and the selection of the minimum value can be efficiently implemented through GPU parallel reduction operations, such as using tensor minimum value reduction instructions to quickly determine the index position corresponding to the minimum weighted sum. For different index space sizes, the processing can be further accelerated through block-level parallelism and index batch distributed computing, which is particularly suitable for real-time processing needs in large-scale codebook scenarios.

[0055] This embodiment combines semantic distance and pixel distance through a weighted and minimization mechanism, achieving a dynamic balance between global semantic consistency and local detail fidelity, thereby improving the representativeness and accuracy of the shared index. This method ensures that index selection considers both the semantic information of the image content and retains rich detail information, providing a unified and optimal basis for subsequent codebook queries and quantized feature retrieval.

[0056] S40, using the shared index to retrieve quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook respectively;

[0057] In this embodiment, retrieving quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook using a shared index is based on the shared index value determined in the previous stage. The standard feature vector corresponding to the index position is located and extracted in both the semantic codebook and pixel codebook. The shared index is an integer position identifier representing the specific index position to be accessed in the semantic codebook and pixel codebook. The semantic codebook is a predefined set of standard semantic features, with each position storing a semantic feature vector to express a certain semantic pattern or high-level abstract information. The pixel codebook is a predefined set of standard pixel features, with each position storing a pixel feature vector to express local detail texture or pixel-level distribution information.

[0058] The retrieval process first parses the shared index value, using it as an access identifier to input into the semantic codebook's storage structure. It then retrieves the semantic codebook feature vector at that location using direct addressing, outputting it as the quantized semantic feature. The same shared index is used to access the pixel codebook's storage structure, retrieving the pixel codebook feature vector at that location, which is then output as the quantized pixel feature. This retrieval operation features random access and can be efficiently implemented using standard arrays or tensor indexing mechanisms. The access latency is logarithmically related to the codebook size rather than linearly.

[0059] During the retrieval process, the consistency of the index space between the semantic codebook and the pixel codebook is ensured, guaranteeing that the shared index accurately corresponds to the feature vectors at the same positions in both codebooks. The quantized semantic features and quantized pixel features output after the retrieval serve as input data for subsequent decoding processing, directly determining the subsequent reconstruction effect and semantic consistency level.

[0060] In practical implementations, semantic codebooks and pixel codebooks can employ various data storage structures such as contiguous memory arrays, dense tensors, or feature matrices, supporting direct index access to improve retrieval efficiency. Shared index parsing ensures consistency and correctness during cross-platform or cross-module calls through data type conversion and index offset adjustment. Adaptation to different codebook capacities can be achieved through dynamically expanding the index space or partitioning storage areas, supporting large-scale feature vector storage and retrieval while maintaining index consistency.

[0061] Retrieval operations can be implemented using tensor indexing instructions in the graphics processing unit (GPU). Combining batch index access with pipelined operations reduces latency and supports high-throughput processing. Furthermore, caching the feature vectors corresponding to frequently used shared indices can reduce the performance overhead of frequent accesses to the full codebook.

[0062] This embodiment achieves efficient and accurate acquisition of standardized quantized features that match the input image in terms of global semantics and local details through a unified retrieval operation of the shared index in the semantic codebook and pixel codebook. These quantized features retain the main semantic information of the input image and contain necessary local detail features, providing high-quality input for the subsequent decoding stage and reducing information loss and bias.

[0063] S50, the quantized pixel features are processed by the pixel decoder to generate a reconstructed image, and the quantized semantic features are processed by the semantic decoder to generate reconstructed semantic features;

[0064] In this embodiment, generating a reconstructed image by processing quantized pixel features through a pixel decoder means inputting the quantized pixel features retrieved from the pixel codebook into the pixel decoder, and having the pixel decoder recover the corresponding image data. Quantized pixel features are low-dimensional discrete vectors, and the pixel decoder is responsible for mapping these discrete vectors into two-dimensional image data in a continuous pixel space. The pixel decoder can internally employ a multi-layer convolutional neural network structure. Such networks use progressive feature mapping and upsampling operations to restore the quantized pixel features into a reconstructed image with spatial resolution. Residual connections, batch normalization, and other modules can be embedded in the convolutional neural network to improve the expressive power of feature recovery and training stability.

[0065] Generating reconstructed semantic features by processing quantized semantic features through a semantic decoder involves taking quantized semantic features retrieved from a semantic codebook as input to a semantic decoder and outputting the decoded semantic representation. The goal is to restore standardized discrete semantic codes into high-level semantic features in a continuous space. The semantic decoder can employ a multilayer perceptron structure, consisting of stacked fully connected layers, to achieve nonlinear mapping and reconstruction of quantized semantic features. Each perceptron layer can introduce activation functions to enhance nonlinear modeling capabilities, while normalization mechanisms can be used to accelerate convergence and improve robustness.

[0066] Throughout the processing, quantized pixel features and quantized semantic features are independently input into their respective decoders. While their decoding operations are independent, they can be parallelized and accelerated during system execution. The reconstructed image and reconstructed semantic features output after processing are used for pixel space reproducibility measurement and semantic space alignment measurement, respectively, forming the basis for subsequent training and loss function calculation.

[0067] In implementation, the pixel decoder can employ a stacked convolutional layer structure, including a combination of standard convolutional layers, deconvolutional layers, and upsampling modules. Different kernel sizes and stride parameters can be selected for different scenarios. For example, for high-resolution images, more convolutional layers and a reduced stride can be used to refine the feature mapping, while for low-resolution images, fewer convolutional layers and a increased stride can be used to improve efficiency. A perceptual loss auxiliary network or a generative adversarial network discriminator can also be integrated into the pixel decoder to improve the visual quality and realism of the decoded image.

[0068] The multilayer perceptron in the semantic decoder can adjust the number of layers and hidden units to adapt to the needs of different semantic feature dimensions and semantic complexity. Regularization mechanisms with dropout can be used during training to prevent overfitting and improve model generalization performance. For high-dimensional quantized semantic features, the decoder can perform dimensionality compression before decoding to reduce computational complexity while maintaining semantic expressiveness.

[0069] The output resolution and semantic space dimension of different decoders can be adjusted according to the needs of downstream tasks. For example, different output image sizes or different semantic feature vector lengths can be configured to support multi-scene adaptation.

[0070] This embodiment, through independent processing by pixel decoder and semantic decoder, can recover high-fidelity images and high-consistency semantic features corresponding to quantized pixel features and quantized semantic features respectively. This enables both the training and inference stages to have the ability to restore details and global semantic consistency at the same time, effectively solving the problem that a single decoder cannot simultaneously take into account visual details and semantic abstraction.

[0071] S60, based on the loss function of the reconstructed image and the reconstructed semantic features, jointly update the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder;

[0072] In this embodiment, the process of jointly updating the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder based on the loss function of the reconstructed image and reconstructed semantic features involves end-to-end gradient optimization of each component of the entire system from the output result. The loss function is used to measure the difference between the model output and the target. The loss of the reconstructed image can be calculated using the mean squared error of the pixel space, perceptual loss, or adversarial loss, reflecting the model's ability to restore visual details. The loss of the reconstructed semantic features is used to measure the difference between the semantic decoder output and the target high-level semantic features, and can use Euclidean distance or cosine similarity as the metric.

[0073] The loss function can also incorporate vector quantization loss related to the semantic codebook and pixel codebook, used to constrain the proximity between the encoder's output features and the vectors in the codebook, ensuring the expressive power and stability of the quantization process. Combining these loss terms forms the overall loss, which is used for training iterations.

[0074] Joint update refers to updating the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder simultaneously through backpropagation of the gradient of the overall loss. During backpropagation, the gradient of each component's parameters with respect to the total loss is calculated, and these gradients are input into an optimization algorithm (such as stochastic gradient descent or the Adam optimizer) to adjust the learnable parameters of each component and reduce the loss value.

[0075] The entire joint update process ensures that the parameters of each component converge together under the same objective, and the trained system can simultaneously improve the quality of visual reconstruction and semantic consistency.

[0076] In practical implementations, the composition of the loss function can be flexibly configured. For example, the pixel reconstruction loss can use L1 loss instead of L2 loss to reduce sensitivity to noise, or a combination of perceptual loss and multi-scale adversarial loss can be added when high reconstruction quality is required to improve perceptual realism. Semantic alignment loss can introduce regularization constraints to maintain the stability of the semantic space distribution.

[0077] The choice of optimization algorithm can be adjusted according to the characteristics of the task. For example, when there is a large amount of training data and a high convergence speed is required, the Adam optimization algorithm with a large learning rate can be used. When the amount of data is limited or the training stability requirement is high, the stochastic gradient descent algorithm with momentum can be used with a lower learning rate. Different learning rates or weight decay parameters can also be set for different components. For example, a lower learning rate can be configured for the semantic codebook and pixel codebook to stabilize codebook updates, while a higher learning rate can be set for the encoder and decoder to improve feature abstraction and decoding capabilities.

[0078] For large-scale data training, a distributed training framework can be used to achieve consistent parameter updates across multiple computing nodes through gradient synchronization mechanisms, reducing training time and improving training efficiency. For memory-constrained environments, gradient accumulation techniques can be used to accumulate gradients from multiple mini-batch calculations and apply them to parameter updates, adapting to hardware limitations.

[0079] Through the joint update operation described above, each component can be trained collaboratively on the goals of pixel restoration and semantic consistency. This enables the overall model to have strong visual detail fidelity and semantic accuracy during inference, overcoming the problem in existing methods where individual optimization makes it difficult to balance detail fidelity and semantic consistency. This significantly improves training efficiency and overall model performance.

[0080] S70, based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, process the target task image data, determine the target task shared index, and output the target task shared index as a unified visual token sequence.

[0081] In this embodiment, the updated semantic encoder, pixel encoder, semantic codebook, and pixel codebook constitute an end-to-end optimized training structure for discretizing the input image, making it particularly suitable for inference processing scenarios. First, target task image data is acquired in this process; this is real-world application data that did not appear during the training process and possesses novel semantic and visual features. The target task image data is input into the updated semantic encoder and pixel encoder. The semantic encoder, through its trained and adjusted deep network structure, extracts high-level semantic features from the global feature level. These features are highly abstract and can represent scene content, object categories, and semantic relationships. The pixel encoder, through its convolutional or other adapted network structures, focuses on extracting low-level features such as texture, edges, and color at the local pixel level from a detailed perspective.

[0082] At this point, the extracted semantic features and pixel features are used to calculate distances with the updated semantic codebook and pixel codebook, respectively. The distance calculation typically uses the squared Euclidean distance form, which measures the difference between the current feature and each feature vector in the codebook. The results of the distance calculation form a semantic distance set and a pixel distance set. These two sets reflect the degree of matching between the target task image data and the current encoder and codebook representation spaces.

[0083] Based on this distance information, a weighted sum-minimization index determination operation is performed. Pixel distances are typically multiplied by weight parameters to balance detail and semantics, while semantic distance is included as a direct term in the weighted sum calculation. All codebook entries are traversed within the index space, and the weighted sum results are compared. Finally, the index position with the smallest corresponding weighted sum is determined and used as the shared index for the target task.

[0084] The concept of shared indexes unifies the index representation of two streams, enabling the coordinated representation of semantic and pixel-level feature encodings. This ensures that downstream multimodal models can understand the target task image data using the same discrete token sequence. The shared indexes are arranged sequentially to form a unified visual token sequence. This sequence describes the semantic and visual integrated features of local regions in the target task image in the form of integer indices, serving as standard discretized data for subsequent multimodal model inputs.

[0085] The unified visual token sequence transforms an input image into an ordered, discrete sequence of index numbers through a series of encoding and discretization operations. These numbers correspond to predefined feature vectors in the semantic codebook and pixel codebook. Each index in this sequence not only represents the pixel details of a local image but also contains trained abstract semantic information, exhibiting compactness, standardization, and scalability. The generation process of the unified visual token sequence ensures that visual information is uniformly mapped to a discrete space compatible with language model input, allowing visual data and text data to be processed using the same model architecture and the same input format.

[0086] In multimodal large-scale models, this sequence can be directly used as input to autoregressive models for image-text alignment tasks, such as requiring the model to generate descriptive text or answer visual questions after inputting an image. In visual understanding tasks, the unified visual token sequence can serve as an intermediate representation for visual summarization, used in downstream scenarios such as multimodal retrieval, cross-modal dialogue, and intelligent question answering. In generation tasks, this sequence can serve as a conditional input to guide the model to progressively generate high-quality images consistent with the original image content, ensuring consistency in detail and semantics during the visual generation process. Furthermore, in cross-domain tasks, the unified visual token sequence can also serve as a compact storage and transmission format for visual information, supporting large-scale image storage optimization and efficient transmission, making it particularly suitable for high-concurrency, high-precision applications such as medical image archiving and digital archiving of financial documents.

[0087] In practical deployments, semantic encoders can use multi-layer Transformer structures, with inputs derived from image representations enhanced by multi-scale feature pyramids, enabling the extraction of global semantic context and cross-regional relationships. Pixel encoders can be based on deep convolutional neural networks and enhance their ability to perceive details through residual connections, feature pyramids, or dynamic convolutional modules.

[0088] The semantic codebook and pixel codebook exist as read-only structures during inference and are typically stored in a cache to reduce latency. Distance calculation operations can be performed in batch parallel processing, for example, by using matrix multiplication and broadcasting to accelerate the distance calculation process between all local blocks of the target task image data and vectors in the codebook.

[0089] The weight parameters can be flexibly adjusted. For example, for scenarios with higher detail requirements, the weight of pixel distance can be increased; for applications where semantic consistency is more important, the weight of pixel distance can be decreased to highlight the importance of semantic distance. The index determination operation can be completed through vectorized procedures, efficient sorting, and minimum value selection functions, improving the throughput efficiency of batch target image data.

[0090] The output format of the unified visual token sequence can be directly interfaced with the input interface of a general autoregressive model, for example, encoded as an integer sequence and sharing the input channel with text tokens, so that a single autoregressive model can process both visual and text inputs at the same time, promoting multimodal interaction.

[0091] This embodiment achieves efficient and unified discrete encoding of novel inputs by jointly processing the target task image data using a trained semantic encoder, pixel encoder, semantic codebook, and pixel codebook during the inference phase. This discrete encoding simultaneously possesses high-level semantic representation capabilities and low-level detail fidelity. A weighted and minimization method is used to determine the shared index, resulting in a unified visual token sequence with autoregressive modeling compatibility. Compared to traditional independent encoding paths or single codebook schemes, this approach improves visual understanding accuracy and visual reconstruction detail without increasing runtime complexity, significantly reduces system latency and resource consumption, and enhances the adaptability of the visual token sequence to different application scenarios.

[0092] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for generating visual tokens based on a shared index, comprising: acquiring an input image and extracting semantic and pixel features using a semantic encoder and a pixel encoder; calculating the distance between each feature and its codebook and performing a weighted sum to determine a shared index; retrieving quantized features from the codebook using the shared index; generating a reconstructed image and reconstructed semantic features using a pixel decoder and a semantic decoder respectively; jointly optimizing the encoder, codebook, and decoder based on the reconstruction results; and generating a unified visual token sequence for the target task image using the optimized components. This invention, through a dual-stream encoder and dual-codebook structure combined with a shared index mechanism, optimizes high-level semantics and low-level details using joint loss during training, thereby achieving the generation of unified visual tokens. It can balance semantic extraction in visual understanding tasks and detail restoration in visual generation tasks, effectively solving the problem that existing models cannot simultaneously satisfy both understanding and generation capabilities.

[0093] In one embodiment, step S10 above includes:

[0094] S101, acquire input image data, and perform multi-scale feature enhancement processing on the input image data to obtain enhanced image data;

[0095] S102, The enhanced image data is processed by the visual Transformer network in the semantic encoder to obtain high-level semantic features;

[0096] S103, the enhanced image data is processed by the residual convolutional network in the pixel encoder to obtain the underlying pixel features;

[0097] S104, Perform cross-modal feature alignment operation on the high-level semantic features and the low-level pixel features to generate aligned semantic features and aligned pixel features.

[0098] In this embodiment, the initial step of the entire processing flow is to acquire input image data. This operation involves the acquisition and import of the original input images. This input image data can come from various scenarios, such as two-dimensional or three-dimensional images acquired by medical imaging systems, or scanned copies of invoices acquired by financial document image acquisition terminals, ensuring that the input data serves as the basis for subsequent calculations under a unified resolution, color space, and file format. After acquiring the input image data, multi-scale feature enhancement processing is immediately performed on the input image data. This processing constructs a multi-level feature pyramid structure, resampling and standardizing the original image at different spatial resolutions, and combining local image details and global contextual information at different scales to form enhanced image data. In this process, methods such as Gaussian pyramids, Laplacian pyramids, or spatial pyramid pooling algorithms can be combined to maximize the preservation of image details and global structure.

[0099] After multi-scale feature enhancement, the enhanced image data is input into a semantic encoder for processing. The semantic encoder uses a visual Transformer network as its core structure. The visual Transformer network utilizes a self-attention mechanism to globally model the image data, dividing image patches into a sequence of uniform-sized patches. Within a high-dimensional embedding space, it calculates and captures semantic features in the image through global contextual relationships. This calculation can flexibly adapt to the spatial topological relationships of different image content, enabling high-level semantic features to cover abstract semantic information including object categories and scene content. The visual Transformer network can further incorporate positional encoding to ensure the integrity of the spatial relationships between image patches.

[0100] Subsequently, the enhanced image data needs to be processed by a pixel encoder, which is based on a residual convolutional network structure. Residual convolutional networks alleviate the vanishing gradient problem in deep network training by introducing residual connections, while also enhancing the ability to capture local spatial details, making them particularly suitable for recovering low-level visual details such as texture and contours. The pixel encoder performs multi-layer convolution operations on the local pixel distribution in the enhanced image data. Through successive convolutions, normalization, and activation units, it extracts low-level pixel features, preserving the complete representation of local high-frequency detail information.

[0101] After extracting high-level semantic features and low-level pixel features, a cross-modal feature alignment operation needs to be performed on the outputs of these two different dimensions and feature spaces. This operation aims to map the high-level semantic features and low-level pixel features into the same or comparable feature spaces, so as to facilitate subsequent shared encoding and index calculation. Cross-modal feature alignment can be achieved through methods such as spatial interpolation, scale normalization, and channel mapping. It can also be combined with multi-head attention mechanisms or two-stream alignment network structures. By calculating the feature similarity matrix between cross-modal features, features from different sources are fused and adjusted, ultimately generating aligned semantic features and aligned pixel features, providing standardized feature inputs for subsequent codebook distance calculation and index minimization.

[0102] This embodiment enriches the expression of the original image at different spatial scales by performing multi-scale feature enhancement processing on the input image data, improving the ability to balance local details and global context. The visual Transformer network processes the enhanced image data, fully capturing high-level semantic relationships and improving the accuracy of semantic understanding. The residual convolutional network enhances the representation and stability of local details for the same enhanced image data, ensuring the integrity of low-level visual features. Cross-modal feature alignment eliminates the differences in spatial dimension, scale, and channel distribution between high-level semantic features and low-level pixel features, laying a unified standard for feature representation in subsequent shared index calculations. Through the close cooperation of these steps, effective alignment of different modal features is achieved while maintaining global semantics and local details, thereby improving the overall efficiency and accuracy of subsequent multi-task visual understanding and generation tasks.

[0103] In one embodiment, step S20 above includes:

[0104] S201, Obtain multiple semantic codebook feature vectors stored in the semantic codebook;

[0105] S202, determine the squared Euclidean distance between the semantic features and each semantic codebook feature vector to obtain a semantic distance set;

[0106] S203, obtain multiple pixel codebook feature vectors stored in the pixel codebook;

[0107] S204, determine the squared Euclidean distance between the pixel feature and the feature vector of each pixel codebook, and obtain the pixel distance set;

[0108] S205, the semantic distance set and the pixel distance set are respectively used as semantic distance and pixel distance.

[0109] In this embodiment, firstly, obtaining multiple semantic codebook feature vectors stored in the semantic codebook provides a standard reference set for distance calculation. As a discrete feature set, the semantic codebook stores a set of semantic codebook feature vectors that can be obtained through pre-training, clustering, or manual design. Each vector represents a standard semantic unit, forming a finite and fixed semantic representation space. This operation needs to ensure that the feature vectors in the codebook maintain consistency with the input semantic features in terms of dimension and numerical range to guarantee comparability. Subsequently, distance calculation is performed between the input semantic features and each semantic codebook feature vector stored in the semantic codebook. The distance metric uses the squared Euclidean distance, specifically calculated by summing the squares of the differences between the corresponding dimensional components of the input semantic features and each semantic codebook feature vector. This distance metric method originates from the similarity metric of traditional vector spaces and can accurately measure the spatial distance between the input semantic features and each standard semantic codebook feature vector, resulting in a semantic distance set used for subsequent optimal index determination.

[0110] Similarly, it is also necessary to obtain multiple pixel codebook feature vectors stored in the pixel codebook. The storage structure of the pixel codebook feature vectors is consistent with that of the semantic codebook. It is a set of standard pixel feature vectors formed through training or clustering, and its function is to provide a pixel-level feature quantization standard. The squared Euclidean distance is calculated for each input pixel feature and each pixel codebook feature vector stored in the pixel codebook. By solving for the square of the difference between each feature component and summing them, a pixel distance set is obtained, which is used to measure the similarity distribution between the input pixel feature and all pixel codebook feature vectors. Finally, the semantic distance set and the pixel distance set are organized and output as the semantic distance and pixel distance, forming the direct input for subsequent shared index minimization calculations. This process requires that the distance values ​​corresponding to each index position in the distance set are strictly ordered to ensure consistency across subsequent cross-set indices.

[0111] This embodiment calculates the squared Euclidean distance between the input semantic features and the semantic codebook feature vectors, accurately measuring the distance relationship between the input semantic features and standard semantic units in a predefined semantic space, thus enhancing the discretization representation capability of semantic encoding. By calculating the squared Euclidean distance between the input pixel features and the pixel codebook feature vectors, it ensures that pixel-level features can finely match standard pixel units during quantization, improving the accuracy of detail representation. Using the semantic distance set and pixel distance set as complete representations of the distance distribution provides standardized similarity measurement inputs for subsequent shared index calculations. Overall, it ensures a strict correspondence between different feature dimensions and the standard set, laying the foundation for data consistency and comparability of expression for subsequent cross-modal distance weighting and minimization processing, thereby improving the accuracy and stability of the unified visual token.

[0112] In one embodiment, step S30 above includes:

[0113] S301, Set the weight parameters to balance semantic importance and pixel importance;

[0114] S302, for each index position shared by the semantic codebook and the pixel codebook, obtain the semantic distance value and pixel distance value corresponding to the current index position;

[0115] S303, determine the product of the weight parameter and the pixel distance value to obtain the pixel weighting result;

[0116] S304, add the semantic distance value to the pixel weighted result to obtain the weighted sum result of the current index position;

[0117] S305, compare the weighted sum of all index positions shared by the semantic codebook and the pixel codebook, and determine the index position corresponding to the minimum weighted sum;

[0118] S306, the index position corresponding to the minimum weighted sum result is used as a shared index.

[0119] In this embodiment, setting a weight parameter that balances semantic importance and pixel importance is a numerical quantification method used to adjust the relative contributions of different modal features in distance-weighted calculation. This weight parameter is typically an adjustable hyperparameter derived from hyperparameter tuning practices during model training, reflecting the proportion of importance of semantic distance and pixel distance in determining the shared index. The purpose of setting this parameter is to achieve flexible modality fusion control, enabling the model to balance the importance of semantic features and pixel details at different training stages or for different datasets.

[0120] First, the weight parameter is usually introduced into the system as a scalar hyperparameter. This scalar defines the relative importance of pixel distance in the weighted sum calculation. The initial value can be set empirically or selected in hyperparameter tuning experiments. For example, the initial value can be set in a common range such as 0.1 to 1.0 to ensure that the numerical scale of pixel distance and semantic distance is comparable.

[0121] Then, the storage and retrieval of weight parameters need to directly participate in numerical calculations during weighted sum computation through a unified interface, typically existing as configurable items in the model configuration or training graph. This can be implemented by storing them as program-level global variables or tensors, and reading them from memory each time an indexed weighted sum is calculated.

[0122] During training, weight parameters can be kept statically as fixed hyperparameters, or they can be dynamically updated by introducing a learning mechanism. For example, by introducing the gradient definition of the weight parameters, they can participate in the optimization of the total loss function and automatically adjust according to the statistical distribution of the training data, making the numerical comparison between semantic distance and pixel distance more balanced. Dynamic weight learning can be achieved by adding regularization loss or jointly optimizing with the main loss function. For example, by limiting the convergence of weight parameters within a certain range, training instability can be prevented.

[0123] In addition, to adapt to different data scenarios, the specific values ​​of the weight parameters can be automatically adjusted according to the statistical characteristics of the input data distribution. For example, by statistically analyzing the mean or standard deviation of semantic distance and pixel distance on the training set before model training, the initial weight values ​​can be dynamically set according to the relative variance of these two distance indicators, so that the contributions of the two are roughly balanced in the weighted sum.

[0124] In engineering implementation, the maintenance of weight parameters is usually managed through training configuration files, model parameter tables, or trainable parameter mechanisms in deep learning frameworks (such as PyTorch or TensorFlow). If set as a trainable parameter, it must be explicitly added to the model's parameter update list to ensure that its gradient is correctly calculated during backpropagation and updated through optimization algorithms (such as Adam, SGD).

[0125] For each index position shared by the semantic codebook and pixel codebook, the semantic distance value and pixel distance value corresponding to the current index position are obtained. This must strictly adhere to the index order to ensure comparability between indices. This process relies on the semantic distance set and pixel distance set output in previous steps. These sets must be organized to ensure a one-to-one correspondence between indices, allowing direct access to distance values ​​in both sets during index traversal. This ensures semantic consistency and data correspondence between semantic distance values ​​and pixel distance values ​​at the same index.

[0126] The product of the weight parameter and the pixel distance value is determined to obtain the pixel-weighted result. The purpose of this multiplication operation is to adjust the weight of the pixel distance value in the overall distance calculation, assigning the pixel distance a scale identical to that of the semantic distance, and adjusting the contribution ratio of the pixel distance in the determination of the shared index through this scale. This calculation is strictly implemented using scalar multiplication with the distance scalar to maintain the numerical comparability and homogeneity of the results.

[0127] The semantic distance value is added to the pixel weighted result to obtain the weighted sum at the current index position. This addition operation is performed separately at each index position, and the resulting weighted sum strictly reflects the weighted composite metric of the importance of semantic and pixel similarity at that index position. This result not only expresses the degree of matching between the current index and the two modalities but also serves as the basis for subsequently selecting the optimal shared index. Therefore, the weighted sum result needs to maintain a strict correspondence with the index position in the data organization.

[0128] To determine the index position corresponding to the minimum weighted sum, the weighted sum of all index positions shared by the semantic codebook and the pixel codebook is compared. This requires performing a global minimum search operation on the entire sequence of weighted sums to ensure that the index that minimizes the weighted sum is accurately found from all candidate indices. This comparison process requires support for traversing the set of weighted sums and recording the minimum value. In practice, computational efficiency can be improved through linear traversal or other efficient sorting algorithms.

[0129] The minimum weighted sum and its corresponding index position are used as a shared index, explicitly pointing to the corresponding entry in the semantic codebook and pixel codebook under this index. Determining this shared index is an explicit way of expressing modality fusion and optimal quantization, representing the optimal consistent index of the current input semantic features and pixel features in the discrete codebook space. The shared index must maintain single-valuedness in the data structure, meaning that the index result, as a unified index, ensures that the access positions of the two codebooks are completely consistent during subsequent retrieval processes.

[0130] This embodiment, by setting weight parameters that balance semantic importance and pixel importance, can flexibly adjust the relative contributions of semantic distance and pixel distance in determining the shared index, effectively solving the problem that traditional single weighting rules are difficult to adapt to different task requirements. By obtaining strictly corresponding semantic distance and pixel distance values ​​at each index position and performing weighted calculations, it ensures that semantic and pixel similarity are equivalently compared in a unified space. By calculating the weighted sum result index by index and globally selecting the index with the smallest weighted sum as the shared index, it can be guaranteed that the selected shared index is optimal under the multimodal similarity fusion metric of semantics and pixels, thereby improving the representativeness and robustness of the shared index.

[0131] In one embodiment, step S40 above includes:

[0132] S401, parse the shared index to determine the target index location;

[0133] S402, access the storage structure of the semantic codebook and find the corresponding semantic codebook feature vector according to the target index position;

[0134] S403, the semantic codebook feature vector is used as a quantized semantic feature;

[0135] S404, access the storage structure of the pixel codebook and find the corresponding pixel codebook feature vector according to the target index position;

[0136] S405, the pixel codebook feature vector is used as the quantized pixel feature.

[0137] In this embodiment, a shared index is used as input to retrieve the corresponding standardized feature vectors from the semantic codebook and the pixel codebook, respectively, forming quantized semantic features and quantized pixel features that can be directly used for subsequent decoding operations.

[0138] First, resolving the shared index is the initial step in the retrieval process. The shared index can be viewed as an integer identifier derived from the previous minimization of the weighted sum of semantic distance and pixel distance. Resolving the shared index requires converting this integer value into an index position to access the semantic codebook and the pixel codebook, typically achieved through direct index mapping in data structures, such as the offset of the corresponding element in an array or tensor. The uniqueness and global consistency of the shared index ensure that it can directly locate the corresponding entry in two different codebooks.

[0139] Next, accessing the semantic codebook's storage structure requires consideration of storage medium, data format, and access latency. The semantic codebook is typically stored as a contiguous block of memory, a tensor, or a database table, with each entry corresponding to a fixed-length high-dimensional vector. Accessing this storage structure involves index addressing, which uses a shared index value to locate the corresponding feature vector in the semantic codebook and retrieve that vector data. In high-performance implementations, this operation may be optimized through low-latency memory addressing, vectorized instructions, or batch read interfaces.

[0140] After retrieving the semantic codebook feature vector, this vector needs to be directly used as the quantized semantic feature output for subsequent input into the semantic decoder processing chain. No additional calculations or mappings are performed here; the original content of the retrieved vector is preserved, ensuring that the quantized semantic features are consistent with the predefined standard semantic feature space in the codebook.

[0141] The process of accessing the pixel codebook is consistent with that of the semantic codebook, including accessing the storage structure, addressing shared index locations, and retrieving corresponding feature vectors. The pixel codebook feature vectors represent the low-level details of the image, such as texture, edges, and other local features, and their storage organization also requires efficient random access capabilities.

[0142] The retrieved pixel codebook feature vectors will be directly used as the quantized pixel feature output, maintaining perfect alignment with the expected input feature space of the pixel decoder. Throughout the process, a shared index acts as a bridge, ensuring consistency and synchronization between semantic and pixel feature retrieval.

[0143] Memory alignment and caching optimizations can be used in large-scale codebooks to reduce access latency, and parallel retrieval of multiple frames or images can be supported through batch query interfaces. Non-linear storage formats (such as hash tables or compressed sparse matrices) can be supported through index mapping functions without changing the definition of the shared index itself.

[0144] This embodiment enables synchronous and efficient retrieval of corresponding semantic codebook feature vectors and pixel codebook feature vectors from a single shared index, ensuring that quantized semantic features and quantized pixel features originate from a unified spatial location and possess strict consistency. The retrieval mechanism corresponds one-to-one with the shared index, guaranteeing the closed-loop reversibility between encoding and decoding. It also supports rapid access and decoupled feature reuse in large-scale, high-concurrency scenarios, providing accurate and standardized input for subsequent pixel and semantic decoding, reducing data bias and storage overhead, and improving the overall efficiency and stability during multi-task joint training and inference.

[0145] In one embodiment, step S50 above includes:

[0146] S501, the quantized pixel features are input into the convolutional neural network layer of the pixel decoder, and the features output by the convolutional neural network layer are processed by a generative adversarial network to generate an initial reconstructed image;

[0147] S502, perform an upsampling operation on the initial reconstructed image to obtain the reconstructed image;

[0148] S503, the quantized semantic features are input into the fully connected layer of the semantic decoder, and the features output by the fully connected layer are processed by a multilayer perceptron to generate reconstructed semantic features.

[0149] In this embodiment, the quantized pixel features and quantized semantic features retrieved from the codebook are respectively input into a dedicated decoder to recover a reconstructable image consistent with the original input data and reconstructed semantic features expressing high-level semantic information, ensuring that the quantization encoding and decoding link forms a closed loop and can support end-to-end joint training.

[0150] First, quantized pixel features are input to the pixel decoder. The convolutional neural network layer within the pixel decoder is the first processing unit, its function being to extract spatial distribution information and local texture patterns from the quantized pixel features through layer-by-layer convolution operations. Here, the convolutional neural network layer performs mapping and feature recovery, gradually mapping the high-dimensional quantized features into a tensor format in the initial image space. Each convolution operation performs weighted summation on the local region through kernel sliding and channel weighting, and achieves non-linear mapping through an activation function, ultimately obtaining a multi-channel, low-resolution feature tensor.

[0151] The output of this convolutional neural network layer is then processed by a generative adversarial network (GAN). In the GAN, the generator module uses deep convolutional networks or deconvolutional modules to further transform the input feature tensor into an output that approximates the distribution of the real image. A discriminator module then evaluates the difference between the generated result and the real image distribution, forming an adversarial optimization signal. During this process, the discriminator weights are fixed, serving as an unbiased reference standard and providing loss feedback during training, rather than directly participating in pixel value calculation. The introduction of the GAN effectively improves the texture detail and visual realism of the initial reconstructed image, making it closer to human perception standards. The output at this stage is the initial reconstructed image, whose resolution may still be lower than the target resolution.

[0152] Subsequently, an upsampling operation is performed on the initial reconstructed image. The purpose of upsampling is to restore the spatial resolution corresponding to the original input image data, so that the generated image has the true physical size and pixel granularity. Upsampling can be achieved through various methods, such as bilinear interpolation, transposed convolution, or subpixel convolution. Among them, transposed convolution has learnable parameters, allowing the model to adaptively adjust the feature rearrangement and interpolation rules during the upsampling process, reducing artifacts and mesh artifacts. In the specific implementation of upsampling, it is necessary to ensure the local continuity of pixel arrangement and the consistency of overall texture, so that the final reconstructed image can be used for reconstruction loss calculation and visual perception evaluation.

[0153] In parallel, the quantized semantic features are input into the fully connected layer of the semantic decoder. The fully connected layer projects the input feature vector onto the intermediate dimensional space required by the semantic decoder through matrix multiplication. This process fully utilizes the compactness of the quantized semantic features through global parameter sharing, forming a high-dimensional dense feature representation.

[0154] The output of the fully connected layer is then fed into a multilayer perceptron for processing. The multilayer perceptron consists of multiple layers of fully connected layers stacked with nonlinear activation functions. Each processing unit alternates between linear mapping and nonlinear transformation, progressively improving the abstractness of semantic representation and the ability to perform nonlinear mapping. The multilayer perceptron ultimately outputs reconstructed semantic features, aiming to fit as closely as possible to the high-level semantic embedding representation obtained from the pre-trained teacher model of the original input image data. During this process, the reconstructed semantic features maintain the same dimensionality and semantic space consistency as the semantic features output by the teacher model to support subsequent semantic alignment loss calculations.

[0155] The pixel decoder and semantic decoder operate in parallel, each handling the reconstruction task corresponding to its input features. The pixel decoder focuses on low-level details and textures, while the semantic decoder focuses on high-level abstract representations. The entire chain ensures that the input codebook features can be reversibly mapped back to the corresponding data representation, providing usable reconstruction results for the pixel reconstruction loss and semantic alignment loss defined during training.

[0156] This embodiment can simultaneously achieve high-fidelity decoding of quantized pixel features and quantized semantic features, enabling the system to balance low-level image detail restoration and high-level semantic information expression without increasing redundant codebooks. The design of the pixel decoder and semantic decoder ensures spatial resolution, visual detail, and semantic consistency of the decoding results. The addition of generative adversarial networks and multilayer perceptrons further enhances the realism and expressiveness of the decoded output. Through this decoding process, the training phase can generate an accurate loss signal based on the difference between the reconstructed output and the original input data, supporting end-to-end joint training of the entire model, thereby improving overall multi-task adaptability and model generalization performance.

[0157] In one embodiment, step S60 above includes:

[0158] S601, determine the pixel reconstruction loss between the reconstructed image and the input image data;

[0159] S602, determine the semantic alignment loss between the reconstructed semantic features and the semantic features extracted by the teacher model;

[0160] S603, determine the vector quantization loss between the semantic codebook feature vector and the semantic encoder output feature;

[0161] S604, determine the vector quantization loss between the pixel codebook feature vector and the pixel encoder output feature;

[0162] S605 combines pixel reconstruction loss, semantic alignment loss, and semantic and pixel vector quantization loss to form the total loss function;

[0163] S606, The gradient of the total loss function with respect to the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder is determined by the backpropagation module;

[0164] S607, Update the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder according to the gradient.

[0165] In this embodiment, multi-component collaborative training is implemented, enabling different encoders, codebooks, and decoders to optimize under shared loss constraints. This ensures consistent model convergence under the global objective, addressing the disconnect between semantic understanding and pixel reconstruction in traditional single-task optimization models. First, the pixel reconstruction loss between the reconstructed image and the input image data is calculated. This can be achieved by comparing the pixel-level differences between the generated and input images. Specifically, this includes using mean squared error to measure the difference in single-pixel intensity, or introducing perceptual loss by using a pre-trained feature extraction network to extract high-dimensional features of the image and then performing difference measurement to capture texture and global structural information. Second, the semantic alignment loss between the reconstructed semantic features and the semantic features extracted by the teacher model is calculated. This step requires aligning the feature embedding output by the semantic decoder with the high-dimensional semantic embedding output by the teacher model. The metric typically uses the squared Euclidean distance, ensuring that the model not only fits the pixel space during training but also aligns with the abstract representation of the high-level semantic space.

[0166] Furthermore, the vector quantization loss between the semantic codebook feature vector and the semantic encoder output features, and the vector quantization loss between the pixel codebook feature vector and the pixel encoder output features are calculated separately. These two losses are designed to ensure that the encoder output converges to its nearest neighbor codebook feature vector in Euclidean space. This mechanism reduces quantization errors and improves the stability and accuracy of subsequent retrieval and reconstruction. Then, the pixel reconstruction loss, semantic alignment loss, and the two types of vector quantization losses are weighted and combined to form the total loss function. Hyperparameter weights can be set for each loss term to adjust its contribution to the overall optimization objective during training, ensuring that each module maintains a balance while considering multiple objectives.

[0167] The backpropagation module calculates the gradient of the total loss function with respect to the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder. The backpropagation process utilizes the chain rule to propagate the loss signal back along the computation graph, accumulating the partial derivatives of each module's parameters layer by layer. Manually calculating each partial derivative is not required; an automatic differentiation mechanism can be used to construct an efficient end-to-end training pipeline. Finally, the parameters of these modules are updated based on the calculated gradient values. Various optimization strategies can be employed for the parameter update process, such as gradient descent algorithms based on a fixed learning rate, or adaptive learner optimizers like Adam and AdaGrad, to ensure optimized convergence speed and accuracy for each module's parameters at different training stages.

[0168] Example Description: This example illustrates the efficient encoding and standardized representation of patient medical images (such as MRI, CT, or ultrasound images) for analysis and processing of medical image data, particularly in medical image diagnostic support systems. First, input image data, such as a patient's head MRI image, is acquired. This input image data undergoes multi-scale feature enhancement during preprocessing. This can be achieved using a pyramid scale construction method, extracting image details and global context at different resolutions to ensure that the subsequent encoder can fully capture local details (such as the texture of small lesions) and overall structure (such as organ anatomical contours) within the image content.

[0169] The enhanced image data is fed into a semantic encoder and a pixel encoder, respectively. The semantic encoder uses a visual Transformer network, employing a self-attention mechanism to capture global dependencies and generate high-level semantic features that characterize potential clinical semantic patterns in the image, such as the shape and location of tumor regions. The pixel encoder uses a residual convolutional network, utilizing deep convolutional layers to extract local detail information, enhancing the ability to model image details. Subsequently, the high-level semantic features and low-level pixel features undergo a cross-modal feature alignment operation, aligning semantic and pixel information in both spatial and channel dimensions to ensure multimodal information consistency, generating aligned semantic features and aligned pixel features.

[0170] Subsequently, the squared Euclidean distances between aligned semantic features and semantic codebook feature vectors, and between aligned pixel features and pixel codebook feature vectors, are determined, resulting in semantic distance sets and pixel distance sets, respectively. This calculation process fully considers the position of each codebook feature vector in the medical image distribution space, ensuring the sensitivity of the distance calculation results to abnormal regions (such as lesions). By using the semantic distance sets and pixel distance sets as inputs for subsequent selection of shared indices, the system sets a weight parameter that balances the importance of semantics and pixels, and weights the pixel distances based on the current task requirements (e.g., diagnostic tasks prioritize semantic information, while image reconstruction tasks prioritize detail information).

[0171] Iterate through each index position shared by the semantic codebook and the pixel codebook, obtain the semantic distance and pixel distance corresponding to the current index, calculate the product of the weight parameter and the pixel distance as the pixel weighted result, and add the semantic distance and the pixel weighted result to obtain the weighted sum. By comparing the weighted sums of all index positions, determine the index position corresponding to the minimum weighted sum, and use it as the shared index.

[0172] By utilizing a shared index, corresponding semantic codebook feature vectors and pixel codebook feature vectors are retrieved from the semantic codebook and pixel codebook, respectively, as quantized semantic features and quantized pixel features. This retrieval process ensures efficient querying and standardized encoding within the global codebook space in medical scenarios, thereby maintaining consistency in the representation and storage of medical images.

[0173] Quantized pixel features are input into the convolutional neural network layer of the pixel decoder, combined with a generative adversarial network to generate an initial reconstructed image, ensuring the decoder can produce realistic medical image output. The initial reconstructed image is then upsampled to restore it to the same resolution as the input image, resulting in the final reconstructed image. Simultaneously, quantized semantic features are input into the fully connected layer of the semantic decoder, processed by a multilayer perceptron to generate reconstructed semantic features, which are then compared with the standard medical semantic features extracted by the teacher model.

[0174] During the training phase, the entire system calculates multiple loss values ​​for each training sample. These include pixel reconstruction loss between the reconstructed image and the input image, used to measure the visual quality and detail fidelity of the reconstructed image; semantic alignment loss between the reconstructed semantic features and the teacher model's semantic features, used to measure semantic consistency; and two types of vector quantization losses between the semantic codebook and the semantic encoder output, and between the pixel codebook and the pixel encoder output, used to optimize the codebook's expressive power. All loss terms are weighted and combined to form the total loss function, and the gradient is automatically solved using the backpropagation algorithm to update the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder. This ensures that the modules can collaboratively optimize after training, taking into account both the semantic interpretation and visual detail reconstruction of medical images.

[0175] Upon completion of training and clinical deployment, the system processes the target task image data (such as new patient MRI images) as input for the inference phase. It directly uses the updated semantic encoder, pixel encoder, semantic codebook, and pixel codebook to repeat the above process, determining the target task shared index and outputting this shared index as a unified visual token sequence. This unified visual token sequence can serve as a standardized feature representation for various downstream medical tasks, such as input to lesion segmentation models, embedded representation input to diagnostic prediction models, or a unified visual feature standard in multimodal electronic medical records, supporting cross-modal information fusion and intelligent clinical decision-making.

[0176] In financial operations, particularly in applications such as intelligent risk control, digital operations, and multi-channel user authentication, it is necessary to process and understand large amounts of multi-source visual data, including images of financial instruments, scanned contract documents, and identity verification photos. Using this system to process financial image data enables unified and standardized encoding of financial visual data, enhancing the model's ability to provide refined support for document authenticity verification, customer profiling, and risk assessment in financial business scenarios.

[0177] First, input image data is acquired, such as a scanned image of a financial instrument containing a seal, signature, and anti-counterfeiting textures. Multi-scale feature enhancement processing is then performed on this image data. Pyramid structures and multi-channel convolution operations are used to extract the anti-counterfeiting details and global layout structure of the instrument at different scales, resulting in enhanced image data. This enhancement process ensures that subsequent calculations can simultaneously capture fine-grained features such as seal outlines and signature handwriting, as well as the overall instrument layout format, providing assurance for the model's understanding of the instrument's authenticity.

[0178] Enhanced image data is input into both a semantic encoder and a pixel encoder. The semantic encoder internally models global dependencies within the image using a visual Transformer network, extracting high-level semantic features and encoding financial semantic patterns such as "bill type, seal type, and region location." The pixel encoder internally extracts low-level pixel features using a residual convolutional network, preserving high-frequency texture information and anti-counterfeiting pattern features. Through cross-modal feature alignment, aligned semantic features and aligned pixel features are generated, ensuring a high degree of consistency between semantic context and visual details at the spatial and channel levels, forming a fundamental representation that can be used for unified processing of financial images.

[0179] Subsequently, the squared Euclidean distances between aligned semantic features and semantic codebook feature vectors, and between aligned pixel features and pixel codebook feature vectors, are calculated to obtain semantic distance sets and pixel distance sets. In financial scenarios, this distance calculation supports comparing the similarity distribution between financial bill images and a standard bill template library, helping to identify abnormal bills. The semantic distance sets and pixel distance sets are used for shared index selection. The system sets weight parameters to balance semantic and detailed information according to business needs; for example, semantic weight can be increased in risk control, while pixel detail weight can be increased in forgery detection.

[0180] For each index position shared by the semantic codebook and the pixel codebook, the semantic distance value and pixel distance value corresponding to the current index are obtained. The product of the pixel distance and the weight parameter is calculated and then added to the semantic distance value to obtain a weighted sum. By comparing the weighted sums of all indices, the index position with the smallest weighted sum is selected as the shared index.

[0181] By utilizing shared indexes, corresponding semantic codebook feature vectors and pixel codebook feature vectors are retrieved from the semantic codebook and pixel codebook, respectively, to serve as quantized semantic features and quantized pixel features. This operation ensures efficient generation of standardized discrete token representations in large-scale financial datasets, supporting cross-system data reconciliation and secure transmission.

[0182] Quantized pixel features are input into the convolutional neural network layer of the pixel decoder. Combined with a generative adversarial network, the features are reconstructed to generate an initial reconstructed image. Subsequently, an upsampling operation is performed to restore the original resolution of the bill image, resulting in the reconstructed image. This reconstructed image can serve as an auxiliary output for bill quality assessment and forgery detection. Simultaneously, quantized semantic features are input into the fully connected layer of the semantic decoder. After further mapping via a multilayer perceptron, reconstructed semantic features are obtained, which are used for extracting structured fields and verifying semantic consistency of financial bills.

[0183] During the training phase, the system calculates multiple losses for each training data point: pixel reconstruction loss between the reconstructed image and the input image, measuring image quality and detail restoration; semantic alignment loss between the reconstructed semantic features and the teacher model's semantic features, measuring financial semantic consistency; vector quantization loss between the semantic codebook and the semantic encoder output, and vector quantization loss between the pixel codebook and the pixel encoder output, used to optimize the accuracy and stability of the codebook representation. These losses are weighted and combined to form a total loss function, and the gradient of the total loss function with respect to the parameters of each module is calculated using the backpropagation algorithm. The parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder are then jointly updated. After training, the entire system forms an efficient and standardized encoding and reconstruction capability that can be directly applied in financial business.

[0184] In practical deployments, financial business systems process target task image data, such as scanned images of new customer contracts collected by financial mobile terminals or bank card images collected by ATMs. Through updated semantic encoders, pixel encoders, semantic codebooks, and pixel codebooks, the computational paths used in the repeated training process are determined to establish a shared index for the target task, outputting a unified visual token sequence. This unified visual token sequence can serve as a standard data structure for multi-channel user identity verification, document anti-counterfeiting authentication, and risk modeling input in the financial sector. It possesses high consistency across businesses and systems, simplifies collaborative processing across multiple financial systems, and improves the efficiency and accuracy of digital financial services.

[0185] This embodiment incorporates the pixel reconstruction loss of the reconstructed image, the semantic alignment loss of the reconstructed semantic features, and the vector quantization loss of semantics and pixels into the loss function, and synchronously updates the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder. This ensures that the model simultaneously optimizes low-level details and high-level semantic performance during training, solving the local optima problem caused by training on a single task. Since the pixel reconstruction loss ensures clear details in the output image, the semantic alignment loss ensures that the high-level abstract representation of the decoder is aligned with the teacher model's output, and the vector quantization loss makes the encoder and codebook more tightly coupled, ultimately enabling coordinated optimization of each module under multi-task conditions, improving the model's overall performance, robustness, and ability to handle diverse inputs in practical applications.

[0186] In one embodiment, a visual token generation apparatus based on a shared index is provided, which corresponds one-to-one with the visual token generation method based on a shared index described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the visual token generation device based on a shared index according to the present invention. The modules include a feature encoding module 10, a distance calculation module 20, a shared index determination module 30, a quantized feature retrieval module 40, a feature decoding module 50, a joint optimization module 60, and a visual token generation module 70. Detailed descriptions of each functional module are as follows:

[0187] Feature encoding module 10 is used to acquire input image data and process the input image data through semantic encoder and pixel encoder to obtain semantic features and pixel features respectively;

[0188] The distance calculation module 20 is used to determine the distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook, so as to obtain the semantic distance and the pixel distance respectively.

[0189] The shared index determination module 30 is used to determine the shared index by minimizing the weighted sum of the semantic distance and the pixel distance;

[0190] The quantization feature retrieval module 40 is used to retrieve quantization semantic features and quantization pixel features from the semantic codebook and pixel codebook respectively using the shared index;

[0191] The feature decoding module 50 is used to process the quantized pixel features through a pixel decoder to generate a reconstructed image, and to process the quantized semantic features through a semantic decoder to generate reconstructed semantic features.

[0192] The joint optimization module 60 is used to jointly update the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder based on the loss function of the reconstructed image and the reconstructed semantic features.

[0193] The visual token generation module 70 is used to process the target task image data based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, determine the target task shared index, and output the target task shared index as a unified visual token sequence.

[0194] In one embodiment, the feature encoding module 10 is specifically used for:

[0195] The input image data is acquired, and multi-scale feature enhancement processing is performed on the input image data to obtain enhanced image data;

[0196] The enhanced image data is processed by a visual Transformer network in a semantic encoder to obtain high-level semantic features;

[0197] The enhanced image data is processed by a residual convolutional network in the pixel encoder to obtain the underlying pixel features;

[0198] Perform a cross-modal feature alignment operation on the high-level semantic features and the low-level pixel features to generate aligned semantic features and aligned pixel features.

[0199] In one embodiment, the distance calculation module 20 is specifically used for:

[0200] Obtain multiple semantic codebook feature vectors stored in the semantic codebook;

[0201] The semantic distance set is obtained by determining the squared Euclidean distance between the semantic features and the feature vectors of each semantic codebook.

[0202] Obtain the feature vectors of multiple pixel codebooks stored in the pixel codebook;

[0203] The squared Euclidean distance between the pixel feature and the feature vector of each pixel codebook is determined to obtain the pixel distance set;

[0204] The semantic distance set and the pixel distance set are respectively used as semantic distance and pixel distance.

[0205] In one embodiment, the shared index determination module 30 is specifically used for:

[0206] Set weight parameters to balance semantic importance and pixel importance;

[0207] For each index position shared by the semantic codebook and the pixel codebook, obtain the semantic distance value and pixel distance value corresponding to the current index position;

[0208] The product of the weight parameter and the pixel distance value is determined to obtain the pixel weighting result;

[0209] The semantic distance value is added to the pixel weighted result to obtain the weighted sum result of the current index position;

[0210] Compare the weighted sum of all index positions shared by the semantic codebook and the pixel codebook, and determine the index position corresponding to the minimum weighted sum.

[0211] The index position corresponding to the minimum weighted sum result is used as a shared index.

[0212] In one embodiment, the quantization feature retrieval module 40 is specifically used for:

[0213] Parse the shared index to determine the target index location;

[0214] Access the storage structure of the semantic codebook and find the corresponding semantic codebook feature vector based on the target index position;

[0215] The semantic codebook feature vector is used as the quantized semantic feature;

[0216] Access the storage structure of the pixel codebook and find the corresponding pixel codebook feature vector according to the target index position;

[0217] The pixel codebook feature vector is used as the quantized pixel feature.

[0218] In one embodiment, the feature decoding module 50 is specifically used for:

[0219] The quantized pixel features are input into the convolutional neural network layer of the pixel decoder, and the features output by the convolutional neural network layer are processed by a generative adversarial network to generate an initial reconstructed image.

[0220] An upsampling operation is performed on the initial reconstructed image to obtain the reconstructed image;

[0221] The quantized semantic features are input into the fully connected layer of the semantic decoder, and the features output by the fully connected layer are processed by a multilayer perceptron to generate reconstructed semantic features.

[0222] In one embodiment, the joint optimization module 60 is specifically used for:

[0223] Determine the pixel reconstruction loss between the reconstructed image and the input image data;

[0224] Determine the semantic alignment loss between the reconstructed semantic features and the semantic features extracted by the teacher model;

[0225] Determine the vector quantization loss between the semantic codebook feature vector and the semantic encoder output features;

[0226] Determine the vector quantization loss between the pixel codebook feature vector and the pixel encoder output feature;

[0227] The pixel reconstruction loss, semantic alignment loss, and semantic and pixel vector quantization loss are combined to form the total loss function;

[0228] The gradient of the total loss function with respect to the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder is determined by the backpropagation module.

[0229] The parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder are updated according to the gradient.

[0230] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side method for generating visual tokens based on a shared index.

[0231] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a visual token generation method based on a shared index.

[0232] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0233] The input image data is acquired and processed by a semantic encoder and a pixel encoder to obtain semantic features and pixel features, respectively.

[0234] The distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook are determined to obtain the semantic distance and the pixel distance, respectively.

[0235] The shared index is determined by minimizing the weighted sum of the semantic distance and the pixel distance;

[0236] The shared index is used to retrieve quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook, respectively.

[0237] The quantized pixel features are processed by a pixel decoder to generate a reconstructed image, and the quantized semantic features are processed by a semantic decoder to generate reconstructed semantic features.

[0238] Based on the loss function of the reconstructed image and reconstructed semantic features, the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder are jointly updated;

[0239] The target task image data is processed based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, the target task shared index is determined and the target task shared index is output as a unified visual token sequence.

[0240] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0241] The input image data is acquired and processed by a semantic encoder and a pixel encoder to obtain semantic features and pixel features, respectively.

[0242] The distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook are determined to obtain the semantic distance and the pixel distance, respectively.

[0243] The shared index is determined by minimizing the weighted sum of the semantic distance and the pixel distance;

[0244] The shared index is used to retrieve quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook, respectively.

[0245] The quantized pixel features are processed by a pixel decoder to generate a reconstructed image, and the quantized semantic features are processed by a semantic decoder to generate reconstructed semantic features.

[0246] Based on the loss function of the reconstructed image and reconstructed semantic features, the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder are jointly updated;

[0247] The target task image data is processed based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, the target task shared index is determined and the target task shared index is output as a unified visual token sequence.

[0248] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0249] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0250] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0251] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A visual token generation method based on a shared index, characterized in that, Includes the following steps: The input image data is acquired and processed by a semantic encoder and a pixel encoder to obtain semantic features and pixel features, respectively. The distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook are determined to obtain the semantic distance and the pixel distance, respectively. The shared index is determined by minimizing the weighted sum of the semantic distance and the pixel distance; The shared index is used to retrieve quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook, respectively. The quantized pixel features are processed by a pixel decoder to generate a reconstructed image, and the quantized semantic features are processed by a semantic decoder to generate reconstructed semantic features. Based on the loss function of the reconstructed image and reconstructed semantic features, the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder and pixel decoder are jointly updated; Based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, the target task image data is processed, the target task shared index is determined, and the target task shared index is output as a unified visual token sequence. The shared index is determined by minimizing the weighted sum of the semantic distance and the pixel distance, including: Set weight parameters to balance semantic importance and pixel importance; setting weight parameters is used to achieve flexible modality fusion control, so as to balance the importance of semantic features and pixel details in different training stages or for different datasets; For each index position shared by the semantic codebook and the pixel codebook, obtain the semantic distance value and pixel distance value corresponding to the current index position; The product of the weight parameter and the pixel distance value is determined to obtain the pixel weighting result, which is used to adjust the weight of the pixel distance value in the overall distance calculation, assigning the pixel distance a scale that is the same as the semantic distance, and adjusting the contribution ratio of the pixel distance in the determination of the shared index through the scale; The semantic distance value is added to the pixel weighted result to obtain the weighted sum result at the current index position. The addition operation is performed separately at each index position, and the resulting weighted sum result reflects the importance weighted synthesis measure of semantic and pixel similarity at the index position. The weighted sum result not only expresses the degree of matching of the current index in the two modalities, but also serves as the basis for subsequent selection of the optimal shared index. Compare the weighted sum of all index positions shared by the semantic codebook and the pixel codebook, and determine the index position corresponding to the minimum weighted sum. The index position corresponding to the minimum weighted sum result is used as a shared index, which explicitly points to the corresponding entry of the semantic codebook and the pixel codebook under this index, ensuring that the selected shared index is optimal under the multimodal similarity fusion measurement of semantics and pixels.

2. The visual token generation method based on a shared index as described in claim 1, characterized in that, The input image data is acquired and processed by a semantic encoder and a pixel encoder to obtain semantic features and pixel features, respectively, including: The input image data is acquired, and multi-scale feature enhancement processing is performed on the input image data to obtain enhanced image data; The enhanced image data is processed by a visual Transformer network in a semantic encoder to obtain high-level semantic features; The enhanced image data is processed by a residual convolutional network in the pixel encoder to obtain the underlying pixel features; Perform a cross-modal feature alignment operation on the high-level semantic features and the low-level pixel features to generate aligned semantic features and aligned pixel features.

3. The visual token generation method based on a shared index as described in claim 1, characterized in that, Determining the distance between the semantic feature and the semantic codebook, and the distance between the pixel feature and the pixel codebook, to obtain the semantic distance and pixel distance respectively, includes: Obtain multiple semantic codebook feature vectors stored in the semantic codebook; The semantic distance set is obtained by determining the squared Euclidean distance between the semantic features and the feature vectors of each semantic codebook. Obtain the feature vectors of multiple pixel codebooks stored in the pixel codebook; The squared Euclidean distance between the pixel feature and the feature vector of each pixel codebook is determined to obtain the pixel distance set; The semantic distance set and the pixel distance set are respectively used as semantic distance and pixel distance.

4. The visual token generation method based on a shared index as described in claim 1, characterized in that, Retrieving quantized semantic features and quantized pixel features from the semantic codebook and pixel codebook respectively using the shared index includes: Parse the shared index to determine the target index location; Access the storage structure of the semantic codebook and find the corresponding semantic codebook feature vector based on the target index position; The semantic codebook feature vector is used as the quantized semantic feature; Access the storage structure of the pixel codebook and find the corresponding pixel codebook feature vector according to the target index position; The pixel codebook feature vector is used as the quantized pixel feature.

5. The visual token generation method based on a shared index as described in claim 1, characterized in that, The image is reconstructed by processing the quantized pixel features using a pixel decoder, and the image is reconstructed by processing the quantized semantic features using a semantic decoder, including: The quantized pixel features are input into the convolutional neural network layer of the pixel decoder, and the features output by the convolutional neural network layer are processed by a generative adversarial network to generate an initial reconstructed image. An upsampling operation is performed on the initial reconstructed image to obtain the reconstructed image; The quantized semantic features are input into the fully connected layer of the semantic decoder, and the features output by the fully connected layer are processed by a multilayer perceptron to generate reconstructed semantic features.

6. The visual token generation method based on a shared index as described in claim 1, characterized in that, Based on the loss function of the reconstructed image and reconstructed semantic features, the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder are jointly updated, including: Determine the pixel reconstruction loss between the reconstructed image and the input image data; Determine the semantic alignment loss between the reconstructed semantic features and the semantic features extracted by the teacher model; Determine the vector quantization loss between the semantic codebook feature vector and the semantic encoder output features; Determine the vector quantization loss between the pixel codebook feature vector and the pixel encoder output feature; The pixel reconstruction loss, semantic alignment loss, and semantic and pixel vector quantization loss are combined to form the total loss function; The gradient of the total loss function with respect to the parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder is determined by the backpropagation module. The parameters of the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder are updated according to the gradient.

7. A visual token generation device based on a shared index, characterized in that, The visual token generation device based on the shared index includes: The feature encoding module is used to acquire input image data and process the input image data through a semantic encoder and a pixel encoder to obtain semantic features and pixel features, respectively. The distance calculation module is used to determine the distance between the semantic feature and the semantic codebook and the distance between the pixel feature and the pixel codebook, so as to obtain the semantic distance and the pixel distance respectively. A shared index determination module is used to determine a shared index by minimizing the weighted sum of the semantic distance and the pixel distance; The quantization feature retrieval module is used to retrieve quantization semantic features and quantization pixel features from the semantic codebook and pixel codebook respectively using the shared index; The feature decoding module is used to process the quantized pixel features through a pixel decoder to generate a reconstructed image, and to process the quantized semantic features through a semantic decoder to generate reconstructed semantic features; The joint optimization module is used to jointly update the semantic encoder, pixel encoder, semantic codebook, pixel codebook, semantic decoder, and pixel decoder based on the loss function of the reconstructed image and the reconstructed semantic features. The visual token generation module is used to process the target task image data based on the updated semantic encoder, pixel encoder, semantic codebook and pixel codebook, determine the target task shared index, and output the target task shared index as a unified visual token sequence. The shared index determination module is specifically used for: Set weight parameters to balance semantic importance and pixel importance; setting weight parameters is used to achieve flexible modality fusion control, so as to balance the importance of semantic features and pixel details in different training stages or for different datasets; For each index position shared by the semantic codebook and the pixel codebook, obtain the semantic distance value and pixel distance value corresponding to the current index position; The product of the weight parameter and the pixel distance value is determined to obtain the pixel weighting result, which is used to adjust the weight of the pixel distance value in the overall distance calculation, assigning the pixel distance a scale that is the same as the semantic distance, and adjusting the contribution ratio of the pixel distance in the determination of the shared index through the scale; The semantic distance value is added to the pixel weighted result to obtain the weighted sum result at the current index position. The addition operation is performed separately at each index position, and the resulting weighted sum result reflects the importance weighted synthesis measure of semantic and pixel similarity at the index position. The weighted sum result not only expresses the degree of matching of the current index in the two modalities, but also serves as the basis for subsequent selection of the optimal shared index. Compare the weighted sum of all index positions shared by the semantic codebook and the pixel codebook, and determine the index position corresponding to the minimum weighted sum. The index position corresponding to the minimum weighted sum result is used as a shared index, which explicitly points to the corresponding entry of the semantic codebook and the pixel codebook under this index, ensuring that the selected shared index is optimal under the multimodal similarity fusion measurement of semantics and pixels.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a shared-index-based visual token generation program stored in the memory and executable on the processor, wherein the shared-index-based visual token generation program, when executed by the processor, implements the steps of the shared-index-based visual token generation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a visual token generation program based on a shared index, which, when executed by a processor, implements the steps of the visual token generation method based on a shared index as described in any one of claims 1-6.

Images