Image generation method and device based on token-level dynamic routing, equipment and medium

Abstract

This invention relates to the field of image detection technology, and discloses an image generation method, apparatus, device, and medium based on token-level dynamic routing. The method includes: constructing a framework and configuring a dynamic routing module, an expert network cluster, and a cross-modal alignment module; acquiring paired text and image samples, generating a token sequence, token features, and an image latent code sequence; inputting the dynamic routing module to generate an expert allocation matrix and sparsifying it to obtain a token-level sparse routing result; distributing tokens to target experts and performing feature transformation to obtain expert features; weighted aggregation of expert features to obtain a text embedding sequence, mapping it to the image latent space through the cross-modal alignment module, and calculating the alignment loss; updating each module based on the loss until convergence, and generating an image corresponding to the target text. This invention can be applied to business scenarios such as fintech and healthcare, achieving fine-grained semantic allocation through token-level sparse routing and cross-modal alignment, thereby improving the semantic consistency between text and generated images.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to an image generation method, apparatus, device, and medium based on token-level dynamic routing. Background Technology

[0002] In recent years, text-to-image generation technology has continued to develop. Existing solutions typically employ a unified text encoding structure to represent the input text holistically, and then use the resulting text representation in the image generation process. This approach is prone to insufficient semantic representation granularity when dealing with complex text containing multiple semantic components, causing interference between object descriptions, attribute descriptions, spatial relationships, and style information in the same representation process. Simultaneously, there is often a lack of efficient alignment mechanisms between text representations and latent image representations, easily leading to cross-modal mapping biases. Furthermore, existing routing methods are prone to expert utilization imbalances, where a few processing units continuously bear the majority of processing tasks, affecting overall semantic expressiveness and training stability.

[0003] In the fintech sector, text-to-image generation technology can be used for scenarios such as marketing page generation, product illustration generation, insurance plan display, and investment advisory content visualization. The input text in these scenarios typically contains financial product information, risk level information, return attribute information, process relationship information, and presentation style information simultaneously, with different tokens carrying significantly different semantic roles. Existing technologies, when uniformly encoding this type of text, often struggle to distinguish between different semantic components with fine granularity, easily leading to the mixing of product attributes, risk warnings, and relationship structures during the representation process. Furthermore, insufficient alignment efficiency between text representation and potential image representation can also cause unstable correspondence between generated content and input text. In solutions employing expert processing structures, there is a tendency for a few experts to be frequently invoked while others are under-participated, affecting the effective expression of diverse semantic information in financial texts.

[0004] In the healthcare field, text-to-image generation technology can be used for scenarios such as generating health education content, creating diagnostic and treatment flowcharts, generating examination instruction pages, and generating rehabilitation guidance content. The input text in these scenarios typically contains information about medical objects, attribute descriptions, spatial relationships, processing procedures, and stylistic expression. Existing technologies often struggle to effectively distinguish the semantic components corresponding to different tokens when processing this type of text, leading to aliasing of key medical semantics during the unified representation process. Furthermore, the lack of efficient alignment between text representations and latent image representations can easily cause discrepancies between the generated content and the input description. Additionally, in expert-involved processing structures, a long-term imbalance in expert input can weaken the ability to differentiate and express different types of medical semantics, thus affecting the overall generation quality. Summary of the Invention

[0005] The main objective of this invention is to provide an image generation method, apparatus, device, and storage medium based on token-level dynamic routing, aiming to solve the technical problem that existing technologies have difficulty in differentiatedly representing the semantic information carried by different tokens in text and in effectively aligning text representations to the latent representation space of images, resulting in an unstable semantic correspondence between the generated content and the input text.

[0006] To achieve the above objectives, the present invention provides an image generation method based on token-level dynamic routing, comprising: A text and image generation framework is constructed, in which a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. Obtain paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples; The token sequence is input into the dynamic routing module, which generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results; Based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster, and each target expert performs feature transformation on the received token to obtain expert features; The expert features are weighted and aggregated according to the token-level sparse routing results to obtain the text embedding sequence; The text embedding sequence is input into the cross-modal alignment module, which maps the text embedding sequence to the image latent space to obtain the projected text representation. The cross-modal alignment loss is then determined based on the projected text representation and the image latent code sequence. The dynamic routing module, the expert network cluster, and the cross-modal alignment module are updated based on the cross-modal alignment loss until the preset convergence condition is met, and the trained image and text generation framework is obtained. Input the target text into the trained image-text generation framework to generate the target image corresponding to the target text.

[0007] Furthermore, to achieve the above objectives, the present invention provides an image generation apparatus based on token-level dynamic routing, comprising: The framework configuration module is used to build a text and image generation framework. In the text and image generation framework, a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. The sample preprocessing module is used to acquire paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples. The routing decision module is used to input the token sequence into the dynamic routing module, and the dynamic routing module generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The sparse filtering module is used to perform sparsification and normalization processing on the expert allocation matrix to obtain token-level sparse routing results. The expert processing module is used to distribute each token to the corresponding target expert in the expert network cluster according to the token-level sparse routing result, and each target expert performs feature transformation on the received token to obtain expert features; The feature aggregation module is used to perform weighted aggregation of each expert feature according to the token-level sparse routing result to obtain a text embedding sequence; An alignment calculation module is used to input the text embedding sequence into the cross-modal alignment module, map the text embedding sequence to the image latent space through the cross-modal alignment module to obtain a projected text representation, and determine the cross-modal alignment loss based on the projected text representation and the image latent code sequence; The parameter update module is used to update the dynamic routing module, the expert network cluster, and the cross-modal alignment module based on the cross-modal alignment loss until a preset convergence condition is met, thus obtaining the trained image and text generation framework. The image generation module is used to input the target text into the trained image-text generation framework and generate the target image corresponding to the target text.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a token-based dynamic routing image generation program stored in the memory and executable on the processor, wherein when the token-based dynamic routing image generation program is executed by the processor, it implements the steps of the token-based dynamic routing image generation method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an image generation program based on token-level dynamic routing, wherein the image generation program based on token-level dynamic routing, when executed by a processor, implements the steps of the image generation method based on token-level dynamic routing as described above.

[0010] Beneficial Effects: This invention relates to the field of image detection technology and discloses an image generation method, apparatus, device, and medium based on token-level dynamic routing. The method includes: constructing an image-text generation framework, configuring a dynamic routing module, an expert network cluster, and a cross-modal alignment module; acquiring paired training text and image samples, generating token sequences, token features, and image latent code sequences; inputting the token sequences into the dynamic routing module to generate an expert allocation matrix, and performing sparsification processing to obtain token-level sparse routing results; distributing tokens to target experts and performing feature transformation to generate expert features; weighted aggregation of expert features to obtain a text embedding sequence, and mapping it to the image latent space through the cross-modal alignment module, calculating the cross-modal alignment loss; updating each module based on the loss until convergence, obtaining a trained image-text generation framework for generating images corresponding to target text. This invention can be applied to business scenarios such as fintech and healthcare. By using token-level sparse routing, it achieves fine-grained semantic allocation of different tokens in the text, enabling each expert sub-network to perform feature transformations for specific semantics. Through weighted aggregation and the cross-modal alignment module, it ensures accurate correspondence between text representation and the image latent space, ultimately improving the semantic consistency and generation effect between the generated image and the input text. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an image generation method based on token-level dynamic routing according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the image generation method based on token-level dynamic routing according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the image generation device based on token-level dynamic routing of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The image generation method based on token-level dynamic routing provided in this invention can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The server can build a text-image generation framework through the client, configuring a dynamic routing module, an expert network cluster, and a cross-modal alignment module; acquire paired training text and image samples, generating token sequences, token features, and image latent code sequences; input the token sequences into the dynamic routing module to generate an expert allocation matrix, and perform sparsification processing to obtain token-level sparse routing results; distribute tokens to target experts and perform feature transformation to generate expert features; weighted aggregate expert features to obtain text embedding sequences, and map them to the image latent space through the cross-modal alignment module, calculating the cross-modal alignment loss; update each module based on the loss until convergence, obtaining the trained text-image generation framework, which is used to generate images corresponding to target text. This invention can be applied to business scenarios such as fintech and healthcare, achieving fine-grained semantic allocation of different tokens in the text through token-level sparse routing, enabling each expert sub-network to perform feature transformation for specific semantics, and ensuring accurate correspondence between text representation and image latent space through weighted aggregation and cross-modal alignment modules, ultimately improving the semantic consistency and generation effect between the generated image and the input text. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the image generation method based on token-level dynamic routing provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the image generation method based on token-level dynamic routing proposed in this invention includes the following steps: S10, construct a text and image generation framework, and configure a dynamic routing module, an expert network cluster, and a cross-modal alignment module in the text and image generation framework. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. In this embodiment, the text generation framework is used to carry the parameter organization and data transmission relationships of the text-side processing unit, expert splitting unit, and cross-modal mapping unit. In implementation, independent parameter areas, input / output interfaces, and update identifiers can be allocated to the dynamic routing module, expert network cluster, and cross-modal alignment module in the unified computation graph, enabling the three processing units to be uniformly scheduled in the same training environment. The framework within the text generation framework corresponds to the connection method, parameter allocation method, and update method under the condition of multiple processing units coexisting. The purpose is to keep text representation computation, expert splitting computation, and cross-modal mapping computation within the same trainable structure, avoiding asynchronous parameter updates caused by the independence of different processing units.

[0016] The dynamic routing module handles token allocation. In implementation, token projection parameters, expert projection parameters, and scoring parameters can be set within the dynamic routing module. After token features enter the module, they are mapped to the routing scoring space via the token projection parameters. The signature vectors corresponding to each expert subnetwork are mapped to the same scoring space via the expert projection parameters. The scoring parameters output the allocation relationship between the token and the expert subnetwork based on these two mapping results. "Dynamic" refers to the allocation result changing with the input token, "routing" refers to the allocation action between the token and the target processing unit, and "module" refers to this action being handled by an independent computation unit. With this configuration, different tokens do not need to go through completely identical processing paths but can enter different expert subnetworks based on their input differences.

[0017] The expert network cluster performs the task of transformation. Multiple expert subnetworks are configured in parallel, each maintaining an independent parameter set. This independent parameter set can include at least self-attention transformation parameters, feedforward transformation parameters, and normalization parameters, and may also include residual adjustment parameters. When multiple expert subnetworks exist in parallel, the allocation relationship output by the dynamic routing module can send different tokens into different parameter spaces, reducing the probability of different semantic components interfering with each other within the same parameter set. The cluster within the expert network cluster corresponds to an organizational method where multiple expert subnetworks are deployed in parallel within a unified framework but their internal parameters are not shared.

[0018] The signature vector is used to identify the trainable semantic preferences of each expert subnetwork. In implementation, each expert subnetwork can be assigned a set of initial vectors with the same dimension as the routing scoring space, establishing a one-to-one correspondence between these vectors and the expert subnetworks. The signature vector is updated during training and is therefore a learnable parameter. The signature corresponds to the identification label of the expert subnetwork in the scoring space, and the vector corresponds to this label as a parameter array. This learnable parameter array is continuously adjusted during loss backpropagation. Since the signature vector directly participates in the scoring calculation of the dynamic routing module, each expert subnetwork will gradually develop different input response biases during training, thus making the expert scoring tool adjustable.

[0019] The cross-modal alignment module handles the mapping and constraint functions between text representations and image representations. In implementation, an alignment layer, a difference metric parameter, and local correspondence constraint parameters can be set within the module. The alignment layer pushes the text-side representation into the image latent space. The difference metric parameter calculates the positional and distributional differences between the mapped text representation and the image latent code sequence. The local correspondence constraint parameters preserve the correspondences between local regions. With this configuration, the text-side output no longer exists merely as an independent vector but can participate in comparable error calculations within the image latent space.

[0020] In practical implementation, either a three-unit parallel configuration or a grouped configuration can be adopted. In the three-unit parallel configuration, the dynamic routing module, the expert network cluster, and the cross-modal alignment module each occupy an independent parameter area and are connected through a fixed interface. In the grouped configuration, the multiple expert sub-networks within the expert network cluster can be further divided into multiple subgroups according to parameter scale or scope of responsibility. Each subgroup is bound to a set of signature vectors. The dynamic routing module calculates the scoring results for different subgroups separately, and the cross-modal alignment module performs a unified mapping on the aggregated results output by different subgroups.

[0021] In one implementation, the dynamic routing module employs a single-stage scoring structure. Token features are mapped to the scoring space through a set of projection parameters, and the signature vectors corresponding to each expert subnetwork are mapped to the same space through another set of projection parameters. The scoring unit outputs an assigned score based on the two types of projection results. Multiple expert subnetworks in the expert network cluster maintain the same number of layers, but their internal parameters are not shared. The cross-modal alignment module uses a single alignment layer configuration, directly participating in the difference measurement after pushing the text-side representation into the image latent space. This implementation is suitable for generation tasks with moderate text length and relatively clear semantic roles, such as fund marketing cards, insurance liability explanation pages, and interest rate explanation charts in fintech businesses.

[0022] In another implementation, the dynamic routing module employs a grouped scoring structure. Multiple expert subnetworks are first divided into several groups based on semantic responsibilities. Each group shares a portion of the scoring parameters, while different groups maintain independent sets of signature vectors. The number of layers in the multiple expert subnetworks within the expert network cluster can vary; deeper expert subnetworks handle relational and stylistic content, while shallower expert subnetworks handle object-oriented and attribute-based content. The cross-modal alignment module uses a two-layer mapping structure, with the first layer performing scale unification and the second layer performing latent space convergence. This type of implementation is suitable for environments with large training sample sizes and significant variations in text structure, such as health management cards, rehabilitation guidance pages, and nutritional intervention graphic pages in healthcare applications.

[0023] Alternatively, an adaptive capacity configuration approach can be used. The initial parameter sizes of multiple expert subnetworks are not fixed; during training, the parameter capacity is adjusted based on the long-term activation distribution of each expert subnetwork. The signature vector remains in a learnable state but is re-bound to the corresponding expert subnetwork after capacity adjustment. The dynamic routing module remains unchanged, and the cross-modal alignment module reduces mapping fluctuations caused by capacity changes by freezing some less sensitive parameters. This type of implementation is suitable for training environments where the input distribution frequently changes.

[0024] In healthcare applications, a framework for generating images and text can be built around family health management visual cards. The text input includes symptom descriptions, behavioral suggestions, frequency limits, temporal relationships, and presentation styles. Multiple expert subnetworks within the expert network cluster are biased towards processing object-type tokens, attribute-type tokens, relationship-type tokens, and style-type tokens, respectively. The signature vectors corresponding to each expert subnetwork gradually develop stable preferences during training. After receiving the text representation, the cross-modal alignment module continuously constrains the correspondence between the text content and local regions of the health icon, making the output image more accurately represent symptom descriptions, action locations, and prompt styles.

[0025] In fintech applications, a framework for generating graphics and text can be built around the product display page. The text input includes the product name, return description, term structure, risk level, fee schedule, and page style requirements. A dynamic routing module distinguishes different semantic responsibilities based on token characteristics and signature vectors. Multiple expert subnetworks handle product entities, attribute modifications, relational expressions, and display style content respectively. A cross-modal alignment module further constrains the text-side representation and image representation, ensuring that the image content in fund cards, insurance description pages, credit flowcharts, or return display pages more closely matches the semantic requirements of the input text.

[0026] This embodiment configures a dynamic routing module, an expert network cluster, and a cross-modal alignment module in parallel within the text-image generation framework, and sets learnable signature vectors for each of the multiple expert sub-networks. The text-side representation no longer undergoes unified processing through a single parameter channel, but can form an allocation relationship for different expert sub-networks based on token semantic differences. After different expert sub-networks retain independent parameters, content with significant differences in semantic roles can form representations in relatively separate parameter spaces. The cross-modal alignment module continues to constrain the correspondence between the text side and the image side, reducing the offset between the text representation and the potential image representation, making it easier to maintain a stable semantic correspondence between the generated content and the input text.

[0027] S20, obtain paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples; In this embodiment, the pairing relationship is used to ensure that the text-side representation and the image-side representation originate from the same semantic object. In implementation, the same pairing identifier can be assigned to both text and image data, and consistency checks are performed on the text and image identifiers during the reading phase. After successful verification, the text and image data are written into the same training unit to avoid mismatches between the sources of text and image representations. The setting of the pairing relationship directly affects whether subsequent text and image latent code representations fall within the same constraint range; therefore, it is a limiting factor affecting the implementation logic.

[0028] Token segmentation is used to break down continuous text into computable semantic units. In implementation, discrete units can be formed based on word boundaries, sub-word boundaries, or phrase boundaries, preserving the original arrangement of these units within the text. Embedding representation is used to convert these discrete units into continuous vectors, which can contain positional encoding, adjacency encoding, and local context encoding. Token sequences correspond to a set of discrete units organized in their original order, while token features correspond to the vectorized result of each discrete unit and its additional structural information. If the text contains object words, attribute words, relation words, and style words simultaneously, category labels can be added to the token features to ensure that different semantic components remain separable in subsequent calculations. In this way, the token sequence is responsible for preserving order information, while the token features are responsible for preserving semantic and structural information; they perform different functions.

[0029] Image latent code sequences are used to convert image content into a compressible, arrangeable, and comparable latent representation to text. In implementation, the image can be divided into multiple local regions, and then an image encoder can map each local region to a low-dimensional latent representation, forming multiple latent code units. Alternatively, the entire image can be directly divided into blocks for encoding, and then the latent codes of each block can be organized in spatial order. The sequence attributes in the image latent code sequence are used to preserve the arrangement relationship of local image content, enabling the correspondence between the token order information on the text side and the local positional relationship on the image side to be established in subsequent calculations. The dimension, granularity of region division, and arrangement order of the latent code affect the degree of image information preservation, and are therefore key aspects that need to be clearly defined during implementation.

[0030] In one implementation, the text side uses a word segmentation method to split the text into multiple word units, then generates an embedding vector for each word unit, and writes the arrangement position of the word units into the vector. The image side uses a region coding method, dividing the image into regular grid regions, then an image encoder compresses each grid region into a latent code unit, and finally obtains the image latent code sequence according to the grid scanning order. This implementation is suitable for situations where the text contains many compound words and the image contains many local objects, such as product illustrations, health science infographics, and business process diagrams.

[0031] In another implementation, the text side employs phrase segmentation, breaking the text down into semantically complete phrase units. These phrase units are then vector-mapped and their adjacency relationships are written into the text. The image side uses multi-scale coding, generating global latent code units at a coarser granularity and local latent code units at a finer granularity. These two types of latent codes are then merged into an image latent code sequence in a predetermined order. This implementation is suitable for situations where text contains dense relational expressions and images simultaneously exhibit overall layout and local details, such as terms and conditions pages, intervention guidance pages, and marketing display pages.

[0032] Alternatively, a hybrid segmentation approach can be used to process text, retaining object-type phrases, attribute-type phrases, and relation-type phrases as discrete units of different lengths, and then uniformly mapping them to token features. On the image side, an irregular region encoding method is used, forming multiple latent code units based on the boundaries of the image content, and then arranging them according to the location of the region center point. This implementation is suitable for scenarios where the boundaries of local image regions are irregular and the semantic components in the text differ significantly.

[0033] In healthcare applications, health tips and corresponding images can be paired and written into the same training unit. When the text contains object descriptions, frequency descriptions, and relationship descriptions, token sequences are formed after word segmentation, and then token features are formed through embedding representation. The images are encoded into image latent codes after local region encoding. This organization ensures that the object information in the text and the local region information in the images maintain a correspondence, facilitating the establishment of stable constraints in subsequent training.

[0034] In fintech operations, product description text and corresponding display images can be paired and written into the same training unit. When the text includes product name, return attributes, term relationships, and format requirements, it is segmented and embedded to form token sequences and token features; the display image is divided into regions and latent codes are extracted to form an image latent code sequence. This organization makes it easier for attribute information, relationship information in the text, and local region information in the image to remain consistent in subsequent training.

[0035] In this embodiment, by binding text and image data to the same training unit through pairing identifiers, the source relationship between text-side and image-side representations becomes more stable. After forming token sequences and generating token features through word segmentation, the sequence information, semantic information, and local structural information in the text can be preserved separately. After forming image latent code sequences through image encoding, the local content of the image is converted into a permutable latent representation. With this setup, the text and image sides can simultaneously retain sequence, semantics, and local positional constraints in subsequent calculations, reducing the probability of mismatch and making it easier to maintain consistent representation relationships.

[0036] S30, the token sequence is input into the dynamic routing module, and the dynamic routing module generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert sub-network; In this embodiment, after the token sequence is fed into the dynamic routing module, the sequential discrete text units need to be converted into comparable routing representations. The token sequence carries the sequential constraint, and the token features carry the semantic constraint; both participate in the allocation calculation within the dynamic routing module. In implementation, token projection parameters and expert projection parameters can be configured in the dynamic routing module. The token features are transformed by the token projection parameters to form a token-side scoring vector, and the signature vectors corresponding to each expert subnetwork are transformed by the expert projection parameters to form expert-side scoring vectors. The role of the signature vector is not to represent the input text content, but to represent the semantic preferences and parameter response directions of the expert subnetworks. Therefore, the signature vectors need to maintain a one-to-one correspondence with the expert subnetworks and be continuously updated during training.

[0037] When the token-side scoring vector and the expert-side scoring vector are in the same scoring space, different tokens can be matched with multiple expert subnetworks. The matching results reflect the degree to which a token is favored by different expert subnetworks. The dynamic characteristic of the dynamic routing module is that the expert allocation result is not fixed to a specific expert subnetwork, but changes with the token characteristics. The expert allocation matrix is ​​used to store the allocation results between all tokens and all expert subnetworks. The rows of the matrix correspond to the token positions, the columns correspond to the expert subnetwork numbers, and the matrix elements correspond to the matching scores or normalized allocation weights.

[0038] To ensure the comparability of routing results for different tokens, the dynamic routing module also needs to be configured with a scoring normalization unit to compress the matching scores to a uniform numerical range. This process preserves both the order information and the allocation relationships across multiple expert subnetworks at different positions in the token sequence, allowing for subsequent selection or filtering based on the matrix results. For product names, return limits, term relationships, risk warnings, and formatting terms in financial texts, or symptom descriptions, frequency descriptions, behavioral constraints, and time relationships in medical and health texts, different tokens will form different expert allocation matrix distributions in the dynamic routing module, thereby reducing semantic ambiguity caused by uniform processing.

[0039] In one implementation, the dynamic routing module employs a dual-projection scoring structure. Token features are mapped to a low-dimensional scoring vector via token projection parameters, and the signature vector is mapped to a same-dimensional scoring vector via expert projection parameters. Then, the matching score between the token and the expert subnetwork is obtained through dot product. Finally, all matching scores are normalized to form the expert allocation matrix. This setup allows for easier control of the number of parameters and is suitable for environments with long text lengths and a moderate number of expert subnetworks. For financial product descriptions in fintech businesses, the product name, return attributes, term descriptions, and risk warnings can be pushed into different matching directions, making the expert allocation matrix exhibit significant differences in entity words and attribute words.

[0040] In another implementation, the dynamic routing module employs a dual-projection, gated scoring structure. Token features pass through a gating unit before projection. The gating unit adjusts the effective dimensions of the token features based on the token's local context, while the signature vector maintains independent projection on the expert side. A matching score is then generated through bilinear matching, followed by normalization to produce the expert assignment matrix. This setup allows the token-side scoring vector to change more significantly with the local context, making it suitable for environments where relational, style, and object terms are highly mixed. For intervention suggestion texts in healthcare, symptom words, frequency words, time words, and precaution words can be assigned to different expert preference regions, improving token-level semantic discriminative capabilities.

[0041] Alternatively, a grouped scoring structure can be used. Multiple expert subnetworks are first divided into several groups, each sharing a portion of the expert projection parameters, while different groups retain independent sets of signature vectors. Token features are first combined with the expert group scoring vectors to form a coarse allocation result, then combined with the signature vectors within each group to form a fine allocation result, ultimately synthesizing into an expert allocation matrix. With this setup, the dynamic routing module is more suitable for environments with a large number of expert subnetworks and can reduce the amount of scoring computation while maintaining allocation accuracy.

[0042] In healthcare applications, health alert texts often simultaneously include object information, frequency information, time information, and behavioral restriction information. After receiving the corresponding token sequence and token features, the dynamic routing module identifies high matching degrees between symptom words, frequency words, and restriction words and different signature vectors. Different columns in the expert assignment matrix exhibit different response intensities, facilitating the guidance of different semantic types to different expert subnetworks.

[0043] In fintech businesses, product presentation text often includes product name, return method, term description, risk level, and presentation style information simultaneously. After receiving the corresponding token sequence and token features, the dynamic routing module generates differentiated matching results between entity-type tokens, attribute-type tokens, and risk warning-type tokens with different signature vectors. The expert allocation matrix can distinguish different semantic categories within the same text.

[0044] This embodiment uses parameterized matching of token features and signature vectors within the dynamic routing module. Different positions in the token sequence no longer share a single routing destination, but instead generate differentiated allocation results for multiple expert sub-networks. After maintaining the correspondence between the signature vector and the expert sub-network, the semantic preferences of the expert sub-networks can be continuously adjusted during training, and the allocation results in the expert allocation matrix will change with the differences in the input tokens. This reduces the scrambling phenomenon caused by different semantic components being pushed into the same processing channel, making the subsequent expert reception relationships clearer.

[0045] S40, the expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results; In this embodiment, the expert allocation matrix contains the allocation strength between tokens and multiple expert subnetworks. The rows of the matrix correspond to token positions, the columns correspond to expert subnetwork numbers, and the matrix elements represent the relative strength of a token being allocated to a particular expert subnetwork. The purpose of sparsification is not to change the semantic source of the matrix, but to compress the number of experts associated with each token, ensuring that a token retains only a small number of expert subnetworks with high allocation strength. In implementation, each row of allocation values ​​can be read along the token dimension, and each row's allocation values ​​can be compared. The top-ranking allocation values ​​or those exceeding a preset limit can be retained, and the unretained positions can be set to zero. After this processing, the matrix changes from a dense distribution to a sparse distribution, and the relationship between tokens and expert subnetworks changes from multi-directional diffusion allocation to few-directional concentrated allocation, thus reducing the amount of subsequent processing.

[0046] Normalization is used to reconstrain the range of allocation values ​​after sparsification, making multiple allocation values ​​retained for the same token comparable. After sparsification, the sum of the retained allocation values ​​for each token usually changes. Without reconstraint, the allocation ratios of different expert subnetworks within the same token are difficult to use directly. In implementation, the retained allocation values ​​for each token can be summed, and then each retained allocation value can be divided by the current sum to obtain a new ratio; alternatively, exponential normalization can be used to map the retained allocation values ​​to a stable range before calculating the ratio. After normalization, the retained allocation values ​​within each token fall back into a unified ratio space, thus forming the token-level sparse routing result. In the token-level sparse routing result, "token-level" reflects that each token retains its own allocation result; "sparse" reflects that each token only retains a portion of the expert subnetworks; and "routing result" reflects that the allocation direction and allocation ratio together constitute the usable output.

[0047] To ensure that sparsity and normalization remain consistent during training, two additional components are typically required: retention rules and anomaly constraints. Retention rules limit the maximum number of expert subnetworks or which assignment values ​​are retained for each token. Anomaly constraints handle cases where all assignment values ​​for a token are cleared after sparsification. In implementation, retention rules can set a fixed or dynamic retention number; anomaly constraints can force the retention of the position corresponding to the maximum assignment value, ensuring that each token retains at least one expert subnetwork. This approach results in token-level sparse routing results that possess both sparsity and stability, allowing them to continue participating in training and inference.

[0048] One implementation uses a fixed number of reservations. The expert allocation matrix is ​​sorted row by row, and for each token, only the positions with the largest allocation values ​​are retained, while the rest are cleared to zero. Then, the retained allocation values ​​are proportionally calculated to obtain the token-level sparse routing results. This method facilitates control over the allocation scale of each token and is suitable for environments with a large number of expert subnetworks and limited resources for single processing. If the matrix size is large, it can be read in blocks before sorting, and then filtering and writing back can be performed within each block.

[0049] Another implementation uses threshold retention. The expert allocation matrix is ​​read row by row, retaining allocation values ​​above a preset threshold and clearing all positions below the threshold to zero. The retained allocation values ​​are then rescaled to obtain token-level sparse routing results. This method is more suitable for inputs with significant differences in allocation strength, allowing different tokens to automatically generate different numbers of retained positions. To prevent some tokens from having no retained positions after threshold filtering, a minimum retention rule can be added, ensuring that each token retains at least one position with the highest allocation value.

[0050] A hybrid retention approach can also be used. First, a threshold is used to filter out significantly lower allocation values. Then, the maximum number of values ​​retained from the remaining positions is limited, and finally, normalization is performed. This approach can suppress interference from weak allocation values ​​while controlling the output size. If a more stable allocation ratio is desired during training, a smoothing term can be added to the retained allocation values ​​before normalization, followed by proportionalization calculation.

[0051] This embodiment sparsifies the expert allocation matrix, compressing the number of expert sub-networks corresponding to a single token and making the allocation relationship between tokens and expert sub-networks more concentrated. Through normalization, the sparsified retained allocation values ​​return to a uniform proportional space, and multiple allocation directions within the same token have clear relative strengths. The resulting token-level sparse routing results reduce invalid allocations while maintaining comparability, helping to reduce semantic clutter and improve the stability of subsequent processing.

[0052] S50, based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster, and each target expert performs feature transformation on the received token to obtain expert features; In this embodiment, after reading the expert identifiers and allocation weights from the token-level sparse routing results by token dimension, the tokens, originally arranged in text order, need to be reorganized into input data categorized by expert. The "token-level" in the token-level sparse routing results indicates that each token retains its own allocation result; "sparse" means that each token points to only a small number of experts; and the routing results represent the available output composed of expert identifiers and allocation relationships. In implementation, a token distribution table can be established based on expert identifiers, and tokens with the same expert identifier can be written into the same expert input buffer. Simultaneously, the original position index and batch index are retained for each token, ensuring that the reorganized data not only retains the expert affiliation relationship but also the information needed for subsequent position recovery. The "correspondence" in "corresponding to target expert" indicates that the target expert is not predetermined but determined by the token-level sparse routing results; the target expert represents the expert sub-network to which the current token is actually delivered in the expert network cluster. After this organization, different tokens are no longer mixed in a unified input sequence but are split into multiple local sequences targeting different experts.

[0053] After a token is sent to the corresponding target expert, a parameterized transformation oriented towards semantic differences needs to be performed within each expert. Feature transformation is not a simple numerical transfer, but rather utilizes the internal parameters of the expert sub-network to transform the token vector from the original representation space to a space more suitable for subsequent representation computation. To ensure the feature transformation is discriminative, each target expert typically maintains an independent parameter set, which at least covers attention mapping parameters, feedforward mapping parameters, and normalization parameters, and may also include gating parameters or residual adjustment parameters. In implementation, multiple layers of transformation can be performed on the received token within each target expert. Each layer recalculates the contextual relationships between tokens and performs a nonlinear mapping on the internal dimensions of a single token. The inter-layer outputs are then written back to the expert output buffer according to their position indices. Expert features are derived from the parameterized transformations performed by each target expert on the received token, preserving the token's inherent semantics while incorporating the bias of the expert sub-network towards that semantic type. To prevent the loss of positional relationships for the same token in a multi-expert environment, the token source identifier also needs to be synchronously stored in the expert output buffer, ensuring that each expert feature can be mapped back to the original token position.

[0054] The cluster attributes in an expert network cluster affect the token distribution and transformation methods. If multiple target experts receive tokens in parallel, the token distribution process needs to support parallel writing and parallel reading. If the expert network cluster adopts a group configuration, target experts within different groups can share some low-level parameters while retaining their own high-level transformation parameters. To avoid insufficient expert utilization due to a small number of tokens, a minimum batch triggering rule can be set within the target experts, allowing tokens with too few tokens to be temporarily stored in the same round before being transformed uniformly. Alternatively, padding and masking control can be performed on token sequences of inconsistent lengths in the expert input buffer, enabling different target experts to complete batch processing in the same computing resource environment. With this setup, token distribution and expert transformation maintain computability and recoverability, and the source relationships of expert features become clearer.

[0055] In one implementation, the token-level sparse routing result adopts a single-expert distribution mode, where each token retains only one expert identifier. The token distribution table splits all tokens into multiple expert input sequences according to the expert identifier, and different expert input sequences are sent to the corresponding target experts in the expert network cluster. Each target expert employs a structure with alternating configurations of multi-layer attention mapping units and feedforward mapping units. The input tokens are transformed layer by layer to output expert features, which are then written to the expert output buffer according to their original position indices. With this configuration, the distribution relationship is clear, expert inputs do not overlap, and it is suitable for environments with long text lengths and limited computational resources.

[0056] In another implementation, the token-level sparse routing result adopts a multi-expert distribution mode, where the same token can retain multiple expert identifiers simultaneously. The token distribution table copies the same token to multiple expert input buffers, and the copies in each buffer are transformed by different target experts. To distinguish the source of the same token in different target experts, an expert number and a copy number can be appended during writing. Each target expert can employ a transformation structure of different depths; target experts with stronger semantic relationships are configured with deeper transformation layers, while those with stronger semantic attributes are configured with shallower transformation layers. With this configuration, the same token can simultaneously accept transformations from multiple parameter spaces, making it suitable for text environments with a high degree of semantic role mixing.

[0057] Another approach is to use gating enhancement. Before the token distribution table is written to the expert input buffer, the corresponding gating parameters are selected based on the expert identifier to suppress or amplify certain dimensions in the token vector before it is sent to the corresponding target expert. After the target expert outputs the expert features, they are written to the expert output buffer according to the original position index and expert number. With this configuration, the tokens received by different target experts already have differentiated activation states before entering the internal transformation, which is suitable for training environments with a large number of experts and high semantic discrimination requirements.

[0058] This embodiment reassembles tokens according to expert identifiers and distributes them to the corresponding target experts. Different semantic contents, originally mixed in a unified sequence, are processed in different parameter spaces, making the correspondence between tokens and experts clearer. Each target expert maintains independent parameters and performs multi-layer transformations on the received tokens. The output not only retains the original token information but also incorporates the bias of the expert subnetwork towards specific semantic categories. With the position index and source identifier synchronously preserved, expert features can stably correspond back to the original token position, reducing the likelihood of misalignment during subsequent use.

[0059] S60, according to the token-level sparse routing results, the expert features are weighted and aggregated to obtain the text embedding sequence; In this embodiment, the token-level sparse routing results already provide the correspondence between each token and the participating experts, as well as the assigned weights for each correspondence. The weighted aggregation is not simply an addition of multiple expert features; instead, it first restores the expert features to their token positions and then calculates the contribution ratio of different expert features to the same token based on the assigned weights. In implementation, an aggregation index can be created for each token, recording the token number, expert number, and weight value. Then, the corresponding expert features are read from the expert output buffer according to the aggregation index, forming a candidate feature set for each token. The weight value controls the strength of different expert features participating in the aggregation; the larger the weight, the greater the influence of the expert feature on the current token. To ensure the comparability of aggregation results between different tokens, it is generally required that the weights associated with the same token be within a uniform range. Then, each expert feature is multiplied by its respective weight and the results are accumulated to obtain the aggregated representation of the token.

[0060] Expert features preserve the transformation results of tokens across different expert parameter spaces. Two issues need to be addressed during aggregation: positional consistency and dimensional consistency. Positional consistency relies on the correspondence between token numbers and expert numbers to prevent cross-writing of expert features from different tokens. Dimensional consistency relies on the uniformity of the output dimensions of each expert to avoid dimensional mismatches during accumulation. In implementation, a uniform dimensional constraint can be set at the expert output end, ensuring all expert features fall into the same vector dimension. Then, in the aggregation unit, they are grouped by token number, placing multiple expert features belonging to the same token into the same computation unit. After weighting and accumulation, each token receives an aggregated representation, which is then arranged according to the original token order to form a text embedding sequence. The sequence attributes in the text embedding sequence come from the token order, and the embedding attributes come from the aggregated vector representation. Therefore, the text embedding sequence simultaneously preserves both sequence information and aggregated semantic information.

[0061] To make the aggregation results more stable, numerical constraints and outlier constraints can be added. Numerical constraints prevent a single expert feature from completely overshadowing other expert features due to an excessively large weight. A smoothing term or truncation range can be set before the weights enter the multiplication operation. Outlier constraints handle cases where a token retains only one expert feature or has no effective expert features. In the case of a single expert, the single expert feature can be directly output; in the case of a missing feature, it can fall back to a preset placeholder vector or the previous level token representation. The resulting text embedding sequence reflects the differentiated contributions of different experts to the same token while maintaining consistency between sequence-level input and output.

[0062] In one implementation, the aggregation unit employs a token-by-token accumulation structure. In the token-level sparse routing result, each token retains a small number of expert IDs and corresponding weights. The aggregation unit reads the corresponding expert features according to the token ID, sequentially performs weight multiplication and vector accumulation, outputs the aggregated representation of a single token, and then concatenates them in token order to obtain the text embedding sequence. This approach is suitable for environments with a large number of experts but a small number of experts retained per token, and its memory usage is relatively controllable.

[0063] In another implementation, the aggregation unit employs a matrix-based parallel structure. First, all expert features are written into a unified feature tensor. Then, the token-level sparse routing results are converted into a weight matrix. Weighted aggregation of multiple tokens is then performed in one step via matrix multiplication, outputting the text embedding sequence. This approach is suitable for batch training environments and can improve parallel computing efficiency. If the number of experts retained for different tokens is inconsistent, the expert numbers and weights can be padded first, and then a mask can be used to prevent invalid positions from participating in the accumulation.

[0064] Another approach is gating enhancement. Before aggregation, a set of aggregation gating values ​​is generated for each token. These gating values, along with routing weights, act on the expert features. The routing weights reflect the allocation strength, while the gating values ​​reflect the readjustment strength during the aggregation stage. Both types of values ​​are multiplied together and then accumulated to obtain the text embedding sequence. This approach is suitable for inputs with high semantic mixing and can further suppress perturbations caused by weakly correlated expert features.

[0065] This embodiment recovers expert features by token number and performs weighted accumulation based on assigned weights. The contributions of different experts to the same token are compressed into the same aggregation result, thus preserving differential semantic information that is difficult to retain in a unified parameter channel. After the aggregation result is arranged according to the original token order, both the sequence structure and the vector structure can remain stable, making it easier to maintain the consistency between text order information and semantic expression in subsequent use.

[0066] S70, the text embedding sequence is input into the cross-modal alignment module, the text embedding sequence is mapped to the image latent space through the cross-modal alignment module to obtain the projected text representation, and the cross-modal alignment loss is determined based on the projected text representation and the image latent code sequence; In this embodiment, after the text embedding sequence is formed, the text-side vector representation needs to be transformed into a space that can be directly compared with the image-side latent representation. The cross-modal alignment module undertakes this mapping and measurement function. In implementation, an alignment layer, a difference measurement unit, and a loss combination unit can be set in the cross-modal alignment module. The alignment layer receives the text embedding sequence and performs a linear or nonlinear transformation on the vector at each token position to make the output dimension consistent with the latent code dimension of the image latent space. After the dimension unification is completed, the text-side representation and the image-side representation are in a comparable state, and the projected text representation is thus formed. The projection in the projected text representation reflects that the vector no longer stays in the text representation space, but is mapped to the corresponding position in the image latent space.

[0067] The image latent space is used to accommodate the image-side compressed representation. Each latent code unit in the image latent code sequence can correspond to a local region, block region, or downsampled region in the image. To ensure comparability between the projected text representation and the image latent code sequence, uniform dimensional and positional constraints need to be maintained within the cross-modal alignment module. In implementation, the image latent code sequence can be organized as a set of latent vectors arranged in spatial order, and the projected text representation can be organized as a set of latent vectors arranged in textual order. With this setup, both the text-side and image-side representations exist in sequence form, which can then be measured based on position, similarity, or distribution differences.

[0068] Cross-modal alignment loss arises from errors in at least two directions. One direction is the correspondence deviation between the projected text representation and the image latent code sequence, and the other direction is the overall distribution deviation between the text-side latent representation set and the image-side latent representation set. In implementation, the positional difference between the projected text representation and the image latent code sequence can be calculated in the difference measurement unit. This positional difference can be measured using distance, similarity, or directional deviation metrics. Simultaneously, the distributional difference between the statistical distributions of the two sets of latent representations can be calculated, determined based on the mean, variance, covariance, or kernel mapping results. The loss combination unit then unifies the different difference measurement results into a single loss value. The resulting cross-modal alignment loss reflects both the proximity of local representations and the consistency of the overall representation.

[0069] To stabilize the loss value during training, numerical constraints are also necessary. This can be achieved by adding a normalization unit after the alignment layer to ensure the projected text representation falls within a predetermined numerical range; alternatively, a smoothing term can be added to the difference metric unit to suppress the excessive amplification of the loss value by individual anomalous latent codes. If the length of the text embedding sequence differs significantly from the length of the image latent code sequence, a sequence compression unit or a sequence expansion unit can be added within the cross-modal alignment module to ensure that the sequences on both sides remain compatible during comparison. This reduces the metric bias caused by dimensionality and length inconsistencies.

[0070] In one implementation, the alignment layer employs a single-layer projection structure. The text embedding sequence is directly converted into a projected text representation after passing through a set of trainable mapping parameters. Then, similarity is calculated position-by-position with the image latent code sequence, and the cross-modal alignment loss is obtained by accumulating the similarity deviations across all positions. This implementation is suitable for environments with short text descriptions and relatively uniform image region division. If the image latent code sequence is obtained by regular block division, the projected text representation can complete the position alignment measurement with the same number of comparison units.

[0071] In another implementation, the alignment layer employs a two-layer mapping structure. The first layer performs dimensionality compression or expansion, while the second layer performs latent space correction. After mapping, instead of directly comparing positions one by one, a similarity matrix between the projected text representation and the image latent code sequence is first calculated. Then, high-response regions are extracted from the similarity matrix for local difference measurement. Simultaneously, the overall distribution difference is calculated for all latent representations, and finally, a cross-modal alignment loss is formed. This implementation is suitable for environments with significant differences in text length and the number of image regions.

[0072] This embodiment maps the text embedding sequence to the image latent space through a cross-modal alignment module, providing a unified basis for comparison between the text-side representation and the image-side representation. After simultaneously measuring the local and global distributional differences between the projected text representation and the image latent code sequence, the offset between the text representation and the image representation is more easily constrained within a stable range, and the correspondence between the text description and the image content is more easily kept consistent.

[0073] S80, update the dynamic routing module, the expert network cluster, and the cross-modal alignment module based on the cross-modal alignment loss until the preset convergence condition is met, and obtain the trained image and text generation framework; In this embodiment, after the cross-modal alignment loss is formed, parameter correction is no longer limited to a single processing unit, but is applied simultaneously to the dynamic routing module, the expert network cluster, and the cross-modal alignment module. This is because the dynamic routing module determines the allocation relationship between tokens and expert sub-networks, the expert network cluster determines the transformation results of different semantic contents, and the cross-modal alignment module determines the mapping relationship between text-side representation and image-side representation. If these three parts are updated in different directions, deviations will continuously accumulate between the text allocation result, expert features, and latent image representation. To avoid this, the cross-modal alignment loss is used as a unified optimization objective during training. Errors are backpropagated from the cross-modal alignment module to the expert network cluster, and then from the expert network cluster to the dynamic routing module, ensuring that the three processing units are synchronously corrected around the same loss value.

[0074] The cross-modal alignment module internally includes at least alignment layer parameters and difference metric parameters. The alignment layer parameters are used to push the text embedding sequence into the image latent space, while the difference metric parameters are used to generate positional and distributional differences. The expert network cluster internally includes at least attention mapping parameters, feedforward mapping parameters, and normalization parameters for each expert subnetwork. The dynamic routing module internally includes at least token projection parameters, expert projection parameters, and routing scoring parameters. During error backpropagation, the cross-modal alignment loss first calculates its gradient with respect to the cross-modal alignment module to obtain the first update information; then, based on the source relationship between expert features and corresponding expert subnetworks, the error is allocated to the expert subnetworks participating in the current training batch to obtain the second update information; finally, based on the allocation relationship in the token-level sparse routing results, the error is backpropagated to the dynamic routing module to obtain the third update information. This update order ensures that the image-side mapping error is perceived first and then backpropagated layer by layer to the text allocation side.

[0075] The preset convergence condition depends not only on the change in loss value but also on the parameter update magnitude. In implementation, the change in cross-modal alignment loss over multiple consecutive training epochs, as well as the average parameter changes of the dynamic routing module, expert network cluster, and cross-modal alignment module, can be recorded. Training is considered complete when the loss change continuously decreases and the average parameter change simultaneously falls within a stable range. Observing only a single loss decrease can easily lead to premature termination under local oscillations. To reduce the impact of frequent fluctuations in the front-end allocation relationship, the three processing units can be divided into different parameter groups with different update magnitudes. The update magnitude of the cross-modal alignment module can be slightly larger, the update magnitude of the expert network cluster moderate, and the update magnitude of the dynamic routing module slightly smaller. This setting helps to first compress the deviation between the text and image sides and then gradually stabilize the semantic allocation results.

[0076] After the training batch is formed, the cross-modal alignment loss is fed into the gradient calculation unit. The gradient calculation unit generates the gradients for the cross-modal alignment module, the expert network cluster, and the dynamic routing module. The parameter update unit corrects the corresponding parameters based on the three types of gradients. The state recording unit records the changes in loss and parameters. The convergence determination unit determines whether to continue training based on the recorded results. After the preset convergence conditions are met, the trained image and text generation framework is output. The batch size can be set to 32, the gradient clipping threshold to 1, the learning rate for the cross-modal alignment module to 0.0003, the learning rate for the expert network cluster to 0.0002, and the learning rate for the dynamic routing module to 0.0001.

[0077] One implementation employs a unified backpropagation structure. The cross-modal alignment loss is fed into the gradient calculation unit after each training epoch. This unit sequentially generates the mapping gradient of the cross-modal alignment module, the expert gradient of the expert network cluster, and the routing gradient of the dynamic routing module. The parameter update unit adjusts the three types of parameters according to a fixed learning rate. The rate of loss decrease and the average change in parameters are recorded over multiple consecutive training epochs. Training stops when both values ​​simultaneously fall below a given threshold. This setup is suitable for environments with a moderate number of parameters and a fixed number of modules, providing a clear update path and easy control over the training process.

[0078] Another implementation uses a grouped update structure. Parameters in the cross-modal alignment module are first updated locally, then parameters in the expert network cluster are updated in batches, and finally, parameters in the dynamic routing module are updated slowly. After grouping the parameters, learning rates and weight decay coefficients can be set separately for different groups, allowing the mapping space to adjust faster than the routing boundary to change. During convergence determination, in addition to recording loss changes, the average gradient magnitude of each parameter group is also recorded. Training stops when the average gradient magnitude simultaneously enters the stable region over several consecutive training epochs. This implementation is suitable for environments with a large number of expert subnetworks and significant differences in the scale of routing parameters and expert parameters.

[0079] In fintech applications, product description text and display images can be used as training data. Text data includes product name, return attributes, term description, risk warnings, and page style information, while image data includes promotional cards, main product images, or flowcharts. During training, cross-modal alignment loss feeds the deviation between the text projection representation and the image latent code representation back to three processing units. The dynamic routing module progressively adjusts the token allocation relationship, the expert network cluster progressively adjusts the transformation parameters for different semantic content, and the cross-modal alignment module progressively compresses the distance between the text representation and the image representation. After training, the output image-text generation framework more easily maintains consistency between the product description and the image content.

[0080] In healthcare applications, health advice texts and accompanying illustrations can be used to form training data. The text data includes object information, frequency information, time information, and precautions information, while the image data includes health infographics, flowcharts, or action illustrations. During training, a cross-modal alignment loss continuously constrains the difference between the text projection representation and the image latent code representation, while a dynamic routing module and an expert network cluster synchronously correct the assignment and transformation parameters. After training, the resulting text-image generation framework more easily maintains the stability of the correspondence between the explanatory text and the illustration content.

[0081] This embodiment uses cross-modal alignment loss as a unified optimization objective, simultaneously correcting the parameters in the dynamic routing module, expert network cluster, and cross-modal alignment module. This allows text assignment results, expert transformation results, and image-side mapping results to be synchronously adjusted around the same error source. With both loss and parameter changes participating in convergence determination, the parameter state at the end of training is more stable. By controlling the update magnitude by module grouping, the fluctuations between text-side assignment relationships and image-side mapping relationships are reduced, making it easier for the trained text-image generation framework to maintain consistent representational capabilities.

[0082] S90, input the target text into the trained image-text generation framework to generate the target image corresponding to the target text.

[0083] In this embodiment, after the target text is fed into the trained image-text generation framework, it needs to be converted into inference input that can be recognized within the framework. This is achieved by segmenting the target text, embedding mapping, and positional encoding to form an inference token sequence and inference token features. These inference token features are then fed into the trained dynamic routing unit. The dynamic routing unit is no longer in a parameter update state; instead, it outputs the allocation results of each token in the current target text to different experts based on fixed parameters. The significance of this setup is that object information, attribute information, relationship information, and style information in the target text will not be uniformly pushed into a single path, but will instead enter different expert channels according to the division of labor formed after training.

[0084] After the expert allocation results are generated, each token is fed into the corresponding expert subnetwork. The expert subnetwork performs parameterized transformation on different tokens in the target text and outputs expert features corresponding to the token positions. Multiple expert features are then aggregated according to the allocation weights to form a text embedding sequence for the current target text. The reason why the trained image-text generation framework can directly accept target text and output target images is that the dynamic routing unit, expert subnetwork, and cross-modal mapping unit have been constrained to a unified parameter set during the training phase. During inference, it is no longer necessary to readjust the parameters; it only needs to use the existing parameters to complete token allocation, feature transformation, and embedding aggregation.

[0085] After the text embedding sequence is formed, it needs to be converted into a conditional representation usable on the image side. In implementation, the text embedding sequence can be fed into a cross-modal mapping unit, and the text-side vectors are pushed into the image latent space through an alignment layer to obtain the inference conditional representation. The inference conditional representation is used to constrain the generation direction of the image-side latent variables, ensuring that the image content maintains a correspondence with the target text. If the image-text generation framework adopts a latent diffusion structure, iterative generation of latent image representations can be performed in the image latent space starting from the initial noise vector, and then the latent image representations can be restored to the target image through an image decoding unit. If the image-text generation framework adopts an autoregressive structure, image-side representations can be generated sequentially according to image blocks or image tags, and then restored to the target image. The target in the target image corresponds to the output image with semantic consistency with the current target text, rather than an arbitrary image.

[0086] To ensure stable output during the inference phase, inference control parameters and output constraints need to be set. Inference control parameters can include the number of iterations, sampling temperature, conditional guidance strength, and resolution settings, used to control the sharpness of the generated image and the text compliance. Output constraints can include size constraints, color gamut constraints, and numerical cropping constraints, used to ensure that the target image is within a displayable, storable, and reusable range. With these settings, the trained image-text generation framework can both generate images based on the target text and ensure stable usability of the generated results.

[0087] In one implementation, the target text is segmented into a fixed-length inference token sequence. This inference token sequence is input into the dynamic routing unit of the trained image-text generation framework. The dynamic routing unit outputs the allocation results between the tokens and the expert sub-networks. Each expert sub-network outputs its expert features, and the aggregation unit summarizes these features into a text embedding sequence. The alignment layer then maps the text embedding sequence into an inference conditional representation in the image latent space. A latent diffusion generation structure is used on the image side, where the inference conditional representation and the initial noise latent variables participate in iterative updates. After iteration, the image decoding unit outputs the target image. This implementation is suitable for scenarios with high image resolution and long text descriptions.

[0088] In another implementation, the target text is segmented into phrases to form an inference token sequence. This inference token sequence, after passing through a dynamic routing unit and an expert subnetwork to obtain a text embedding sequence, does not directly participate in the diffusion iteration. Instead, it first generates multiple sets of local condition vectors, which are then applied to an image patch generation unit. The image patch generation unit generates the image-side latent representation block by block. All image patches are then concatenated to form a complete image, which is then processed by a post-processing unit to output the target image. This implementation is suitable for scenarios where there are many local objects in the image and densely packed relational descriptions in the text.

[0089] In one implementation, the target text undergoes word segmentation, embedding mapping, dynamic routing allocation, expert transformation, and weighted aggregation to form a text embedding sequence. This text embedding sequence is then mapped across modalities to obtain a low-resolution conditional vector. The low-resolution conditional vector and initial noise latent variables are input into a low-resolution generation unit. Within a preset number of iterations, the noise latent variables are denoised and updated to obtain a low-resolution latent image representation, which is then restored to a low-resolution image by a low-resolution decoding unit. The low-resolution image provides the subject outline, region layout, object location, and main semantic relationships. The low-resolution image is then re-encoded into high-resolution guiding features, and the text embedding sequence corresponding to the target text is mapped back into a high-resolution conditional vector. The high-resolution guiding features, high-resolution conditional vector, and high-resolution initial noise latent variables are input into a high-resolution thinning unit. Within a preset number of thinning iterations, the high-resolution initial noise latent variables are denoised and updated to obtain a high-resolution latent image representation, which is then output by a high-resolution decoding unit as the target image. The output resolution of the low-resolution generation unit can be set to 256. 256 or 512 The output resolution of the high-resolution refinement unit can be set to 512. 512, 768 768 or 1024 1024; The number of iterations for low-resolution generation units can be less than the number of iterations for high-resolution refinement units, so that the main layout is stabilized first, and then texture details, edge transitions and local structures are supplemented.

[0090] This embodiment feeds the target text into a trained image-text generation framework. Different semantic components in the text can enter different expert subnetworks according to the allocation relationships formed after training. After aggregation and cross-modal mapping, they are transformed into image-side conditional representations. The image generation process no longer relies solely on a unified text vector for constraint. The resulting target image is more likely to maintain consistent object information, attribute information, relational information, and style information with the target text. The output stability and text compliance during the inference stage are also more easily kept within a controllable range.

[0091] In one embodiment, step S20 above includes: S201, Obtain paired training text samples and training image samples, perform pairing consistency verification on the paired training text samples and training image samples, and generate sample pairing identifiers; S202, perform multi-granularity word segmentation on the training text samples according to the sample pairing identifier to obtain a set of text fragments; S203, map the set of text segments to a unified embedding space, and write the arrangement order and contextual adjacency relationship of each text segment into the mapping result to obtain the token sequence and token features; S204, after the token sequence is formed, the object description component, attribute description component, spatial relationship description component and style description component corresponding to each token are distinguished to obtain the semantic component distinction result, and the semantic component distinction result is written into the token feature; S205, perform image region segmentation and regional latent code extraction on the training image samples according to the sample pairing identifier to obtain multiple latent code units; S206, the multiple latent code units are arranged according to the spatial arrangement order of each image region to obtain the image latent code sequence.

[0092] In this embodiment, the pairing relationship directly determines whether text-side information and image-side information fall within the same training constraint unit. If text data and image data come from different semantic objects, there will be no effective correspondence between the subsequently generated token sequences, token features, and image latent code sequences, and cross-modal constraints will be interfered with by erroneous samples. In implementation, a unified pairing identifier can be assigned to text data and image data during the data access phase, and consistency verification can be performed before writing them into the training set. Consistency verification can be completed based on file number, sample index, business tag, timestamp combination, or manual annotation relationship, or it can be automatically verified based on the similarity threshold between text summary and image tag. After successful verification, text data and image data are written into the same training record, and the sample pairing identifier is saved along with the record, enabling subsequent text-side output and image-side output to continuously reference the same source relationship. With this setting, errors occurring during training will converge around the same semantic object, rather than pushing irrelevant text and irrelevant images into the same constraint space.

[0093] Word segmentation serves to divide discrete semantic units. When continuous text is directly vectorized, local semantic boundaries are easily blurred, especially when object information, attribute information, spatial relationship information, and style information coexist. Uniform segmentation weakens the separability between different semantic roles. Multi-granularity word segmentation preserves language units at different scales. In implementation, character-level, word-level, and phrase-level units can be generated simultaneously, and then the multi-granularity results are merged into a text fragment set according to preset mapping rules. Character-level units are beneficial for preserving fine-grained differences, word-level units are beneficial for preserving stable semantics, and phrase-level units are beneficial for preserving local combination relationships. After the text fragment set is formed, each text fragment has a fragment number and original position index, which can continue to preserve order information when subsequently mapped to a unified embedding space. If the input text contains object names, quantity modifiers, color modifiers, positional relationships, and style constraints, the segmentation results at different granularities will simultaneously preserve this information in the text fragment set, making it easier for subsequent vector representations to distinguish local semantic roles.

[0094] The unified embedding space is responsible for vector normalization across text fragments of different granularities. Text fragments formed at different granularities have inconsistent lengths and contextual ranges. If they are directly used in subsequent calculations, fragments of different granularities will be on different metric scales. In implementation, an embedding matrix and a positional encoding matrix can be set to map each text fragment in the text fragment set to a fixed-dimensional vector, and then the order of arrangement can be written into the mapping result through additional positional encoding. To preserve the relative adjacency relationships between fragments, an adjacency tag matrix or a local window relationship matrix can be superimposed on the mapping result to ensure that the dependency information between adjacent text fragments is not severed. The token sequence here serves as the order organization function, and the token features carry the semantic vector and structural vector functions. Each token in the token sequence corresponds to a position index, and each vector in the token features is bound to the corresponding position index. After this formation, the order information and semantic information are not mixed into a single scalar, but are stored separately in the index and vector, and subsequent calculations can call both simultaneously.

[0095] The distinction between object descriptive components, attribute descriptive components, spatial relationship descriptive components, and style descriptive components serves to separate different semantic responsibilities within the same text from a unified vector. Object descriptive components are used to label entity names or main content; attribute descriptive components are used to label decorative information such as color, size, quantity, and state; spatial relationship descriptive components are used to label layout relationships such as top / bottom, left / right, front / back, inside / outside, and near / far; and style descriptive components are used to label visual constraints such as realistic, flat, minimalist, business, and medical / science popularization. In implementation, after the token sequence is formed, each token can be semantically categorized based on lexical tags, dependency relations, phrase role labels, or a trained semantic classifier. The determination results are organized into semantic component distinction results and then written into the token features. The writing method can employ additional category vectors, additional category bit labels, or additional category indices. The resulting token features not only retain the original embedding values ​​but also the category information of each token in terms of semantic responsibility. If subsequent assignments or aggregations are performed for different semantic responsibilities, the category information in the token features can directly participate in the calculation, reducing mutual interference between object information and attribute information, and between relationship information and style information.

[0096] Image region segmentation serves the function of local semantic separation in images. If the entire image is directly compressed into a single latent representation, local objects, local attributes, and local spatial relationships will be compressed into a single global vector, which is not conducive to subsequent correspondence with local fragments in the text. In implementation, the image can be divided into regular regions or content-adaptive regions. Regular region segmentation can use a uniform grid to divide the image into multiple sub-regions according to width and height; content-adaptive region segmentation can form irregular regions based on saliency detection, boundary detection, or candidate region extraction results. Each image region retains its region number, spatial coordinates, and adjacency relationships. Regional latent code extraction is completed after region segmentation. An image encoder can map each region to a fixed-dimensional latent vector, and multiple latent vectors constitute a set of latent code units. With this setup, the image no longer only has a global representation, but rather a set of vectors oriented towards different local regions.

[0097] Image latent code sequences are formed during the latent code unit arrangement stage. Without a unified order, it's difficult to establish a stable correspondence between multiple latent code units and subsequent text-side sequence information or local relational information. In implementation, latent code units can be arranged according to the spatial arrangement order of image regions. This spatial arrangement order can be a grid order from top to bottom and left to right, or an order based on the coordinates of the region's center point. After arrangement, each latent code unit has a fixed sequence number, thus forming the image latent code sequence. Each position in the image latent code sequence not only represents a latent vector but also indirectly represents the local region position to which that latent vector belongs. After this formation, both the text-side token sequence and the image-side image latent code sequence possess sequential attributes, allowing for comparison of both semantic vectors and local positional distributions.

[0098] After the token sequence, token features, and image latent code sequence are formed simultaneously in this stage, the text side and the image side are organized into two sets of inputs that can be referenced in parallel. The text side retains discrete order, semantic category, and contextual adjacency, while the image side retains region order, local latent representation, and spatial location information. If the text contains mixed expressions of objects, attributes, relationships, and styles, or if the image contains multiple objects, multiple regions, and multi-layered layouts, the two sets of outputs formed in the current stage can still remain comparable in terms of order, category, and local region.

[0099] This embodiment generates sample pairing identifiers through pairing consistency verification, confining text and image data to the same semantic object scope, thus reducing the interference of mismatched samples on subsequent constraints. Through multi-granularity word segmentation and unified embedding mapping, the sequence information, local combination information, and vector information in the text are preserved separately, and the token sequence and token features have a clear division of labor. By writing the semantic component differentiation results into the token features, text units with different semantic responsibilities obtain additional distinguishing information in the same vector space, mitigating the situation where mixed semantics are compressed into a single representation. Through image region segmentation, regional latent code extraction, and spatial order arrangement, an image latent code sequence with local regional meaning is formed on the image side. The resulting text-side output and image-side output maintain consistency in source relationship, sequence relationship, category relationship, and local region relationship, making it easier to maintain the correspondence stability between text content and image content during subsequent use.

[0100] In one embodiment, step S30 above includes: S301, input the token sequence into the dynamic routing module and obtain the token feature corresponding to the token sequence; S302, Obtain the signature vector corresponding to each expert sub-network from the expert network cluster; S303, the token features are mapped to the shared attention space through the learnable token projection matrix in the dynamic routing module to obtain the token query vector; S304, the signature vectors corresponding to each expert subnetwork are mapped to the shared attention space through the learnable expert projection matrix in the dynamic routing module to obtain the expert key vectors; S305, perform query key-value attention calculation based on the token query vector and the expert key vector to obtain the original association score matrix; S306, perform attention normalization on the original association score matrix and arrange it according to the token dimension and expert dimension to generate an expert allocation matrix.

[0101] In this embodiment, after the token sequence is sent to the dynamic routing module, the module simultaneously receives two types of input information: one is the token feature corresponding one-to-one with the token position, and the other is the signature vector corresponding one-to-one with the expert subnetwork. The token feature carries the vector representation of the text unit in the semantic space, and the signature vector carries the learnable identifier of the expert subnetwork in the division space. If the two types of input are directly compared, there are often problems such as inconsistent dimensions, inconsistent numerical scales, and inconsistent semantic coordinate systems. Therefore, two sets of projection parameters need to be set inside the dynamic routing module to push the token feature and the signature vector into the same comparison space. The token projection matrix performs a linear mapping or a biased affine mapping on the token feature, so that each token obtains a set of query vectors; the expert projection matrix performs a same-dimensional mapping on the signature vector, so that each expert subnetwork obtains a set of key vectors. The role of the shared attention space is to put the semantic distribution on the text side and the ability distribution on the expert side into the same comparable scale, so that different tokens and different expert subnetworks can form a unified scoring result.

[0102] After query vector generation, different tokens are no longer just sequential text units, but are transformed into scoring entities that can participate in allocation calculations. After key vector generation, different expert subnetworks are no longer just parallel parameter units, but are transformed into target entities that can be matched with tokens. Query key-value attention calculation occurs between the query vector and the key vector, and the calculation result reflects the compatibility between each token and each expert subnetwork. In implementation, a dot product can be performed on each token query vector and each expert key vector separately, or a scaling or bias term can be added before the dot product to control the numerical amplitude. The resulting original association score matrix retains the complete distribution of token-to-expert subnetwork relationships. Each row in the matrix corresponds to the original score of a token across all expert subnetworks, and each column corresponds to the response of an expert subnetwork to all tokens. At this stage, the original association score matrix is ​​still unconstrained, and the total scores between different tokens are not comparable, therefore it cannot be directly used as the basis for subsequent allocation.

[0103] Attention normalization serves two purposes: numerical constraint and proportional output. Before normalization, the original scores of some tokens may be generally too high, while those of others may be generally too low. Even if the relative order is correct, this is not conducive to subsequent screening on a uniform scale. After normalization, the association results of the same token across multiple expert sub-networks are compressed into a set of comparable proportional values. In implementation, this can be achieved by exponentially normalizing the original association score matrix row by row along the token dimension, or by using a normalization method with a temperature parameter to adjust the steepness of the distribution. When the temperature parameter is small, the allocation results are more concentrated, and a few expert sub-networks will receive higher weights; when the temperature parameter is large, the allocation results are smoother, and multiple expert sub-networks will participate together. After normalization, each token corresponds to a set of allocation weights with a controlled total amount, and the dynamic routing module can output the allocation relationship on a uniform scale.

[0104] The expert assignment matrix is ​​formed during the matrix arrangement stage of the normalized results. Arrangement is not simply formatting; it involves rebinding the normalized assignment results to the token and expert dimensions, allowing subsequent modules to directly read the output. In implementation, each row of the normalized result retains three types of information: token index, expert index, and assignment weight, and is then written into the target matrix in the order of token position and expert number. This results in an expert assignment matrix with three attributes: First, a sequential attribute, preserving the order of tokens in the original text. Second, an assignment attribute, representing the weight relationship between each token and multiple expert subnetworks. Third, an updatable attribute, continuously changing as token features, signature vectors, and projection parameters are adjusted. Once the expert assignment matrix is ​​formed, subsequent modules do not need to return to the original text or original signature vectors to recalculate matching relationships; instead, they can directly perform subsequent filtering, distribution, and aggregation operations based on the matrix results.

[0105] There are explicit generation and dependency relationships among token features, signature vectors, query vectors, key vectors, the original association score matrix, and the expert assignment matrix. Token features are mapped to the query vector via the token projection matrix, the signature vector is mapped to the key vector via the expert projection matrix, the query vector and key vector are processed through attention calculation to form the original association score matrix, and the original association score matrix is ​​normalized and arranged to form the expert assignment matrix. Any change in the numerical scale, dimension setting, or index binding method of any intermediate result will affect the final expert assignment result. To avoid breakage of intermediate results, the dynamic routing module typically needs to maintain a unified index management mechanism. The token index is used to bind the query vector to the token position, the expert index is used to bind the key vector to the expert subnetwork number, and the matrix writing rules are used to ensure the stability of the row and column meanings in the final matrix. In this way, the assignment result can reflect semantic differences while maintaining the traceability of the calculation process.

[0106] The role of the signature vector in this step is not only to provide expert identification, but more importantly, to explicitly introduce the semantic preferences of the expert sub-networks into the allocation computation. Without the signature vector, the dynamic routing module can only perform aimless internal clustering based on token-side features, making it difficult to stably bind the allocation results to specific expert sub-networks. After the signature vector is added, the preference direction of the expert sub-network is reflected in the shared attention space through the projection matrix, and the relative position between the token query vector and the expert key vector has a clear meaning. If the same token consistently approaches the same expert key vector in different training rounds, it indicates that the expert sub-network has gradually assumed the corresponding semantic responsibility during training; if different tokens tend towards different expert key vectors, it indicates that a division of labor has formed between the expert sub-networks. The expert allocation matrix obtained in this way is not a random distribution result, but rather the output result of the combined effect of textual semantics and expert responsibilities.

[0107] If multiple semantic roles appear together in the input text, the responses of different tokens in the expert assignment matrix will differ significantly. Object-type tokens are more likely to generate higher matching values ​​with expert subnetworks oriented towards entity extraction and subject recognition; attribute-type tokens are more likely to generate higher matching values ​​with expert subnetworks oriented towards modification relationships and state constraints; spatial relationship-type tokens are more likely to generate higher matching values ​​with expert subnetworks oriented towards layout relationships and relative positions; and style-type tokens are more likely to generate higher matching values ​​with expert subnetworks oriented towards visual constraints and expressive preferences. This resulting expert assignment matrix reflects token differences at the numerical level, expert specialization at the structural level, and an adjustable assignment strategy at the training level.

[0108] For example, the formula for generating the expert allocation matrix is:

[0109]

[0110]

[0111]

[0112] in, This represents the expert allocation matrix, used to characterize the allocation weights between each token and each expert subnetwork. This represents the token feature matrix corresponding to the input token sequence. This represents the token query vector matrix. This represents the expert key vector matrix. This represents the token projection matrix. Represents the expert projection matrix. This represents a matrix composed of the signature vectors corresponding to each expert subnetwork. , ,…, Let represent the signature vectors corresponding to the 1st to nth expert subnetworks, respectively. d represents the vector dimension of the shared attention space. n represents the number of expert subnetworks. Softmax represents the normalization operation along the expert dimension, transforming the allocation result of a single token across all expert subnetworks into a comparable weight.

[0113] This embodiment maps token features and signature vectors to a shared attention space, enabling the dynamic routing module to measure the matching degree between tokens and expert sub-networks on a unified scale. This allows different semantic units, originally mixed within the same text, to receive differentiated allocation results. After attention normalization, the weights assigned by multiple experts to the same token are compressed to a comparable range, making the values ​​in the expert allocation matrix more stable and facilitating consistency in subsequent filtering and distribution. As the signature vector continues to participate in matching calculations, the division of labor within the expert sub-networks gradually becomes clearer, the association between tokens and experts is more likely to remain stable, and the situation where different semantic components in the text are compressed into a single processing channel is reduced.

[0114] In one embodiment, step S40 above includes: S401, Read the expert allocation weights in the expert allocation matrix according to the token dimension, and determine a high-weight expert subset for each token based on the sorting result of the expert allocation weights; S402, retain the expert allocation weights in the high-weight expert subsets corresponding to each token, and remove the expert allocation weights that have not entered the high-weight expert subsets corresponding to each token, to obtain a sparse candidate matrix; S403, normalize the retention weights corresponding to each token in the sparse candidate matrix to obtain the initial sparse routing matrix; S404: Count the number of times each expert subnetwork is retained in the current training batch to obtain the expert call statistics for the current training batch. S405, Based on the expert call statistics of the current training batch, the retained weights in the initial sparse routing matrix are balanced and corrected to obtain a balanced correction matrix; S406, the balance correction matrix is ​​arranged according to the token dimension and the expert dimension to obtain the token-level sparse routing result.

[0115] In this embodiment, after the expert allocation matrix is ​​formed, each token corresponds to a set of allocation weights for all expert subnetworks. Rows in the matrix distinguish different tokens, columns distinguish different expert subnetworks, and matrix elements reflect the relative strength of a token assigned to a particular expert subnetwork. If all allocation weights are directly retained, a single token would be associated with too many expert subnetworks simultaneously, resulting in overly dispersed allocations, insufficient concentration of the correspondence between tokens and expert subnetworks, and excessive overlap in content across different expert subnetworks in the same batch. After reading the expert allocation weights by token dimension, all expert allocation weights corresponding to each token are sorted in descending order, and a high-weight expert subset is determined according to a preset retention quantity or a preset retention threshold. When using a preset retention quantity, the expert subnetworks ranked in the top 2, top 3, or top 4 are selected as high-weight expert subsets from the sorting results; when using a preset retention threshold, expert subnetworks with expert allocation weights greater than or equal to the threshold are selected as high-weight expert subsets. If multiple expert subnetworks have the same expert allocation weight and are located at the retention boundary, the expert subnetwork with the smaller number is retained according to the order of their expert subnetwork numbers. If all expert allocation weights corresponding to a certain token are lower than the preset retention threshold, then the expert subnetwork with the largest expert allocation weight is retained to prevent the token from losing effective allocation results after filtering.

[0116] Once the high-weight expert subset is determined, retention and elimination actions need to be performed simultaneously. Retention targets matrix elements that have entered the high-weight expert subset, while elimination targets those that haven't. The resulting sparse candidate matrix retains its original row and column positions structurally, but many low-weight positions have been set to zero or marked as invalid. The sparse candidate matrix retains two key pieces of information: which expert subnetworks each token ultimately retained, and the relative strength of these retained positions in the original matrix. The matrix positions are not rearranged, meaning that the correspondence can still be directly restored using token and expert indices, avoiding the loss of allocation sources after filtering. To prevent a token from having all positions eliminated after filtering, a minimum retention constraint is usually set to ensure that each token retains at least one high-weight position, thus avoiding empty allocations.

[0117] Normalization occurs after the sparse candidate matrix is ​​formed. Before filtering, each row in the original expert allocation matrix already has a uniform numerical scale; after filtering, some weights are removed, and the total remaining weights of the same token will change. If normalization is not performed, the remaining weights will no longer constitute a stable proportion, making direct comparison difficult in subsequent use. Normalization is performed on a token-by-token basis. For each token, the total weights remaining are calculated first, and then each remaining weight is divided by this total to form a new proportion. If a token has many remaining positions, the contribution of each remaining position will be redistributed according to the remaining total after normalization; if a token has only a few remaining positions, these positions will concentrate on carrying the entire allocation strength of that token after normalization. The normalized result is organized into an initial sparse routing matrix. The difference between the initial sparse routing matrix and the sparse candidate matrix is ​​that the former not only preserves the sparse structure but also restores the directly usable proportional relationships within each token, transforming the numerical meaning from the original allocation strength to the effective allocation proportion after filtering.

[0118] Relying solely on the initial sparse routing matrix is ​​insufficient to guarantee stable utilization of different expert sub-networks across batches. If several tokens are consistently concentrated in a few expert sub-networks, these sub-networks will bear a higher load for an extended period, while other sub-networks will participate insufficiently, leading to a gradual imbalance in the allocation structure. Therefore, it is necessary to count the number of times each expert sub-network is retained in the current training batch. The retention count is not based on all responses in the original allocation matrix, but rather on the still valid positions in the sparse candidate matrix or the initial sparse routing matrix. After the statistics are completed, the expert call statistics for the current training batch are obtained. These statistics describe the frequency with which each expert sub-network is selected in the current batch; a higher value indicates that the expert sub-network receives more tokens in the current batch. The statistical results and the allocation weights of individual tokens represent different levels of information. The former reflects the global utilization status at the batch scale, while the latter reflects the local allocation status at the token scale. Only by combining both can local selection and overall balance be simultaneously adjusted.

[0119] The balancing correction is based on the expert call statistics of the current training batch. If the retention count of a certain expert subnetwork is significantly high, it indicates that the current batch relies heavily on that expert subnetwork, and the retention weights of that expert subnetwork in the initial sparse routing matrix need to be suppressed. If the retention count of a certain expert subnetwork is significantly low, it indicates that the participation of that expert subnetwork in the current batch is insufficient, and the corresponding retention weights can be appropriately compensated. The balancing correction does not recalculate the high-weight expert subsets, but rather redistributes the values ​​within the range of the existing retention positions in the initial sparse routing matrix. In implementation, correction coefficients can be constructed based on the retention counts, with smaller correction coefficients for expert subnetworks with higher retention counts and larger correction coefficients for expert subnetworks with lower retention counts. These correction coefficients are then applied to the effective weight positions in the initial sparse routing matrix. After correction, the balancing correction matrix is ​​obtained. The balancing correction matrix preserves the sparse structure and the effective correspondence between tokens and expert subnetworks, but the numerical distribution has simultaneously absorbed the overall usage status of the current batch.

[0120] For example, in a training batch, assuming the expert network cluster has four expert subnetworks, after high-weight filtering and normalization, a certain token retains only two valid expert positions. The initial retention weight for expert subnetwork 1 is 0.60, and for expert subnetwork 3 it is 0.40. After statistically analyzing all tokens in the current training batch, the expert call statistics are as follows: expert subnetwork 1 retains 120 times, expert subnetwork 2 retains 45 times, expert subnetwork 3 retains 60 times, and expert subnetwork 4 retains 35 times. This statistical result shows that expert subnetwork 1 is used more frequently in the current training batch, while expert subnetwork 3 is used relatively less frequently. Based on this statistical state, a correction coefficient of 0.90 is assigned to expert subnetwork 1, and a correction coefficient of 1.10 is assigned to expert subnetwork 3. These correction coefficients are then applied to the valid weight positions of the token in the initial sparse routing matrix, resulting in a correction value of 0.54 for expert subnetwork 1 and 0.44 for expert subnetwork 3. After renormalizing the corrected values, the weights of the token in the balanced correction matrix are approximately 0.551 for expert subnetwork 1 and 0.449 for expert subnetwork 3. This result maintains a high matching degree between the current token and expert subnetwork 1, while compensating for the less frequently used expert subnetwork 3, thus narrowing the gap in retained weights compared to before correction. This mitigates the impact of unbalanced expert utilization in the current training batch without altering the high-weight expert subset.

[0121] Token-level sparse routing results are generated during the orchestration stage of the balancing correction matrix. During orchestration, the token dimension and expert dimension are still used as the basic organizing coordinates, ensuring that the output maintains the same index meaning as the original expert allocation matrix and intermediate matrices. The resulting token-level sparse routing results are matrix outputs in form, but simultaneously contain three levels of information in meaning. One level is the token-level sparse expert selection result; another is the effective allocation ratio formed after normalization and filtering; and the third is the balancing correction result after combining the statistics of expert calls in the current training batch. This output not only suppresses the scattered response caused by low-weight allocations but also avoids long-term overload of a few expert subnetworks in the current training batch. Because the output still maintains the fixed positional relationship between the token dimension and expert dimension, subsequent use can directly read the expert subnetworks retained for each token and their corresponding corrected weights based on the index, without needing to re-parse the filtering process.

[0122] This embodiment filters a subset of high-weight experts from the expert allocation matrix by token dimension, transforming the allocation relationship of a single token to an expert subnetwork from a decentralized state to a centralized state, effectively compressing low-weight responses. By normalizing the retained weights in the sparse candidate matrix, the effective allocation ratio after filtering returns to a uniform numerical scale, and the retained weights within different tokens have stable comparability. By statistically analyzing the number of expert calls in the current training batch and balancing the initial sparse routing matrix, the imbalance in expert usage at the batch scale is suppressed, and the continuous overload of a few expert subnetworks is alleviated. The final token-level sparse routing result simultaneously retains the token index, expert index, and corrected allocation weights, resulting in a more centralized allocation relationship, more stable values, and a more balanced expert participation state.

[0123] In one embodiment, step S50 above includes: S501, Read the expert identifiers corresponding to each token in the token-level sparse routing results according to the token dimension, and generate a token expert distribution table; S502, according to the token expert distribution table, each token is recombined according to the expert identifier to obtain multiple expert input token sequences; S503, input the multiple expert input token sequences into the corresponding target experts in the expert network cluster respectively; S504, each corresponding target expert performs multi-layer feature transformation based on its own independent parameters. Each layer of feature transformation includes self-attention processing of the received token to obtain context interaction features, and feedforward transformation processing of the context interaction features. The output of the previous layer and the output of the current layer are merged with the residual to obtain intermediate expert features. S505, based on the token arrangement order in each of the expert input token sequences, the intermediate expert features are backfilled and recombined to obtain expert features.

[0124] In this embodiment, after the token-level sparse routing result is formed, the effective positions in the matrix have already limited the scope of experts each token actually participates in. At this point, it is no longer necessary for all tokens to simultaneously enter all expert subnetworks. Instead, effective expert identifiers are read according to the token dimension to generate a token expert allocation table. The token expert allocation table stores at least three types of information: a token index, used to identify the current token's position in the original token sequence; an expert identifier, used to identify which expert subnetwork the current token is assigned to; and a batch index or sequence index, used to distinguish tokens at the same position in different text samples. After the token expert allocation table is formed, the token set originally arranged in text order is re-split into a local input set clustered by expert affiliation. The significance of this setup is that different semantic components in the text are sent to expert subnetworks with different parameter responsibilities, reducing mutual interference caused by mixed transformations in the same parameter space.

[0125] When reorganizing tokens according to expert identifiers, tokens belonging to the same expert need to be aggregated into the same cache while preserving the original positional mapping. The expert input token sequence is not a simple truncation of the original text; rather, it involves reorganizing tokens scattered across different positions in the original sequence according to expert identifiers. During the reorganization process, two additional steps are typically required: length alignment and position recording. Length alignment addresses situations where different experts receive inconsistent numbers of tokens, which can be standardized using padding vectors or mask matrices. Position recording stores the token's sequence number in the original text, ensuring accurate backfilling of subsequently generated intermediate expert features. If the same token is assigned to multiple experts, multiple copies of that token need to be created in the expert input cache, each with an expert ID, to distinguish the source of the transformation results from different experts. After this reorganization process, multiple expert input token sequences are sent to their corresponding target experts within the expert network cluster. The input boundaries between expert subnetworks become clear, and subsequent parameter calculations no longer mix the responsibilities of different experts.

[0126] Each target expert undertakes parameterized transformations oriented towards a specific semantic category. Each expert performs multi-layer feature transformations based on their own independent parameters, which at least cover self-attention mapping parameters, feedforward mapping parameters, normalization parameters, and residual adjustment parameters. Self-attention processing reallocates contextual weights based on the token relationships within the current expert's input token sequence, enabling the tokens received by the current expert to establish new semantic dependencies locally. If the expert's input token sequence contains both subject words and modifiers, after self-attention computation, the response strength of the subject word to the modifier and the constraint strength of the modifier on the subject word are requantized, forming contextual interaction features. Contextual interaction features are no longer simply copies of the original token features but rather the result of local semantic interaction after incorporating the effects of expert parameters. Because different experts use independent parameters, the contextual interaction features obtained for the same token among different experts often show significant differences.

[0127] Feedforward transformation processes the dimensionality remapping within the contextual interaction features. While self-attention addresses the relationship allocation between tokens, feedforward transformation addresses the representational expansion within a single token vector. In implementation, the contextual interaction features can be mapped to a higher-dimensional space first, then compressed back to the target dimension via non-linear activation. Alternatively, gating units can be added to control the passing ratio of different dimensions. This results in output vectors that are more sensitive to the semantic direction of the current expert's preference. If an expert favors entity-type tokens, the feedforward transformed vector will strengthen entity-related dimensions; if an expert favors relation-type tokens, the feedforward transformed vector will strengthen relation-related dimensions. When residual merging is performed between the previous layer output and the current layer output, the original representation is not completely overwritten but is retained along with the newly generated transformation result. The purpose of residual merging is to suppress numerical degradation and excessive semantic shift that may occur in deep transformations, ensuring that each layer output retains the basic features of the original tokens while also incorporating the newly added contextual and semantic preference information from the current layer. Multi-layer feature transformation is carried out layer by layer on this basis. The results between different layers are accumulated through residuals, so that the intermediate expert features simultaneously carry local semantic relationships, deep transformation results and expert parameter biases.

[0128] After the intermediate expert features are formed, it is necessary to restore the original token organization. Because the preceding reassembly was done according to expert identifiers, the token order in the multiple expert input token sequences differs from the original text order. Without backfilling and reassembly, these results cannot be used subsequently according to the original token positions. During backfilling, based on the token arrangement order in each expert input token sequence and the previously saved original position mapping relationship, each intermediate expert feature is written back to the position buffer of the corresponding token. If the same token has multiple expert sources, the intermediate expert feature corresponding to each expert number is registered in the position buffer. After reassembly, the output result is no longer a local sequence separated by expert, but rather returns to the expert feature set organized by token position. Each vector in the expert features retains both the original token position and the transformation traces from the specific expert, thus enabling simultaneous identification of both the position source and the expert source when used subsequently.

[0129] This embodiment distributes different tokens to corresponding target experts based on token-level sparse routing results. Multiple semantic components, originally mixed in the same parameter space, are split into different expert sub-networks for processing, making the division of labor between tokens and experts clearer. After multiple expert input token sequences undergo multi-layer feature transformation with their own independent parameters, the output not only retains the original token information but also superimposes contextual interaction information and deep representational information related to expert semantic preferences. After intermediate expert features are backfilled and recombined according to the original token positions, the expert features maintain both positional consistency and source traceability. The resulting output reduces mutual interference between different semantic components in the same transformation space, making it easier to maintain a stable correspondence between token semantics and expert responsibilities during subsequent processing.

[0130] In one embodiment, step S60 above includes: S601, Read the expert identifier and route weight corresponding to each token in the token-level sparse routing result according to the token dimension, and generate a token expert alignment table; S602, according to the token expert alignment table, each expert feature is backfilled into the position corresponding to each token to obtain the aggregated candidate feature group corresponding to each token; S603, apply the routing weights corresponding to each token to the expert features in the aggregated candidate feature group corresponding to each token respectively, to obtain the weighted expert feature group corresponding to each token; S604, accumulate and aggregate the weighted expert feature groups corresponding to each token to obtain the token aggregate feature corresponding to each token; S605, based on the semantic component differentiation results in the token features corresponding to each token, extract the aggregated sub-features corresponding to the object description component, attribute description component, spatial relationship description component and style description component from the token aggregate features corresponding to each token, and generate a local semantic summary based on each aggregated sub-feature; S606, the local semantic summary and the token aggregation features corresponding to each token are fused and arranged according to the order of the tokens in the token sequence to obtain the text embedding sequence.

[0131] In this embodiment, after the token-level sparse routing results are formed, the effective correspondence between a single token and multiple expert subnetworks has been preserved. Subsequently, the expert features scattered across different expert subnetworks need to be reorganized back into the token dimension. When reading expert identifiers and routing weights by token dimension, the reading object is no longer the original expert allocation matrix, but rather a set of effective positions that has already been filtered and corrected. This reading result directly limits which expert features the current token can take values ​​from, and what contribution proportion different expert features should bear when participating in aggregation. The token expert alignment table here performs index reconstruction, storing at least three types of information: token index, expert index, and routing weight. The token index is used to restore the original text order, the expert index is used to locate the source of expert features, and the routing weight is used to control the contribution share of different expert features within the same token. If the same token retains multiple expert positions, the token expert alignment table will establish multiple mapping records for that token, so that subsequent aggregation does not depend on the expert output order, but on a clear index binding relationship.

[0132] Expert feature backfilling occurs after index recovery. The expert features output by the expert network cluster were originally stored scattered along the expert dimension, and the token arrangement within different experts might differ. Without backfilling, the output results of a single token from different experts cannot be uniformly aggregated. During backfilling, based on the token index in the token expert alignment table, the expert features belonging to the same token are written to the same aggregation position, forming an aggregated candidate feature group. Each member in the aggregated candidate feature group retains its original expert source; therefore, the candidate group is not an indiscriminate stack of vectors, but a set of vectors with source identifiers. With this setup, the transformation results of a single token in multiple expert parameter spaces can be accessed simultaneously in a unified location, avoiding misalignment between expert output and token position.

[0133] When routing weights are applied to the aggregation of candidate feature groups, each expert source needs to be weighted individually. The goal of weighting is not simply to amplify or reduce the numerical value, but to carry over the level of expert participation formed in the previous allocation stage to the aggregation stage. In implementation, this can be achieved by multiplying the routing weights element-wise with the corresponding expert features, or by first expanding the routing weights to the same dimension as the expert features before performing vector multiplication. In the resulting weighted expert feature group, the numerical magnitude of different expert features already reflects their relative strength of participation. If an expert feature corresponds to a higher routing weight, it retains a larger numerical contribution in subsequent accumulation; if it corresponds to a lower routing weight, it retains only a smaller influence in the accumulation. This approach allows the differentiated judgments from the allocation stage to be passed to the representation stage, ensuring that expert outputs with closer semantic responsibilities occupy a larger proportion of the aggregation result.

[0134] Accumulation and aggregation occur within a single token. The aggregation object is not all expert features, but rather the weighted expert feature group corresponding to the current token. In implementation, vector accumulation can be performed group by group according to token number, or all weighted expert features can be organized into tensors and then reduced in parallel. After accumulation, the token aggregated feature is obtained. The token aggregated feature retains the original token position while incorporating the different transformation results of multiple expert subnetworks for that token; therefore, it is no longer a single-path output but a comprehensive representation based on the contributions of multiple experts. If the same token is mainly processed by a single expert, that expert's features will dominate the accumulated result; if the same token requires multiple experts to express it, the accumulated result will simultaneously retain information from multiple semantic directions. The resulting token aggregated feature is more suitable as a local basic unit for the subsequent overall text representation.

[0135] The preceding token features already contain the distinction results for object description components, attribute description components, spatial relationship description components, and style description components. At this stage, these distinction results can be used to perform targeted splitting of the token aggregation features. In implementation, corresponding dimensions or sub-vectors can be extracted from the token aggregation features based on semantic category labels, forming aggregated sub-features corresponding to object description components, attribute description components, spatial relationship description components, and style description components. This extraction does not re-divide the original text, but rather preserves the separable parts of different semantic roles within the already formed aggregated representation. After each aggregated sub-feature is generated, it can be further locally summarized to form a local semantic summary. The role of the local semantic summary is to compress the prominent information of similar semantic components in the current token neighborhood into a more stable local representation, reducing the impact of occasional fluctuations in a single token on the overall text representation.

[0136] Local semantic summaries and token aggregation features need to be fused and orchestrated. During fusion, it's crucial to preserve the fine-grained differences already formed within individual token aggregation features while simultaneously writing the category summary information provided by the local semantic summaries back into the sequence representation. This can be achieved through concatenation mapping, element-wise weighted fusion, or gated fusion methods, combining the local semantic summaries with the corresponding token aggregation features and then reorganizing them according to the tokens' order in the token sequence. The resulting text embedding sequence retains the original token order, but each position in the sequence now includes both multi-expert aggregation results and local semantic summaries. Once the text embedding sequence is formed, order information, expert contribution information, and semantic category information are uniformly preserved in the same sequence output. Subsequent cross-modal mapping can then utilize both local token representations and compressed results of similar semantic categories within a local scope.

[0137] For example, the formula for generating token aggregation features is:

[0138] in, This represents the token aggregation feature corresponding to the t-th token. This represents the routing weight of the t-th token in the token-level sparse routing result corresponding to the i-th expert subnetwork. Let represent the sparse routing weight vector of the t-th token facing all expert subnetworks. This represents the set of k expert subnetwork indices with the highest weights selected from the routing weight vector of the t-th token. This indicates that the i-th expert subnetwork inputs the t-th token. The expert features obtained after feature transformation. Let represent the input representation corresponding to the t-th token. k represents the number of high-weight expert subnetworks retained.

[0139] In this embodiment, after rebinding expert outputs to token positions using a token expert alignment table, expert features scattered across different expert sub-networks can be uniformly processed under the same token dimension, reducing position mismatches and source confusion. By continuing to apply routing weights to the aggregated candidate feature group, the strength of expert participation formed in the previous allocation stage is carried over to the representation stage, ensuring that the contribution ratio of multiple expert outputs to a single token remains consistent. By accumulating the weighted expert feature group, the token aggregated feature absorbs effective information from multiple expert parameter spaces, allowing the convergence of differentiated semantics that are difficult to retain in a single path representation. By extracting aggregated sub-features based on the semantic component differentiation results and generating a local semantic summary, object description, attribute description, spatial relationships, and style information remain separable in the aggregated representation. After the local semantic summary is fused with the token aggregated feature in its original order, the text embedding sequence simultaneously retains order information, expert contribution information, and semantic category information, making it easier to maintain the integrity and stability of the text semantic expression in subsequent use.

[0140] In one embodiment, step S70 above includes: S701, The text embedding sequence is input into the cross-modal alignment module, and the text embedding sequence is mapped to the image latent space through the alignment layer in the cross-modal alignment module to obtain the projected text representation; S702, Based on the arrangement structure of the image latent code sequence, the image latent code sequence is organized at the regional level to obtain an image region latent code set; S703, the text embedding sequence is divided into multiple text segments to obtain the text segmentation result; S704, the image region latent code set is divided into multiple image regions to obtain the image region division result; S705, establish a soft correspondence between text segments and image regions based on the text segmentation results and the image region segmentation results; S706, Based on the positional difference between the projected text representation and the image latent code sequence in the image latent space, determine the mapping difference term; S707, Based on the degree of deviation between the distribution of the projected text representation and the distribution of the image latent code sequence, determine the distribution difference term; S708, determine the local alignment constraint term based on the soft correspondence; S709, the mapping difference term, the distribution difference term, and the local alignment constraint term are combined to obtain the cross-modal alignment loss.

[0141] In this embodiment, after the text embedding sequence is formed, it remains in the text-side representation space. If directly compared with the image latent code sequence, the vector dimension, numerical scale, and local structural meaning may be inconsistent, making it difficult to stably establish cross-modal constraints. The cross-modal alignment module undertakes two tasks at this stage: one is to push the text embedding sequence into the image latent space to form a comparable projected text representation; the other is to construct positional differences, distribution differences, and local correspondence constraints between the projected text representation and the image latent code sequence, and unify these differences into a cross-modal alignment loss. With this setting, the text-side output is no longer just an independent semantic sequence, but is transformed into a latent representation that can directly participate in image-side metrics.

[0142] The alignment layer performs the mapping function. Each position vector in the text embedding sequence first undergoes dimensionality transformation and then scale adjustment to ensure that the output vector maintains the same or compatible dimensions as the latent code vector in the image latent space. The mapping process can employ a single-layer linear transformation or a multi-layer nonlinear transformation. If a multi-layer structure is used, the middle layers compress local redundant information, and the final layer pushes the text-side representation into the image-side latent distribution. After the projected text representation is formed, each text position no longer only represents the original language semantics but also includes latent coordinate information that can be directly compared with the image latent code vector. The resulting projected text representation preserves the text order and possesses comparability with the image latent space.

[0143] At this stage, the image latent code sequence needs to be organized at the region level. Although the spatial order of the image latent code sequence has been preserved, without further extraction of the region structure, there is still a lack of clear reference units between the local text content and the local image content. Region-level organization can reorganize the continuous latent codes into a set of image region latent codes according to their spatial location, adjacency relationship, and block boundaries in the image latent code sequence. Each image region not only retains a set of latent code vectors, but also retains the position index and local range of the region in the overall image. In this way, the image side is no longer just a one-dimensional sequence, but is reconstructed into a set of latent representations with local region semantics, establishing a computable foundation for the correspondence between subsequent text fragments and image regions.

[0144] Text embedding sequences also require segmentation. Multiple consecutive positions in a text embedding sequence are not always independently expressed; some semantic units are composed of multiple adjacent positions. For example, object names and modifiers, relational words and positional words, style words and qualifiers often need to be considered as local combinations. The purpose of segmentation is to reorganize continuous vector sequences into several text segments with locally complete semantics. During segmentation, adjacent vectors can be merged into a single text segment based on positional adjacency, semantic category continuity, or similarity aggregation results. After the text segmentation results are formed, each text segment has a segment range and a set of segment vectors. After this processing, the alignment unit of local content on the text side is no longer a single isolated token, but a segment unit that is closer to a semantically complete expression.

[0145] After the text segmentation and image region segmentation results are formed, no single text segment is fixedly assigned to a specific image region. Instead, the association strength between each text segment and all image regions is calculated separately, forming a continuous mapping result from segment to region. In implementation, the projected text representation corresponding to each text segment and the latent code representation corresponding to each image region can be input into the similarity calculation unit to obtain a correspondence matrix consisting of the number of segments multiplied by the number of regions. Each element in the correspondence matrix represents the matching strength between a text segment and an image region. This matrix is ​​then normalized along the text segment dimension so that the sum of the association strengths of the same text segment with all image regions satisfies a unified constraint. After normalization, each text segment receives a set of continuous weights. These continuous weights are not a unique assignment result but rather indicate that the current text segment may simultaneously have varying degrees of correspondence with multiple image regions. In this soft correspondence, positions with higher weights indicate that the text segment is more likely to constrain the corresponding image region, while positions with lower weights indicate that the text segment and the corresponding image region only have a weak or background association.

[0146] When a text fragment contains a single object semantic, soft correspondences typically exhibit a locally concentrated distribution. When a text fragment contains object modifiers, relational modifiers, or layout modifiers simultaneously, soft correspondences typically exhibit a multi-regional distribution. For example, a text fragment could be a blue bank card icon. The image region segmentation result could include the top-left icon region, the central number region, the bottom-right description region, and the background decoration region. If the original matching strengths obtained after similarity calculation are 0.80, 0.10, 0.05, and 0.05 respectively, after normalization, the text fragment forms the highest association strength with the top-left icon region and a lower association strength with the other regions. This indicates that the blue bank card icon mainly constrains the top-left icon region, while retaining only a weak influence on other regions. If the text fragment is a benefit curve located below the title, this fragment contains both object information and spatial relational information. The normalized association strength may be distributed across the curve region and the adjacent region below the title, rather than being concentrated in a single region. This prevents the relational semantics in the text from being forcibly compressed into a single image region.

[0147] The introduction of soft correspondences can also distinguish between strongly constrained regions and auxiliary constrained regions. For high-risk warning bars for text fragments, they can be associated with the top banner region, the prominent identification region, and the main text description region simultaneously within the image region set. However, the correspondence strength can be set to high, medium, and low, respectively, so that during training, the top banner region bears the primary constraint, the prominent identification region bears the auxiliary constraint, and the main text description region retains only the weak constraint. In this way, the relationship between text fragments and image regions is no longer a simple one-to-one binding, but a weighted correspondence with primary and secondary differences. When calculating local alignment constraint terms later, these continuous weights can be directly used, so that highly associated regions bear stronger alignment errors and low-associated regions bear weaker alignment errors, thereby establishing more detailed cross-modal constraints without losing complex semantics.

[0148] The mapping difference term is built upon the latent positional differences between the projected text representation and the image latent code sequence. The projected text representation has entered the image latent space, and the image latent code sequence itself also resides within this space; therefore, the difference between the two can be obtained through inverse measurements such as vector distance, angular deviation, or similarity. The mapping difference term reflects the degree of offset between each text position or fragment in the latent space and its corresponding vector on the image side. If the projected text representation and the image latent code sequence are sufficiently close at similar semantic positions, the mapping difference term will decrease; if they are scattered in different regions of the latent space, the mapping difference term will increase. This resulting metric directly constrains the text representation to converge towards the image representation.

[0149] The distribution difference term constrains the overall distribution, not just individual locations. Even if the projected text representation and the image latent code sequence are close in local locations, if the overall distribution is still offset, the subsequent generated results may still exhibit local accuracy but overall imbalance. The distribution difference term measures the global deviation between the latent representation set on the text side and the latent representation set on the image side by comparing the mean distribution, variance distribution, covariance structure, or kernel mapping results of the two sets of vectors. When local correspondence is strong and the overall distribution offset is large, the distribution difference term will still maintain a high value; the distribution difference term will only decrease when both local correspondence and the overall distribution are close. This setting avoids the overall semantic structure imbalance caused by relying solely on local similarity.

[0150] The local alignment constraint is built upon soft correspondences. Soft correspondences already provide a continuous weight distribution between text fragments and image regions, but these weights still need to be transformed into constraints that can be used for training. In implementation, high-weight correspondences can bear stronger alignment penalties, while low-weight correspondences can bear weaker penalties. When a text fragment corresponds to multiple image regions, the differences between local regions are weighted and summarized according to the soft correspondence weights. The local alignment constraint reflects the fine-grained consistency between local semantic units in the text and local regional units in the image. If an object fragment is close to an incorrect image region in the latent space, the local alignment constraint will amplify this deviation; if a relation fragment maintains stable consistency with its corresponding image region, the local alignment constraint will decrease accordingly.

[0151] The mapping difference term, distribution difference term, and local alignment constraint term each play a different role at the numerical level. The mapping difference term controls the proximity of points, the distribution difference term controls the overall distribution consistency, and the local alignment constraint term controls the consistency of local details between segments and regions. When these three are combined, they are unified into a single cross-modal alignment loss. During the combination process, different weights can be assigned to the three terms, or range constraints can be applied to each term first, and then the results can be aggregated. The resulting cross-modal alignment loss preserves both local accuracy and overall distribution constraints, reflecting both the offset between individual text positions and image positions, as well as the overall consistency between the entire text segment and the entire image in the latent space.

[0152] For example, the formula for cross-modal alignment loss is:

[0153] in, This represents the cross-modal alignment loss, used to characterize the overall deviation between the text-side representation and the image-side representation under a unified training objective. [] indicates the expectation operation or average operation, which is used to summarize the difference results between all text fragments and image regions to obtain a stable global difference value. This represents the text segment representation matrix obtained based on the text segmentation results. The text segment representation matrix is ​​composed of text embedding sequences reorganized according to segment boundaries. Each row or each segment unit corresponds to the representation of a text segment in the image latent space. This represents the image region representation matrix obtained based on the image region segmentation results. The image region representation matrix here is composed of the image latent code sequence reorganized according to the region boundaries. Each row or each region unit corresponds to the representation of an image region in the image latent space. The text side alignment matrix is ​​used to further map the text fragment representation to a cross-modal comparison space. This matrix is ​​a learnable parameter that adjusts the coordinate distribution of the text fragment representation to make it more suitable for difference measurement with the image region representation. The image-side alignment matrix is ​​used to map the image region representation to a comparison space consistent with the text side. This matrix is ​​also a learnable parameter, and its function is to perform coordinate transformation on the image region representation so that the text side and the image side have a unified and comparable scale.

[0154] This represents the mapping difference term, which measures the Euclidean distance or L2 norm difference between the mapped text fragment representation and the image region representation. The smaller this value, the closer the text fragment and the image region are in the latent space.

[0155] λ represents the weighting coefficient of the distribution variance term, used to control the degree of influence of the distribution variance term in the total loss. λ is a non-negative scalar parameter; the larger the value, the stronger the constraint on the overall distribution consistency.

[0156] MMD(H,Z) represents the maximum mean difference between the text representation set and the image representation set. This term measures the deviation between the overall distribution on the text side and the overall distribution on the image side, and belongs to the distribution difference term. Here, H represents the overall text representation set corresponding to the text embedding sequence, and Z represents the overall image representation set corresponding to the image latent code sequence. H and Z are used instead of... and This is because this study emphasizes the overall consistency of distribution, focusing on the global statistical characteristics of the two types of representations rather than the one-to-one differences between individual segments and individual regions.

[0157] β represents the weighting coefficient of the local alignment constraint term, used to control the influence of the local alignment constraint term on the total loss. β is also a non-negative scalar parameter; the larger the value, the stronger the local correspondence constraint between the text fragment and the image region.

[0158] This represents a local alignment constraint term, which is determined based on the soft correspondence between text fragments and image regions. It is used to constrain highly correlated text fragments and highly correlated image regions to maintain stronger consistency. This term reflects the alignment quality at the local level, rather than the alignment quality at the overall distribution level.

[0159] This embodiment maps the text embedding sequence to the image latent space through an alignment layer, achieving a unified comparison scale between text side vectors and image side latent codes, thus alleviating the problem of direct comparison between text and image representations. By organizing the image latent code sequence into a set of image region latent codes and dividing the text embedding sequence into text segments, a soft correspondence is established between text segments and image regions, resulting in continuously adjustable correspondence constraints between local text content and local image content. By simultaneously introducing mapping difference terms, distribution difference terms, and local alignment constraint terms, the cross-modal alignment loss no longer relies solely on a single positional error but can simultaneously constrain local proximity, overall distribution consistency, and segment region correspondence. This cross-modal alignment loss more easily compresses the potential offset between text and image representations, ensuring more stable consistency in both local semantics and overall structure of the representations obtained through subsequent training.

[0160] In one embodiment, step S80 above includes: S801, the cross-modal alignment loss is backpropagated along the parameter link of the cross-modal alignment module to generate the first update information corresponding to the cross-modal alignment module; S802, update the parameters in the cross-modal alignment module according to the first update information, and send the updated error signal back to the expert network cluster; S803, based on the source relationship between the error signal and the characteristics of each expert, determine the set of expert sub-networks participating in the current training batch processing in the expert network cluster, and generate the second update information corresponding to the expert sub-network set; S804, update the transformation parameters of the expert sub-network set participating in the current training batch processing in the expert network cluster according to the second update information, and send the updated error signal back to the dynamic routing module along the routing chain corresponding to the token-level sparse routing result; S805 generates third update information based on the error signal returned to the dynamic routing module, and updates the token projection matrix, expert projection matrix and routing parameters in the dynamic routing module based on the third update information. S806 records the changes in cross-modal alignment loss and module parameter changes after each update of the dynamic routing module, expert network cluster, and cross-modal alignment module. S807 determines whether the preset convergence condition is met based on the change information of cross-modal alignment loss and the change information of module parameters. S808, when the preset convergence conditions are met, combines the updated dynamic routing module, expert network cluster, and cross-modal alignment module into a trained image and text generation framework.

[0161] In this embodiment, after the cross-modal alignment loss is formed, the training process no longer remains within a single mapping unit, but instead forms a unified parameter update relationship around the dynamic routing module, the expert network cluster, and the cross-modal alignment module. The cross-modal alignment module is located between the text embedding sequence and the image latent space, undertaking parameter calculations for the mapping layer, local correspondence constraint unit, and difference measurement unit. The expert network cluster is located between the routing assignment result and the text embedding formation process, undertaking multi-expert semantic transformation. The dynamic routing module is located between the token feature and the expert assignment matrix, undertaking assignment parameter calculations. The three processing units maintain a dependency relationship in the training graph through forward computation results. After the cross-modal alignment loss is determined, the error is propagated back from the cross-modal alignment module, then to the expert network cluster, and finally to the dynamic routing module, thus enabling the mapping space, expert transformation space, and routing assignment space to converge synchronously around the same error source. The training input can be batches of paired text and paired images, and the output is the parameter-updated image-text generation framework.

[0162] The first update information is derived by backpropagating the cross-modal alignment loss along the parameter connections of the cross-modal alignment module. The cross-modal alignment module contains at least alignment layer parameters, local alignment constraint parameters, and distribution difference metric parameters. The alignment layer parameters are responsible for embedding the text into the image latent space; the local alignment constraint parameters are responsible for transforming the soft correspondence between text fragments and image regions into optimizable quantities; and the distribution difference metric parameters are responsible for constraining the overall statistical deviation between the text side and the image side. During backpropagation, the gradients of these parameters are first calculated based on the cross-modal alignment loss, and then the gradients are converted into parameter corrections to form the first update information. The first update information can be represented as the gradient value, mean gradient, gradient direction, and corresponding update magnitude for each set of parameters. After the parameter updates are completed, the error signal of the cross-modal alignment module continues to retain index information related to the text embedding position, so that when backpropagated to the expert network cluster, the source location of the current expert feature can be accurately found.

[0163] The update of the expert network cluster is not performed simultaneously on all expert subnetworks, but is limited to the set of expert subnetworks actually involved in the current training batch based on the source relationships of expert features. The source relationships are jointly provided by the expert feature index, token index, and expert index formed in the forward phase, which clearly identifies which expert subnetworks directly contribute to the text embedding formation of the current batch. Based on the error signal and these source relationships, the set of expert subnetworks participating in the current training batch can be screened out, and a second update information can be generated. The parameters covered by the second update information include at least the self-attention mapping parameters, feedforward mapping parameters, normalization parameters, and residual adjustment parameters within each participating expert subnetwork. During the update, different expert subnetworks maintain independent parameter corrections, avoiding parameter mixing, thus ensuring that the transformation spaces of different semantic responsibilities remain distinct. If only some expert subnetworks are invoked in the current training batch, the parameters of expert subnetworks not participating in the current batch remain unchanged, avoiding parameter drift caused by invalid updates.

[0164] When the error signal is propagated back to the dynamic routing module, it is not regenerated detached from the forward allocation relationship, but rather propagated backward along the valid allocation relationships already retained in the token-level sparse routing results. The purpose of this is to constrain the backward update to the positions that actually participated in the forward computation, ensuring that unretained route positions do not participate in the current round's allocation parameter correction. Based on the propagated error signal, third update information can be generated, covering at least the token projection matrix, expert projection matrix, and route scoring parameters. Updating the token projection matrix changes the coordinate distribution of token features after entering the scoring space; updating the expert projection matrix changes the representation position of the signature vector after entering the scoring space; and updating the route scoring parameters changes the strength of the response between the token and the expert. After this update, the expert allocation matrix generated by the dynamic routing module in the next training round will be closer to the cross-modal alignment target, rather than merely close to the local attention response.

[0165] After each round of parameter adjustment, it is also necessary to record the changes in cross-modal alignment loss and module parameter changes. The cross-modal alignment loss change information can include the loss value of the current round, the difference in loss between adjacent rounds, and the moving average loss value across multiple rounds. The module parameter change information can include the average update magnitude, parameter norm change value, and mean gradient value of the cross-modal alignment module, expert network cluster, and dynamic routing module. The preset convergence condition does not rely on a single loss decrease but considers both loss and parameter changes simultaneously. If the loss value decreases continuously over several rounds, and the average parameter change value of the three processing units is simultaneously below the stability threshold, then the training can be considered to have entered a stable state. To improve feasibility, a small number of parameter examples can be set, such as a batch size of 32, a learning rate of 0.0003 for the cross-modal alignment module, a learning rate of 0.0002 for the expert network cluster, a learning rate of 0.0001 for the dynamic routing module, a gradient clipping threshold of 1, a loss change threshold of 0.001, and a parameter change threshold of 0.0005. With this configuration, the convergence speed of the mapping space is higher than that of the expert transformation space, which in turn is higher than that of the routing allocation space. This reduces the instability caused by rapid oscillations in front-end allocation.

[0166] During training, parameter update rhythms can be set to allow the three processing units to use different update magnitudes within the same round. The alignment layer parameters and metric parameters in the cross-modal alignment module can maintain a relatively fast update rhythm to promptly compress the offset between the text and image sides; the multi-layer attention parameters and feedforward parameters in the expert network cluster maintain a moderate update rhythm to gradually stabilize the transformation results for different semantic types; the token projection matrix, expert projection matrix, and route scoring parameters in the dynamic routing module maintain a relatively slow update rhythm to maintain a smooth change in the expert assignment boundary. If an abnormal increase in loss occurs for several consecutive rounds during training, the most recently stable parameter snapshot can be retained, and training can continue with a smaller update magnitude after recovery. Thus, the trained image-text generation framework is not the output of a single converged module, but rather a set of parameters jointly formed by the three processing units under a unified objective constraint.

[0167] This embodiment applies the cross-modal alignment loss uniformly to the cross-modal alignment module, expert network cluster, and dynamic routing module. This allows text mapping results, expert transformation results, and route allocation results to be adjusted collectively around the same error source, reducing the likelihood of correct local mappings but global allocation offsets. By updating only the expert sub-networks actually involved in the current training batch based on expert feature source relationships, the parameter correction range is limited to the effective contribution positions, suppressing the drift of irrelevant expert parameters. By backpropagating error signals along the effective positions in the token-level sparse routing results, the dynamic routing module's updates are consistent with the forward allocation, making the allocation boundaries more stable. By simultaneously recording loss change information and module parameter change information, and then determining the convergence state based on both, the parameter set at the end of training is more stable, and the resulting image-text generation framework maintains high consistency in text allocation, semantic transformation, and cross-modal mapping.

[0168] In one embodiment, an image generation apparatus based on token-level dynamic routing is provided, which corresponds one-to-one with the image generation method based on token-level dynamic routing described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the image generation device based on token-level dynamic routing of the present invention. The modules include a framework configuration module 10, a sample preprocessing module 20, a routing decision module 30, a sparse filtering module 40, an expert processing module 50, a feature aggregation module 60, an alignment calculation module 70, a parameter update module 80, and an image generation module 90. Detailed descriptions of each functional module are as follows: The framework configuration module 10 is used to construct a text and image generation framework. In the text and image generation framework, a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. The sample preprocessing module 20 is used to acquire paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples. The routing decision module 30 is used to input the token sequence into the dynamic routing module, and the dynamic routing module generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert sub-network. The sparse filtering module 40 is used to perform sparsification and normalization processing on the expert allocation matrix to obtain token-level sparse routing results. The expert processing module 50 is used to distribute each token to the corresponding target expert in the expert network cluster according to the token-level sparse routing result, and each target expert performs feature transformation on the received token to obtain expert features; Feature aggregation module 60 is used to perform weighted aggregation of each expert feature according to the token-level sparse routing result to obtain a text embedding sequence; Alignment calculation module 70 is used to input the text embedding sequence into the cross-modal alignment module, map the text embedding sequence to the image latent space through the cross-modal alignment module to obtain the projected text representation, and determine the cross-modal alignment loss based on the projected text representation and the image latent code sequence; The parameter update module 80 is used to update the dynamic routing module, the expert network cluster and the cross-modal alignment module based on the cross-modal alignment loss until the preset convergence condition is met, and the trained image and text generation framework is obtained. The image generation module 90 is used to input the target text into the trained image-text generation framework and generate the target image corresponding to the target text.

[0169] In one embodiment, the sample preprocessing module 20 is specifically used for: Obtain paired training text samples and training image samples, perform pairing consistency verification on the paired training text samples and training image samples, and generate sample pairing identifiers; Based on the sample pairing identifiers, multi-granularity word segmentation is performed on the training text samples to obtain a set of text fragments; The set of text fragments is mapped to a unified embedding space, and the arrangement order and contextual adjacency of each text fragment are written into the mapping result to obtain the token sequence and token features; After the token sequence is formed, the object description components, attribute description components, spatial relationship description components and style description components corresponding to each token are distinguished to obtain semantic component distinction results, and the semantic component distinction results are written into the token feature; Based on the sample pairing identifier, the training image samples are divided into image regions and regional latent codes are extracted to obtain multiple latent code units; The multiple latent code units are arranged according to the spatial arrangement order of each image region to obtain the image latent code sequence.

[0170] In one embodiment, the routing decision module 30 is specifically used for: The token sequence is input into the dynamic routing module, and the token characteristics corresponding to the token sequence are obtained; Obtain the signature vector corresponding to each expert subnetwork from the expert network cluster; The token features are mapped to the shared attention space using the learnable token projection matrix in the dynamic routing module to obtain the token query vector. The signature vectors corresponding to each expert subnetwork are mapped to the shared attention space using the learnable expert projection matrix in the dynamic routing module to obtain the expert key vectors. Based on the token query vector and the expert key vector, query key value attention calculation is performed to obtain the original association score matrix; Attention normalization is performed on the original association score matrix, and the matrix is ​​arranged according to the token dimension and the expert dimension to generate an expert assignment matrix.

[0171] In one embodiment, the sparse screening module 40 is specifically used for: Read the expert allocation weights in the expert allocation matrix according to the token dimension, and determine a high-weight expert subset for each token based on the sorting result of the expert allocation weights; The expert allocation weights in the high-weight expert subsets corresponding to each token are retained, and the expert allocation weights that have not entered the high-weight expert subsets corresponding to each token are removed to obtain a sparse candidate matrix. The retention weights corresponding to each token in the sparse candidate matrix are normalized to obtain the initial sparse routing matrix. The number of times each expert subnetwork is retained in the current training batch is counted to obtain the expert call statistics for the current training batch. Based on the expert call statistics of the current training batch, the retained weights in the initial sparse routing matrix are balanced and corrected to obtain a balanced correction matrix. The balance correction matrix is ​​arranged according to the token dimension and the expert dimension to obtain the token-level sparse routing result.

[0172] In one embodiment, the expert processing module 50 is specifically used for: Read the expert identifiers corresponding to each token in the token-level sparse routing results according to the token dimension, and generate a token expert distribution table; Based on the token expert distribution table, each token is recombined according to the expert identifier to obtain multiple expert input token sequences; The multiple expert input token sequences are respectively input into the corresponding target experts in the expert network cluster; Each corresponding target expert performs multi-layer feature transformation based on their own independent parameters. Each layer of feature transformation includes self-attention processing of the received token to obtain context interaction features, and feedforward transformation processing of the context interaction features. The output of the previous layer is merged with the output of the current layer to obtain intermediate expert features. The intermediate expert features are backfilled and recombined according to the token arrangement order in the expert input token sequence to obtain the expert features.

[0173] In one embodiment, the feature aggregation module 60 is specifically used for: Read the expert identifier and route weight corresponding to each token in the token-level sparse routing results according to the token dimension, and generate a token expert alignment table; Based on the token expert alignment table, each expert feature is backfilled into the position corresponding to each token to obtain the aggregated candidate feature group corresponding to each token; The routing weights corresponding to each token are applied to the expert features in the aggregated candidate feature group corresponding to each token to obtain the weighted expert feature group corresponding to each token. The weighted expert feature groups corresponding to each token are accumulated and aggregated to obtain the token aggregated feature corresponding to each token. Based on the semantic component differentiation results in the token features corresponding to each token, aggregate sub-features corresponding to object description components, attribute description components, spatial relationship description components and style description components are extracted from the token aggregate features corresponding to each token, and local semantic summaries are generated based on each aggregate sub-feature. The local semantic summary and the token aggregation features corresponding to each token are fused and arranged according to the order of the tokens in the token sequence to obtain the text embedding sequence.

[0174] In one embodiment, the alignment calculation module 70 is specifically used for: The text embedding sequence is input into the cross-modal alignment module, and the text embedding sequence is mapped to the image latent space through the alignment layer in the cross-modal alignment module to obtain the projected text representation; Based on the arrangement structure of the image latent code sequence, the image latent code sequence is organized at the regional level to obtain an image region latent code set; The text embedding sequence is divided into multiple text segments to obtain the text segmentation result; The image region latent code set is divided into multiple image regions to obtain the image region division result; A soft correspondence between text segments and image regions is established based on the text segmentation results and the image region segmentation results; Based on the positional difference between the projected text representation and the image latent code sequence in the image latent space, a mapping difference term is determined; Based on the degree of deviation between the distribution of the projected text representation and the distribution of the image latent code sequence, a distribution difference term is determined; Local alignment constraints are determined based on the soft correspondence; The cross-modal alignment loss is obtained by combining the mapping difference term, the distribution difference term, and the local alignment constraint term.

[0175] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the server-side functions or steps of a token-level dynamic routing-based image generation method.

[0176] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a token-level dynamic routing-based image generation method.

[0177] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: A text and image generation framework is constructed, in which a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. Obtain paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples; The token sequence is input into the dynamic routing module, which generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results; Based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster, and each target expert performs feature transformation on the received token to obtain expert features; The expert features are weighted and aggregated according to the token-level sparse routing results to obtain the text embedding sequence; The text embedding sequence is input into the cross-modal alignment module, which maps the text embedding sequence to the image latent space to obtain the projected text representation. The cross-modal alignment loss is then determined based on the projected text representation and the image latent code sequence. The dynamic routing module, the expert network cluster, and the cross-modal alignment module are updated based on the cross-modal alignment loss until the preset convergence condition is met, and the trained image and text generation framework is obtained. Input the target text into the trained image-text generation framework to generate the target image corresponding to the target text.

[0178] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: A text and image generation framework is constructed, in which a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. Obtain paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples; The token sequence is input into the dynamic routing module, which generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results; Based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster, and each target expert performs feature transformation on the received token to obtain expert features; The expert features are weighted and aggregated according to the token-level sparse routing results to obtain the text embedding sequence; The text embedding sequence is input into the cross-modal alignment module, which maps the text embedding sequence to the image latent space to obtain the projected text representation. The cross-modal alignment loss is then determined based on the projected text representation and the image latent code sequence. The dynamic routing module, the expert network cluster, and the cross-modal alignment module are updated based on the cross-modal alignment loss until the preset convergence condition is met, and the trained image and text generation framework is obtained. Input the target text into the trained image-text generation framework to generate the target image corresponding to the target text.

[0179] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0182] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0183] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image generation method based on token-level dynamic routing, characterized in that, Includes the following steps: A text and image generation framework is constructed, in which a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. Obtain paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples; The token sequence is input into the dynamic routing module, which generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results; Based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster, and each target expert performs feature transformation on the received token to obtain expert features; The expert features are weighted and aggregated according to the token-level sparse routing results to obtain the text embedding sequence; The text embedding sequence is input into the cross-modal alignment module, which maps the text embedding sequence to the image latent space to obtain the projected text representation. The cross-modal alignment loss is then determined based on the projected text representation and the image latent code sequence. The dynamic routing module, the expert network cluster, and the cross-modal alignment module are updated based on the cross-modal alignment loss until the preset convergence condition is met, and the trained image and text generation framework is obtained. Input the target text into the trained image-text generation framework to generate the target image corresponding to the target text.

2. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, Obtain paired training text samples and training image samples; perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features; and extract image latent code sequences from the training image samples, including: Obtain paired training text samples and training image samples, perform pairing consistency verification on the paired training text samples and training image samples, and generate sample pairing identifiers; Based on the sample pairing identifiers, multi-granularity word segmentation is performed on the training text samples to obtain a set of text fragments; The set of text fragments is mapped to a unified embedding space, and the arrangement order and contextual adjacency of each text fragment are written into the mapping result to obtain the token sequence and token features; After the token sequence is formed, the object description components, attribute description components, spatial relationship description components and style description components corresponding to each token are distinguished to obtain semantic component distinction results, and the semantic component distinction results are written into the token feature; Based on the sample pairing identifier, the training image samples are divided into image regions and regional latent codes are extracted to obtain multiple latent code units; The multiple latent code units are arranged according to the spatial arrangement order of each image region to obtain the image latent code sequence.

3. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, The token sequence is input into the dynamic routing module, which generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork, including: The token sequence is input into the dynamic routing module, and the token characteristics corresponding to the token sequence are obtained; Obtain the signature vector corresponding to each expert subnetwork from the expert network cluster; The token features are mapped to the shared attention space using the learnable token projection matrix in the dynamic routing module to obtain the token query vector. The signature vectors corresponding to each expert subnetwork are mapped to the shared attention space using the learnable expert projection matrix in the dynamic routing module to obtain the expert key vectors. Based on the token query vector and the expert key vector, query key value attention calculation is performed to obtain the original association score matrix; Attention normalization is performed on the original association score matrix, and the matrix is ​​arranged according to the token dimension and the expert dimension to generate an expert assignment matrix.

4. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, The expert allocation matrix is ​​subjected to sparsification and normalization to obtain token-level sparse routing results, including: Read the expert allocation weights in the expert allocation matrix according to the token dimension, and determine a high-weight expert subset for each token based on the sorting result of the expert allocation weights; The expert allocation weights in the high-weight expert subsets corresponding to each token are retained, and the expert allocation weights that have not entered the high-weight expert subsets corresponding to each token are removed to obtain a sparse candidate matrix. The retention weights corresponding to each token in the sparse candidate matrix are normalized to obtain the initial sparse routing matrix. The number of times each expert subnetwork is retained in the current training batch is counted to obtain the expert call statistics for the current training batch. Based on the expert call statistics of the current training batch, the retained weights in the initial sparse routing matrix are balanced and corrected to obtain a balanced correction matrix. The balance correction matrix is ​​arranged according to the token dimension and the expert dimension to obtain the token-level sparse routing result.

5. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, Based on the token-level sparse routing results, each token is distributed to the corresponding target expert in the expert network cluster. Each target expert then performs feature transformation on the received token to obtain expert features, including: Read the expert identifiers corresponding to each token in the token-level sparse routing results according to the token dimension, and generate a token expert distribution table; Based on the token expert distribution table, each token is recombined according to the expert identifier to obtain multiple expert input token sequences; The multiple expert input token sequences are respectively input into the corresponding target experts in the expert network cluster; Each corresponding target expert performs multi-layer feature transformation based on their own independent parameters. Each layer of feature transformation includes self-attention processing of the received token to obtain context interaction features, and feedforward transformation processing of the context interaction features. The output of the previous layer is merged with the output of the current layer to obtain intermediate expert features. The intermediate expert features are backfilled and recombined according to the token arrangement order in the expert input token sequence to obtain the expert features.

6. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, The expert features are weighted and aggregated according to the token-level sparse routing results to obtain a text embedding sequence, including: Read the expert identifier and route weight corresponding to each token in the token-level sparse routing results according to the token dimension, and generate a token expert alignment table; Based on the token expert alignment table, each expert feature is backfilled into the corresponding position of each token to obtain the aggregated candidate feature group corresponding to each token; The routing weights corresponding to each token are applied to the expert features in the aggregated candidate feature group corresponding to each token to obtain the weighted expert feature group corresponding to each token. The weighted expert feature groups corresponding to each token are accumulated and aggregated to obtain the token aggregated feature corresponding to each token. Based on the semantic component differentiation results in the token features corresponding to each token, aggregate sub-features corresponding to object description components, attribute description components, spatial relationship description components and style description components are extracted from the token aggregate features corresponding to each token, and local semantic summaries are generated based on each aggregate sub-feature. The local semantic summary and the token aggregation features corresponding to each token are fused and arranged according to the order of the tokens in the token sequence to obtain the text embedding sequence.

7. The image generation method based on token-level dynamic routing as described in claim 1, characterized in that, The text embedding sequence is input into the cross-modal alignment module, which maps the text embedding sequence to the image latent space to obtain a projected text representation. The cross-modal alignment loss is then determined based on the projected text representation and the image latent code sequence, including: The text embedding sequence is input into the cross-modal alignment module, and the text embedding sequence is mapped to the image latent space through the alignment layer in the cross-modal alignment module to obtain the projected text representation; Based on the arrangement structure of the image latent code sequence, the image latent code sequence is organized at the regional level to obtain an image region latent code set; The text embedding sequence is divided into multiple text segments to obtain the text segmentation result; The image region latent code set is divided into multiple image regions to obtain the image region division result; A soft correspondence between text fragments and image regions is established based on the text fragment segmentation results and the image region segmentation results; Based on the positional difference between the projected text representation and the image latent code sequence in the image latent space, a mapping difference term is determined; Based on the degree of deviation between the distribution of the projected text representation and the distribution of the image latent code sequence, a distribution difference term is determined; Local alignment constraints are determined based on the soft correspondence; The cross-modal alignment loss is obtained by combining the mapping difference term, the distribution difference term, and the local alignment constraint term.

8. An image generation device based on token-level dynamic routing, characterized in that, The image generation device based on token-level dynamic routing includes: The framework configuration module is used to build a text and image generation framework. In the text and image generation framework, a dynamic routing module, an expert network cluster, and a cross-modal alignment module are configured. The expert network cluster includes multiple expert sub-networks, and the signature vector corresponding to each expert sub-network is a learnable parameter. The sample preprocessing module is used to acquire paired training text samples and training image samples, perform word segmentation and embedding representation on the training text samples to obtain token sequences and token features, and extract image latent code sequences from the training image samples. The routing decision module is used to input the token sequence into the dynamic routing module, and the dynamic routing module generates an expert allocation matrix based on the token features and the signature vectors corresponding to each expert subnetwork. The sparse filtering module is used to perform sparsification and normalization processing on the expert allocation matrix to obtain token-level sparse routing results. The expert processing module is used to distribute each token to the corresponding target expert in the expert network cluster according to the token-level sparse routing result, and each target expert performs feature transformation on the received token to obtain expert features; The feature aggregation module is used to perform weighted aggregation of each expert feature according to the token-level sparse routing result to obtain a text embedding sequence; An alignment calculation module is used to input the text embedding sequence into the cross-modal alignment module, map the text embedding sequence to the image latent space through the cross-modal alignment module to obtain a projected text representation, and determine the cross-modal alignment loss based on the projected text representation and the image latent code sequence; The parameter update module is used to update the dynamic routing module, the expert network cluster, and the cross-modal alignment module based on the cross-modal alignment loss until a preset convergence condition is met, thus obtaining the trained image and text generation framework. The image generation module is used to input the target text into the trained image-text generation framework and generate the target image corresponding to the target text.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a token-based dynamic routing image generation program stored in the memory and executable on the processor. When executed by the processor, the token-based dynamic routing image generation program implements the steps of the token-based dynamic routing image generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an image generation program based on token-level dynamic routing, which, when executed by a processor, implements the steps of the image generation method based on token-level dynamic routing as described in any one of claims 1-7.

Images