Method, system and storage medium for generating image description using machine learning model

Abstract

This disclosure describes techniques for generating an image description using a machine learning model. Expert mix (MoE) blocks are incorporated into a plurality of sub-models of a machine learning model. A first sub-model of the machine learning model includes at least one first MoE block, the first MoE block including a first plurality of experts. A second sub-model of the machine learning model includes at least one second MoE block, the second MoE block including a second plurality of experts. Only a subset of the first plurality of experts is activated to generate a visual marker based on the input image. Only a subset of the second plurality of experts is activated to project visual markings into an input space of the third sub-model. And outputting the text description of the input image through a third sub-model of the machine learning model.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 639,969, filed April 29, 2024, and U.S. Non-Provisional Application No. 18 / 827,521, filed September 6, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application generally relates to the field of computer science, and more specifically to methods, systems, and storage media for generating image descriptions using machine learning models. Background Technology

[0003] Machine learning models are increasingly being used across various industries to perform a wide range of tasks. These tasks can include generating image descriptions. Improved techniques for leveraging machine learning models for image description generation are desired. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for generating image descriptions using a machine learning model is provided, comprising: configuring the machine learning model by incorporating expert hybrid MoE blocks into multiple sub-models of the machine learning model, wherein a first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein a second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts. Visual markers are generated through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate visual markers based on an input image. The visual markers are projected through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual markers into the input space of a third sub-model of the machine learning model. A textual description of the input image is output through the third sub-model of the machine learning model, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

[0005] In a second aspect of this disclosure, a system for generating image descriptions using a machine learning model is provided, comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and including computer-readable instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include: configuring the machine learning model by incorporating expert-mixed MoE blocks into a plurality of sub-models of the machine learning model, wherein a first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein a second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts; generating visual markers through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual markers based on an input image; projecting the visual markers through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual markers into an input space of a third sub-model of the machine learning model; and outputting a textual description of the input image through the third sub-model of the machine learning model, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

[0006] In a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations including: configuring a machine learning model by incorporating expert hybrid MoE blocks into multiple sub-models of a machine learning model, wherein a first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein a second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts; generating visual markers through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual markers based on an input image; projecting the visual markers through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual markers into the input space of a third sub-model of the machine learning model; and outputting a textual description of the input image through the third sub-model of the machine learning model, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The following detailed description will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, exemplary embodiments of various aspects of this disclosure are shown in the drawings; however, the invention is not limited to the specific methods and means disclosed.

[0009] Figure 1 An example system for generating image descriptions using a machine learning model, according to this disclosure, is shown.

[0010] Figure 2 An example converter block according to the first sub-model of this disclosure is shown.

[0011] Figure 3 An example system for generating image descriptions using a machine learning model, according to this disclosure, is shown.

[0012] Figure 4 An example three-stage training process according to this disclosure is shown.

[0013] Figure 5 An example process for generating image descriptions using a machine learning model, according to this disclosure, is shown.

[0014] Figure 6 An example process for generating visual markers from a first sub-model, according to this disclosure, is shown.

[0015] Figure 7 An example process for generating visual markers from a first sub-model, according to this disclosure, is shown.

[0016] Figure 8 An example process for processing visual markers via a second sub-model, according to this disclosure, is shown.

[0017] Figure 9 An example process for generating image descriptions using a machine learning model, according to this disclosure, is shown.

[0018] Figure 10 An example process for generating image descriptions using a machine learning model, according to this disclosure, is shown.

[0019] Figure 11 An example process for training a machine learning model according to this disclosure is shown.

[0020] Figure 12 An example process for training a machine learning model according to this disclosure is shown.

[0021] Figure 13 Example tables illustrating performance data according to this disclosure are shown.

[0022] Figure 14A An example table showing the scoring data according to this disclosure is shown.

[0023] Figure 14B An example table showing the scoring data according to this disclosure is shown.

[0024] Figure 15 Example tables illustrating performance data according to this disclosure are shown.

[0025] Figure 16A An example table illustrating performance data according to the present invention is shown.

[0026] Figure 16B An example table showing performance data according to this disclosure is provided.

[0027] Figure 16C An example table illustrating performance data according to the present invention is shown.

[0028] Figure 17 An example computing device is shown that can be used to perform any of the techniques disclosed herein. Detailed Implementation

[0029] In the field of natural language processing, large language models can be based on transformer architectures. Many multimodal machine learning models utilize pre-trained visual encoders to provide visual content and enable their visual capabilities. However, scaling up large multimodal language models can be challenging. Therefore, techniques for improving large multimodal language models are needed.

[0030] This paper describes an improved technique for enhancing multimodal machine learning models. The technique described incorporates Top-K sparse-gated expert hybrid (MoE) blocks into each sub-model of the multimodal machine learning model. For example, MoE blocks are incorporated into visual encoders, multilayer perceptron (MLP) connectors, and large language models of multimodal machine learning models to enhance the capabilities of the multimodal machine learning model.

[0031] A three-stage training process with auxiliary loss can be used to train multimodal machine learning models to stabilize training and maintain a balanced loading of experts. The first stage of the three-stage training process may include an MLP connector for a pre-trained multimodal machine learning model. The second stage may include warming up the entire multimodal machine learning model through pre-fine-tuning. Pre-fine-tuning stabilizes the third stage of the three-stage training process with added MoE blocks. During the third stage, each MLP block in each sub-model can be replaced with a Top-K sparsely gated MoE block through a co-upgrade. Each MoE block of each sub-model can be initialized from a corresponding MLP that has been well-trained through the first and / or second stages. Each MoE block may include a Top-K router trained from scratch to select the MLP expert during the third stage.

[0032] Figure 1 An example system 100 according to this disclosure is illustrated. System 100 may include a machine learning model 101. Machine learning model 101 may include a first sub-model 102. First sub-model 102 may include a contrastive language image pre-trained (CLIP) visual encoder. Machine learning model 101 may include a second sub-model 104. Second sub-model 104 may include an MLP connector. Machine learning model 101 may include a third sub-model 106. Third sub-model 106 may include a large language model.

[0033] Machine learning model 101 can be configured by incorporating expert hybrid (MoE) blocks into each of the first sub-models 102, 104, and 106. For example, at least one first MoE block 112 can be incorporated into the first sub-model 102. Each of the multiple first MoE blocks 112 may include a first plurality of experts. MoE blocks can be incorporated into each layer of the first sub-model 102. At least one second MoE block 114 can be incorporated into the second sub-model 104. Each of the multiple second MoE blocks 114 may include a second plurality of experts. MoE blocks can be incorporated into each layer of the second sub-model 104. At least one third MoE block 116 can be incorporated into the third sub-model 106. Each of the multiple third MoE blocks 116 may include a third plurality of experts. MoE blocks can be incorporated into each layer of the third sub-model 106.

[0034] Machine learning model 101 may receive input image 103 as input. First sub-model 102 may receive input image 103. First sub-model 102 may generate visual labels based on input image 103. To generate visual labels based on input image 103, first sub-model 102 may generate a representation of input image 103 by performing self-attention and normalization. The representation of input image 103 may be routed (e.g., via a router of at least one first MoE block 112) to a subset of first plurality of experts. Only a subset of the first plurality of experts in at least one first MoE block 112 may be activated to process the representation for generating visual labels. The subset of the first plurality of experts may include those experts from the first plurality of experts most capable of performing the visual label generation task (e.g., experts from the first plurality of experts capable of generating the best visual labels). The subset of the first plurality of experts may include any number K experts from the first plurality of experts (such as Top-K experts). Visual labels may be generated by computing a weighted sum of the outputs from the activated subsets of the first plurality of experts. The remainder of the experts in the first group can remain deactivated (e.g., idle) during the generation of visual tags.

[0035] The second sub-model 104 can receive visual tags. The second sub-model 104 can project the visual tags into the latent input space of the third sub-model 106, making them available for consumption by the third sub-model 106. To project the visual tags into the latent input space of the third sub-model 106, the visual tags can be routed (e.g., via a router of at least one second MoE block 114) to a subset of the second plurality of experts. Only a subset of the second plurality of experts in at least one second MoE block 114 can be activated to process the visual tags and project them into the latent input space of the third sub-model 106. The subset of the second plurality of experts can include those experts from the second plurality of experts most capable of processing the visual tags. The subset of the second plurality of experts can include any number K experts from the second plurality of experts (such as Top-K experts). A weighted sum of the outputs from the activated subsets of the second plurality of experts can be computed. The weighted sum of the outputs can be projected into the latent input space of the third sub-model 106. The remaining experts in the second plurality of experts can remain deactivated (e.g., idle) during the processing of the visual tags.

[0036] The third sub-model 106 may receive the projected visual markers. The third sub-model 106 may also receive embeddings associated with the text query 130. The embeddings associated with the text query 130 may reside in the same space as the projected visual markers. The text query 130 may include user queries instructing answers to questions about the input image 103 and / or any other natural language task to be performed on the input image 103.

[0037] The third sub-model 106 can generate a text description 140 of the input image 103 based on the projected visual markers and / or embeddings associated with the text query 130. The text description 140 of the input image 103 can respond to the text query 130. To generate the text description, the projected visual markers and / or embeddings (e.g., via a router of at least one third MoE block 116) can be routed to a subset of third-order experts. Only a subset of third-order experts in at least one third MoE block 116 can be activated to process the projected visual markers and / or embeddings used to generate the text description 140. The subset of third-order experts can include those experts from the third-order experts who are best able to process the projected visual markers and / or embeddings (e.g., experts from the third-order experts who are best able to generate the text description 140). The subset of third-order experts can include any number K experts from the third-order experts (such as Top-K experts). A weighted sum of the outputs from the activated subsets of the third-order experts can be computed. The weighted sum of the outputs can be used to generate the text description 140. The third sub-model 106 can output the generated text description 140. During the generation of the text description 140, the remaining experts among the third plurality of experts can remain inactive (e.g., idle).

[0038] Figure 2 An exemplary MoE converter block 204 of a first sub-model 102 is shown. The first sub-model 102 may include multiple converter blocks having MoE blocks (e.g., one in each layer of the first sub-model 102). Each converter block may be similar to... Figure 2 The example shown is based on MoE converter block 204. Each converter block can be configured to perform self-attention and normalization before the representation reaches MoE block 112 to generate a representation of the input image (e.g., input image 103). MoE block 112 may include Top-K router 205. Top-K router 205 can select Top-K MLP expert candidates. Figure 2 In the example, Top-K router 205 can select MLP1 and MLP3 as Top-K MLP expert candidates. Only MLP1 and MLP3 can be activated to process the representation used to generate visual tags. Visual tags can be generated by computing a weighted sum of the outputs from MLP1 and MLP3. The rest of the experts in the first plurality of experts (e.g., MLP2 and MLP4) can remain deactivated (e.g., idle) during the generation of visual tags.

[0039] In one embodiment, the Top-K router 205 can select Top-K MLP expert candidates from a total of S experts, employing a linear layer to compute a normalized weight matrix based on the input W for voting: W = Softmax(Linear(W)) ∈ R N×S Then, a Top-K MLP expert can be selected for each label based on W, and the renormalized weights W are calculated as follows. K ∈R N×K : W K =Softmax(TopK(W))∈R N×K Each selected expert can be an MLP block, and the final output can be a reweighted sum: The output has the same dimensions as the output of a single dense MLP block.

[0040] Figure 3 An example system 300 according to this disclosure is shown. System 300 may include a first sub-model 102, a second sub-model 104, and a third sub-model 106. As mentioned above... Figures 1 to 2As described, the first sub-model 102 may include one or more first MoE blocks 112. For example, the visual encoding portion of the first sub-model 102 may include a converter model, which may include consecutive MLP blocks in a converter encoder. Each individual MLP block may be replaced by a Top-K sparse MoE block. Skipped connections and the output of the MoE blocks may be maintained.

[0041] The second sub-model 104 may include one or more second MoE blocks 114. Each MoE block 114 may include a Top-K router 305. The Top-K router 305 may select Top-K MLP expert candidates. Figure 3 In the example, Top-K router 305 can select MLP2 and MLP4 as Top-K MLP expert candidates. Only MLP2 and MLP4 can be activated to process the visual tags generated by the first sub-model 102 and project the visual tags into the latent input space of the third sub-model 106. The rest of the experts in the second set of experts (e.g., MLP1 and MLP3) can remain deactivated (e.g., idle) during the processing of the visual tags.

[0042] For example, the Top-K router 305 can select Top-K MLP expert candidates from a total of S experts, employing a linear layer to compute a normalized weight matrix based on the input W for voting: W = Softmax(Linear(W)) ∈ R N×S Then, Top-K MLP experts can be selected for each label based on W, and the renormalized weights can be calculated as follows: W K =Softmax(TopK(W))∈R n×K Each selected expert can be an MLP block, and the final output can be a reweighted sum: The output has the same dimensions as the output of a single dense MLP block.

[0043] The third sub-model 106 can generate a text description 140 of the input image 103 based on the projected visual markers and / or embeddings 302 associated with the text query 130. The text description 140 of the input image 103 can be in response to the text query 130. To generate the text description, the projected visual markers and / or embeddings 302 (e.g., via a router of at least one third MoE block 116) can be routed to a subset of third multiple experts. Only a subset of third multiple experts in at least one third MoE block 116 can be activated to process the projected visual markers and / or embeddings 302 for generating the text description 140.

[0044] In this embodiment, high-resolution input is necessary for the third sub-model 106 to understand the details of the input image 103. However, simply expanding the number of visual labels by receiving high-resolution input significantly increases training and inference costs. For example, given a 336×336 image as input, the first sub-model 102 might convert it into 576 labels, while a 672×672 input would equal 2304 labels. To remedy this problem, the input image 103 can be scaled to a multi-resolution pyramid image (e.g., image 333). The multi-resolution pyramid image can be sent to the first sub-model 102. The first sub-model 102 can return a pyramid of multi-resolution visual features. The high-resolution feature map can then be downsampled and stitched before being sent to the second sub-model 104. As a result, the number of visual labels (e.g., 576) is maintained while utilizing the multi-resolution input.

[0045] Figure 4 An example three-stage training process 400 for training machine learning model 101 is shown. To smooth training stability during the training of machine learning model 101, a three-stage training process 400 can be employed. The three-stage training process 400 includes a pre-training phase 402, a pre-fine-tuning phase 404, and a visual instruction tuning phase 406. During the pre-training phase 402, a second sub-model 104 (e.g., an MLP connector) can be pre-trained while the first sub-model 102 (e.g., a visual encoder) and the third sub-model 106 (e.g., a large language model) remain frozen. The first sub-model 102 and the third sub-model 106 can be pre-trained on large-scale data. During the pre-fine-tuning phase 404, the parameters of machine learning model 101 can be fine-tuned using image illustration data to warm up the entire machine learning model 101 before adding the MoE block in the visual instruction tuning phase 406. For example, the parameters of each of the first sub-model 102, the second sub-model 104, and the third sub-model 106 can be fine-tuned during the pre-fine-tuning phase 404.

[0046] During the visual instruction scaling phase 406, the machine learning model 101 is scaled using upgraded MoE blocks and trained on visual instruction scaling data. Scaled-up the machine learning model 101 with upgraded MoE blocks may include adding at least one MoE block to each of the first sub-model 102, the second sub-model 104, and the third sub-model 106. Adding at least one MoE block to each of the first sub-model 102, the second sub-model 104, and the third sub-model 106 may include generating initially well-trained experts for each of the first sub-model 102, the second sub-model 104, and the third sub-model 106 based on pre-fine-tuned parameters.

[0047] For example, the initial expert in each MoE block of the first sub-model can be an MLP of the first sub-model that has been well trained in the second pre-fine-tuning stage. The initial expert in the second sub-model can be an MLP of the second sub-model that has been well trained in the first pre-training stage and the second pre-fine-tuning stage. The initial well-trained expert for each of the first sub-models 102, the second sub-model 104, and the third sub-model 106 can be copied (e.g., duplicated) N times to generate at least one initial expert block 405 for each of the first sub-models 102, the second sub-model 104, and the third sub-model 106.

[0048] Before training the machine learning model 101 using visual instruction-adjusted data, the initial expert block 405 in each of the first sub-models 102, second sub-model 104, and third sub-model 106 may include N exact copies of the corresponding initial expert. Then, each of the first sub-models 102, second sub-model 104, and third sub-model 106 can be trained iteratively using the visual instruction-adjusted data. For example, at least one first MoE block 112 can be obtained by iteratively training at least one initial expert block for the first sub-model 102. At least one second MoE block 114 can be obtained by iteratively training at least one initial expert block for the second sub-model 104. At least one third MoE block can be obtained by iteratively training at least one initial expert block for the third sub-model. During each iteration, different experts in the initial expert block 405 can be activated to process different data. In this way, at the end of the visual instruction-adjusted phase 406, the experts in each expert block will have different parameters.

[0049] In an embodiment, to maintain load balancing among experts in each MoE block during the visual instruction adjustment phase, an auxiliary loss can be employed based on language modeling cross-entropy loss. The auxiliary loss may include load balancing loss and router z-loss. As a result, the total loss L can be: L = L ce +α b L b +α z L z . Language modeling loss L ce It is the cross-entropy of the next label prediction, while α b and α z These are the load balancing losses L b And router z loss L z The coefficients. For simplicity, the auxiliary loss can be referred to as "bz loss" in this paper. The auxiliary loss can be applied individually to the first sub-model 102, the second sub-model 104, and the third sub-model 106.

[0050] Figure 5 An example process 500 for generating image descriptions using a machine learning model is shown. Although in Figure 5 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0051] At 502, a machine learning model (e.g., machine learning model 101) can be configured. The machine learning model can be configured by incorporating expert hybrid (MoE) blocks into multiple sub-models of the machine learning model (e.g., first sub-model 102, second sub-model 104, and / or third sub-model 106). The first sub-model may include a contrastive language image pre-trained (CLIP) visual encoder. The first sub-model of the machine learning model may include at least one first MoE block (e.g., multiple first MoE blocks 112). At least one first MoE block may include a first plurality of experts. The second sub-model may include an MLP connector. The second sub-model of the machine learning model may include at least one second MoE block (e.g., multiple second MoE blocks 114). At least one second MoE block may include a second plurality of experts. The third sub-model may include a large language model.

[0052] At point 504, visual tags can be generated by the first sub-model. Visual tags can be generated based on the input image (e.g., input image 103). Only a subset of the first plurality of experts in at least one first MoE block can be activated to generate visual tags. The subset of the first plurality of experts can include those experts from the first plurality of experts most capable of performing the visual tag generation task (e.g., the experts from the first plurality of experts capable of generating the best visual tags). The subset of the first plurality of experts can include any number K experts from the first plurality of experts (such as Top-K experts). The remaining experts in the first plurality of experts can remain deactivated (e.g., idle) during visual tag generation.

[0053] At point 506, the visual marker can be projected through the second sub-model. Only a subset of the second plurality of experts in at least one second MoE block can be activated to project the visual marker into the input space of the third sub-model. The subset of the second plurality of experts can include those experts from the second plurality of experts most capable of projecting the visual marker into the input space of the third sub-model. The subset of the second plurality of experts can include any number K experts from the second plurality of experts (such as Top-K experts). The remaining experts in the second plurality of experts can remain deactivated (e.g., idle) during the projection of the visual marker.

[0054] At point 508, a text description of the input image can be output via a third sub-model (e.g., text description 140). The third sub-model can generate the text description of the input image based on the projected visual markers. The third sub-model can be configured to generate and output a description of the input image based on the projected markers.

[0055] Figure 6 An example process 600 for generating visual labels using a first sub-model of a machine learning model is shown. Although in Figure 6 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0056] A machine learning model (e.g., machine learning model 101) may receive an input image (e.g., input image 103) as input. A first sub-model of the machine learning model (e.g., first sub-model 102) may receive the input image. At 602, a representation of the input image may be generated. The representation of the input image may be generated based on self-attention and normalization performed through the first sub-model. The first sub-model may include at least one first MoE block (e.g., first MoE block 112). At least one first MoE block may include a first plurality of experts.

[0057] At 604, the representation of the input image can be routed to the activated subset of the first plurality of experts. The representation of the input image can be routed to the activated subset of the first plurality of experts via a router (e.g., Top-K router 205) of at least one first MoE block. The representation can only be processed by activating a subset of the first plurality of experts in at least one first MoE block. The subset of the first plurality of experts can include those experts from the first plurality of experts most capable of performing the visual tag generation task (e.g., experts from the first plurality of experts capable of generating the best visual tags). The subset of the first plurality of experts can include any number K experts from the first plurality of experts (such as Top-K experts). At 606, the representation can be processed via the activated subset of the first plurality of experts. At 608, visual tags can be generated. Visual tags can be generated by computing a weighted sum of the outputs from the activated subset of the first plurality of experts in the first sub-model. The remaining experts in the first plurality of experts can remain deactivated (e.g., idle) during the generation of visual tags via the first sub-model.

[0058] Figure 7 An example process 700 for generating visual labels using a first sub-model of a machine learning model is shown. Although in Figure 7 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0059] In this embodiment, high-resolution input is necessary for a third sub-model (e.g., third sub-model 106) of a machine learning model (e.g., machine learning model 101) to understand the details of the input image (e.g., input image 103). At 702, the input image can be divided into blocks (e.g., multi-resolution pyramid images). The blocks can be sent to a first sub-model (e.g., first sub-model 102). At 704, the first sub-model can generate a pyramid of multi-resolution visual features based on the blocks. For example, only a subset of the activations of the first plurality of experts of the first sub-model can generate a pyramid of multi-resolution visual features based on the blocks. At 706, the high-resolution feature map can be downsampled and stitched channel-by-channel before being sent to a second sub-model (e.g., second sub-model 104). As a result, the number of visual labels can be maintained while utilizing the multi-resolution input.

[0060] Figure 8 An example process 800 for processing visual markers via a second sub-model is shown. Although in Figure 8 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0061] A machine learning model (e.g., machine learning model 101) may receive an input image (e.g., input image 103) as input. A first sub-model (e.g., first sub-model 102) may receive the input image. The first sub-model may generate visual labels based on the input image. A second sub-model of the machine learning model (e.g., second sub-model 104) may receive the visual labels. The second sub-model may include at least one second MoE block (e.g., second MoE block 114). At least one second MoE block of the second sub-model may include a second plurality of experts.

[0062] At 802, visual tags can be routed to an activated subset of second-multiple experts via a router (e.g., Top-K router 305) of at least one second MoE block. Only a subset of the second-multiple experts in at least one second MoE block can be activated to process the visual tags. The subset of second-multiple experts can include those experts from the second-multiple experts most capable of processing visual tags. The subset of second-multiple experts can include any number K experts from the second-multiple experts (such as Top-K experts). At 804, the subset of second-multiple experts can process the visual tags. The remaining experts in the second-multiple experts can remain deactivated (e.g., idle) during the processing of the visual tags. At 806, a weighted sum of the outputs from the activated subsets of the second-multiple experts can be computed as tags projected into the input space of a third sub-model. The projected tags can be consumed by the third sub-model of the machine learning model.

[0063] Figure 9 An example process 900 for generating image descriptions using a machine learning model is shown. Although in Figure 9 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0064] At 902, a machine learning model (e.g., machine learning model 101) can be configured. The machine learning model can be configured by incorporating expert hybrid (MoE) blocks into multiple sub-models of the machine learning model (e.g., first sub-model 102, second sub-model 104, and / or third sub-model 106). The first sub-model may include a contrastive language image pre-trained (CLIP) visual encoder. At least one first MoE block (e.g., first MoE block 112) may be incorporated into the first sub-model. At least one first MoE block may include a first plurality of experts. The second sub-model may include an MLP connector. At least one second MoE block (e.g., second MoE block 114) may be incorporated into the second sub-model. At least one second MoE block may include a second plurality of experts. The third sub-model may include a large language model. At least one third MoE block (e.g., (multiple) third MoE blocks 116) may be incorporated into the third sub-model. At least one third MoE block may include a third plurality of experts.

[0065] At 904, visual tags can be generated by the first sub-model. Visual tags can be generated based on the input image (e.g., input image 103). Only a subset of the first plurality of experts in at least one first MoE block can be activated to generate visual tags. The subset of the first plurality of experts can include those experts from the first plurality of experts most capable of performing the visual tag generation task (e.g., the experts from the first plurality of experts capable of generating the best visual tags). The subset of the first plurality of experts can include any number K experts from the first plurality of experts (such as Top-K experts). The remaining experts in the first plurality of experts can remain deactivated (e.g., idle) during visual tag generation.

[0066] At position 906, visual markers can be projected into the input space of the third sub-model via the second sub-model. Only a subset of the second plurality of experts in at least one second MoE block can be activated to project the visual markers into the input space of the third sub-model. The subset of the second plurality of experts can include those experts from the second plurality of experts most capable of projecting the visual markers into the input space of the third sub-model. The subset of the second plurality of experts can include any number K experts from the second plurality of experts (such as Top-K experts). The remaining experts among the second plurality of experts can remain deactivated (e.g., idle) during the projection of the visual markers by the second sub-model.

[0067] At point 908, a text description of the input image can be generated via a third sub-model (e.g., text description 140). Only a subset of the third plurality of experts in at least one third MoE block can be activated to generate the text description based on the projected labels. The subset of the third plurality of experts can include those experts from the third plurality of experts most capable of generating the text description. The subset of the third plurality of experts can include any number K experts from the third plurality of experts (such as Top-K experts). During the generation of the text description via the third sub-model, the remaining experts among the third plurality of experts can remain deactivated (e.g., idle).

[0068] Figure 10 An example process 1000 for generating image descriptions using a machine learning model is shown. Although in... Figure 10 The operations are described as a series of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder or modify the operations described.

[0069] At 1002, a machine learning model (e.g., machine learning model 101) can be configured. The machine learning model can be configured by incorporating expert hybrid (MoE) blocks into multiple sub-models of the machine learning model (e.g., first sub-model 102, second sub-model 104, and / or third sub-model 106). The first sub-model may include a contrastive language image pre-trained (CLIP) visual encoder. At least one first MoE block (e.g., multiple first MoE blocks 112) can be incorporated into at least one first MoE block. At least one first MoE block may include a first plurality of experts. The second sub-model may include an MLP connector. At least one second MoE block (e.g., multiple second MoE blocks 114) can be incorporated into the second sub-model. At least one second MoE block may include a second plurality of experts. The third sub-model may include a large language model. At least one third MoE block (e.g., multiple third MoE blocks 116) can be incorporated into the third sub-model. At least one third MoE block may include a third plurality of experts.

[0070] At point 1004, visual tags can be generated via a first sub-model. Visual tags can be generated based on an input image (e.g., input image 103). A subset of a first plurality of experts in at least one first MoE block can be activated to generate visual tags. The subset of the first plurality of experts can include those experts from the first plurality of experts most capable of performing the visual tag generation task (e.g., the experts from the first plurality of experts capable of generating the best visual tags). The subset of the first plurality of experts can include any number K experts from the first plurality of experts (such as Top-K experts). The remaining experts in the first plurality of experts can remain deactivated (e.g., idle) during visual tag generation.

[0071] At point 1006, visual markers can be projected into the input space of the third sub-model via the second sub-model. Only a subset of the second plurality of experts in at least one second MoE block can be activated to project the visual markers into the input space of the third sub-model. The subset of the second plurality of experts can include those experts from the second plurality of experts most capable of projecting the visual markers into the input space of the third sub-model. The subset of the second plurality of experts can include any number K experts from the second plurality of experts (such as Top-K experts). The remaining experts in the second plurality of experts can remain deactivated (e.g., idle) during the projection of the visual markers.

[0072] The third sub-model can receive the projected visual labels. At 1008, a text query can be received by the machine learning model (e.g., text query 130). The text query can include user queries instructing answers to questions about the input image and / or performing any other natural language task about the input image. The text query can be converted into an embedding (e.g., embedding 302). The embedding can reside in the same space as the projected visual labels. The text embedding can be fed into the third sub-model. At 1010, a text description of the input image can be generated by the third sub-model (e.g., text description 140). The third sub-model can generate a text description of the input image based on the projected visual labels and the text embedding.

[0073] Figure 11 An example process 1100 for training a machine learning model is shown. Although in Figure 11 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0074] To smooth training stability during the training of a machine learning model (e.g., machine learning model 101), a three-stage training process can be employed. The three-stage training process includes a first training stage, a second training stage, and a third training stage. For example, the three-stage training process may include a pre-training stage as the first training stage (e.g., pre-training stage 402). At 1102, a second sub-model (e.g., second sub-model 104) of the machine learning model can be pre-trained while the first sub-model (e.g., first sub-model 102) and the third sub-model (e.g., third sub-model 106) of the machine learning model are frozen.

[0075] The three-stage training process may include a pre-fine-tuning stage as a second training stage (e.g., pre-fine-tuning stage 404). At 1104, the parameters of the machine learning model can be pre-fine-tuned to warm up the entire machine learning model before adding MoE blocks. The parameters of each of the first, second, and third sub-models can be pre-fine-tuned. The three-stage training process may include a visual instruction tuning stage as a third training stage (e.g., visual instruction tuning stage 406). At 1106, at least one MoE block can be added to each of the first, second, and third sub-models during the visual instruction tuning stage. The machine learning model can be trained on the visual instruction tuning data. At 1108, an auxiliary loss can be employed to maintain load balancing among experts in each MoE block during the visual instruction tuning stage. During the training of the machine learning model, a load balancing loss and a router-z loss can be applied to each of the first, second, and third sub-models to maintain load balancing among experts in each of the first, second, and third sub-models.

[0076] Figure 12 An example process 1200 for training a machine learning model is shown. Although in Figure 12 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the described operations.

[0077] A machine learning model (e.g., machine learning model 101) can be scaled up with upgraded MoE blocks (e.g., co-upgraded MoE405) and trained on visually instructed data in a third training phase. Scale-up the machine learning model with upgraded MoE blocks may include adding at least one MoE block to each of the first sub-models (e.g., first sub-model 102), second sub-models (e.g., second sub-model 104), and third sub-models (e.g., third sub-model 106) of the machine learning model. Adding at least one MoE block to each of the first, second, and third sub-models may include generating an initially well-trained expert for each of the first, second, and third sub-models based on pre-tuned parameters. At 1202, initial experts for each of the first, second, and third sub-models of the machine learning model may be generated based on the pre-tuned parameters of the machine learning model. For example, the initial expert in each MoE block of the first sub-model may be an MLP of the first sub-model that has been well-trained through a second pre-tuning phase. The initial expert for the second sub-model can be an MLP that has been well trained through the first pre-training phase and the second pre-fine-tuning phase. The initial expert for each MoE block of the third sub-model can be an MLP that has been well trained through the second pre-fine-tuning phase.

[0078] At point 1204, at least one initial expert block can be generated for each of the first, second, and third sub-models. At least one initial expert block for each of the first, second, and third sub-models can be generated or obtained by copying the initial expert for that sub-model. For example, the initially well-trained expert for each of the first, second, and third sub-models can be copied (e.g., duplicated) N times to generate at least one initial expert block for each of the first, second, and third sub-models.

[0079] Before training the machine learning model using visual instructions to adjust the data, the initial expert block in each of the first, second, and third sub-models can include N exact copies of the corresponding initial expert. Then, the initial expert blocks in each of the first, second, and third sub-models can be trained iteratively during the third training phase. At 1206a, at least one first MoE block (e.g., multiple MoE blocks 112) can be generated by iteratively training at least one initial expert block for the first sub-model. At 1206b, at least one second MoE block (e.g., multiple MoE blocks 114) can be obtained by iteratively training at least one initial expert block for the second sub-model. At 1206c, at least one third MoE block (e.g., multiple MoE blocks 116) can be generated by iteratively training at least one initial expert block for the third sub-model. During each iteration, different experts can be activated to process different data. In this way, at the end of the third training phase, the experts in each expert block will have different parameters.

[0080] Experiments were conducted to evaluate the performance of machine learning model 101. The performance of machine learning model 101 was evaluated on multiple competitive benchmarks. Furthermore, ablation studies were performed on each of the first sub-models 102, the second sub-model 104, and the third sub-model 106 with upgraded MoE blocks. Figure 13 Table 1300 shows a comparison of the performance of machine learning model 101 with other prior art instruction-following-based multimodal large language models.

[0081] Ablation studies were conducted on the second sub-model 104 with upgraded MoE blocks. Results were... Figure 14A The results are shown in Table 1400. After establishing the baseline model, the MLP connector was replaced with an MLP having an upgraded MoE block. The study began with a Top2-in4 router, which showed a slight improvement over the baseline considering only two linear layers per expert. Adding bzloss to enable balancing of experts loaded in the MLP with the upgraded MoE block demonstrated a significant improvement over the baseline. Ablation studies were performed on the first sub-model 102 with the upgraded MoE block. Results are in Figure 14B As shown in Table 1401, firstly, the CLIP based on the MLP with upgraded MoE blocks was unfrozen. A significant improvement over the VQA-based benchmark was observed. Then, during the visual instruction tuning phase, the MLP layers in each feedforward block with top2-in-4 routers and bzloss were replaced with sparsely gated MoE layers. As shown in Table 1401, this further improved performance. The learning rate was reduced to 2e-6, which is consistent with the learning rate of large language models, as 2e-5 could lead to training instability.

[0082] To further evaluate the effectiveness of the upgraded sparse-gated MoE block in the first sub-model 102 and the second sub-model 104, machine learning model 101 was evaluated on limited training data. Results are... Figure 15 As shown in Table 1500, machine learning model 101 outperforms the other 7B models.

[0083] After replacing all MLP blocks in the vision part with sparsely gated MoE blocks, the performance of using the MoE architecture in the third sub-model 106 is evaluated. Each MLP block in the third sub-model 106 is upgraded using sparsely gated MoE blocks. The weights of each expert are initialized from the pre-trained MLP blocks. Figure 16A The results shown in Table 1600 demonstrate that the upgraded model consistently outperforms the other models.

[0084] As mentioned above, multi-resolution input is necessary for multimodal large language models to understand the content of images. Multi-resolution image features are applied as input to the first sub-model 102. The multi-resolution image features are concatenated channel-by-channel to maintain the same total number of visual markers as the low-resolution input. Figure 16B Table 1601 shows that the combination of 1008 (3x) and 336 (1x) is empirically optimal for the performance of machine learning model 101.

[0085] Previous ablation studies were based on visual instruction tuning directly after pre-training on the MLP connector. Due to training instability during the process, a warm-up / pre-fine-tuning phase prior to visual instruction tuning is preferable. As a result, a pre-fine-tuning phase was added to pre-tun the machine learning model on high-quality image illustration data, and all parameters of the machine learning model were not frozen during the pre-fine-tuning phase. Figure 16C Table 1602 shows the results of the ablation study on the pre-fine-tuning phase.

[0086] Figure 17 The diagram illustrates computing devices that can be used in various aspects, such as Figures 1 to 4 The services, networks, modules, and / or devices described in any of them. About Figures 1 to 4 Any or all components can be free. Figure 17 One or more instances of the computing device 1700 are implemented. Figure 17 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet computer, network device, PDA, e-reader, digital cellular phone, or other computing node, and can be used to perform any aspect of the computer described herein, such as to implement the methods described herein.

[0087] The computing device 1700 may include a substrate or “motherboard,” which is a printed circuit board to which multiple components or devices may be connected via a system bus or other electrical communication path. One or more central processing units (CPUs) 1704 may operate in conjunction with a chipset 1706. The CPUs(multiple) 1704 may be standard programmable processors that perform the arithmetic and logic operations necessary for the operation of the computing device 1700.

[0088] Multiple CPU1704s can perform necessary operations by manipulating switching elements that distinguish and change these states, transitioning from one discrete physical state to the next. Switching elements typically include electronic circuitry (e.g., flip-flops) that maintains one of two binary states, and electronic circuitry (e.g., logic gates) that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, etc.

[0089] The multiple CPUs 1704 can be expanded or replaced by other processing units (e.g., multiple GPUs 1705). The multiple GPUs 1705 may include processing units dedicated to, but not limited to, highly parallel computing, such as graphics and other visualization-related processing.

[0090] Chipset 1706 can provide an interface between CPU(s) 1704 and the rest of the components and devices on the substrate. Chipset 1706 can provide an interface to random access memory (RAM) 1708, which serves as the main memory in computing device 1700. Chipset 1706 can also provide an interface to computer-readable storage media, such as read-only memory (ROM) 1720 or non-volatile RAM (NVRAM) (not shown), for storing basic routines that help start computing device 1700 and transfer information between various components and devices. ROM 1720 or NVRAM can also store other software components necessary for the operation of computing device 1700 according to the aspects described herein.

[0091] Computing device 1700 can operate in a networked environment using a logical connection to remote computing nodes and computer systems via a local area network (LAN). Chipset 1706 may include functionality for providing network connectivity via a network interface controller (NIC) 1722 (such as a Gigabit Ethernet adapter). NIC 1722 enables computing device 1700 to connect to other computing nodes via network 1716. It should be understood that multiple NICs 1722 may exist in computing device 1700, connecting the computing device to other types of networks and remote computer systems.

[0092] Computing device 1700 can be connected to mass storage device 1728, which provides non-volatile storage for the computer. Mass storage device 1728 can store system programs, application programs, other program modules, and data, as described in more detail herein. Mass storage device 1728 can be connected to computing device 1700 via storage controller 1724, which is connected to chipset 1706. Mass storage device 1728 can consist of one or more physical storage units. Mass storage device 1728 may include management components. Storage controller 1724 can interface with physical storage units via a Serial Amount SCSI (SAS) interface, a Serial Advanced Technology Attachment (SATA) interface, a Fibre Channel (FC) interface, or other types of interfaces used for physical connection and data transfer between the computer and physical storage units.

[0093] The computing device 1700 can store data on the mass storage device 1728 by changing the physical state of the physical storage units to reflect the stored information. The specific changes in physical state may depend on various factors and on different embodiments of this specification. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage units and whether the mass storage device 1728 is characterized as primary or secondary storage.

[0094] For example, computing device 1700 can store information in mass storage device 1728 by issuing instructions via storage controller 1724 to change the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification, and the foregoing examples are for facilitating this description only. Computing device 1700 can also read information from mass storage device 1728 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0095] In addition to the aforementioned high-capacity storage device 1728, the computing device 1700 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium can be any available medium that provides storage for non-transitory data and can be accessed by the computing device 1700.

[0096] By way of example and not limitation, computer-readable storage media can include volatile and non-volatile, transient and non-transitory computer-readable storage media implemented in any method or technology, as well as removable and non-removable media. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, optical disc ROM (“CD-ROM”), digital versatile disc (“DVD”), high-definition DVD (“HD-DVD”), Blu-ray or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0097] Such as Figure 17 The mass storage device 1728 depicted herein can store an operating system used to control the operation of the computing device 1700. The operating system may include a version of the Linux operating system. The operating system may include a version of the Windows Server operating system from Microsoft Corporation. Depending on another aspect, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as iOS and Android, may also be used. It should be understood that other operating systems may also be used. The mass storage device 1728 can store other systems, applications, and data utilized by the computing device 1700.

[0098] Mass storage device 1728 or other computer-readable storage media may also be encoded with computer-executable instructions that, when loaded into computing device 1700, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. These computer-executable instructions transform computing device 1700 by specifying how CPU 1704 transitions between states, as described above. Computing device 1700 can access computer-readable storage media storing computer-executable instructions that, when executed by computing device 1700, can perform the methods described herein.

[0099] Such as Figure 17 The computing device 1700 depicted may also include an input / output controller 1732 for receiving and processing input from multiple input devices, such as a keyboard, mouse, touchpad, touchscreen, electronic stylus, or other types of input devices. Similarly, the input / output controller 1732 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, plotter, or other types of output device. It should be understood that the computing device 1700 may not include... Figure 17All components shown may include Figure 17 Other components not explicitly shown in the document, or those that can be utilized with Figure 17 The architecture shown is completely different from the one shown.

[0100] As described in this article, a computing device can be a physical computing device, such as... Figure 17 The computing device 1700. A computing node may also include virtual machine host processes and one or more virtual machine instances. Computer-executable instructions may be executed indirectly by the physical hardware of the computing device through the interpretation and / or execution of instructions stored and executed in the context of a virtual machine.

[0101] It should be understood that the methods and systems described herein are not limited to any particular method, component, or implementation. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0102] As used in the specification and appended claims, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” include plural referents. A range may be expressed herein as “about” a particular value, and / or “about” another particular value. When such a range is expressed, another embodiment includes from a particular value and / or another particular value. Similarly, when a value is expressed as an approximation using the antecedent “about,” it will be understood that the particular value forms another embodiment. It should also be understood that each endpoint of a range is significant relative to and independent of the other endpoint.

[0103] "Optional" or "optionally" means that the event or situation described below may or may not occur, and the description includes instances where the event or situation occurs as well as instances where it does not occur.

[0104] Throughout this specification and claims, the word "comprising" and variations thereof, such as "comprising" and "including," mean "including but not limited to" and are not intended to exclude, for example, other components, integers, or steps. "Exemplary" means "example" and is not intended to convey indications of preferred or ideal embodiments. "This" is used not in a limiting sense but for purposes of explanation.

[0105] Components that can be used to perform the described methods and systems are described. When combinations, subsets, interactions, groups, etc., of these components are described, it should be understood that while specific references to each of the various individual and collective combinations and substitutions of these components may not be explicitly described, they are particularly contemplated and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to operations in the described methods. Therefore, if various additional operations exist that can be performed, it should be understood that each of these additional operations can be performed using any particular embodiment or combination of embodiments of the described methods.

[0106] The method and system can be more readily understood by referring to the preferred embodiments and the examples included therein, as well as the following detailed description of the accompanying drawings and their descriptions.

[0107] As those skilled in the art will understand, the methods and systems may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. More specifically, the methods and systems may take the form of computer software implemented in the web. Any suitable computer-readable storage medium may be used, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0108] Embodiments of the methods and systems are described below with reference to block diagrams and flowcharts illustrating the methods, systems, apparatus, and computer program products. It will be understood that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented by computer program instructions. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart blocks.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can instruct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of writing comprising computer-readable instructions for implementing the functions specified in the flowchart block. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in the flowchart block.

[0110] The various features and processes described above can be used independently of each other or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular sequence, and the associated blocks or states may be executed in other suitable sequences. For example, the described blocks or states may be executed in a different order than specifically described, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the described example embodiments. The example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the described example embodiments.

[0111] It should also be understood that various items are shown as being stored in memory or, when in use, in a storage device, and these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software modules and / or system may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Furthermore, in some embodiments, some or all of the system and / or modules may be implemented or provided in other ways, such as at least in part in firmware and / or hardware, including, but not limited to, one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on computer-readable media, such as hard disks, memory, networks, or portable media articles that will be read by appropriate devices or via appropriate connections. Systems, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media, including wireless and wired / cable-based media, and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products can also take other forms. Therefore, the invention can be practiced with other computer system configurations.

[0112] While methods and systems have been described in conjunction with preferred embodiments and specific examples, they are not intended to limit the scope to the particular embodiments illustrated, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0113] Unless otherwise expressly stated, any method described herein should never be construed as requiring its operations to be performed in a particular order. Therefore, no inference of order is intended in any respect where a method claim does not actually describe the order of its operations or where the claims or description do not specifically state that the operations are limited to a particular order. This preserves any possible non-expressive basis for interpretation, including: logical questions concerning the arrangement of steps or operational flows; general meanings derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0114] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of this disclosure. Other embodiments will be apparent to those skilled in the art upon consideration of the description and practice described herein. This description and the example drawings are to be considered exemplary only, and the true scope and spirit are indicated by the appended claims.

Claims

1. A method for generating image descriptions using a machine learning model, comprising: The machine learning model is configured by incorporating expert hybrid MoE blocks into multiple sub-models of the machine learning model, wherein the first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein the second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts. Visual labels are generated through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual labels based on the input image; The visual labels are projected through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual labels into the input space of the third sub-model of the machine learning model; as well as The third sub-model of the machine learning model outputs a textual description of the input image, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

2. The method according to claim 1, further comprising: A representation of the input image is generated based on self-attention and normalization performed through the first sub-model; as well as The representation of the input image is routed to the activated subset of the first plurality of experts via the router of the at least one first MoE block.

3. The method according to claim 1, further comprising: The visual tags are generated by calculating a weighted sum of the outputs from the activated subset of the first plurality of experts.

4. The method according to claim 1, further comprising: The input image is divided into blocks; A high-resolution visual feature pyramid is generated based on the blocks using the activated subsets of the first plurality of experts; as well as The high-resolution visual features are downsampled, and the downsampled high-resolution visual features are then stitched together channel by channel.

5. The method according to claim 1, further comprising: The visual markers are routed to the activated subset of the second plurality of experts via the router of the at least one second MoE block; The visual markers are processed through the activated subset of the second plurality of experts; as well as Calculate the weighted sum of the outputs from the activated subset of the second plurality of experts.

6. The method of claim 1, wherein the third sub-model comprises at least one third MoE block, wherein the at least one third MoE block comprises a third plurality of experts, and the method further comprises: The text description of the input image is generated only from a subset of the third set of experts.

7. The method according to claim 1, further comprising: Receive input text for querying; The input text query is converted into an embedding in the input space of the third sub-model; as well as The text description of the input image is generated by the third sub-model based on the projected visual markers and the embedding.

8. The method of claim 1, wherein the machine learning model is trained using a three-stage training process, and the three-stage training process includes: While freezing the first sub-model and the third sub-model, the second sub-model is pre-trained. Fine-tuning the parameters of the machine learning model; as well as At least one MoE block is added to each of the first, second, and third sub-models, and the machine learning model is trained on visual instruction-adjusted data.

9. The method of claim 8, wherein adding at least one MoE block to each of the first sub-model, the second sub-model, and the third sub-model comprises: Based on the pre-tuned parameters, an initial expert is generated for each of the first, second, and third sub-models; as well as By replicating the initial expert, at least one initial expert block is generated for each of the first, second, and third sub-models.

10. The method of claim 9, further comprising: The at least one first MoE block is generated by iteratively training the at least one initial expert block for the first sub-model; The at least one second MoE block is generated by iteratively training the at least one initial expert block for the second sub-model; as well as The at least one third MoE block is generated by iteratively training the at least one initial expert block for the third sub-model.

11. The method of claim 8, further comprising: During the training of the machine learning model, load balancing loss and router-z loss are applied to each of the first, second, and third sub-models to maintain load balancing among the experts in each of the first, second, and third sub-models.

12. A system for generating image descriptions using a machine learning model, comprising: At least one processor; as well as At least one memory, communicatively coupled to the at least one processor, and including computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: The machine learning model is configured by incorporating expert hybrid MoE blocks into multiple sub-models of the machine learning model, wherein the first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein the second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts. Visual labels are generated through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual labels based on the input image; The visual labels are projected through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual labels into the input space of the third sub-model of the machine learning model; and The third sub-model of the machine learning model outputs a textual description of the input image, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

13. A non-transitory computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, cause the processor to perform operations including: The machine learning model is configured by incorporating expert hybrid MoE blocks into multiple sub-models of the machine learning model, wherein the first sub-model of the machine learning model includes at least one first MoE block, wherein the at least one first MoE block includes a first plurality of experts, wherein the second sub-model of the machine learning model includes at least one second MoE block, and wherein the at least one second MoE block includes a second plurality of experts. Visual labels are generated through the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual labels based on the input image; The visual labels are projected through the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual labels into the input space of the third sub-model of the machine learning model; as well as The third sub-model of the machine learning model outputs a textual description of the input image, wherein the third sub-model is configured to generate a description of the input image based on markers projected into the input space of the third sub-model.

Images