Task execution method, device and equipment, computer readable medium and program product

Abstract

The embodiment of the invention discloses a task execution method, device and equipment, a computer readable medium and a program product. According to one specific embodiment, the method comprises the steps that task input information corresponding to a target task is obtained, and the task input information comprises information in at least one modal form; performing modal feature alignment on information in various modal forms in the task input information to obtain feature alignment information; according to the task type corresponding to the target task, determining an autoregression generation normal form corresponding to the target task, and taking the autoregression generation normal form as a target autoregression generation normal form; and generating a task output result corresponding to the target task by utilizing a generation model corresponding to the target autoregression generation normal form according to the feature alignment information. The embodiment is related to artificial intelligence, and different suitable autoregressive generation normal forms can be selected based on different task types so as to accurately and efficiently generate the task output result of the target task.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to task execution methods, apparatus, devices, computer-readable media, and program products. Background Technology

[0002] Currently, with the continuous development of multimodal technologies, feature fusion based on multimodal information can acquire multifaceted feature information, enabling the generation of more accurate output results. For task execution in the modal domain, there are numerous task types, and selecting a suitable generation paradigm often requires relevant experts to choose based on historical experience. This not only results in inaccurate generation paradigms but also inefficient task execution and long execution times. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure provide task execution methods, apparatuses, devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a task execution method, including: acquiring task input information corresponding to a target task, wherein the task input information includes at least one modal form of information; performing modal feature alignment on the information of various modal forms in the task input information to obtain feature alignment information; determining an autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as a target autoregressive generation paradigm; and generating a task output result corresponding to the target task using a generation model corresponding to the target autoregressive generation paradigm based on the feature alignment information.

[0006] Optionally, the task input information includes: target instructions in text modality and target images in image modality; and the above-mentioned modal feature alignment of the information in various modal forms in the task input information to obtain feature alignment information includes: performing word segmentation and encoding processing on the target instructions to obtain instruction feature information; performing image encoding processing on the target images to obtain image feature information; and using an attention-based modal alignment model to perform modal feature alignment on the instruction feature information and the image feature information to obtain feature alignment information.

[0007] Optionally, determining the autoregressive generation paradigm corresponding to the target task as the target autoregressive generation paradigm based on the task type corresponding to the target task includes: in response to determining that the task type corresponding to the target task is a semantic understanding type, determining the autoregressive generation paradigm corresponding to the target task as a first generation paradigm of single-granularity stepwise prediction.

[0008] Optionally, determining the autoregressive generation paradigm corresponding to the target task as the target autoregressive generation paradigm based on the task type corresponding to the target task includes: in response to determining that the task type corresponding to the target task is an image generation type, determining the autoregressive generation paradigm corresponding to the target task as a second generation paradigm of phased stepwise prediction.

[0009] Optionally, the above-mentioned method of generating the task output result corresponding to the target task using the target autoregressive generation paradigm and the corresponding generation model based on the feature alignment information includes: inputting the feature alignment information into the generation model corresponding to the second generation paradigm to obtain multiple initial image prediction feature information; performing residual accumulation processing on the multiple initial image prediction feature information to obtain image prediction feature information; and inputting the image prediction feature information into a pre-trained image feature decoding model to obtain a predicted image as the task output result.

[0010] Optionally, the above-mentioned method of generating the task output result corresponding to the target task based on the above-mentioned feature alignment information and using the generation model corresponding to the above-mentioned target autoregressive generation paradigm includes: inputting the above-mentioned feature alignment information into the generation model corresponding to the above-mentioned first generation paradigm to obtain semantic understanding feature prediction information; inputting the above-mentioned semantic understanding feature prediction information into a pre-trained semantic decoding model to obtain semantic understanding information, which is used as the above-mentioned task output result.

[0011] Secondly, some embodiments of this disclosure provide a task execution apparatus, including: an acquisition unit configured to acquire task input information corresponding to a target task, wherein the task input information includes at least one modal form of information; a feature alignment unit configured to perform modal feature alignment on the information of various modal forms in the task input information to obtain feature alignment information; a determination unit configured to determine an autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as a target autoregressive generation paradigm; and a generation unit configured to generate a task output result corresponding to the target task based on the feature alignment information and using a generation model corresponding to the target autoregressive generation paradigm.

[0012] Optionally, the above task input information includes: target instructions in text modality and target images in image modality; and the feature alignment unit can be further configured to: perform word segmentation and encoding processing on the target instructions to obtain instruction feature information; perform image encoding processing on the target images to obtain image feature information; and, based on the instruction feature information and the image feature information, use an attention-based modality alignment model to perform modality feature alignment on the instruction feature information and the image feature information to obtain feature alignment information.

[0013] Optionally, the determining unit can be configured to: in response to determining that the task type corresponding to the above target task is a semantic understanding type, determine that the autoregressive generation paradigm corresponding to the above target task is the first generation paradigm of single-granularity stepwise prediction.

[0014] Optionally, the determining unit can be configured to: in response to determining that the task type corresponding to the above target task is an image generation type, determine that the autoregressive generation paradigm corresponding to the above target task is a second generation paradigm of phased stepwise prediction.

[0015] Optionally, the generation unit can be configured to: input the above feature alignment information into the generation model corresponding to the second generation paradigm to obtain multiple initial image prediction feature information; perform residual accumulation processing on the above multiple initial image prediction feature information to obtain image prediction feature information; input the above image prediction feature information into a pre-trained image feature decoding model to obtain a predicted image, which is used as the task output result.

[0016] Optionally, the generation unit can be configured to: input the above feature alignment information into the generation model corresponding to the first generation paradigm to obtain semantic understanding feature prediction information; input the above semantic understanding feature prediction information into a pre-trained semantic decoding model to obtain semantic understanding information, which is used as the output result of the above task.

[0017] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0018] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0019] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0020] The above embodiments of this disclosure have the following beneficial effects: Through the task execution methods of some embodiments of this disclosure, different suitable autoregressive generation paradigms can be selected based on different task types to accurately and efficiently generate the task output results of the target task. Specifically, the reason for the low execution efficiency of related target tasks is that: there are many task types, and selecting a suitable generation paradigm often requires relevant experts to select based on historical experience. This not only results in inaccurate generation paradigms but also in low efficiency and long execution time. Based on this, the task execution methods of some embodiments of this disclosure first obtain the task input information corresponding to the target task, wherein the task input information includes at least one modal form of information. Here, obtaining the task input information allows for the execution of the target task based on the task input information. Here, the inclusion of at least one modal form of information in the task input information allows the subsequent generation model to obtain more task content about the target task, achieving more accurate generation of task output results. Then, modal feature alignment is performed on the various modal forms of information in the above task input information to map the semantic features of different modalities to a unified space, realizing the establishment of cross-modal correspondence. Simultaneously, redundant information can be suppressed, improving the accuracy of subsequent model outputs. Furthermore, based on the task type corresponding to the aforementioned target task, an autoregressive generation paradigm corresponding to the target task is determined as the target autoregressive generation paradigm. Here, selecting a suitable autoregressive generation paradigm based on different task types can significantly improve the generation efficiency and accuracy of task output results, thereby improving the execution efficiency of the target task. Finally, based on the aforementioned feature alignment information, using the generation model corresponding to the aforementioned target autoregressive generation paradigm, the task output results corresponding to the aforementioned target task can be accurately generated. In summary, modal feature alignment enables the mapping of feature semantic content from various modalities to the same space and the suppression of redundant information. By setting an autoregressive generation paradigm corresponding to the task type, the generation efficiency and accuracy of task output results can be significantly improved, thereby enhancing the execution efficiency of the target task. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0022] Figure 1 This is a schematic diagram of an application scenario of a task execution method according to some embodiments of the present disclosure; Figure 2These are flowcharts of some embodiments of the task execution method according to this disclosure; Figure 3 These are flowcharts of other embodiments of the task execution method according to this disclosure; Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the task execution apparatus according to this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] Before performing any of the operations involving the collection, storage, or use of user personal information (such as task input information) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0029] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1This is a schematic diagram of an application scenario of a task execution method according to some embodiments of the present disclosure.

[0031] exist Figure 1 In this application scenario, firstly, the electronic device 101 can acquire the task input information 103 corresponding to the target task 102. The task input information 103 includes information in at least one modal form. In this application scenario, the target task 102 can be a "face recognition task." The task input information 103 can include: instruction information 1031 in text modality and an image 1032 in image modality. The instruction information 1031 can be "determine identity information based on face information in the image." Then, the electronic device 101 can perform modal feature alignment on the various modal forms of information in the task input information 103 to obtain feature alignment information 104. Next, the electronic device 101 can determine the autoregressive generation paradigm corresponding to the target task 102 based on the task type 105, as the target autoregressive generation paradigm 106. In this application scenario, the task type 105 can be an "information recognition type." The target autoregressive generation paradigm 106 can be the "nextscale paradigm". Finally, the electronic device 101 can align the information 104 according to the above features and use the generation model 107 corresponding to the above target autoregressive generation paradigm to generate the task output result 108 corresponding to the above target task 102. In this application scenario, the task output result 108 can be "Zhang**".

[0032] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0033] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0034] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of a task execution method according to the present disclosure. The task execution method includes the following steps: Step 201: Obtain the task input information corresponding to the target task.

[0035] In some embodiments, the execution subject of the above task execution method (e.g.) Figure 1The electronic device 101 shown can acquire task input information corresponding to the target task via a wired or wireless connection. The target task can be a task to be executed. For example, the target task could be a "face recognition task" or an "image generation task." In practice, the target task is a task executed according to relevant input instructions. These relevant input instructions are contained in the task input information. These instructions indicate the task objective corresponding to the target task. For example, the relevant input instruction could be "generate a description of a landscape scene." The task input information can be the input content during the execution of the target task. In practice, the input instructions in the task input information determine the execution objective of the target task. In addition, the task input information can also include information in at least one modal form. For example, at least one modal form can include: text modality, audio modality, image modality, and video modality.

[0036] Step 202: Perform modal feature alignment on the information of various modal forms in the above task input information to obtain feature alignment information.

[0037] In some embodiments, the execution entity may perform modal feature alignment on the information of various modalities in the task input information to obtain feature alignment information. Modal feature alignment may involve aligning and fusing the feature semantic content under various modalities. The feature alignment information may be the result of aligning and fusing the feature semantic content under various modalities.

[0038] As an example, firstly, for each modality form corresponding to at least one modality form in the task input information, the modality content under the above modality form is encoded to obtain modality encoded information. Then, using explicit alignment or implicit alignment, the obtained at least one modality encoded information is aligned and fused with feature semantic content to obtain feature alignment information.

[0039] Optionally, after step 202, the steps further include: In response to the determination that the task input information includes information in unimodal form, the encoded information corresponding to the unimodal information is directly determined as the feature alignment information.

[0040] Step 203: Based on the task type corresponding to the above target task, determine the autoregressive generation paradigm corresponding to the above target task, and use it as the target autoregressive generation paradigm.

[0041] In some embodiments, the aforementioned executing entity can determine the corresponding autoregressive generation paradigm for the target task based on the task type corresponding to the target task, and use it as the target autoregressive generation paradigm. The task type can be the content type of the task execution content. In practice, each task type can be pre-set. For different task types, corresponding autoregressive generation paradigms are set to generate feature prediction information for the next time step. For example, the task type can be, but is not limited to, one of the following: image recognition, image segmentation, image generation, text generation, and text adjustment. An autoregressive generation paradigm is a generation method based on sequential stepwise prediction. Its core idea is to deduce future content stepwise from historical information. In practice, for task types that classify according to the generated object, the corresponding autoregressive generation paradigms can include: text autoregressive generation paradigm, image autoregressive generation paradigm, and multimodal autoregressive generation paradigm. A text autoregressive generation paradigm can be a word-by-word / character generation paradigm, where each prediction step is based on the context of the already generated sequence. Image autoregressive generation paradigms can generate images pixel-by-pixel or token-by-token, modeling spatial dependencies between pixels / tokens. Multimodal autoregressive generation paradigms can fuse multimodal inputs such as text and images to generate cross-modal content. In practice, the classification is based on the task execution method, and the corresponding autoregressive generation paradigms can include: Strict Autoregressive, Masked Autoregressive, and Hybrid Autoregressive.

[0042] In practice, the task type can be either stored in the task input information or determined based on the instruction content of the relevant input instructions in the task input information.

[0043] As an example, the aforementioned executing entity can query the autoregressive generation paradigm corresponding to the task type through the mapping table between task types and autoregressive generation paradigms, and use it as the target autoregressive generation paradigm.

[0044] In some optional implementations of certain embodiments, the execution entity may determine the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, and use it as the target autoregressive generation paradigm, including the following steps: In response to determining that the target task corresponds to an image generation type, the executing entity can determine that the autoregressive generation paradigm corresponding to the target task is a second generation paradigm with phased, progressive prediction. The image generation type can be a task whose content is related to image generation. The second generation paradigm with phased, progressive prediction can be a next-scale paradigm. Based on the next-scale paradigm, image tokens are generated to generate images. In practice, tasks corresponding to the image generation type can include: image background replacement, text-to-image generation, and image subject replacement. In practice, the generation models corresponding to the next-scale paradigm can include: VAR models and Emu3 models. The next-scale paradigm is suitable for high-resolution, detailed image generation, but often not suitable for text generation.

[0045] Step 204: Based on the above feature alignment information, use the above target autoregressive generation paradigm corresponding to the generation model to generate the task output results corresponding to the above target task.

[0046] In some embodiments, the execution entity can generate the task output corresponding to the target task based on the aforementioned feature alignment information and the corresponding generative model of the aforementioned target autoregressive generation paradigm. Each autoregressive generation paradigm has a corresponding generative model for content generation. For example, for strict autoregression, the corresponding generative model can be a Transformer decoder-based model. For masked autoregression, the corresponding generative model can be a MAE (Masked Autoencoder) model or a MAR (Masked Autoregressive Models) model. For text autoregressive generation paradigms, the corresponding generative model can be a large model from the GPT series. For image autoregressive generation paradigms, the corresponding generative model can be a Q-VAE+autoregressive model. The task output can be the output content of the target task. For example, if the target task requires generating an image, the corresponding task output is the generated image. If the target task requires generating text, the corresponding task output is the generated text.

[0047] As an example, the aforementioned execution entity can input feature alignment information into the generation model corresponding to the aforementioned target autoregressive generation paradigm to obtain the task output result corresponding to the aforementioned target task.

[0048] In some optional implementations of certain embodiments, the execution entity can generate the task output result corresponding to the target task based on the feature alignment information and the target autoregressive generation paradigm corresponding to the generation model, including the following steps: The first step involves inputting the aforementioned feature alignment information into the generative model corresponding to the second generation paradigm to obtain multiple initial image prediction feature information. This initial image prediction feature information can be the semantic content of the image features of the predicted image at the corresponding resolution, generated initially. In practice, the initial image prediction feature information can be in vector form. There is a one-to-one correspondence between the initial image prediction feature information and the resolutions among the multiple resolutions. That is, each initial image prediction feature information corresponds to a specific resolution to generate a predicted image at that resolution. In practice, the generative model corresponding to the second generation paradigm can be VAR (Visual Autoregressive Modeling).

[0049] The second step involves residual accumulation processing on the multiple initial image prediction feature information to obtain the image prediction feature information. Here, in the field of multimodal generation, Next-Scale Residual Accumulation is a technique that improves generation quality by optimizing multi-scale features in stages. Its core idea is to refine the error step by step, transferring the generation error from the low-resolution stage to the high-resolution stage for compensation, ultimately achieving more accurate detail reconstruction. The image prediction feature information can be the semantic content of the predicted image. In practice, the image prediction feature information can be in vector form. The predicted image can be an image generated according to the instructions of the target command.

[0050] The third step involves inputting the aforementioned image prediction feature information into a pre-trained image feature decoding model to obtain the predicted image, which serves as the task output. The image feature decoding model can be a decoding model that decodes feature information to generate the image. In practice, there is a one-to-one correspondence between image feature decoding models and image encoding models. In practice, the image feature decoding model can be the Decoder model in VQVAE.

[0051] The above embodiments of this disclosure have the following beneficial effects: Through the task execution methods of some embodiments of this disclosure, different suitable autoregressive generation paradigms can be selected based on different task types to accurately and efficiently generate the task output results of the target task. Specifically, the reason for the low execution efficiency of related target tasks is that: there are many task types, and selecting a suitable generation paradigm often requires relevant experts to select based on historical experience. This not only results in inaccurate generation paradigms but also in low efficiency and long execution time. Based on this, the task execution methods of some embodiments of this disclosure first obtain the task input information corresponding to the target task, wherein the task input information includes at least one modal form of information. Here, obtaining the task input information allows for the execution of the target task based on the task input information. Here, the inclusion of at least one modal form of information in the task input information allows the subsequent generation model to obtain more task content about the target task, achieving more accurate generation of task output results. Then, modal feature alignment is performed on the various modal forms of information in the above task input information to map the semantic features of different modalities to a unified space, realizing the establishment of cross-modal correspondence. Simultaneously, redundant information can be suppressed, improving the accuracy of subsequent model outputs. Furthermore, based on the task type corresponding to the aforementioned target task, an autoregressive generation paradigm corresponding to the target task is determined as the target autoregressive generation paradigm. Here, selecting a suitable autoregressive generation paradigm based on different task types can significantly improve the generation efficiency and accuracy of task output results, thereby improving the execution efficiency of the target task. Finally, based on the aforementioned feature alignment information, using the generation model corresponding to the aforementioned target autoregressive generation paradigm, the task output results corresponding to the aforementioned target task can be accurately generated. In summary, modal feature alignment enables the mapping of feature semantic content from various modalities to the same space and the suppression of redundant information. By setting an autoregressive generation paradigm corresponding to the task type, the generation efficiency and accuracy of task output results can be significantly improved, thereby enhancing the execution efficiency of the target task.

[0052] Further reference Figure 3 The flowchart 300 illustrates another embodiment of the task execution method according to this disclosure. The task execution method includes the following steps: Step 301: Obtain the task input information corresponding to the target task.

[0053] In some embodiments, the specific implementation of step 301 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 201 in the corresponding embodiment will not be repeated here.

[0054] Step 302: Perform word segmentation and encoding on the above target instructions to obtain instruction feature information.

[0055] In some embodiments, the execution entity (e.g. Figure 1 The electronic device 101 shown can perform word segmentation and encoding processing on the aforementioned target instructions to obtain instruction feature information. The task input information includes: target instructions in text modality and target images in image modality. The target instruction can be the task objective corresponding to the target task. That is, the target instruction indicates the completion objective and execution process of the target task. In practice, the target instruction can be in text modality. Similarly, for instructions in audio modality, the audio modality instruction can be converted into a text modality target instruction. The target image can be an image involved in the execution of the target task. The execution of the target task requires utilizing the image semantic content of the target image to finally generate the task output result. For example, a text description of the target image. Another example is background replacement processing of the target image. The instruction feature information can characterize the feature semantic content corresponding to the target instruction. In practice, the instruction feature information can be in vector form.

[0056] As an example, firstly, the aforementioned execution entity can use a word segmenter to segment the target instruction and obtain the segmentation result. Then, the segmentation result is input into the text encoding layer to obtain the instruction feature information.

[0057] Step 303: Perform image encoding processing on the target image to obtain image feature information.

[0058] In some embodiments, the execution entity may perform image encoding processing on the target image to obtain image feature information. This image feature information can characterize the semantic content of the image features. In practice, the image feature information can be in vector form.

[0059] As an example, the aforementioned execution entity can input the target image into an image coding layer (e.g., a multi-layer concatenated convolutional layer) to obtain image feature information.

[0060] Step 304: Based on the above instruction feature information and the above image feature information, use an attention-based modal alignment model to perform modal feature alignment on the above instruction feature information and the above image feature information to obtain feature alignment information.

[0061] In some embodiments, the execution entity can perform modal feature alignment on the instruction feature information and the image feature information using an attention-based modal alignment model, thereby obtaining feature alignment information. Modal feature alignment can involve aligning and fusing the semantic content of features across different modalities. The feature alignment information can be the result of aligning and fusing the semantic content of features across different modalities. The attention-based modal alignment model can be a model that aligns the semantic content of features across different modalities. In practice, the attention-based modal alignment model can be a bidirectional attention interaction model. A bidirectional attention interaction model can establish bidirectional attention channels from image to text and from text to image, achieving bidirectional feature mapping through alternating optimization.

[0062] As an example, the aforementioned executing entity can input the aforementioned instruction feature information and the aforementioned image feature information into the modal alignment model of the attention mechanism to perform modal feature alignment on the aforementioned instruction feature information and the aforementioned image feature information to obtain feature alignment information.

[0063] Step 305: In response to determining that the task type corresponding to the above target task is semantic understanding type, determine that the autoregressive generation paradigm corresponding to the above target task is the first generation paradigm of single-granularity stepwise prediction.

[0064] In some embodiments, in response to determining that the task type corresponding to the target task is a semantic understanding type, the executing entity may define the autoregressive generation paradigm corresponding to the target task as a first generation paradigm of single-granularity stepwise prediction. Here, the semantic understanding type can be a task type related to semantic content understanding. For example, a task under the semantic understanding type can be a text semantic extraction task, an image semantic extraction task, or a text generation task. In practice, the first generation paradigm of single-granularity stepwise prediction can be a generation paradigm based on the Next-Token mechanism.

[0065] Step 306: Input the above feature alignment information into the generative model corresponding to the first generative paradigm to obtain semantic understanding feature prediction information.

[0066] In some embodiments, the executing entity can input the feature alignment information into the generative model corresponding to the first generation paradigm to obtain semantic understanding feature prediction information. The semantic understanding feature prediction information can be the feature semantic information of the predicted semantic understanding content at the next time step. In practice, the semantic understanding feature prediction information can be in vector form. In practice, by decoding the semantic understanding feature prediction information, the semantic understanding result (i.e., the task output result) can be obtained. The semantic understanding information can be the task output result of the target task.

[0067] Step 307: Input the semantic understanding feature prediction information into the pre-trained semantic decoding model to obtain semantic understanding information, which is used as the output result of the above task.

[0068] In some embodiments, the execution entity can input the semantic understanding feature prediction information into a pre-trained semantic decoding model to obtain semantic understanding information, which is then used as the output of the task. In practice, a semantic decoding model is a decoding model that decodes the semantic understanding feature prediction information to obtain semantic understanding information in text form. For example, a semantic decoding model can be a multi-layered cascaded Long Short-Term Memory (LSTM) network model.

[0069] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 In some corresponding embodiments, the task execution method flow 300, when the task type is determined to be semantic understanding type, can accurately generate semantic understanding information by using the generative model corresponding to the first generation paradigm of single-granularity stepwise prediction, so as to achieve efficient execution of the target task.

[0070] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a task execution device, which are similar to... Figure 2 Corresponding to the method embodiments shown, this task execution device can be specifically applied to various electronic devices.

[0071] like Figure 4 As shown, a task execution device 400 includes: an acquisition unit 401, a feature alignment unit 402, a determination unit 403, and a generation unit 404. The acquisition unit is configured to acquire task input information corresponding to a target task, wherein the task input information includes at least one modal form of information; the feature alignment unit 402 is configured to perform modal feature alignment on the various modal forms of information in the task input information to obtain feature alignment information; the determination unit 403 is configured to determine the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as the target autoregressive generation paradigm; and the generation unit 404 is configured to generate the task output result corresponding to the target task using the feature alignment information and the generation model corresponding to the target autoregressive generation paradigm.

[0072] In some optional implementations of some embodiments, the task input information includes: a target instruction in text modality and a target image in image modality; and the feature alignment unit 402 can be configured to: perform word segmentation and encoding processing on the target instruction to obtain instruction feature information; perform image encoding processing on the target image to obtain image feature information; and perform modal feature alignment on the instruction feature information and the image feature information using an attention-based modal alignment model to obtain feature alignment information.

[0073] In some optional implementations of some embodiments, the determining unit 403 may be configured to: in response to determining that the task type corresponding to the above target task is a semantic understanding type, determine that the autoregressive generation paradigm corresponding to the above target task is a first generation paradigm of single-granularity stepwise prediction.

[0074] In some optional implementations of some embodiments, the determining unit 403 may be configured to: in response to determining that the task type corresponding to the target task is an image generation type, determine that the autoregressive generation paradigm corresponding to the target task is a second generation paradigm of phased stepwise prediction.

[0075] In some optional implementations of certain embodiments, the generation unit 404 may be configured to: input the aforementioned feature alignment information into the generation model corresponding to the aforementioned second generation paradigm to obtain multiple initial image prediction feature information; perform residual accumulation processing on the aforementioned multiple initial image prediction feature information to obtain image prediction feature information; input the aforementioned image prediction feature information into a pre-trained image feature decoding model to obtain a predicted image, which is used as the task output result.

[0076] In some optional implementations of some embodiments, the generation unit 404 may be configured to: input the above-mentioned feature alignment information into the generation model corresponding to the above-mentioned first generation paradigm to obtain semantic understanding feature prediction information; input the above-mentioned semantic understanding feature prediction information into a pre-trained semantic decoding model to obtain semantic understanding information, which is used as the output result of the above-mentioned task.

[0077] It is understandable that the units described in the task execution device 400 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the task execution device 400 and the units contained therein, and will not be repeated here.

[0078] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0079] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0080] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0081] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0082] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0083] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0084] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire task input information corresponding to a target task, wherein the task input information includes information in at least one modal form; perform modal feature alignment on the information in the task input information to obtain feature alignment information; determine the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as the target autoregressive generation paradigm; and generate the task output result corresponding to the target task using the generation model corresponding to the target autoregressive generation paradigm based on the feature alignment information.

[0085] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a feature alignment unit, a determination unit, and a generation unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may also be described as "a unit that acquires task input information corresponding to a target task."

[0088] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0089] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described task execution methods.

[0090] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A task execution method, comprising: Obtain task input information corresponding to the target task, wherein the task input information includes information in at least one modal form; Modal feature alignment is performed on the various modal forms of information in the task input information to obtain feature alignment information; Based on the task type corresponding to the target task, determine the autoregressive generation paradigm corresponding to the target task, and use it as the target autoregressive generation paradigm; Based on the feature alignment information, the target autoregressive generation paradigm is used to generate the corresponding task output result of the target task using the target autoregressive generation paradigm.

2. The method according to claim 1, wherein, The task input information includes: target instructions in text mode and target images in image mode; and The modal feature alignment of the information in the task input information of various modal forms to obtain feature alignment information includes: The target instruction is segmented and encoded to obtain instruction feature information; The target image is subjected to image encoding processing to obtain image feature information; Based on the instruction feature information and the image feature information, a modal feature alignment model based on an attention mechanism is used to align the instruction feature information and the image feature information to obtain feature alignment information.

3. The method according to claim 2, wherein, The step of determining the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as the target autoregressive generation paradigm, includes: In response to determining that the task type corresponding to the target task is a semantic understanding type, the autoregressive generation paradigm corresponding to the target task is determined to be the first generation paradigm of single-granularity stepwise prediction.

4. The method according to claim 2, wherein, The step of determining the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, as the target autoregressive generation paradigm, includes: In response to determining that the task type corresponding to the target task is an image generation type, the autoregressive generation paradigm corresponding to the target task is determined to be a second generation paradigm of phased stepwise prediction.

5. The method according to claim 4, wherein, The step of generating the task output result corresponding to the target task using the target autoregressive generation paradigm corresponding to the generation model based on the feature alignment information includes: The feature alignment information is input into the generation model corresponding to the second generation paradigm to obtain multiple initial image prediction feature information; Residual accumulation processing is performed on the multiple initial image prediction feature information to obtain image prediction feature information; The image prediction feature information is input into a pre-trained image feature decoding model to obtain a predicted image, which is then used as the task output.

6. The method according to claim 3, wherein, The step of generating the task output result corresponding to the target task using the target autoregressive generation paradigm corresponding to the generation model based on the feature alignment information includes: The feature alignment information is input into the generative model corresponding to the first generation paradigm to obtain semantic understanding feature prediction information; The semantic understanding feature prediction information is input into a pre-trained semantic decoding model to obtain semantic understanding information, which is then used as the task output.

7. A task execution device, comprising: The acquisition unit is configured to acquire task input information corresponding to the target task, wherein the task input information includes information in at least one modal form; The feature alignment unit is configured to perform modal feature alignment on information of various modal forms in the task input information to obtain feature alignment information; The determining unit is configured to determine the autoregressive generation paradigm corresponding to the target task based on the task type corresponding to the target task, and use it as the target autoregressive generation paradigm. The generation unit is configured to generate the task output result corresponding to the target task based on the feature alignment information and using the target autoregressive generation paradigm corresponding to the generation model.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images