Method, apparatus, device, storage medium and program product for data processing

Abstract

According to embodiments of the present disclosure, a method, an apparatus, a device, a storage medium and a program product for data processing are provided. The method comprises: obtaining a token sequence corresponding to a multi-modal input for a machine learning model, the multi-modal input comprising a text part and a non-text part; determining, based on information associated with the text part, a token type of each token in the token sequence corresponding to the text part, the token type comprising a valid type and an invalid type; updating the token sequence by pruning the tokens in the token sequence determined as the invalid type to obtain an updated token sequence; determining attention weight information corresponding to the updated token sequence, the attention weight information comprising an attention weight value of each token in the updated token sequence; and processing, by the machine learning model, the updated token sequence based on the attention weight information to obtain a model output corresponding to the multi-modal input.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, electronic devices, computer-readable storage media, and computer program products for data processing. Background Technology

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in various application environments. A machine learning model can receive a batch of token sequences at a time. To enable the machine learning model to process multiple token sequences within a batch, each token sequence in the batch is pre-paddinged to a predetermined length. The length of the token sequence is the number of tokens it contains. This ensures that the multiple token sequences obtained by the model have the same number of tokens. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for data processing is provided. The method includes: acquiring a word sequence corresponding to a multimodal input to a machine learning model, the multimodal input including a text portion and a non-text portion; determining the word type of each word in the word sequence corresponding to the text portion based on information associated with the text portion, the word type including valid types and invalid types; updating the word sequence by pruning words determined to be invalid types from the word sequence to obtain an updated word sequence; determining attention weight information corresponding to the updated word sequence, the attention weight information including the attention weight value of each word in the updated word sequence; and processing the updated word sequence using a machine learning model based on the attention weight information to obtain a model output corresponding to the multimodal input.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: a lexical sequence acquisition module configured to acquire a lexical sequence corresponding to a multimodal input to a machine learning model, the multimodal input including a text portion and a non-text portion; a lexical type determination module configured to determine the lexical type of each lexical in the lexical sequence corresponding to the text portion based on information associated with the text portion, the lexical type including valid types and invalid types; a lexical sequence update module configured to update the lexical sequence by pruning lexicals determined to be invalid types from the lexical sequence, obtaining an updated lexical sequence; a weight information determination module configured to determine attention weight information corresponding to the updated lexical sequence, the attention weight information including attention weight values ​​of each lexical in the updated lexical sequence; and a lexical sequence processing module configured to process the updated lexical sequence using a machine learning model based on the attention weight information, obtaining a model output corresponding to the multimodal input.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, cause the processor to perform the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1A A block diagram of an application environment according to an exemplary implementation of this disclosure is shown; Figure 1B A schematic diagram of an example architecture of a machine learning model according to an embodiment of the present disclosure is shown; Figure 1C A schematic diagram of an example environment for model training and application according to some embodiments of the present disclosure is shown; Figure 2 A flowchart of a method for data processing according to some embodiments of the present disclosure is shown; Figure 3A Examples of lexical sequences according to some embodiments of this disclosure are shown; Figure 3B Examples of attention weight information according to some embodiments of this disclosure are shown; Figure 3C Examples of attention weight information according to other embodiments of this disclosure are shown; Figure 4An exemplary structural block diagram of an apparatus for data processing according to some embodiments of the present disclosure is shown; and Figure 5 A block diagram of an electronic device that can implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0010] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0011] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0012] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0013] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0015] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0016] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0017] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0018] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0019] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0020] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0021] For ease of understanding, some of the terms used in the embodiments of this disclosure will be explained below.

[0022] Vision-Language-Action (VLA) model: a general-purpose robot neural network architecture that integrates perception, understanding, and motion control.

[0023] Token: Also known as a token, it is the basic unit for machine learning models to understand information.

[0024] Token padding mechanism: Also known as token filling mechanism, padding mechanism, etc. In sequence modeling, to align data of different lengths, a token padding mechanism is often used to add meaningless placeholders to each sequence. These meaningless placeholders can also be called padding tokens.

[0025] Padding Tax: The waste of computational resources caused by machine learning models being forced to process a large number of meaningless padding words during training.

[0026] Attention Mask: A Boolean matrix that controls which positions in a Transformer can see each other, used to maintain the causality or modal isolation of sequences.

[0027] Figure 1A A block diagram 100A of an application environment according to an exemplary implementation of this disclosure is shown. For example... Figure 1A As shown, robot device 110 and user 120 can be located in environment 160, and user 120 can control robot device 110 to perform a variety of tasks.

[0028] Environment 160 may include, but is not limited to, one or more objects. For example, in a kitchen / dining environment, environment 160 may include, but is not limited to, various items, such as edible food, kitchen utensils, etc., or a combination of one or more of the above. In a wardrobe environment, environment 160 may include, but is not limited to, various types of clothing. Environment 160 may include, but is not limited to, one or more rooms. For example, in a home environment, environment 160 may include, but is not limited to, a living room, bedroom, study, kitchen, toilet, etc., or a combination of one or more of the above.

[0029] like Figure 1A As shown, the robot device 110 may include multiple parts. For example, the control unit 111 can serve as the control center of the robot device 110, and an application can be loaded into the control unit 111 to control the various parts of the robot device. The user 120 can use the interaction unit 112 to interact with the robot device 110, for example, by inputting control commands to the robot device 110 to perform desired tasks. The robot device 110 may include an arm 113 for performing actions such as grasping and releasing. For example, the arm 113 can grasp an object and move it to a desired position, and so on.

[0030] Alternatively and / or additionally, the robot device 110 may also include a data acquisition unit 114. Here, the data acquisition unit 114 may include various types, such as an image acquisition unit, a sound acquisition unit, etc. Alternatively and / or additionally, the robot device 110 may further include a sensing unit for detecting surrounding objects, for example, detecting the distance between the robot and surrounding objects based on laser light, etc. The robot device 110 may also include a drive unit 115, for example, the drive unit 115 can drive the robot device 110 to move along a desired path.

[0031] Environment 160 may include one or more acquisition units 130, ..., and 132. For example, one or more image acquisition devices may be deployed in a room to acquire images of the room from various angles. Environment 160 may include a control device 140, which can control one or more acquisition units 130, ..., and 132, etc., via a network (not shown). Alternatively and / or additionally, in a smart home environment, control device 140 can control various electrical devices in environment 160.

[0032] Alternatively and / or additionally, a machine learning model (e.g., machine learning model 150) may be provided to manage environment 160. It should be understood that, although... Figure 1AThe machine learning model 150 is shown to be located within environment 160, but it can also be deployed in robotic device 110. Alternatively and / or additionally, the machine learning model 150 can also be located at a remote device outside environment 160, and the control device 140, robotic device 110, or other device can access the remote machine learning model 150 via a network.

[0033] Machine learning model 150 may include one or more models. If machine learning model 150 includes multiple models, these multiple models may include multiple types of models. Machine learning model 150 may include a Vision-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. Machine learning model 150 may, for example, include at least a Vision-Language Model (VLM) and an action model. The Vision-Language Model has the ability to process images and natural language. The Action Model can control the robotic device 110 to perform various actions. Machine learning model 150 may also include, for example, an image processing model, a text processing model, and so on.

[0034] In some embodiments, the machine learning model 150 may include a content-generating model capable of generating corresponding outputs based on model inputs. In some implementations, the machine learning model may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and may obtain corresponding model outputs based on the model inputs and prompts, thereby completing the task execution.

[0035] The machine learning model 150 may include an encoder module 151, a backbone network 152, and a decoder module 156. The encoder module 151 performs feature encoding on the model input (e.g., encoding an image or video into a feature space). The encoder module 151 may include, but is not limited to, a visual encoder, a speech encoder, and an action encoder. The backbone network 152 performs feature extraction on the encoded model input to obtain a feature representation 155 for performing a task. In some embodiments, the backbone network 152 may include one or more attention modules (e.g., transformer blocks) capable of performing attention-based processing on the encoded model input to extract feature representations. The decoder module 156 then decodes the output feature representation 155 to obtain the model output. The decoder module 156 may include at least an action decoder. For example, for a VLA model, the model output may be, for example, the next action the robot will perform. It should be understood that... Figure 1AThe illustration shows an example application of machine learning model 150, which is not limited in this disclosure.

[0036] like Figure 1A As shown, user 120 can instruct robot device 110 to manipulate various objects in environment 160 and complete corresponding tasks. Here, objects can be various items in environment 160. For example, user 120 can instruct robot device 110 to find a specific object in environment 160. Or, user 120 can instruct robot device 110 to place the found object in a designated location, and so on. In some examples, user 120 can instruct robot device 110 to perform a task in environment 160. For example, in a life service scenario, robot device 110 can perform tasks such as indoor cleaning, item organization, home control, and voice interaction in environment 160. In an industrial manufacturing scenario, robot device 110 can perform tasks such as industrial inspection and production operations in environment 160. However, these are merely examples, and this disclosure does not limit the scope of the invention.

[0037] Figure 1B A schematic diagram of an example architecture of a machine learning model 150 according to an embodiment of the present disclosure is shown. As mentioned above, the machine learning model 150 according to an embodiment of the present disclosure may include a Visual-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. Given visual input 153a (e.g., an image or video, etc.) and language input 153b (e.g., text commands or voice commands) in the environment space (e.g., a physical environment or a virtual environment) in which the robot device 110 is located, the VLA model can output action commands 157. The action commands 157 can be used to control the robot device 110 to perform the task requested by the visual input 153a or the language input 153b.

[0038] In some embodiments, such as Figure 1B As shown, the VLA model may include a visual encoder 151a, a language encoder 151b, a backbone network 152, and an action decoder 156a. The visual encoder 151a performs feature encoding on the visual input 153a (e.g., encoding an image or video into a feature space) to obtain a visual representation 154a. The language encoder performs feature encoding on the language input 153b, such as text instructions or speech instructions, to obtain a language representation 154b. The backbone network 152 performs feature extraction based on the visual representation 154a and the language representation 154b to obtain a feature representation 155 for action decision-making. Then, the action decoder 156a performs feature decoding on the feature representation 155 to obtain an action instruction 157 (or a sequence of action instructions).

[0039] The motion command 157 here can be implemented in several ways. As an example, motion command 157 can indicate the relative change in the end-effector posture of the robotic arm (i.e., robot device 110). Based on this relative change using the inverse dynamics model of the robotic arm, motion parameters such as angles, velocities, and accelerations for multiple degrees of freedom of the robotic arm (e.g., joints or end-effector grippers) can be determined. Alternatively, motion command 157 can also indicate the motion parameters for multiple degrees of freedom of the robot device 110. In other words, motion decoder 156a can also directly output the motion parameters for multiple degrees of freedom of a specific robot device 110, allowing the robot device 110 controller to directly drive its movement based on these motion parameters.

[0040] In some embodiments, the VLA model may further include a motion encoder 151c. The model input of the VLA model may also include motion input 153c, which may include, but is not limited to, motion commands 157 provided to the robot device 110 in the previous control cycle or motion sensing results of the robot device 110 in the previous control cycle. The motion encoder 151c may perform feature encoding on the motion input 153c to obtain a motion representation 154c. The backbone network 152 may perform feature extraction based on the visual representation 154a, language representation 154b, and motion representation 154c to obtain a feature representation 155. Of course, the model input of the VLA model described above is merely exemplary. In practical applications, the model input of the VLA model may also include, for example, depth maps, point clouds, force feedback information, or haptic feedback information. In this case, the VLA model may also include, for example, a depth map encoder, a point cloud encoder, a force encoder, or a haptic encoder.

[0041] In some embodiments, the backbone network 152 can perform multimodal fusion and feature extraction on the visual representation 154a, language representation 154b, and action representation 154c based on a cross-attention mechanism to obtain feature representation 155. In some embodiments, the visual encoder 151a, language encoder 151b, and backbone network 152 can be implemented based on a pre-trained visual-language model (VLM). In other words, a VLA model can be constructed by combining the action encoder 151c and action decoder 156a on the basis of a pre-trained VLM. Of course, the backbone network 152 described above is only exemplary. In practical applications, any other suitable multimodal model can be used to construct the backbone network 152. The embodiments of this disclosure do not limit this.

[0042] Figure 1CA schematic diagram of an example environment 100C for model training and application according to some embodiments of the present disclosure is shown. Figure 1C The example environment 100C illustrates three distinct phases of the machine learning model 150: a pre-training phase 172, a fine-tuning phase 174, and an application phase 176. A testing phase, not shown in the figure, may also occur after the pre-training phase 172 or the fine-tuning phase 174.

[0043] Example environment 100C involves a model pre-training system 192, a model fine-tuning system 194, and a model application system 196. In the pre-training phase 172, the model pre-training system 192 is configured to perform pre-training of a machine learning model 150 using a pre-training dataset 182. At the start of pre-training, the individual components of the machine learning model 150 may have initial parameter values. The pre-training process involves updating the parameter values ​​of the machine learning model 150 to desired values ​​based on data from the pre-training dataset 182. The pre-training task is used to assist in updating the parameters of the machine learning model 150.

[0044] During the pre-training phase 172, the machine learning model 150 can learn strong generalization capabilities using a pre-training dataset 182 containing a large amount of data. After pre-training, the parameter values ​​of the machine learning model 150 have been updated to include the pre-trained parameter values. In some embodiments, the machine learning model 150 may include a Visual Language Action (VLA) model, and the pre-training dataset 182 used to train the machine learning model 150 may include multiple pre-training samples, each of which may include sample language input, sample visual input, and sample action instructions. The pre-trained machine learning model 120 can generate corresponding action instructions relatively accurately based on the received language input and visual input.

[0045] The pre-trained machine learning model 150 can be provided to the fine-tuning stage 174, where the model fine-tuning system 194 fine-tunes it for different downstream tasks. In the fine-tuning stage 174, the parameter values ​​of the machine learning model 150 are further adjusted using the training dataset 184. During fine-tuning, the parameters of the machine learning model 150 are also updated and adjusted using the corresponding training algorithm. Since the model has learned a great deal from the training data in the pre-training stage, a downstream task model that meets expectations can be obtained using only a small amount of training data in the fine-tuning stage 174. The training samples included in the training dataset 184 can be of the same type as the pre-training samples included in the pre-training dataset 182; that is, the training dataset 184 can include multiple training samples, each of which can include sample language input, sample visual input, and sample action instructions. In some cases, the pre-training dataset 182 can include multiple pre-training samples for a general scenario, while the training dataset 184 can include multiple training samples for a specific scenario or specific task.

[0046] In some embodiments, a testing phase may be included after the fine-tuning phase 174, where the performance of the machine learning model 150 can be further tested using a test dataset. The dataset used in the testing phase is of the same type as that used in the fine-tuning phase.

[0047] In application phase 176, the obtained machine learning model 150, with trained parameter values, can be provided to the model application system 196 for use. In application phase 176, the machine learning model 150 can be used to process corresponding model inputs 186 in the real-world scenario and provide corresponding model outputs 188. As an example only, model inputs 186 may include verbal input and visual input, and model outputs 188 may include corresponding action instructions.

[0048] exist Figure 1C In this system, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0049] The server-side device can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc. It is understood that the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on the same electronic device or on different electronic devices; this disclosure does not limit this.

[0050] It should be understood that Figure 1CThe components and arrangements shown in the example environment 100C are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 may be integrated in the same system or device. For example, at least the model pre-training system 192 and the model fine-tuning system 194 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect.

[0051] As mentioned earlier, a machine learning model can receive a batch of word sequences at a time. To enable the model to process multiple word sequences within a batch, each word sequence in the batch is pre-padded to a predetermined length (e.g., 200), ensuring that the multiple word sequences obtained by the model have the same number of words. However, since the original length of each word sequence before padding may be relatively small (typically between 10 and 30), the word sequences obtained by the model will include a large number of meaningless padding words. This fixed-length padding mechanism generates a significant padding tax. When processing such word sequences, the machine learning model consumes substantial computational resources to handle these meaningless padding words. Since the computational complexity of the self-attention mechanism increases quadratically with the sequence length, this redundancy not only wastes valuable computational power but also limits the size of the batches that the machine learning model can process simultaneously.

[0052] In view of this, according to embodiments of the present disclosure, an improved data processing scheme is provided. According to the scheme of the embodiments of the present disclosure, a word sequence corresponding to a multimodal input for a machine learning model is obtained. The multimodal input includes a text portion and a non-text portion. Based on information associated with the text portion, the word type of each word in the word sequence corresponding to the text portion is determined. The word type includes valid types and invalid types. The word sequence is updated by pruning words determined to be invalid types, resulting in an updated word sequence. Attention weight information corresponding to the updated word sequence is determined, including the attention weight value of each word in the updated word sequence. Using a machine learning model, the updated word sequence is processed based on the attention weight information to obtain the model output corresponding to the multimodal input.

[0053] In this way, the lexical type of each lexical unit in the lexical sequence corresponding to the text portion can be determined, and the lexical sequence can be updated by pruning lexical units determined to be invalid types. This can effectively reduce the number of lexical units included in the lexical sequence. Furthermore, determining attention weight information based on such an updated lexical sequence can reduce the amount of information contained in the attention weight information. Determining the final model output based on such attention weight information can reduce the resource consumption of model computation and improve computational efficiency.

[0054] The following description continues with reference to the accompanying drawings, illustrating some exemplary embodiments of this disclosure. The data processing procedures involved in this disclosure can be implemented to generate videos for training machine learning models that generate images or videos containing single objects.

[0055] Figure 2 A flowchart of a method 200 for data processing according to some embodiments of the present disclosure is shown. Method 200 can be implemented in... Figure 1C The model pre-training system 192 / model fine-tuning system 194, or an electronic device on which the model pre-training system 192 / model fine-tuning system 194 is deployed, will be described exemplarily herein with the method 200 implemented at the model fine-tuning system 194 as an example. Reference will be made to Figures 1A to 1C Let's describe method 200.

[0056] In box 210, the model fine-tuning system 194 acquires the word sequence corresponding to the multimodal input for the machine learning model, the multimodal input including text and non-text parts. The machine learning model here can be, for example, the aforementioned machine learning model 150, which may include, for example, the Visual Language Action (VLA) model.

[0057] In some embodiments, a machine learning model can acquire multiple word sequences corresponding to multiple multimodal inputs at once. As mentioned earlier, to enable the machine learning model to process these multiple word sequences in batches, each word sequence can be pre-padded to a predetermined length. The data processing procedure described below can be a data processing procedure for any word sequence among the multiple word sequences.

[0058] In some embodiments, the multimodal input for a machine learning model may include at least an image portion (e.g., the visual input 153a mentioned above) and a text portion (e.g., the language input 153b mentioned above). The image portion may include at least one image or at least one video frame. The text portion may include at least one text unit. The lexical sequence corresponding to such a multimodal input may include at least a plurality of lexical units corresponding to the image portion and a plurality of lexical units corresponding to the text portion. In some embodiments, the multimodal input may further include an action portion (e.g., the language input 153c mentioned above). In this case, the lexical sequence may further include a plurality of lexical units corresponding to the action portion.

[0059] In some embodiments, the lexical sequence corresponding to the multimodal input may include pre-filled filler words in both the multiple words corresponding to the image portion and the multiple words corresponding to the text portion. These filler words can be considered as meaningless placeholders.

[0060] In box 220, the model fine-tuning system 194 determines the lexical type of each lexical in the lexical sequence corresponding to the text portion based on information associated with the text portion. The lexical type includes valid type and invalid type.

[0061] In some embodiments, information associated with a text portion indicates the importance of each word corresponding to that text portion. For each word corresponding to a text portion, the model fine-tuning system 194 can determine the comparison result between the importance of that word and a threshold. If the comparison result indicates that the importance of that word is below the threshold, the model fine-tuning system 194 can determine that the word type is an invalid word. If the comparison result indicates that the importance of that word reaches the threshold, the model fine-tuning system 194 can determine that the word type is a valid word.

[0062] Specifically, the information associated with the text portion may include, for example, at least padding mask information, which can indicate whether each word corresponding to the text portion is a padding word. For each word associated with the text portion, if the word is a padding word, the model fine-tuning system 194 can determine that the importance of the word is below a threshold, and thus determine that the word type is an invalid word. If the word is not a padding word, the model fine-tuning system 194 can determine that the importance of the word reaches a threshold, and thus determine that the word type is a valid word.

[0063] The information associated with the text portion can also include any other suitable information. For example, semantic information can indicate whether a lexical unit has semantic meaning. For instance, the text portion of a multimodal input can include multiple text units (e.g., characters, words) and multiple symbols. At least one lexical unit corresponding to each text unit in the lexical sequence can have relatively complete semantic meaning, while the lexical unit corresponding to each symbol may have no semantic meaning or a vague semantic meaning. It is understood that filler lexical units do not have semantic meaning.

[0064] Therefore, the importance of each word can be determined based on its semantics corresponding to the text portion. For example, for each word associated with a text portion, if the word has semantic meaning, the model fine-tuning system 194 can determine that the word's importance reaches a threshold, thus determining that the word's word type is a valid word. If the word does not have semantic meaning or its semantic meaning is too ambiguous, the model fine-tuning system 194 can determine that the word's importance is below a threshold, thus determining that the word's word type is an invalid word.

[0065] In box 230, the model fine-tuning system 194 updates the lexical sequence by pruning lexical segments that are determined to be invalid, thus obtaining the updated lexical sequence.

[0066] Taking a multimodal input that includes at least an image and a text portion as an example, the word sequence corresponding to the multimodal input can include words that are determined to be invalid, both those corresponding to the image portion and those corresponding to the text portion. The method for determining the word type of words corresponding to the image portion may differ from the method for determining the word type of words corresponding to the text portion. The invalid word types pruned by the model fine-tuning system 194 can include words corresponding to the text portion or words corresponding to the image portion. The following explanation uses only the example of pruned words that are determined to be invalid, specifically those corresponding to the text portion, for illustration.

[0067] refer to Figure 3A , Figure 3A Example 300A of a lexical sequence according to some embodiments of the present disclosure is shown. For example... Figure 3A As shown in Figure 302, the lexical sequence (represented as multiple squares in the figure, each square representing a lexical) can include 978 lexical units. For example only, units 1-768 in this sequence can correspond to image portions, units 769-968 to text portions, and units 969-978 to action portions. In the following description, unless otherwise specified, each lexical unit in the sequence will be used as an example.

[0068] If the 778th to 968th lexical units in the lexical unit sequence are invalid (marked in black in the diagram), the model fine-tuning system 194 can update the lexical unit sequence by pruning these units (i.e., units 778 to 968) to obtain an updated lexical unit sequence. In this case, the updated lexical unit sequence includes 787 units, which is significantly less than the number of units in the original lexical unit sequence. This reduces the computational resources required for the machine learning model to process the lexical unit sequence, thus improving computational efficiency.

[0069] It should be noted that those skilled in the art will understand that this disclosure does not limit the individual lexical units in the lexical sequence. For example, in the lexical sequence shown in box 302, the 1st to 512th lexical units can be lexical units corresponding to the action part, the 513th to 768th lexical units and the 969th to 978th ​​lexical units can both be lexical units corresponding to the image part, and the 769th to 968th lexical units can be lexical units corresponding to the text part.

[0070] In box 240, the model fine-tuning system 194 determines the attention weight information corresponding to the updated lexical sequence, which includes the attention weight value of each lexical in the updated lexical sequence.

[0071] Each word in a word sequence has corresponding positional information. The model fine-tuning system 194, for example, can obtain the positional encoding identifier (position_id) of each word in the word sequence. This positional encoding identifier indicates the position of each word in the word sequence. (Reference) Figure 3A The word sequence shown in box 302 comprises 978 words, as illustrated in the figure. The position codes of these 978 words can be sequentially assigned as 1, 2, 3, ..., 977, 978, which represent the position of the corresponding word in the word sequence. For example, if the position code of a word is 2, then that word is the second word in the word sequence.

[0072] In some embodiments, the model fine-tuning system 194 can divide the updated lexical sequence into a first lexical segment (also referred to as the prefix portion of the lexical sequence) and a second lexical segment (also referred to as the suffix portion of the lexical sequence). The first lexical segment includes at least one lexical corresponding to the text portion and the lexical preceding the at least one lexical, and the second lexical segment includes the lexical following the at least one lexical. (Continue to the next step) Figure 3ATaking the 778th to 968th lexical units in the lexical unit sequence shown in box 302 as invalid lexical units, the terminal device 110 can obtain the updated lexical unit sequence by cutting out the lexical units that are determined to be invalid (i.e., the 778th to 968th lexical units).

[0073] Box 304 illustrates an example of an updated lexical sequence. As shown in Box 304, the lexical sequence can include a total of 787 lexical units. The positional codes of the first 777 lexical units can be sequentially 1, 2, 3, ..., 777, and the positional codes of the last 10 lexical units can be sequentially 969, 967, ..., 978. Taking the lexical sequence shown in Box 302, where multiple lexical units with positional codes 1-768 correspond to the image portion, multiple lexical units with positional codes 769-968 correspond to the text portion, and multiple lexical units with positional codes 969-978 correspond to the action portion, as an example, the first lexical segment in the lexical sequence shown in Box 304 can include multiple lexical units with positional codes 1-777, and the second lexical segment can include multiple lexical units with positional codes 969-968.

[0074] At this point, there are words with discontinuous positional codes in the word sequence. For example, after a word with positional code 777, there is a word with positional code 969. This affects the machine learning model's perception of time steps and action order, increasing the probability of data processing errors. To address this issue, the model fine-tuning system 194 can remap the original positional information of the second word segment in the word sequence to its positional information in the updated word sequence. For example, the model fine-tuning system 194 can remap the original positional information of the second word segment in the word sequence to its positional information in the updated word sequence by remapping the positional code identifiers of each word in the second word segment. After remapping, the positional code identifier of the first word in the second word segment is continuous with the positional code identifier of the last word in the first word segment.

[0075] Continue to refer to Figure 3A The model fine-tuning system 194 can remap the positional encoding identifiers of each word in the second word segment of the word sequence shown in box 304 (i.e., multiple words with positional encoding identifiers of 969-968, which are marked as squares filled with diagonal lines in the figure) to obtain the word sequence shown in box 306. As shown in box 306, the positional encoding identifiers of the second word segment after remapping can be 778-787 in sequence.

[0076] In some embodiments, the model fine-tuning system 194 can determine the attention weight information corresponding to the updated word sequence based on the original position information of the first word segment and the remapped position information of the second word segment. As an example only, the attention weight information can be represented as an attention mask matrix, where the rows and columns can be, for example, multiple positional encoding identifiers for multiple words. (See reference...) Figure 3B and Figure 3C , Figure 3B and Figure 3C Examples 300B and 300C illustrate attention weight information according to some embodiments of this disclosure. In examples 300B and 300C, the symbol T indicates that the machine learning model needs to pay attention to a word at that position, and the symbol F indicates that the machine learning model does not need to pay attention to a word at that position. The attention weight of each word marked F can be set to infinity to shield the propagation of invalid information.

[0077] Example 300B shows an unupdated sequence of lexical terms (e.g.) Figure 3A The attention weight information corresponding to the lexical sequence shown in the middle box 302. Taking the multiple lexical units with positional encoding identifiers of 769-968 in the lexical sequence as corresponding to the text part of the multimodal input, and the lexical unit type of the multiple lexical units with positional encoding identifiers of 778-968 being invalid, in the attention weight information shown in Example 300B, although the attention weights of the multiple lexical units corresponding to rows 778-968 and columns 778-968 (i.e., the multiple lexical units with positional encoding identifiers of 778-968) enclosed by the dashed line can be set to infinity, these lexical units still participate in the linear transformation and attention calculation of the machine learning model. This not only consumes computing resources but also affects the efficiency of model calculation.

[0078] Example 300C shows an updated sequence of lexical segments whose positional information of the second lexical segment has been remapped (e.g., Figure 3A The attention weight information corresponds to the lexical sequence shown in the middle box 306. It can be seen that compared with the lexical sequence shown in Example 300B, this attention weight information is more concise. Without affecting the lexical sequences that the machine learning model needs to focus on, it can reduce the computational cost of the machine learning model and improve its computational efficiency.

[0079] In some embodiments, the attention weight information includes bidirectional attention information and causal attention information. The bidirectional attention information indicates the attention information within the first lexical segment, and the causal attention information indicates the attention information of the second lexical segment on the first lexical segment. The model fine-tuning system 194 can determine the boundary between the bidirectional attention information and the causal attention information in the attention weight information based on the division of the first and second lexical segments (the division method of the first and second lexical segments will not be described in detail here).

[0080] Specifically, the boundary between bidirectional attention information and causal attention information in the attention weight information can be determined based on the number of lexical units included in the first lexical unit segment. For example... Figure 3C As shown, the first lexical segment may include 777 lexical units with positional encoding identifiers from 1 to 777. The boundary between bidirectional attention information and causal attention information may lie between the 777th and 778th lexical units (e.g., the bold solid line shown in the figure). The model fine-tuning system 194 may apply an attention mechanism to the updated lexical sequence based on the determined boundary to determine the bidirectional attention information and the causal attention information.

[0081] In box 250, the model fine-tuning system 194 uses a machine learning model to process the updated word sequence based on attention weight information to obtain the model output corresponding to the multimodal input.

[0082] Machine learning models can, for example, determine the attention weights between each word in the updated word sequence based on attention weight information, and then process multiple words in the updated word sequence based on the determined attention weights to obtain the model output corresponding to the multimodal input.

[0083] The model output can be the predicted action command corresponding to the multimodal input. During the model training phase, the samples used to train the machine learning model can include multimodal input and the corresponding sample action commands. The model fine-tuning system 194 can calculate the difference (e.g., mean squared error) between the sample action commands in the samples and the predicted action commands in the model output to train the machine learning model. Since invalid terms have been pruned, the machine learning model does not need to use additional masking operations to block invalid information, which can improve the training efficiency while ensuring the training effect of the model.

[0084] Therefore, the lexical type of each lexical in the lexical sequence corresponding to the text portion can be determined, and the lexical sequence can be updated by pruning lexical types determined to be invalid. This can effectively reduce the number of lexical elements included in the lexical sequence. Furthermore, dynamically determining attention weight information based on such an updated lexical sequence can reduce the amount of information in the attention weight information without affecting the tokens that the model needs to focus on. Determining the final model output based on such attention weight information can reduce the resource consumption of model computation and improve computational efficiency. Especially in the model training process, this token processing method and attention weight information determination method can greatly improve the training speed of the model.

[0085] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 An exemplary structural block diagram of an apparatus 400 for data processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a model pre-training system 192 / model fine-tuning system 194. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0086] like Figure 4 As shown, the device 400 includes: a lexical sequence acquisition module 410, configured to acquire a lexical sequence corresponding to a multimodal input for a machine learning model, the multimodal input including a text part and a non-text part; a lexical type determination module 420, configured to determine the lexical type of each lexical in the lexical sequence corresponding to the text part based on information associated with the text part, the lexical type including a valid type and an invalid type; a lexical sequence update module 430, configured to update the lexical sequence by pruning lexicals determined to be invalid types in the lexical sequence to obtain an updated lexical sequence; a weight information determination module 440, configured to determine the attention weight information corresponding to the updated lexical sequence, the attention weight information including the attention weight value of each lexical in the updated lexical sequence; and a lexical sequence processing module 450, configured to use a machine learning model to process the updated lexical sequence based on the attention weight information to obtain the model output corresponding to the multimodal input.

[0087] In some embodiments, the weight information determination module 440 is further configured to: divide a first lexical segment and a second lexical segment in the updated lexical sequence, wherein the first lexical segment includes at least one lexical segment corresponding to the text portion and a lexical segment preceding the at least one lexical segment, and the second lexical segment includes a lexical segment following the at least one lexical segment; remap the original position information of the second lexical segment in the lexical sequence to position information in the updated lexical sequence; and determine attention weight information corresponding to the updated lexical sequence based on the original position information of the first lexical segment and the remapped position information of the second lexical segment.

[0088] In some embodiments, the attention weight information includes bidirectional attention information and causal attention information, and the weight information determination module 440 is further configured to: divide the updated lexical sequence into a first lexical segment and a second lexical segment, the first lexical segment including at least one lexical corresponding to the text portion and a lexical preceding the at least one lexical, and the second lexical segment including a lexical following the at least one lexical; determine the boundary between bidirectional attention information and causal attention information in the attention weight information based on the division of the first lexical segment and the second lexical segment; and apply an attention mechanism to the updated lexical sequence based on the determined boundary to determine the bidirectional attention information and the causal attention information.

[0089] In some embodiments, bidirectional attention information indicates attention information within the first lexical segment, and causal attention information indicates attention information of the second lexical segment on the first lexical segment.

[0090] In some embodiments, the multimodal input includes an image portion, a text portion, and an action portion, wherein a first lexical segment includes lexical units corresponding to the image portion and the text portion, and a second lexical segment includes lexical units corresponding to the action portion.

[0091] In some embodiments, the information associated with a text portion indicates the importance of each lexical unit corresponding to the text portion.

[0092] In some embodiments, the lexical type determination module 420 is further configured to: for each lexical corresponding to a text portion, in response to the importance of the lexical being lower than a threshold, determine that the lexical type of the lexical is an invalid lexical, or in response to the importance of the lexical reaching a threshold, determine that the lexical type of the lexical is a valid lexical.

[0093] In some embodiments, the information associated with the text portion includes padding mask information indicating whether each word corresponding to the text portion is a padding word, and the apparatus 400 further includes: a first importance determination module configured to determine that the importance of the word is below a threshold in response to the word being a padding word; or a second importance determination module configured to determine that the importance of the word reaches a threshold in response to the determination that the word is not a padding word.

[0094] In some embodiments, the machine learning model includes a visual language action model.

[0095] The units and / or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0096] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices can, for example, be used to implement... Figure 1C The model pre-training system 192 and the model fine-tuning system 194 are mentioned.

[0097] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1C The model pre-training system 192 / model fine-tuning system 194.

[0098] like Figure 5As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors 510 or processing units, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0099] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 500.

[0100] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0101] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0102] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0103] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0104] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0106] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for data processing, comprising: Obtain the word sequence corresponding to the multimodal input for the machine learning model, wherein the multimodal input includes a text part and a non-text part; Based on information associated with the text portion, the lexical type of each lexical in the lexical sequence corresponding to the text portion is determined, and the lexical type includes valid type and invalid type; The word sequence is updated by pruning words that are determined to be invalid, resulting in an updated word sequence. Determine the attention weight information corresponding to the updated lexical sequence, wherein the attention weight information includes the attention weight value of each lexical in the updated lexical sequence; as well as Using the machine learning model, the updated lexical sequence is processed based on the attention weight information to obtain the model output corresponding to the multimodal input.

2. The method according to claim 1, wherein determining the attention weight information corresponding to the updated lexical sequence includes: The updated lexical sequence is divided into a first lexical segment and a second lexical segment. The first lexical segment includes at least one lexical corresponding to the text portion and the lexical preceding the at least one lexical, and the second lexical segment includes the lexical following the at least one lexical. The original position information of the second word segment in the word sequence is remapped to the position information in the updated word sequence; as well as Based on the original position information of the first lexical segment and the remapped position information of the second lexical segment, the attention weight information corresponding to the updated lexical sequence is determined.

3. The method according to claim 1, wherein the attention weight information includes bidirectional attention information and causal attention information, and wherein determining the attention weight information corresponding to the updated lexical sequence includes: The updated lexical sequence is divided into a first lexical segment and a second lexical segment. The first lexical segment includes at least one lexical corresponding to the text portion and the lexical preceding the at least one lexical, and the second lexical segment includes the lexical following the at least one lexical. Based on the division of the first lexical segment and the second lexical segment, the boundary between the bidirectional attention information and the causal attention information in the attention weight information is determined; as well as An attention mechanism is applied to the updated lexical sequence based on the determined boundaries to determine the bidirectional attention information and the causal attention information.

4. The method according to claim 3, wherein the bidirectional attention information indicates attention information within the first lexical segment, and the causal attention information indicates attention information of the second lexical segment on the first lexical segment.

5. The method according to claim 2 or 3, wherein the multimodal input includes an image portion, the text portion, and an action portion, and wherein the first lexical segment includes lexical units corresponding to the image portion and the text portion, and the second lexical segment includes lexical units corresponding to the action portion.

6. The method of claim 1, wherein the information associated with the text portion indicates the importance of each lexical unit corresponding to the text portion.

7. The method of claim 6, wherein determining the lexical type of each lexical in the lexical sequence corresponding to the text portion based on information associated with the text portion comprises: For each word unit corresponding to the aforementioned text portion, In response to the lexical element's importance being below a threshold, the lexical element type is determined to be an invalid lexical element, or When the importance of a word reaches a threshold, the word type is determined to be a valid word.

8. The method of claim 7, wherein the information associated with the text portion includes padding mask information indicating whether each word corresponding to the text portion is a padding word, and the method further comprises: In response to the fact that the word is a filler word, the importance of the word is determined to be lower than the threshold. or In response to determining that the lexical is not a filler lexical, the importance of the lexical is determined to have reached the threshold.

9. The method of claim 1, wherein the machine learning model comprises a visual language action model.

10. An apparatus for data processing, comprising: The lexical sequence acquisition module is configured to acquire the lexical sequence corresponding to the multimodal input of the machine learning model, wherein the multimodal input includes a text part and a non-text part; The lexical type determination module is configured to determine the lexical type of each lexical in the lexical sequence corresponding to the text portion based on information associated with the text portion, wherein the lexical type includes valid type and invalid type; The lexical sequence update module is configured to update the lexical sequence by pruning lexical elements that are determined to be invalid, thereby obtaining an updated lexical sequence. The weight information determination module is configured to determine the attention weight information corresponding to the updated lexical sequence, wherein the attention weight information includes the attention weight value of each lexical in the updated lexical sequence. as well as The lexical sequence processing module is configured to use the machine learning model to process the updated lexical sequence based on the attention weight information to obtain the model output corresponding to the multimodal input.

11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images