Method, apparatus, device, storage medium and program product for data processing

Abstract

According to embodiments of the present disclosure, a method, an apparatus, a device, a storage medium and a program product for data processing are provided. The method comprises: obtaining a token sequence corresponding to a multi-modal input for a machine learning model, the multi-modal input comprising an image part and a non-image part; determining, based on information associated with the image part, a token type of each token in the token sequence corresponding to the image part, the token type comprising a valid type and an invalid type; updating the token sequence by cropping off the tokens in the token sequence determined as the invalid type to obtain an updated token sequence; determining attention weight information corresponding to the updated token sequence, the attention weight information comprising an attention weight value of each token in the updated token sequence; and processing, by the machine learning model, the updated token sequence based on the attention weight information to obtain a model output corresponding to the multi-modal input.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, electronic devices, computer-readable storage media, and computer program products for data processing. Background Technology

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in various application environments. Vision-Language-Action (VLA) models are widely used in the field of robot control. The VLA model is a general-purpose robot neural network architecture that integrates perception, understanding, and motion control. It can simultaneously process multimodal model inputs and achieve multimodal collaborative decision-making. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for data processing is provided. The method includes: acquiring a word sequence corresponding to a multimodal input to a machine learning model, the multimodal input including an image portion and a non-image portion; determining the word type of each word in the word sequence corresponding to the image portion based on information associated with the image portion, the word type including valid types and invalid types; updating the word sequence by pruning words determined to be invalid types from the word sequence to obtain an updated word sequence; determining attention weight information corresponding to the updated word sequence, the attention weight information including the attention weight value of each word in the updated word sequence; and processing the updated word sequence using a machine learning model based on the attention weight information to obtain a model output corresponding to the multimodal input.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: a lexical sequence acquisition module configured to acquire a lexical sequence corresponding to a multimodal input to a machine learning model, the multimodal input including an image portion and a non-image portion; a lexical type determination module configured to determine the lexical type of each lexical in the lexical sequence corresponding to the image portion based on information associated with the image portion, the lexical type including valid type and invalid type; a lexical sequence update module configured to update the lexical sequence by pruning lexicals determined to be of invalid type, obtaining an updated lexical sequence; a weight information determination module configured to determine attention weight information corresponding to the updated lexical sequence, the attention weight information including the attention weight value of each lexical in the updated lexical sequence; and a lexical sequence processing module configured to process the updated lexical sequence using a machine learning model based on the attention weight information, obtaining a model output corresponding to the multimodal input.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, cause the processor to perform the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1A A block diagram of an application environment according to an exemplary implementation of this disclosure is shown; Figure 1B A schematic diagram of an example architecture of a machine learning model according to an embodiment of the present disclosure is shown; Figure 1C A schematic diagram of an example environment for model training and application according to some embodiments of the present disclosure is shown; Figure 2 A flowchart of a method for data processing according to some embodiments of the present disclosure is shown; Figure 3A Examples of lexical sequences according to some embodiments of this disclosure are shown; Figure 3B Examples of attention weight information according to some embodiments of this disclosure are shown; Figure 3C Examples of attention weight information according to other embodiments of this disclosure are shown; Figure 4 An exemplary structural block diagram of an apparatus for data processing according to some embodiments of the present disclosure is shown; and Figure 5 A block diagram of an electronic device that can implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0010] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0011] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0012] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0013] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0015] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0016] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0017] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0018] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0019] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0020] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0021] Figure 1A A block diagram 100A of an application environment according to an exemplary implementation of this disclosure is shown. For example... Figure 1A As shown, robot device 110 and user 120 can be located in environment 160, and user 120 can control robot device 110 to perform a variety of tasks.

[0022] Environment 160 may include, but is not limited to, one or more objects. For example, in a kitchen / dining environment, environment 160 may include, but is not limited to, various items, such as edible food, kitchen utensils, etc., or a combination of one or more of the above. In a wardrobe environment, environment 160 may include, but is not limited to, various types of clothing. Environment 160 may include, but is not limited to, one or more rooms. For example, in a home environment, environment 160 may include, but is not limited to, a living room, bedroom, study, kitchen, toilet, etc., or a combination of one or more of the above.

[0023] like Figure 1A As shown, the robot device 110 may include multiple parts. For example, the control unit 111 can serve as the control center of the robot device 110, and an application can be loaded into the control unit 111 to control the various parts of the robot device. The user 120 can use the interaction unit 112 to interact with the robot device 110, for example, by inputting control commands to the robot device 110 to perform desired tasks. The robot device 110 may include an arm 113 for performing actions such as grasping and releasing. For example, the arm 113 can grasp an object and move it to a desired position, and so on.

[0024] Alternatively and / or additionally, the robot device 110 may also include a data acquisition unit 114. Here, the data acquisition unit 114 may include various types, such as an image acquisition unit, a sound acquisition unit, etc. Alternatively and / or additionally, the robot device 110 may further include a sensing unit for detecting surrounding objects, for example, detecting the distance between the robot and surrounding objects based on laser light, etc. The robot device 110 may also include a drive unit 115, for example, the drive unit 115 can drive the robot device 110 to move along a desired path.

[0025] Environment 160 may include one or more acquisition units 130, ..., and 132. For example, one or more image acquisition devices may be deployed in a room to acquire images of the room from various angles. Environment 160 may include a control device 140, which can control one or more acquisition units 130, ..., and 132, etc., via a network (not shown). Alternatively and / or additionally, in a smart home environment, control device 140 can control various electrical devices in environment 160.

[0026] Alternatively and / or additionally, a machine learning model (e.g., machine learning model 150) may be provided to manage environment 160. It should be understood that, although... Figure 1A The machine learning model 150 is shown to be located within environment 160, but it can also be deployed in robotic device 110. Alternatively and / or additionally, the machine learning model 150 can also be located at a remote device outside environment 160, and the control device 140, robotic device 110, or other device can access the remote machine learning model 150 via a network.

[0027] Machine learning model 150 may include one or more models. If machine learning model 150 includes multiple models, these multiple models may include multiple types of models. Machine learning model 150 may include a Vision-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. Machine learning model 150 may, for example, include at least a Vision-Language Model (VLM) and an action model. The Vision-Language Model has the ability to process images and natural language. The Action Model can control the robotic device 110 to perform various actions. Machine learning model 150 may also include, for example, an image processing model, a text processing model, and so on.

[0028] In some embodiments, the machine learning model 150 may include a content-generating model capable of generating corresponding outputs based on model inputs. In some implementations, the machine learning model may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and may obtain corresponding model outputs based on the model inputs and prompts, thereby completing the task execution.

[0029] The machine learning model 150 may include an encoder module 151, a backbone network 152, and a decoder module 156. The encoder module 151 performs feature encoding on the model input (e.g., encoding an image or video into a feature space). The encoder module 151 may include, but is not limited to, a visual encoder, a speech encoder, and an action encoder. The backbone network 152 performs feature extraction on the encoded model input to obtain a feature representation 155 for performing a task. In some embodiments, the backbone network 152 may include one or more attention modules (e.g., transformer blocks) capable of performing attention-based processing on the encoded model input to extract feature representations. The decoder module 156 then decodes the output feature representation 155 to obtain the model output. The decoder module 156 may include at least an action decoder. For example, for a VLA model, the model output may be, for example, the next action the robot will perform. It should be understood that... Figure 1A The illustration shows an example application of machine learning model 150, which is not limited in this disclosure.

[0030] like Figure 1A As shown, user 120 can instruct robot device 110 to manipulate various objects in environment 160 and complete corresponding tasks. Here, objects can be various items in environment 160. For example, user 120 can instruct robot device 110 to find a specific object in environment 160. Or, user 120 can instruct robot device 110 to place the found object in a designated location, and so on. In some examples, user 120 can instruct robot device 110 to perform a task in environment 160. For example, in a life service scenario, robot device 110 can perform tasks such as indoor cleaning, item organization, home control, and voice interaction in environment 160. In an industrial manufacturing scenario, robot device 110 can perform tasks such as industrial inspection and production operations in environment 160. However, these are merely examples, and this disclosure does not limit the scope of the invention.

[0031] Figure 1BA schematic diagram of an example architecture of a machine learning model 150 according to an embodiment of the present disclosure is shown. As mentioned above, the machine learning model 150 according to an embodiment of the present disclosure may include a Visual-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. Given visual input 153a (e.g., an image or video, etc.) and language input 153b (e.g., text commands or voice commands) in the environment space (e.g., a physical environment or a virtual environment) in which the robot device 110 is located, the VLA model can output action commands 157. The action commands 157 can be used to control the robot device 110 to perform the task requested by the visual input 153a or the language input 153b.

[0032] In some embodiments, such as Figure 1B As shown, the VLA model may include a visual encoder 151a, a language encoder 151b, a backbone network 152, and an action decoder 156a. The visual encoder 151a performs feature encoding on the visual input 153a (e.g., encoding an image or video into a feature space) to obtain a visual representation 154a. The language encoder performs feature encoding on the language input 153b, such as text instructions or speech instructions, to obtain a language representation 154b. The backbone network 152 performs feature extraction based on the visual representation 154a and the language representation 154b to obtain a feature representation 155 for action decision-making. Then, the action decoder 156a performs feature decoding on the feature representation 155 to obtain an action instruction 157 (or a sequence of action instructions).

[0033] The motion command 157 here can be implemented in several ways. As an example, motion command 157 can indicate the relative change in the end-effector posture of the robotic arm (i.e., robot device 110). Based on this relative change using the inverse dynamics model of the robotic arm, motion parameters such as angles, velocities, and accelerations for multiple degrees of freedom of the robotic arm (e.g., joints or end-effector grippers) can be determined. Alternatively, motion command 157 can also indicate the motion parameters for multiple degrees of freedom of the robot device 110. In other words, motion decoder 156a can also directly output the motion parameters for multiple degrees of freedom of a specific robot device 110, allowing the robot device 110 controller to directly drive its movement based on these motion parameters.

[0034] In some embodiments, the VLA model may further include a motion encoder 151c. The model input of the VLA model may also include motion input 153c, which may include, but is not limited to, motion commands 157 provided to the robot device 110 in the previous control cycle or motion sensing results of the robot device 110 in the previous control cycle. The motion encoder 151c may perform feature encoding on the motion input 153c to obtain a motion representation 154c. The backbone network 152 may perform feature extraction based on the visual representation 154a, language representation 154b, and motion representation 154c to obtain a feature representation 155. Of course, the model input of the VLA model described above is merely exemplary. In practical applications, the model input of the VLA model may also include, for example, depth maps, point clouds, force feedback information, or haptic feedback information. In this case, the VLA model may also include, for example, a depth map encoder, a point cloud encoder, a force encoder, or a haptic encoder.

[0035] In some embodiments, the backbone network 152 can perform multimodal fusion and feature extraction on the visual representation 154a, language representation 154b, and action representation 154c based on a cross-attention mechanism to obtain feature representation 155. In some embodiments, the visual encoder 151a, language encoder 151b, and backbone network 152 can be implemented based on a pre-trained visual-language model (VLM). In other words, a VLA model can be constructed by combining the action encoder 151c and action decoder 156a on the basis of a pre-trained VLM. Of course, the backbone network 152 described above is only exemplary. In practical applications, any other suitable multimodal model can be used to construct the backbone network 152. The embodiments of this disclosure do not limit this.

[0036] Figure 1C A schematic diagram of an example environment 100C for model training and application according to some embodiments of the present disclosure is shown. Figure 1C The example environment 100C illustrates three distinct phases of the machine learning model 150: a pre-training phase 172, a fine-tuning phase 174, and an application phase 176. A testing phase, not shown in the figure, may also occur after the pre-training phase 172 or the fine-tuning phase 174.

[0037] Example environment 100C involves a model pre-training system 192, a model fine-tuning system 194, and a model application system 196. In the pre-training phase 172, the model pre-training system 192 is configured to perform pre-training of a machine learning model 150 using a pre-training dataset 182. At the start of pre-training, the individual components of the machine learning model 150 may have initial parameter values. The pre-training process involves updating the parameter values ​​of the machine learning model 150 to desired values ​​based on data from the pre-training dataset 182. The pre-training task is used to assist in updating the parameters of the machine learning model 150.

[0038] During the pre-training phase 172, the machine learning model 150 can learn strong generalization capabilities using a pre-training dataset 182 containing a large amount of data. After pre-training, the parameter values ​​of the machine learning model 150 have been updated to include the pre-trained parameter values. In some embodiments, the machine learning model 150 may include a Visual Language Action (VLA) model, and the pre-training dataset 182 used to train the machine learning model 150 may include multiple pre-training samples, each of which may include sample language input, sample visual input, and sample action instructions. The pre-trained machine learning model 120 can generate corresponding action instructions relatively accurately based on the received language input and visual input.

[0039] The pre-trained machine learning model 150 can be provided to the fine-tuning stage 174, where the model fine-tuning system 194 fine-tunes it for different downstream tasks. In the fine-tuning stage 174, the parameter values ​​of the machine learning model 150 are further adjusted using the training dataset 184. During fine-tuning, the parameters of the machine learning model 150 are also updated and adjusted using the corresponding training algorithm. Since the model has learned a great deal from the training data in the pre-training stage, a downstream task model that meets expectations can be obtained using only a small amount of training data in the fine-tuning stage 174. The training samples included in the training dataset 184 can be of the same type as the pre-training samples included in the pre-training dataset 182; that is, the training dataset 184 can include multiple training samples, each of which can include sample language input, sample visual input, and sample action instructions. In some cases, the pre-training dataset 182 can include multiple pre-training samples for a general scenario, while the training dataset 184 can include multiple training samples for a specific scenario or specific task.

[0040] In some embodiments, a testing phase may be included after the fine-tuning phase 174, where the performance of the machine learning model 150 can be further tested using a test dataset. The dataset used in the testing phase is of the same type as that used in the fine-tuning phase.

[0041] In application phase 176, the obtained machine learning model 150, with trained parameter values, can be provided to the model application system 196 for use. In application phase 176, the machine learning model 150 can be used to process corresponding model inputs 186 in the real-world scenario and provide corresponding model outputs 188. As an example only, model inputs 186 may include verbal input and visual input, and model outputs 188 may include corresponding action instructions.

[0042] exist Figure 1C In this system, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0043] The server-side device can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc. It is understood that the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on the same electronic device or on different electronic devices; this disclosure does not limit this.

[0044] It should be understood that Figure 1C The components and arrangements shown in the example environment 100C are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 may be integrated in the same system or device. For example, at least the model pre-training system 192 and the model fine-tuning system 194 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect.

[0045] As mentioned earlier, VLA models are widely used in robot control, enabling simultaneous processing of multimodal model inputs and achieving multimodal collaborative decision-making. Multimodal model inputs typically include at least multiple images, comprising a portion of the images actually captured or acquired, and another portion obtained by processing (e.g., mirroring) this portion. Traditionally, an automatic image addition mechanism is used to generate an image with the opposite perspective from the actually captured or acquired portion. For example, if the captured image is a left-hand perspective image, an automatic image addition mechanism can be used to add a right-hand perspective image based on the left-hand perspective image. The other portion of the image, obtained by processing the portion of the image, does not contain new valid information and is therefore usually considered meaningless. VLA models typically employ a masking mechanism to attempt to ignore the meaningless other portion of the image, but this meaningless other portion is still provided to the VLA model as part of the model input. Since the model input includes both meaningful and meaningless parts of the image, the meaningless part is also included in the model's processing flow, consuming GPU memory. This not only depletes the model's computational resources but also affects its computational efficiency. Currently, there is no specific mechanism for filtering out invalid images.

[0046] In view of this, according to embodiments of the present disclosure, an improved data processing scheme is provided. According to the scheme of the embodiments of the present disclosure, a word sequence corresponding to a multimodal input for a machine learning model is obtained. The multimodal input includes an image portion and a non-image portion. Based on information associated with the image portion, the word type of each word in the word sequence corresponding to the image portion is determined. The word type includes valid types and invalid types. The word sequence is updated by pruning words determined to be invalid types, resulting in an updated word sequence. Attention weight information corresponding to the updated word sequence is determined, including the attention weight value of each word in the updated word sequence. Using a machine learning model, the updated word sequence is processed based on the attention weight information to obtain the model output corresponding to the multimodal input.

[0047] In this way, the lexical type of each lexical unit in the lexical sequence corresponding to the image portion can be determined, and the lexical sequence can be updated by pruning lexical units determined to be invalid types. This can effectively reduce the number of lexical units included in the lexical sequence. Furthermore, determining attention weight information based on such an updated lexical sequence can reduce the amount of information contained in the attention weight information. Determining the final model output based on such attention weight information can reduce the resource consumption of model computation and improve computational efficiency.

[0048] The following description continues with reference to the accompanying drawings, illustrating some exemplary embodiments of this disclosure. The data processing procedures involved in this disclosure can be implemented to generate videos for training machine learning models that generate images or videos containing single objects.

[0049] Figure 2 A flowchart of a method 200 for data processing according to some embodiments of the present disclosure is shown. Method 200 can be implemented in... Figure 1C The model pre-training system 192 / model fine-tuning system 194, or an electronic device on which the model pre-training system 192 / model fine-tuning system 194 is deployed, will be described exemplarily herein with the method 200 implemented at the model fine-tuning system 194 as an example. Reference will be made to Figures 1A to 1C Let's describe method 200.

[0050] In box 210, the model fine-tuning system 194 acquires the token sequence corresponding to the multimodal input for the machine learning model. The multimodal input includes both image and non-image components. The machine learning model here can be, for example, the aforementioned machine learning model 150, which may include, for example, a visual language action (VLA) model. A token is the basic unit by which a machine learning model understands information. A token sequence is a sequence containing multiple tokens.

[0051] In some embodiments, the multimodal input for a machine learning model may include at least an image portion (e.g., the visual input 153a mentioned above) and a text portion (e.g., the language input 153b mentioned above). The image portion may include at least one image or at least one video frame. The text portion may include at least one text unit. The lexical sequence corresponding to such a multimodal input may include at least a plurality of lexical units corresponding to the image portion and a plurality of lexical units corresponding to the text portion. In some embodiments, the multimodal input may further include an action portion (e.g., the language input 153c mentioned above). In this case, the lexical sequence may further include a plurality of lexical units corresponding to the action portion.

[0052] In box 220, the model fine-tuning system 194 determines the lexical type of each lexical in the lexical sequence corresponding to the image part based on information associated with the image part. The lexical type includes valid type and invalid type.

[0053] In some embodiments, the image portion may include at least one image, each image corresponding to at least one lexical in a lexical sequence. In some examples, the at least one image may include multiple images captured from different perspectives of the robot. By way of example only, the at least one image may include an image captured from the robot's left-hand perspective and an image captured from the robot's right-hand perspective. It is understood that this is merely an example, and in real-world scenarios, images from more perspectives may be included, such as images captured from the robot's frontal perspective, images captured from the robot's rear perspective, and so on.

[0054] As mentioned above, in some embodiments, at least one image may include a portion of the image actually captured and another portion of the image obtained by processing (e.g., mirroring) this portion. As an example only, an image captured from the robot's left-hand perspective may be an actually captured image, while an image captured from the robot's right-hand perspective may be an image obtained by copying, mirroring, or otherwise processing the image captured from the robot's left-hand perspective. In this case, the image captured from the robot's left-hand perspective can be considered a meaningful image, and the image captured from the robot's right-hand perspective can be considered a meaningless image.

[0055] The model fine-tuning system 194 can, for example, determine the image type of at least one image based on information associated with image portions, where the image type can include valid and invalid types. It is understood that images corresponding to valid image types are meaningful to the machine learning model, while images corresponding to invalid image types are meaningless. In some examples, the image type of a genuinely acquired image can be determined as a valid type, while images obtained by processing a genuinely acquired image can be determined as invalid types. Of course, this is just an example; invalid image types could also include, for example, damaged images, blurred images, etc.

[0056] The model fine-tuning system 194 can determine the image type of an image in any appropriate way according to the actual scene requirements and user needs. It is understandable that the image type of the same image may differ when determined under different conditions. For example, if determination condition A indicates that an image from a specific acquisition viewpoint is determined as an invalid type, then even if an image from a non-specific acquisition viewpoint is incomplete, that image will still be determined as a valid type under determination condition A. If determination condition B indicates that an image with quality problems such as damage, blurriness, or excessive noise is determined as an invalid type, then that image will be determined as an invalid type under determination condition B.

[0057] The image type of an image can be associated with the lexical type of at least one lexical unit corresponding to that image. Specifically, if the image type of an image is a valid type, the model fine-tuning system 194 can determine that the lexical type of at least one lexical unit in the lexical sequence corresponding to that image is a valid type. If the image type of an image is an invalid type, the model fine-tuning system 194 can determine that the lexical type of at least one lexical unit in the lexical sequence corresponding to that image is an invalid type.

[0058] Regarding the specific method of determining the image type, in some examples, the model fine-tuning system 194 can directly use a trained classification model to determine the image type of at least one image based on information associated with the image portion. In some examples, the model fine-tuning system 194 can determine the image type of at least one image based on predetermined conditions, according to the content included in the information associated with the image portion.

[0059] In some examples, the information associated with an image portion may include viewpoint information corresponding to each image in at least one image. Viewpoint information may, for example, indicate the acquisition viewpoint of the corresponding image. The model fine-tuning system 194 may, for example, determine whether the viewpoint information of each image in at least one image is predetermined viewpoint information based on the information associated with the image portion. The predetermined viewpoint information may indicate one or more acquisition viewpoints. The model fine-tuning system 194 may, for example, determine that the image type of an image is invalid if it determines that the viewpoint information of an image is predetermined viewpoint information (i.e., the acquisition viewpoint corresponding to the image belongs to one or more acquisition viewpoints indicated by the predetermined viewpoint information), and may determine that the image type of an image is valid if it determines that the viewpoint information of an image is not predetermined viewpoint information (i.e., the acquisition viewpoint corresponding to the image does not belong to one or more acquisition viewpoints indicated by the predetermined viewpoint information). For example, if the predetermined viewpoint information indicates a robot's right-hand viewpoint, the viewpoint information of image A indicates that the acquisition viewpoint of image A is the robot's right-hand viewpoint, and the viewpoint information of image B indicates that the acquisition viewpoint of image B is the robot's left-hand viewpoint, then the model fine-tuning system 194 may determine that the image type of image A is invalid and the image type of image B is valid.

[0060] In some examples, the information associated with an image portion may include device information of the acquisition device corresponding to one or more images in at least one image. It is understood that not every image in at least one image may have been acquired by the acquisition device; one or more images acquired by the acquisition device may have the corresponding device information of the acquisition device, while images not acquired by the acquisition device do not have the corresponding device information of the acquisition device. The acquisition device may be, for example, a camera, and the device information may indicate, for example, the name, number, etc., of the corresponding acquisition device. The model fine-tuning system 194 may, for example, determine whether each image in at least one image has the corresponding device information of the acquisition device based on the information associated with the image portion. The model fine-tuning system 194 may, for example, determine that the image type of an image is a valid type in response to determining that an image has the corresponding device information, and may determine that the image type of an image is an invalid type in response to determining that an image does not have the corresponding device information.

[0061] In some examples, the information associated with an image portion may include feature information of each image in at least one image. The feature information of each image indicates at least one of the pose information of an object in the corresponding image and environmental information in the image. For each image in at least one image, the model fine-tuning system 194 may determine the viewpoint corresponding to that image based on the feature information of that image in any suitable manner. The model fine-tuning system 194 may determine whether the viewpoint corresponding to each image in the at least one image is a predetermined viewpoint, which may include one or more viewpoints. For example, in response to determining that the viewpoint of an image is a predetermined viewpoint (or that the viewpoint of the image belongs to a predetermined viewpoint), the model fine-tuning system 194 may determine that the image type of that image is an invalid type, and in response to determining that the viewpoint of an image is not a predetermined viewpoint (or that the viewpoint of the image does not belong to a predetermined viewpoint), the model fine-tuning system 194 may determine that the image type of that image is a valid type.

[0062] In some examples, to maximize the accuracy of image type determination, the information related to the image portion may include one or more of the following: viewpoint information corresponding to each image in at least one of the at least one images, device information of the acquisition device corresponding to one or more images in at least one image, and feature information of each image in at least one image. It is understood that the information related to the image portion may also include other information, such as image identification information, object identification information in the image, etc., and this disclosure does not limit this. When the information related to the image portion includes multiple types of information, the model fine-tuning system 194 can use various methods to determine the image type.

[0063] In box 230, the model fine-tuning system 194 updates the lexical sequence by pruning lexical segments that are determined to be invalid, thus obtaining the updated lexical sequence.

[0064] Taking a multimodal input that includes at least an image and a text portion as an example, the word sequence corresponding to the multimodal input can include words that are determined to be invalid, both those corresponding to the image portion and those corresponding to the text portion. The method for determining the word type of words corresponding to the image portion can differ from the method for determining the word type of words corresponding to the text portion. The invalid word types pruned by the model fine-tuning system 194 can include words corresponding to either the text portion or the image portion. The following explanation uses only the example of pruned words determined to be invalid corresponding to the image portion for illustration.

[0065] refer to Figure 3A , Figure 3A Example 300A of a lexical sequence according to some embodiments of the present disclosure is shown. For example... Figure 3A As shown in Figure 302, the lexical sequence (represented as multiple squares in the figure, each square representing a lexical) can include 978 lexical units. For example only, units 1-768 in this sequence can correspond to image portions, units 769-968 to text portions, and units 969-978 to action portions. In the following description, unless otherwise specified, each lexical unit in the sequence will be used as an example.

[0066] In the 1st to 768th lexical units of the lexical unit sequence, if the 513th to 768th lexical units are invalid (marked in black in the diagram), the model fine-tuning system 194 can update the lexical unit sequence by pruning these lexical units (i.e., the 513th to 768th lexical units) to obtain an updated lexical unit sequence. In this case, the updated lexical unit sequence includes 722 lexical units, which is significantly less than the number of lexical units in the original lexical unit sequence. This reduces the computational resources required for the machine learning model to process the lexical unit sequence, improving computational efficiency.

[0067] It should be noted that those skilled in the art will understand that this disclosure does not limit the individual lexical units in the lexical sequence. For example, in the lexical sequence shown in box 302, the 1st to 512th lexical units can be lexical units corresponding to the action part, the 513th to 768th lexical units and the 969th to 978th ​​lexical units can both be lexical units corresponding to the image part, and the 769th to 968th lexical units can be lexical units corresponding to the text part.

[0068] In box 240, the model fine-tuning system 194 determines the attention weight information corresponding to the updated lexical sequence, which includes the attention weight value of each lexical in the updated lexical sequence.

[0069] Each word in a word sequence has corresponding positional information. The model fine-tuning system 194, for example, can obtain the positional encoding identifier (position_id) of each word in the word sequence. This positional encoding identifier indicates the position of each word in the word sequence. (Reference) Figure 3A The word sequence shown in box 302 comprises 978 words, as illustrated in the figure. The position codes of these 978 words can be sequentially assigned as 1, 2, 3, ..., 977, 978, which represent the position of the corresponding word in the word sequence. For example, if the position code of a word is 2, then that word is the second word in the word sequence.

[0070] In some embodiments, the model fine-tuning system 194 can divide a first lexical segment and a second lexical segment into an updated lexical sequence. The first lexical segment includes at least one lexical corresponding to the image portion and a lexical preceding the at least one lexical, and the second lexical segment includes lexical following the at least one lexical. In some examples, the order of lexical segments in the lexical sequence can be: lexical corresponding to the image portion - lexical corresponding to the text portion - lexical corresponding to the action portion. That is, the lexical corresponding to the image portion is placed before the lexical corresponding to the non-image portion (i.e., the lexical corresponding to the text portion and the lexical corresponding to the action portion). In this case, the first lexical segment can include the lexical corresponding to the image portion, and the second lexical segment can include the lexical corresponding to the non-image portion.

[0071] Continue to refer to Figure 3A Taking the 513th to 768th lexical units in the lexical unit sequence shown in box 302 as invalid lexical units, the terminal device 110 can obtain the updated lexical unit sequence by cutting out the lexical units that are determined to be invalid (i.e., the 513th to 768th lexical units).

[0072] Box 304 illustrates an example of an updated lexical sequence. As shown in Box 304, the lexical sequence shown in Box 304 can include a total of 722 lexical units. The positional codes of the first 512 lexical units in this sequence can be 1, 2, 3, ..., 512, and the positional codes of the last 210 lexical units can be 769, 767, ..., 978, respectively. Taking the lexical sequence shown in Box 302, where multiple lexical units with positional codes 1-768 correspond to the image portion, multiple lexical units with positional codes 769-968 correspond to the text portion, and multiple lexical units with positional codes 969-978 correspond to the action portion as an example, the first lexical segment in the lexical sequence shown in Box 304 can include multiple lexical units with positional codes 1-512, and the second lexical segment can include multiple lexical units with positional codes 769-968.

[0073] At this point, there are words with discontinuous positional codes in the word sequence. For example, a word with positional code 512 is followed by a word with positional code 769. This affects the machine learning model's perception of time steps and action order, increasing the probability of data processing errors. To address this issue, the model fine-tuning system 194 can remap the original positional information of the second word segment in the word sequence to its positional information in the updated word sequence. For example, the model fine-tuning system 194 can remap the original positional information of the second word segment in the word sequence to its positional information in the updated word sequence by remapping the positional code identifiers of each word in the second word segment. After remapping, the positional code identifier of the first word in the second word segment is continuous with the positional code identifier of the last word in the first word segment.

[0074] Continue to refer to Figure 3A The model fine-tuning system 194 can remap the positional encoding identifiers of each word in the second word segment of the word sequence shown in box 304 (i.e., multiple words with positional encoding identifiers of 769-968, which are marked as squares filled with diagonal lines in the figure) to obtain the word sequence shown in box 306. As shown in box 306, the positional encoding identifiers of the second word segment after remapping can be 513-722 in sequence.

[0075] In some embodiments, the model fine-tuning system 194 can determine the attention weight information corresponding to the updated lexical sequence based on the original position information of the first lexical segment and the remapped position information of the second lexical segment. As an example only, the attention weight information can be represented as an attention mask matrix, whose rows and columns can be, for example, multiple positional identifiers for multiple lexical segments. The attention mask matrix is ​​a Boolean matrix that controls which positions in the Transformer can see each other, and it can be used to maintain the causality or modal isolation of the sequence.

[0076] refer to Figure 3B and Figure 3C , Figure 3B and Figure 3C Examples 300B and 300C illustrate attention weight information according to some embodiments of this disclosure. In examples 300B and 300C, the symbol T indicates that the machine learning model needs to pay attention to a word at that position, and the symbol F indicates that the machine learning model does not need to pay attention to a word at that position. The attention weight of each word marked F can be set to infinity to shield the propagation of invalid information.

[0077] Example 300B shows an unupdated sequence of lexical terms (e.g.) Figure 3A The attention weight information corresponding to the word sequence shown in the middle box 302. Taking the multiple words with position encoding labels 1-768 in the word sequence corresponding to the multimodal input image portion, and the multiple words with position encoding labels 513-768 having invalid word types as an example, in the attention weight information shown in Example 300B, although the attention weights of the multiple words corresponding to rows 513-768 and columns 513-768 (that is, the multiple words with position encoding labels 513-768) enclosed by the dashed lines can be set to infinity, these words still participate in the linear transformation and attention calculation of the machine learning model. This not only consumes computing resources but also affects the efficiency of model calculation.

[0078] Example 300C shows an updated sequence of lexical segments whose positional information of the second lexical segment has been remapped (e.g., Figure 3A The attention weight information corresponds to the lexical sequence shown in the middle box 306. It can be seen that compared with the lexical sequence shown in Example 300B, this attention weight information is more concise. Without affecting the lexical sequences that the machine learning model needs to focus on, it can reduce the computational cost of the machine learning model and improve its computational efficiency.

[0079] In some embodiments, the model fine-tuning system 194 may divide the updated lexical sequence into a third lexical segment (also referred to as the prefix portion of the lexical sequence) and a fourth lexical segment (also referred to as the suffix portion of the lexical sequence). The third lexical segment may, for example, include at least one lexical corresponding to an image portion and at least one lexical corresponding to a text portion, and the fourth lexical segment may, for example, include at least one lexical corresponding to an action portion.

[0080] Attention weight information includes bidirectional attention information and causal attention information. Bidirectional attention information indicates the attention information within the third lexical segment, while causal attention information indicates the attention information of the fourth lexical segment on the third lexical segment. The model fine-tuning system 194 can determine the boundary between bidirectional attention information and causal attention information in the attention weight information based on the division of the third and fourth lexical segments.

[0081] Specifically, the boundary between bidirectional attention information and causal attention information in attention weight information can be determined based on the number of lexical units included in the third lexical unit fragment. For example... Figure 3C As shown, the first lexical segment may include 712 lexical units with positional encoding identifiers from 1 to 712. The boundary between bidirectional attention information and causal attention information may lie between the 712th and 713th lexical units (e.g., the bold solid line shown in the figure). The model fine-tuning system 194 may apply an attention mechanism to the updated lexical sequence based on the determined boundary to determine the bidirectional attention information and the causal attention information.

[0082] In box 250, the model fine-tuning system 194 uses a machine learning model to process the updated word sequence based on attention weight information to obtain the model output corresponding to the multimodal input.

[0083] Machine learning models can, for example, determine the attention weights between each word in the updated word sequence based on attention weight information, and then process multiple words in the updated word sequence based on the determined attention weights to obtain the model output corresponding to the multimodal input.

[0084] The model output can be the predicted action command corresponding to the multimodal input. During the model training phase, the samples used to train the machine learning model can include multimodal input and the corresponding sample action commands. The model fine-tuning system 194 can calculate the difference (e.g., mean squared error) between the sample action commands in the samples and the predicted action commands in the model output to train the machine learning model. Since invalid terms have been pruned, the machine learning model does not need to use additional masking operations to block invalid information, which can improve the training efficiency while ensuring the training effect of the model.

[0085] In some examples, the model fine-tuning system 194 and the machine learning model can perform dimensionality calibration on the data involved in the data processing flow. For instance, the model fine-tuning system 194 and the machine learning model can perform dimensionality calibration on the updated lexical sequence to ensure that the dimension of the final model output is consistent with the dimension of the model output obtained by processing the unupdated lexical sequence. This avoids changes in the dimension of the model output and prevents subsequent processing of the model output from being affected.

[0086] Therefore, the lexical type of each lexical unit in the lexical sequence corresponding to the image portion can be determined, and the lexical sequence can be updated by pruning lexical units determined to be invalid types. This can effectively reduce the number of lexical units included in the lexical sequence. Furthermore, dynamically determining attention weight information based on such an updated lexical sequence can reduce the amount of information in the attention weight information without affecting the tokens that the model needs to focus on. Determining the final model output based on such attention weight information can reduce the resource consumption of model computation and improve computational efficiency. Especially in the model training process, this token processing method and attention weight information determination method can greatly improve the training speed of the model.

[0087] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 An exemplary structural block diagram of an apparatus 400 for data processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a model pre-training system 192 / model fine-tuning system 194. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0088] like Figure 4 As shown, the device 400 includes: a lexical sequence acquisition module 410, configured to acquire a lexical sequence corresponding to a multimodal input for a machine learning model, the multimodal input including an image portion and a non-image portion; a lexical type determination module 420, configured to determine the lexical type of each lexical in the lexical sequence corresponding to the image portion based on information associated with the image portion, the lexical type including valid type and invalid type; a lexical sequence update module 430, configured to update the lexical sequence by pruning lexicals determined to be invalid types in the lexical sequence to obtain an updated lexical sequence; a weight information determination module 440, configured to determine the attention weight information corresponding to the updated lexical sequence, the attention weight information including the attention weight value of each lexical in the updated lexical sequence; and a lexical sequence processing module 450, configured to use a machine learning model to process the updated lexical sequence based on the attention weight information to obtain the model output corresponding to the multimodal input.

[0089] In some embodiments, the weight information determination module 440 is further configured to: divide a first word segment and a second word segment in the updated word sequence, wherein the first word segment includes at least one word corresponding to the image portion and a word preceding the at least one word, and the second word segment includes a word following the at least one word; remap the original position information of the second word segment in the word sequence to position information in the updated word sequence; and determine the attention weight information corresponding to the updated word sequence based on the original position information of the first word segment and the remapped position information of the second word segment.

[0090] In some embodiments, the non-image portion of the multimodal input includes a text portion and an action portion, wherein a first lexical segment includes lexical units corresponding to the image portion, and a second lexical segment includes lexical units corresponding to the non-image portion.

[0091] In some embodiments, the non-image portion includes a text portion and an action portion, the attention weight information includes bidirectional attention information and causal attention information, and the weight information determination module 440 is further configured to: divide the updated lexical sequence into a third lexical segment and a fourth lexical segment, the third lexical segment including at least one lexical corresponding to the image portion and at least one lexical corresponding to the text portion, and the fourth lexical segment including at least one lexical corresponding to the action portion; determine the boundary between bidirectional attention information and causal attention information in the attention weight information based on the division of the third lexical segment and the fourth lexical segment; and apply an attention mechanism to the updated lexical sequence based on the determined boundary to determine bidirectional attention information and causal attention information.

[0092] In some embodiments, bidirectional attention information indicates attention information within the third lexical segment, and causal attention information indicates attention information of the fourth lexical segment on the third lexical segment.

[0093] In some embodiments, the image portion includes at least one image, and the lexical type determination module 420 is further configured to: determine the image type of each of the at least one image based on information associated with the image portion, the image type including a valid type and an invalid type; for each of the at least one image, in response to the image type of the image being determined to be a valid type, determine the lexical type of at least one lexical in the lexical sequence corresponding to the image as a valid type, or in response to the image type of the image being determined to be an invalid type, determine the lexical type of at least one lexical in the lexical sequence corresponding to the image as an invalid type.

[0094] In some embodiments, at least one image includes multiple images captured from different perspectives of the robot.

[0095] In some embodiments, the information associated with the image portion includes at least one of the following: viewpoint information corresponding to each image in at least one image, and device information of the corresponding acquisition device, and the lexical type determination module 420 is further configured to: for each image in at least one image, determine that the image type of the image is an invalid type based on at least one of the following: the viewpoint information corresponding to the image is predetermined viewpoint information, or the image does not have device information.

[0096] In some embodiments, the information associated with the image portion includes feature information of each image in at least one image, the feature information indicating at least one of pose information of an object in the corresponding image and environmental information in the image, and the lexical type determination module 420 is further configured to: determine the viewpoint corresponding to the image based on the feature information of each image in at least one image; and determine the image type of the image as invalid in response to the viewpoint corresponding to the image being a predetermined viewpoint.

[0097] In some embodiments, the lexical type determination module 420 is further configured to: determine the image type of at least one image based on information associated with the image portion using a trained classification model.

[0098] In some embodiments, the machine learning model includes a visual language action model.

[0099] The units and / or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0100] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices can, for example, be used to implement... Figure 1C The model pre-training system 192 and the model fine-tuning system 194 are mentioned.

[0101] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1C The model pre-training system 192 / model fine-tuning system 194.

[0102] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors 510 or processing units, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0103] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 500.

[0104] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0105] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0106] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0107] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0108] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for data processing, comprising: Obtain the word sequence corresponding to the multimodal input for the machine learning model, wherein the multimodal input includes an image part and a non-image part; Based on information associated with the image portion, the lexical type of each lexical in the lexical sequence corresponding to the image portion is determined, and the lexical type includes valid type and invalid type; The word sequence is updated by pruning words that are determined to be invalid, resulting in an updated word sequence. Determine the attention weight information corresponding to the updated lexical sequence, wherein the attention weight information includes the attention weight value of each lexical in the updated lexical sequence; as well as Using the machine learning model, the updated lexical sequence is processed based on the attention weight information to obtain the model output corresponding to the multimodal input.

2. The method according to claim 1, wherein determining the attention weight information corresponding to the updated lexical sequence includes: The updated lexical sequence is divided into a first lexical segment and a second lexical segment. The first lexical segment includes at least one lexical corresponding to the image portion and a lexical preceding the at least one lexical, and the second lexical segment includes a lexical following the at least one lexical. The original position information of the second lexical segment in the lexical sequence is remapped to the position information in the updated lexical sequence; as well as Based on the original position information of the first lexical segment and the remapped position information of the second lexical segment, the attention weight information corresponding to the updated lexical sequence is determined.

3. The method according to claim 2, wherein the non-image portion of the multimodal input includes a text portion and an action portion, and wherein the first lexical segment includes lexical units corresponding to the image portion, and the second lexical segment includes lexical units corresponding to the non-image portion.

4. The method of claim 1, wherein the non-image portion comprises a text portion and an action portion, the attention weight information comprises bidirectional attention information and causal attention information, and wherein determining the attention weight information corresponding to the updated lexical sequence comprises: The updated lexical sequence is divided into a third lexical segment and a fourth lexical segment. The third lexical segment includes at least one lexical corresponding to the image part and at least one lexical corresponding to the text part, and the fourth lexical segment includes at least one lexical corresponding to the action part. Based on the division of the third and fourth word segments, the boundary between the bidirectional attention information and the causal attention information in the attention weight information is determined; as well as An attention mechanism is applied to the updated lexical sequence based on the determined boundaries to determine the bidirectional attention information and the causal attention information.

5. The method according to claim 4, wherein the bidirectional attention information indicates attention information within the third lexical segment, and the causal attention information indicates attention information of the fourth lexical segment on the third lexical segment.

6. The method of claim 1, wherein the image portion comprises at least one image, and determining the lexical type of each lexical in the lexical sequence corresponding to the image portion based on information associated with the image portion comprises: Based on information associated with the image portion, the image type of each of the at least one image is determined, the image type including valid type and invalid type; For each of the at least one images, In response to the image type being determined to be a valid type, the word type of at least one word in the word sequence corresponding to the image is determined to be a valid type, or In response to the image type being determined to be invalid, the lexical type of at least one lexical in the lexical sequence corresponding to the image is determined to be invalid.

7. The method of claim 6, wherein the at least one image comprises multiple images acquired from different perspectives of the robot.

8. The method of claim 7, wherein the information associated with the image portion includes at least one of the following: viewpoint information corresponding to each image in the at least one image, and device information of the corresponding acquisition device, and Determining the image type of each of the at least one image based on information associated with the image portion includes: For each of the at least one images, the image type of that image is determined to be invalid based on at least one of the following: The viewpoint information corresponding to this image is the predetermined viewpoint information, or... The image does not contain device information.

9. The method of claim 7, wherein the information associated with the image portion includes feature information of each of the at least one image, the feature information indicating at least one of pose information of an object in the corresponding image and environmental information in the image, and Determining the image type of each of the at least one image based on information associated with the image portion includes: For each of the at least one images, the viewpoint corresponding to that image is determined based on the feature information of that image; as well as In response to the fact that the viewpoint corresponding to the image is a predetermined viewpoint, the image type of the image is determined to be invalid.

10. The method of claim 6, wherein determining the image type of each of the at least one image based on information associated with the image portion comprises: Using a trained classification model, the image type of each of the at least one image is determined based on information associated with the image portion.

11. The method of claim 1, wherein the machine learning model comprises a visual language action model.

12. An apparatus for data processing, comprising: The lexical sequence acquisition module is configured to acquire the lexical sequence corresponding to the multimodal input of the machine learning model, wherein the multimodal input includes an image part and a non-image part; The lexical type determination module is configured to determine the lexical type of each lexical in the lexical sequence corresponding to the image portion based on information associated with the image portion, wherein the lexical type includes valid type and invalid type; The lexical sequence update module is configured to update the lexical sequence by pruning lexical elements that are determined to be invalid, thereby obtaining an updated lexical sequence. The weight information determination module is configured to determine the attention weight information corresponding to the updated lexical sequence, wherein the attention weight information includes the attention weight value of each lexical in the updated lexical sequence. as well as The lexical sequence processing module is configured to use the machine learning model to process the updated lexical sequence based on the attention weight information to obtain the model output corresponding to the multimodal input.

13. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processor.

14. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 11.

Images