Visual data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium. The method comprises the following steps: respectively extracting a plurality of visual lexical elements and at least one text lexical element from visual data and a text instruction aiming at the visual data, wherein the text instruction indicates a processing task related to the visual data; utilizing a visual module in the trained multi-modal model to extract visual features from the plurality of visual lexical elements; extracting text features from the at least one text lexical element by using a text module in the multi-modal model; based on the visual features and the text features, respectively executing a self-attention mechanism on the visual features and the text features to determine visual weights corresponding to a plurality of visual lexical elements and a text weight corresponding to at least one text lexical element; and determining a result for the processing task based on the plurality of visual lexical elements and the visual weight, and the at least one text lexical element and the text weight. In this way, the ability of the multi-modal model to process visual information can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for visual data processing. Background Art

[0002] In recent years, multimodal language models have attracted much attention due to their ability to understand and generate language guided by visual input. The model is able to process and fuse multiple types of data inputs, supporting applications across a variety of fields, including image captioning, visual question answering, and video analysis. However, multimodal language models still face challenges in combining high-resolution, long-duration visual input with complex language. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for visual data processing is provided. The method comprises: extracting a plurality of visual words and at least one text word from visual data and a text instruction for the visual data, respectively, the text instruction indicating a processing task related to the visual data; extracting visual features from the plurality of visual words using a visual module in a trained multimodal model; extracting text features from at least one text word using a text module in the multimodal model; performing a self-attention mechanism on the visual features and the text features, respectively, based on the visual features and the text features, to determine visual weights corresponding to the plurality of visual words and text weights corresponding to the at least one text word; and determining a result for the processing task based on the plurality of visual words and the visual weights, and the at least one text word and the text weights.

[0004] In a second aspect of the present disclosure, a device for visual data processing is provided. The device includes: a word-unit extraction module, configured to extract multiple visual word-units and at least one text word-unit from visual data and text instructions for visual data, respectively, the text instructions indicating processing tasks related to visual data; a visual feature extraction module, configured to extract visual features from multiple visual word-units using a visual module in a trained multimodal model; a text feature extraction module, configured to extract text features from at least one text word-unit using a text module in a multimodal model; a self-attention mechanism execution module, configured to execute a self-attention mechanism on visual features and text features, respectively, based on visual features and text features, to determine visual weights corresponding to multiple visual word-units and text weights corresponding to at least one text word-unit; and a result determination module, configured to determine a result for the processing task based on multiple visual word-units and visual weights, and at least one text word-unit and text weights.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A schematic diagram showing the architecture of a multimodal model according to some embodiments of the present disclosure;

[0012] Figure 3 A schematic diagram showing processing of visual data according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing an aggregation operation on multiple visual word-grams according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram showing the assignment of position identifiers to a plurality of visual word-grams according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic diagram showing a self-attention mechanism for visual features and text features according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A schematic diagram showing a training process of a multimodal model according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A schematic diagram showing an environment in which embodiments of the present disclosure can be implemented;

[0018] Fig. 9 A schematic diagram showing a process for visual data processing according to some embodiments of the present disclosure;

[0019] Fig.10 A block diagram showing an apparatus for visual data processing according to some embodiments of the present disclosure; and

[0020] Fig.11 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0029] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0030] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0031] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, an electronic device 110 can utilize a multimodal model 120 to perform a data processing task. In some implementations, the electronic device 110 can utilize the multimodal model 120 to generate a processing result 112 based on input information 102.

[0032] In some embodiments, the input information 102 may include multimodal data, such as visual data, text data, and voice data.

[0033] exist Figure 1 In the embodiment, the electronic device 110 can be any type of device with computing capabilities, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0034] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

[0035] As mentioned earlier, multimodal language models still face challenges in combining high-resolution, long-duration visual input with complex language.

[0036] In order to improve the efficiency of multimodal data processing, in an embodiment of the present disclosure, a data processing scheme is proposed. Specifically, multiple visual words and at least one text word are extracted from visual data and text instructions for visual data, respectively, and the text instructions indicate processing tasks related to visual data. Visual features are extracted from multiple visual words using a visual module in a trained multimodal model. Text features are extracted from at least one text word using a text module in a multimodal model. Based on the visual features and the text features, a self-attention mechanism is performed on the visual features and the text features, respectively, to determine the visual weights corresponding to the multiple visual words and the text weights corresponding to the at least one text word. Based on the multiple visual words and the visual weights, and the at least one text word and the text weights, the result for the processing task is determined.

[0037] According to the scheme disclosed in the present invention, on the one hand, the introduction of separate visual modules and text modules can better process visual-related data and text-related data, and extract more accurate visual words and text words. On the other hand, the self-attention mechanism is performed on visual features and text features based on a shared self-attention module, which can save model parameters and reduce computational overhead. In this way, the ability of the multimodal model to process visual information can be improved without compromising the ability to process language.

[0038] Figure 2 A schematic diagram of the architecture of a multimodal model 120 according to some embodiments of the present disclosure is shown. The multimodal model 120 can be configured to process multimodal input data, including text data, visual data. The multimodal model 120 can be configured to generate corresponding model outputs based on the model input.

[0039] like Figure 2 As shown, the input information of the multimodal model 120 includes visual data 205 and text instructions 210 for the visual data 205. The text instructions 210 indicate processing tasks related to the visual data 205. The text instructions 210 can be configured to guide or instruct the multimodal model 120 to generate model outputs based on the visual data 205. For example, the text instructions 210 can indicate a content understanding task for the visual data 205, a task of classifying the visual data 205, and the like.

[0040] During the processing of the multimodal model 120, a plurality of visual tokens 215 and at least one textual token 220 may be extracted from the visual data 205 and the textual instruction 210, respectively. Each textual token may correspond to a text unit in the textual instruction 210, such as a word or a phrase, etc. Each visual token may correspond to a visual unit in the visual data 205, such as at least a portion of a tile in an image or video frame.

[0041] In some embodiments, the visual data 205 may include a video or an image. The multimodal model 120 may include a vision transformer (ViT) 225. The visual data 205 needs to be processed to improve the accuracy of the extracted visual words. Figure 3 The processing of visual data 205 is described.

[0042] Figure 3 Schematic diagram 300 of processing visual data 205 according to some embodiments of the present disclosure is shown. Figure 3 As shown, the target frame 302 or image of the video can be scaled to a target image 305 having a predetermined size, and the predetermined size is a multiple of the processing size of the visual transformer 225. In one example, the visual transformer 225 is capable of processing an image of 336*336 pixels (i.e., the length and width of the processing size of the visual transformer 225 are 336 pixels), and the original size of the target frame can be upsampled to an integer multiple of 336 to obtain a target image 305 having a predetermined size (e.g., the predetermined size is 1008*672 pixels).

[0043] After the target frame 302 is scaled to a target image 305 having a predetermined size, the target image 305 may be divided into a plurality of tiles (also referred to as partial views) 306-1 to 306-6 having a processing size (for ease of description, collectively referred to as a plurality of tiles 306). For example, the processing size is 336*336 pixels and the predetermined size is 1008*672 pixels. The target image 305 may be divided into 336*336 regions to obtain six tiles 306. The size of the predetermined size may be configured according to actual applications.

[0044] In some embodiments, the maximum number of the plurality of tiles 306 may be limited to H. Taking the target frame 302 as having a size of [h, w] and a processing size of 336 pixels as an example, the target frame 302 may be scaled to [p h ×336,p w ×336], where p w ×p h ≤H,p h =[p w ×h / w].

[0045] In addition to dividing the target image 305, the target frame 302 or image may be scaled to a scaled image (also referred to as a global view) 307 having a processing size. Continuing to use the processing size of 336*336 pixels as an example, the target frame 302 may be directly scaled to a scaled image 307 of 336*336 pixels.

[0046] After obtaining the plurality of image blocks 306 and the zoomed image 307, the plurality of visual word units 215 may be extracted therefrom by using the visual transformer 225. In this way, it is ensured that the global information and the detailed local information are retained and used for subsequent processing, thereby improving the accuracy of data processing.

[0047] In some embodiments, after extracting the plurality of visual word-grams 215, the visual word-grams 215 may be processed to reduce the computational load. Figure 4 The processing of visual word-grams 215 is described.

[0048] Figure 4 FIG. 4 is a schematic diagram 400 showing an aggregation operation on a plurality of visual word-grams 215 according to some embodiments of the present disclosure. Figure 4 As shown, first, multiple visual words 405-1 to 405-n (for the convenience of description, they can be collectively referred to as multiple visual words 405) of each frame or image of the video can be extracted. Each frame or image here can refer to each block in the multiple blocks 306 or a scaled image 307. For each frame of the video or for the image, an aggregation operation is performed on the multiple visual words 405 to obtain multiple visual words 410-1 to 410-n (for the convenience of description, they can be collectively referred to as multiple visual words 410) corresponding to each frame or image. Taking the multiple visual words in the window 412 as an example, the aggregation operation is performed on the multiple visual words in the window 412 to obtain the aggregated visual word 414. Similarly, the aggregation operation is performed on the multiple visual words 405 according to the window to obtain the aggregated multiple visual words 410. In this way, the number of visual words can be reduced, the computational overhead can be reduced, and the integrity of the visual information can be retained, so that the multimodal model 120 can ensure efficient processing and robust performance when processing high-resolution visual features.

[0049] In some embodiments, the aggregation operation includes an average pooling operation. First, a pooling window may be set, and then an average pooling operation is performed on multiple visual word units 405 within the pooling window to aggregate the multiple visual word units 405. The present disclosure does not limit the size of the pooling window, which may be a 2x2 pooling window or a 3x3 pooling window.

[0050] Continue to refer Figure 2 After extracting the plurality of visual words 215, the visual module (also called visual expert) 230 in the trained multimodal model 120 can be used to extract visual features from the plurality of visual words 215. The visual module 230 is good at processing vision-related data and can generate corresponding outputs.

[0051] In some embodiments, the multimodal model 120 further includes a tokenizer 240 for text, and the tokenizer 240 can be used to extract at least one text word 220 from the text instruction 210 .

[0052] After extracting at least one text word 220, a text module (also called a text expert) 235 in the trained multimodal model 120 may be used to extract text features from the at least one text word 220. The text module 235 is good at processing text-related data and can generate corresponding outputs.

[0053] In some embodiments, a routing module may be further provided before the visual module 230 and the text module 235. The routing module is configured to determine the type of input data, and route the visual data to the visual module 230, and route the text data to the text module 235. In one example, the routing module may obtain a plurality of visual word units 215 and at least one text word unit 220, and route the plurality of visual word units 215 to the visual module 230, and route at least one text word unit 220 to the text module 235.

[0054] In some embodiments, the multimodal model 120 includes a language model (LM), which includes at least a visual module 230 and a text module 235. The language model is trained to have a predetermined number of position IDs. The language model is usually designed for a limited input length. In one example, the length of a common language model is 4K. For video processing tasks, if position IDs are directly assigned according to the visual word units of the video, and a unique position ID is assigned to each visual word unit, this may cause a length overflow problem, resulting in a decrease in the performance of the multimodal model.

[0055] In order to solve the above-mentioned length overflow problem, the same position identifier may be assigned to multiple visual words. Figure 5 FIG. 5 is a schematic diagram 500 showing how to assign position identifiers to a plurality of visual word-grams according to some embodiments of the present disclosure. Figure 5 As shown, at least one position identifier of a predetermined number of position identifiers may be assigned to a visual word 510 corresponding to each frame (eg, frame 505) in the video, and two or more visual words corresponding to each frame may be assigned the same position identifier. Figure 5In the example, a position identifier (e.g., 1) can be assigned to the visual word 510 corresponding to the frame 505, that is, all the visual words share one position identifier. In one example, three position identifiers can also be assigned to the visual word corresponding to each frame. For example, if each frame corresponds to 144 visual words, a position identifier can be assigned to every 48 visual words, that is, every 48 visual words share one position identifier. The present application does not limit the number of assigned position identifiers. In this way, by making multiple visual words share one position identifier, the context length of the multimodal model 120 input can be improved without affecting the performance of the multimodal model 120.

[0056] In some embodiments, the number of position identifiers assigned to each frame is determined based on the total number of frames of the video. When the total number of frames is small, more position identifiers may be assigned to each frame. When the total number of frames is large, fewer position identifiers may be assigned to each frame.

[0057] In some embodiments, a predetermined number of position identifiers are also assigned to at least one text word 220. Figure 2 Since the length of the text instruction 210 is short, the length of the at least one text word unit 220 is also short, and each text word unit in the at least one text word unit 220 may be assigned a different position identifier. Alternatively or additionally, two or more visual word units in the at least one text word unit 220 are assigned the same position identifier.

[0058] After extracting the visual features and the text features, a self-attention mechanism can be performed on the visual features and the text features respectively based on the visual features and the text features to determine the visual weights corresponding to the multiple visual words 215 and the text weight corresponding to the at least one text word 220. Figure 6 Describes performing self-attention on visual and textual features.

[0059] Figure 6 600 is a schematic diagram showing a self-attention mechanism for visual features and text features according to some embodiments of the present disclosure. Figure 6 As shown, the multimodal model 120 also includes a self-attention module 605. The input of the self-attention module 605 is defined as the query (Q) feature, key (K) feature, and value (V) feature input to each transformer layer. The processing of the self-attention module 605 can be expressed as follows:

[0060] Attetion(Q,K,V)=Vsoftmax(K T ) (1)

[0061] Where Q represents the query feature, K represents the key feature, and V represents the value feature. The above process can be understood as using the query feature Q and the key feature K to calculate the self-attention weight, and using the self-attention weight to perform weighted summation on the value feature V.

[0062] In terms of visual features, the self-attention module 605 can be used to perform a self-attention mechanism on the visual features based on the text features to determine the visual weights corresponding to the multiple visual word units 215. The self-attention mechanism performed on the visual features can be generalized as follows:

[0063]

[0064] Among them, v represents visual weight, O v represents the query feature for visual features, K v represents the key feature for visual features, K t represents the key feature for text features, V v Represents the value feature for visual features, V t Represents the value feature for a text feature. v , K v and V v Representing different projections of visual features.

[0065] In terms of text features, the self-attention module 605 can be used to perform a self-attention mechanism on the text features based on the visual features to determine the text weight corresponding to at least one text word 220. The self-attention mechanism performed on the text features can be generally expressed as follows:

[0066]

[0067] Among them, t represents the text weight, Q t Represents the query feature for the text feature. t , K t and V t Represent different projections of text features.

[0068] After the visual weight and the text weight are determined, a result 245 for the processing task may be determined based on the plurality of visual word-grams 215 and the visual weights, and the at least one text word-gram 220 and the text weight. Figure 2 In the example of , when the text instruction 210 is “Please describe the video content in detail”, the result 245 is “playing football”, which describes the content presented by the input visual data 205.

[0069] Continue to refer Figure 6In some embodiments, the multimodal model 120 includes a feedforward neural network 610. Based on the plurality of visual words 215 and the visual weights, a weighted visual word-gram may be obtained. Based on the at least one text word-gram 220 and the text weight, a weighted text word-gram may be obtained. Using the feedforward neural network 610, the result of the processing task may be determined based on the weighted visual word-gram and the weighted text word-gram.

[0070] In some embodiments, the output results of the feedforward neural network 610 can be re-input into the visual module 230 and the text module 235 for iterative processing, so as to generate more accurate visual word units and text word units.

[0071] The following will refer to Figure 7 Describe the training process of the multimodal model 120, Figure 7 FIG. 7 is a schematic diagram of a training process 700 of a multimodal model 120 according to some embodiments of the present disclosure. Figure 7 As shown, the multimodal model 120 includes at least a visual transformer 225, a multi-layer perceptron 705, and a language model 710 including a visual module 230 and a text module 235. The training process 700 includes a first training stage 715, a second training stage 720, and a third training stage 725.

[0072] In the first training phase 715, the parameters of the multilayer perceptron 705 in the multimodal model 120 are updated, and the parameters of the visual transformer 225 and the language model 710 are kept unchanged. The multilayer perceptron 705 is configured to map the visual word units output by the visual transformer 225 to the feature space of the language model 710, so that the language model 710 can interpret and express the information in the image.

[0073] In the second training phase 720, the parameters of the visual module 230 and the multilayer perceptron 705 of the language model 710 are updated, and the parameters of the visual transformer 225 and the text module 235 of the language model 710 are kept unchanged. The training data of the second training phase 720 may include bilingual subtitles, interlaced text and image pairs, video subtitles, etc.

[0074] In the third training stage 725, supervised training is used to fine-tune the parameters of the visual transformer 225, the multi-layer perceptron 705, and the language model 710. The training data of the third training stage 725 includes fine-grained subtitles, multi-turn dialogues based on visual content, visual question answering (VQA), math problems and diagrams, etc.

[0075] Figure 8 A schematic diagram of an environment 800 in which embodiments of the present disclosure can be implemented is shown. Figure 8In the environment 800 of FIG. 8 , the model is generally shown to involve different stages, including a training stage 802 and an application stage 806. After the training stage is completed, there may also be a testing stage, which is not shown in the figure.

[0076] In the training phase 802, the model training system 810 is configured to perform training of the model 805 using the training data set 812. The model 805 may be, for example, Figure 1 The multimodal model 120 in FIG. 805 may have initial parameter values ​​at the beginning of training. The training process is to update the parameter values ​​of the model 805 to the expected values ​​based on the training data.

[0077] In the application phase 806, the obtained model 805 has trained parameter values ​​and can be provided to the model application system 830 for use. In the application phase 806, the model 805 can be used to process the corresponding target input 832 in the actual scene and provide the corresponding target output 834. The model application system 830 can be configured to implement Figure 1 An electronic device 110.

[0078] exist Figure 8 In the embodiment, the model training system 810 and the model application system 830 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0079] It should be understood that Figure 8 The components and arrangements in the environment 800 shown are merely examples, and a computing system suitable for implementing the exemplary implementation described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as being separate, the model training system 810 and the model application system 830 may be integrated into the same system or device. The implementation of the present disclosure is not limited in this respect.

[0080] Fig. 9 FIG. 9 is a schematic diagram of a process 900 for visual data processing according to some embodiments of the present disclosure. The process 900 may be implemented in Figure 1 The electronic device 110 is located at the electronic device 110.

[0081] In block 910 , the electronic device 110 extracts a plurality of visual word-grams and at least one text word-gram from visual data and text instructions for the visual data, respectively, where the text instructions indicate a processing task related to the visual data.

[0082] In block 920 , the electronic device 110 extracts visual features from a plurality of visual word-grams using a visual module in a trained multimodal model.

[0083] In block 930 , the electronic device 110 extracts text features from at least one text word using a text module in the multimodal model.

[0084] In box 940, the electronic device 110 performs a self-attention mechanism on the visual features and the text features respectively based on the visual features and the text features to determine visual weights corresponding to multiple visual words and a text weight corresponding to at least one text word.

[0085] In block 950 , the electronic device 110 determines a result for the processing task based on the plurality of visual word-grams and the visual weights, and the at least one text word-gram and the text weight.

[0086] In some embodiments, the visual data includes a video or an image, the multimodal model includes a visual transformer, and wherein extracting multiple visual words from the visual data includes: scaling a target frame or image of the video to a target image having a predetermined size, the predetermined size being a multiple of a processing size of the visual transformer; dividing the target image into a plurality of tiles having the processing size; scaling the target frame or image to a scaled image having the processing size; and extracting multiple visual words from the plurality of tiles and the scaled image using the visual transformer.

[0087] In some embodiments, the multimodal model includes a feedforward neural network, and determining the result of a processing task includes: obtaining weighted visual words based on multiple visual words and visual weights; obtaining weighted text words based on at least one text word and a text weight; and determining the result of the processing task based on the weighted visual words and weighted text words using the feedforward neural network.

[0088] In some embodiments, the visual data includes a video or an image, and extracting multiple visual words from the visual data includes: extracting multiple visual words for each frame of the video or the image; and for each frame of the video or for the image, performing an aggregation operation on the multiple visual words to obtain multiple visual words corresponding to each frame or image.

[0089] In some embodiments, the aggregation operation includes an average pooling operation.

[0090] In some embodiments, the multimodal model includes a language model, the language model includes at least a visual module and a text module, and the language model is trained to have a predetermined number of place identifiers, wherein the visual data includes a video, and process 900 also includes: assigning at least one place identifier of the predetermined number of place identifiers to the visual word element corresponding to each frame in the video, wherein two or more visual word elements corresponding to each frame are assigned the same place identifier.

[0091] In some embodiments, the number of location identifiers assigned to each frame is determined based on the total number of frames in the video.

[0092] In some embodiments, the multimodal model includes at least a visual transformer, a multilayer perceptron, and a language model including a visual module and a text module, and the training process of the multimodal model includes: a first training stage, in which the parameters of the multilayer perceptron in the multimodal model are updated and the parameters of the visual transformer and the language model are kept unchanged, and the multilayer perceptron is configured to map the visual word units output by the visual transformer to the feature space of the language model; a second training stage, in which the parameters of the visual module and the multilayer perceptron of the language model are updated and the parameters of the text module in the visual transformer and the language model are kept unchanged; a third training stage, in which the parameters of the visual transformer, the multilayer perceptron, and the language model are fine-tuned through supervised training.

[0093] Fig.10 1 is a block diagram of an apparatus 1000 for visual data processing according to some embodiments of the present disclosure. The apparatus 1000 may be implemented as or included in Figure 1 The electronic device 110 is located at the device 1000. Each module / component in the device 1000 may be implemented by hardware, software, firmware, or any combination thereof.

[0094] As shown in the figure, the device 1000 includes a word-unit extraction module 1010, which is configured to extract multiple visual word-units and at least one text word-unit from visual data and text instructions for visual data, respectively, and the text instructions indicate processing tasks related to visual data. The device 1000 includes a visual feature extraction module 1020, which is configured to extract visual features from multiple visual word-units using a visual module in a trained multimodal model. The device 1000 includes a text feature extraction module 1030, which is configured to extract text features from at least one text word-unit using a text module in a multimodal model. The device 1000 includes a self-attention mechanism execution module 1040, which is configured to perform a self-attention mechanism on the visual features and the text features, respectively, based on the visual features and the text features, to determine the visual weights corresponding to the multiple visual word-units and the text weights corresponding to the at least one text word-unit. In addition, the device 1000 includes a result determination module 1050, which is configured to determine the result for the processing task based on the multiple visual word-units and the visual weights, and the at least one text word-unit and the text weights.

[0095] In some embodiments, the visual data includes a video or an image, the multimodal model includes a visual transformer, and the word-gram extraction module 1010 is further configured to scale a target frame or image of the video into a target image having a predetermined size, wherein the predetermined size is a multiple of a processing size of the visual transformer; divide the target image into a plurality of tiles having the processing size; scale the target frame or image into a scaled image having the processing size; and extract a plurality of visual words from the plurality of tiles and the scaled image using the visual transformer.

[0096] In some embodiments, the multimodal model includes a feedforward neural network, and the result determination module 1050 is further configured to obtain weighted visual words based on multiple visual words and visual weights; obtain weighted text words based on at least one text word and a text weight; and determine the result of the processing task based on the weighted visual words and weighted text words using the feedforward neural network.

[0097] In some embodiments, the visual data includes videos or images, and the word-gram extraction module 1010 is further configured to extract multiple visual word-grams for each frame of the video or the image; and for each frame of the video or for the image, perform an aggregation operation on the multiple visual word-grams to obtain multiple visual word-grams corresponding to each frame or image.

[0098] In some embodiments, the aggregation operation includes an average pooling operation.

[0099] In some embodiments, the multimodal model includes a language model, the language model includes at least a visual module and a text module, and the language model is trained to have a predetermined number of location identifiers, the visual data includes a video, and the device 1000 also includes a location identifier assignment unit, configured to assign at least one location identifier of the predetermined number of location identifiers to the visual word element corresponding to each frame in the video, wherein two or more visual word elements corresponding to each frame are assigned the same location identifier.

[0100] In some embodiments, the number of location identifiers assigned to each frame is determined based on the total number of frames in the video.

[0101] In some embodiments, a predetermined number of position identifiers are also assigned to the at least one text word.

[0102] In some embodiments, the multimodal model includes at least a visual transformer, a multilayer perceptron, and a language model including a visual module and a text module, and the training process of the multimodal model includes: a first training stage, in which the parameters of the multilayer perceptron in the multimodal model are updated and the parameters of the visual transformer and the language model are kept unchanged, and the multilayer perceptron is configured to map the visual word units output by the visual transformer to the feature space of the language model; a second training stage, in which the parameters of the visual module and the multilayer perceptron of the language model are updated and the parameters of the text module in the visual transformer and the language model are kept unchanged; and a third training stage, in which the parameters of the visual transformer, the multilayer perceptron, and the language model are fine-tuned through supervised training.

[0103] Fig.11 1 shows a block diagram of an electronic device 1100 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig.11 The electronic device 1100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig.11 The electronic device 1100 shown can be used to implement Figure 1 electronic device 110, or Fig.10 Device 1000.

[0104] like Fig.11As shown, the electronic device 1100 is in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to, one or more processors or processing units 1110, a memory 1120, a storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. The processing unit 1110 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 1120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1100.

[0105] The electronic device 1100 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1120 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1130 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 1100.

[0106] The electronic device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.11 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1120 may include a computer program product 1125 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0107] The communication unit 1140 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 1100 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 1100 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0108] The input device 1150 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 1160 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 1100 may also communicate with one or more external devices (not shown) through the communication unit 1140 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 1100, or communicate with any device that allows the electronic device 1100 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0109] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0110] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0113] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0114] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: extracting a plurality of visual word-grams and at least one text word-gram from visual data and text instructions for the visual data, respectively, the text instructions indicating a processing task associated with the visual data; Extracting visual features from the plurality of visual words using a visual module in the trained multimodal model; Extracting text features from the at least one text word using a text module in the multimodal model; Based on the visual features and the text features, performing a self-attention mechanism on the visual features and the text features respectively to determine visual weights corresponding to the multiple visual word-grams and a text weight corresponding to the at least one text word-gram; as well as An outcome for the processing task is determined based on the plurality of visual word-grams and the visual weights, and the at least one textual word-gram and the textual weight.

2. The method of claim 1 , wherein the visual data comprises a video or an image, the multimodal model comprises a visual transformer, and wherein extracting the plurality of visual tokens from the visual data comprises: Scaling the target frame of the video or the image into a target image with a predetermined size, wherein the predetermined size is a multiple of a processing size of the visual transformer; Dividing the target image into a plurality of tiles having the processing size; scaling the target frame or the image into a scaled image having the processing size; as well as The plurality of visual words are extracted from the plurality of tiles and the zoomed image using the visual transformer.

3. The method of claim 1 , wherein the multimodal model comprises a feed-forward neural network, and determining the result of the processing task comprises: Obtaining a weighted visual word-metaphor based on the multiple visual word-metaphors and the visual weights; Obtaining a weighted text word based on the at least one text word and the text weight; as well as The result of the processing task is determined based on the weighted visual word-grams and the weighted text word-grams using the feed-forward neural network.

4. The method of claim 1, wherein the visual data comprises a video or an image, and extracting a plurality of visual words from the visual data comprises: Extracting multiple visual words from each frame of the video or the image; as well as For each frame of the video or for the image, an aggregation operation is performed on the multiple visual words to obtain multiple visual words corresponding to each frame or the image. The method of claim 4 , wherein the aggregation operation comprises an average pooling operation.

6. The method according to claim 1, wherein the multimodal model comprises a language model, the language model comprises at least the visual module and the text module, and the language model is trained to have a predetermined number of location identifiers, Wherein the visual data comprises a video, and the method further comprises: At least one position identifier among the predetermined number of position identifiers is assigned to a visual word corresponding to each frame in the video, wherein two or more visual words corresponding to each frame are assigned the same position identifier. The method of claim 6 , wherein the number of location identifiers assigned to each frame is determined based on a total number of frames in the video.

8. The method according to claim 6, wherein the predetermined number of position identifiers are also assigned to the at least one text word.

9. The method according to claim 1, wherein the multimodal model comprises at least a visual transformer, a multilayer perceptron, and a language model comprising the visual module and the text module, and the training process of the multimodal model comprises: A first training stage, in which parameters of a multilayer perceptron in the multimodal model are updated and parameters of the visual transformer and the language model are kept unchanged, wherein the multilayer perceptron is configured to map visual word units output by the visual transformer to a feature space of the language model; A second training stage, in which the parameters of the visual module and the multilayer perceptron of the language model are updated, and the parameters of the visual transformer and the text module in the language model are kept unchanged; as well as The third training stage is to fine-tune the parameters of the visual transformer, the multi-layer perceptron and the language model through supervised training.

10. A data processing device, comprising: a word-gram extraction module configured to extract a plurality of visual word-grams and at least one text word-gram from visual data and text instructions for the visual data, respectively, the text instructions indicating a processing task associated with the visual data; A visual feature extraction module configured to extract visual features from the plurality of visual words using a visual module in the trained multimodal model; A text feature extraction module, configured to extract text features from the at least one text word using a text module in the multimodal model; A self-attention mechanism execution module is configured to execute a self-attention mechanism on the visual features and the text features respectively based on the visual features and the text features to determine visual weights corresponding to the multiple visual words and a text weight corresponding to the at least one text word; as well as The result determination module is configured to determine a result for the processing task based on the multiple visual word-grams and the visual weights, and the at least one text word-gram and the text weight.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit. 12 . A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to implement the method according to claim 1 .

13. A computer program product comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

Images