Method, apparatus, device and storage medium for information processing

Abstract

According to embodiments of the present disclosure, a method, apparatus, device and storage medium for information processing are provided. The method comprises: obtaining a processing request for a target model; in response to the processing request comprising input visual content, determining first visual features of the input visual content by using a first visual encoding model, and determining second visual features of the input visual content by using a second visual encoding model, the first visual encoding model being different from the second visual encoding model; constructing an input feature cell sequence corresponding to the processing request based on at least the first visual features and the second visual features; and obtaining at least one output feature cell generated by the target model based on the input feature cell sequence to generate response content for the processing request. Embodiments of the present disclosure can improve the output accuracy of the target model by extracting multiple visual features.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for information processing. Background Technology

[0002] With the development of computer technology, people have higher and higher demands for network models. Current models are becoming more and more mature in their image processing technology. For example, models can generate corresponding text descriptions based on input images, or models can generate images based on descriptions, etc. However, the accuracy of current model response results still needs to be improved. Summary of the Invention

[0003] In a first aspect of this disclosure, an information processing method is provided. The method includes: acquiring a processing request for a target model; in response to the processing request including input visual content, determining first visual features of the input visual content using a first visual encoding model, and determining second visual features of the input visual content using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; constructing an input feature unit sequence corresponding to the processing request based at least on the first and second visual features; and acquiring at least one output feature unit generated by the target model based on the input feature unit sequence to generate response content for the processing request.

[0004] In a second aspect of this disclosure, an information processing apparatus is provided. The apparatus includes: an acquisition module configured to acquire a processing request for a target model; a determination module configured to, in response to the processing request including input visual content, determine first visual features of the input visual content using a first visual encoding model and determine second visual features of the input visual content using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; a construction module configured to construct an input feature unit sequence corresponding to the processing request based at least on the first and second visual features; and a generation module configured to acquire at least one output feature unit generated by the target model based on the input feature unit sequence to generate response content for the processing request.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A first flowchart of an information processing procedure according to some embodiments of the present disclosure is shown;

[0012] Figure 3A A second flowchart illustrating an information processing procedure according to some embodiments of the present disclosure is shown;

[0013] Figure 3B A third flowchart illustrating an information processing procedure according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram of the position embedding according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A frequency allocation diagram according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic diagram of a training feature unit sequence according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A schematic diagram of attention masks according to some embodiments of the present disclosure is shown;

[0018] Figure 8 Interaction diagrams according to some embodiments of this disclosure are shown;

[0019] Figure 9 A schematic structural block diagram of an information processing apparatus according to certain embodiments of the present disclosure is shown;

[0020] Figure 10 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0025] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0026] To meet users' diverse requirements for models, various generative models have emerged, such as text generation models and image generation models. However, these models all take unimodal data as input and generate unimodal outputs. While this approach can accomplish the corresponding generation tasks, the models cannot handle multimodal information, greatly limiting their application effectiveness. Consequently, some research has been conducted on models targeting multimodal inputs, but these models have failed to generate responses with high accuracy.

[0027] Embodiments of this disclosure propose an information processing scheme. According to the scheme, a processing request for a target model is obtained; in response to the processing request including input visual content, a first visual feature of the input visual content is determined using a first visual encoding model, and a second visual feature of the input visual content is determined using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; an input feature unit sequence corresponding to the processing request is constructed based at least on the first and second visual features; and at least one output feature unit generated by the target model based on the input feature unit sequence is obtained to generate response content for the processing request.

[0028] Based on this approach, the embodiments of this disclosure can extract features of the input visual content from different angles using a first visual encoding model and a second visual encoding model, thereby extracting more comprehensive features and generating more accurate response content accordingly.

[0029] Example Environment

[0030] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, the example environment 100 may include an electronic device 110. In some embodiments, the electronic device 110 may receive a processing request for a target model 120; in response to the processing request including input visual content, it may determine first visual features of the input visual content using a first visual encoding model and determine second visual features of the input visual content using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; construct an input feature unit sequence corresponding to the processing request based at least on the first and second visual features; and obtain at least one output feature unit generated by the target model 120 based on the input feature unit sequence to generate response content for the processing request. The target model 120 may be deployed on the electronic device 110 or on other devices, which will not be elaborated here.

[0031] In some embodiments, the electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 may also support any type of interface for the target user (such as "wearable" circuitry).

[0032] Electronic device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Electronic device 110 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0033] It should be understood that the structure and function of the various embedded elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0035] Example process

[0036] Figure 2 A flowchart of an information processing procedure 200 according to some embodiments of the present disclosure is shown. Procedure 200 can be implemented at electronic device 110. Reference is made below. Figure 1 Describe the process 200.

[0037] In box 210, electronic device 110 receives a processing request for the target model.

[0038] In some embodiments, to meet the needs of processing multimodal information, a target model can be constructed. This target model includes multiple processing blocks, each including a first attention unit, a second attention unit, and a fusion unit. The first attention unit is configured to process first visual features and text feature units in the input feature unit sequence. The second attention unit is configured to process second visual features. The fusion unit is configured to fuse the first output of the first attention unit and the second output of the second attention unit. This target model can process multimodal information; therefore, the processing request can include both input visual content and input text content.

[0039] For example, refer to Figure 3A The target model includes multiple processing blocks 304, see reference. Figure 3B Each processing block 304 includes an attention layer, which comprises a first attention unit, a second attention unit, and a fusion unit. The first attention unit includes a normalization layer 321 and a query-key-value layer 320. The second attention unit includes a normalization layer 323 and a query-key-value layer 322. The attention layer includes a fusion unit 319, which is a multimodal multi-head self-attention layer. Furthermore, the first set of parameters of the first attention unit is independent of the second set of parameters of the second attention unit.

[0040] In some embodiments, reference Figure 3B The processing block 304 may further include a first feedforward unit and a second feedforward unit. The first feedforward unit includes a normalization layer 316 and a feedforward layer 315, and the second feedforward unit includes a normalization layer 318 and a feedforward layer 317. In this way, when the target model processes multimodal input information including text and images, it can use a unified attention layer to process image feature units corresponding to images and text feature units corresponding to text simultaneously. Furthermore, due to the differences between image modalities and text modalities, different feedforward units are applied for processing, thereby improving the model's processing performance for multimodal information.

[0041] In box 220, in response to a processing request including input visual content, electronic device 110 determines a first visual feature of the input visual content using a first visual coding model and determines a second visual feature of the input visual content using a second visual coding model.

[0042] In some embodiments, if the processing request includes input visual content, the target model needs to process the visual content. Since different visual encoding models have varying capabilities, their effectiveness in extracting visual content features differs. To improve the target model's understanding of visual content, different visual encoding models can be applied to determine the visual features of the visual content. Therefore, the first visual encoding model differs from the second visual encoding model.

[0043] In some embodiments, the first visual coding model includes a coding model based on a visual transducer, and the second visual coding model includes a coding model based on a variational autoencoder. Since visual transducer coding models such as the Vision Transformer (ViT) model segment the image into multiple image patches and perform global modeling through the transformer's self-attention mechanism, thereby capturing long-range dependencies in the image, the ViT model is suitable for the field of image understanding. On the other hand, variational autoencoder-based coding models such as the Variational Autoencoder (VAE) model model and generate complex data distributions by mapping the image to a probability distribution in a latent space; therefore, the VAE model is more suitable for the field of image generation. To simultaneously possess image generation and image understanding capabilities, the first visual coding model can be determined as the ViT model, and the second visual coding model can be determined as the VAE model. For example, if the processing request includes an input image, the ViT model can be applied to determine the first visual features of the input image, and the VAE model can be used to determine the second visual features of the input image.

[0044] In box 230, electronic device 110 constructs a sequence of input feature units corresponding to the processing request, based at least on the first visual feature and the second visual feature.

[0045] In some embodiments, since the processing request includes input text content and input visual content, text features corresponding to the input text content can be determined, and then, based on the first visual features, the second visual features, and the text content, a sequence of input feature units corresponding to the processing request can be constructed. (See reference) Figure 3A The input text content 306 can be processed by the text tagger 305 to obtain text features. The input visual content 309 is then processed by the first visual encoding model 307 to obtain first visual features, and by the second visual encoding model 308 to obtain second visual features. Next, using the first visual features, the second visual features, and the text content, an input feature unit sequence corresponding to the processing request is constructed, and this sequence is input into the processing block 304.

[0046] In some embodiments, the target model can handle multiple types of tasks. If the target model's task is an image generation task, and the processing request includes input text content, then an initial noisy image can be obtained, and the text feature units corresponding to the text content and the visual feature units corresponding to the noisy image can be determined. Then, the normalization layer 321 and query-key-value layer 320 in the first attention unit are applied to process the text feature units, generating a feature matrix corresponding to the text feature units. Simultaneously, the normalization layer 323 and query-key-value layer 322 in the second attention unit are applied to process the visual feature units, generating a feature matrix corresponding to the visual feature units.

[0047] Then, the feature matrices corresponding to the text feature units and the visual feature units are fused across modally by the fusion unit 319 to obtain the fused feature units. The obtained fused feature units include the feature matrices corresponding to the text modality and the feature matrices corresponding to the image modality. The normalization layer 316 and the feedforward layer 315 in the first feedforward unit are used to process the feature matrices corresponding to the text modality, and the normalization layer 318 and the feedforward layer 317 in the second feedforward unit are used to process the feature matrices corresponding to the image modality. Then, the outputs of the first feedforward unit and the second feedforward unit are input into the next processing block.

[0048] In this way, text features and image features in the target model are deeply fused in each processing block, enabling the target model to achieve high-quality image generation without relying on the text encoder required by traditional text-to-image models.

[0049] In some embodiments, visual features can also be adjusted using Adaptive Layer Normalization (AdaLN). For example, referencing Figure 3B In each processing block of the target model, the time step embedding dynamically adjusts the statistical properties of the visual feature map of that processing block through adaptive layer normalization 324, thereby controlling the behavior of the generation process at different time steps, so that the target model can adaptively predict the velocity field according to the current time step.

[0050] In some embodiments, if the processing request includes input visual content, which serves as a model condition, then a first visual feature and a second visual feature corresponding to the input visual content can be obtained. The first visual feature is generated using a first visual encoding model, and the second visual feature is generated using a second visual encoding model. Then, the normalization layer 321 and the query-key-value layer 320 in the first attention unit are applied to process the feature units corresponding to the first visual feature, generating a feature matrix corresponding to the first visual feature. Simultaneously, the normalization layer 323 and the query-key-value layer 322 in the second attention unit are applied to process the feature units corresponding to the second visual feature, generating a feature matrix corresponding to the second visual feature.

[0051] Then, the feature matrix corresponding to the first visual feature and the feature matrix corresponding to the second visual feature are fused across modally by the fusion unit 319 to obtain the fused feature unit. The obtained fused feature unit includes the feature matrix corresponding to the first visual feature and the feature matrix corresponding to the second visual feature. The normalization layer 316 and the feedforward layer 315 in the first feedforward unit are used to process the feature matrix corresponding to the first visual feature, and the normalization layer 318 and the feedforward layer 317 in the second feedforward unit are used to process the feature matrix corresponding to the second visual feature. Then, the output results of the first feedforward unit and the second feedforward unit are input into the next processing block.

[0052] In some embodiments, to enhance the target model's ability to perceive location, location information can be embedded into the feature units before attention calculation. The electronic device 110 can determine the index information of a target feature unit in the input feature unit sequence based on its position and type within the sequence. Then, based on the index information, attention information associated with the target feature unit is determined. The index information is essentially a location embedding identifier corresponding to the target feature unit, used to provide location information for each feature unit in the sequence.

[0053] In some embodiments, since the processing request includes input text content, and since the text content is only related to the time dimension, the electronic device 110 can, in response to the target feature unit corresponding to the text content, determine a first value of the first dimension of the index information based on the sequence position, and then set the other dimensions of the index information to preset values. For example, refer to... Figure 4 For the target feature unit 402 corresponding to the text content, the index information [4,0,0] corresponding to the target feature unit 402 can be determined according to the sequence position. The 4 in the index information [4,0,0] is the first value of the first dimension, and the [0,0] in the index information [4,0,0] is the preset value.

[0054] In some embodiments, the first dimension can be the time dimension. Then, the index information corresponding to the target feature unit 402 can be sequentially increased according to the time dimension. For example, the index information corresponding to the next feature unit of the target feature unit 402 is [5,0,0], the index information corresponding to the next feature unit is [6,0,0], and so on. Furthermore, since the text content is only related to the time dimension, the values ​​of other dimensions of the index information can be set to preset values ​​to facilitate model processing.

[0055] In some embodiments, since the processing request includes input visual content, and since the visual content is related to both temporal and spatial dimensions, the electronic device 110 can, in response to a target feature unit corresponding to an image patch in an image, determine a second value of the first dimension of the indexing information based on the sequence position, wherein multiple feature units corresponding to the image are associated with the same sequence position. The electronic device 110 can then determine a third value of the second dimension and a fourth value of the third dimension of the indexing information based on the two-dimensional position of the image patch in the image.

[0056] For example, refer to Figure 4 If the input visual content is an image, and the image is related to both time and space dimensions, the time dimension T can be defined as the first dimension. Since an image is two-dimensional, the space dimension includes width and height. The height H in the space dimension can be defined as the second dimension, and the width W in the space dimension can be defined as the third dimension. The time dimension represents the temporal positional relationship between image feature units and text feature units, while the space dimension represents the spatial positional relationship between image feature units.

[0057] For the target feature unit 401 corresponding to the image, the target feature unit 401 corresponds to multiple image patches in the image. Based on the sequence position, the multiple index information [2,0,0]~[2,2,2] corresponding to the target feature unit 401 can be determined. The value ranked first in the index information [2,0,0]~[2,2,2] is the second value of the first dimension, the value ranked second in the index information [2,0,0]~[2,2,2] is the third value of the second dimension, and the value ranked third in the index information [2,0,0]~[2,2,2] is the fourth value of the third dimension. Taking the index information [2,1,0] in the target feature unit 401 as an example, 2 is the second value of the first dimension, 1 is the third value of the second dimension, and 0 is the fourth value of the third dimension.

[0058] In some embodiments, since the model needs to refer to historical images to perform the generation task in tasks such as image editing and multi-image generation, if the input visual content is an image, the spatial dimension value in the index information of the target feature unit corresponding to the image can be set to start from 0, i.e., [0,0] to [2,2]. This also associates multiple feature units corresponding to the image with the same sequence position, for example, by referencing... Figure 4 In the index information corresponding to target feature unit 401 and target feature unit 403, the values ​​corresponding to the spatial dimension are the same. For example, in [2,0,0] and [9,0,0], the spatial dimension corresponds to [0,0]. In this way, it can be ensured that all images have consistent positional information in space.

[0059] In some embodiments, to ensure that each dimension can obtain long-distance semantic associations or local semantic associations, a rotational position embedding method can be applied to alternately incorporate frequency information from multiple dimensions into the embedding vector. For example, refer to... Figure 5 Frequency information from multiple dimensions can be alternately incorporated into the embedding vector in the order of time dimension T, spatial dimension height H, and spatial dimension width W, as shown in the following formula:

[0060]

[0061] θ represents the rotation frequency applied to a specific size, β represents the fundamental frequency, d represents the number of dimensions, and i and j are variables. Alternatively, the first 48 channels can be alternately assigned to the time dimension, the height in the spatial dimension, and the width in the spatial dimension, and then the remaining 16 channels can be assigned to the time dimension to model long sequences.

[0062] In box 240, electronic device 110 acquires at least one output feature unit generated by the target model based on the input feature unit sequence to generate response content for the processing request.

[0063] In some embodiments, at least one output feature unit includes a set of text feature units and a set of visual feature units, and the response content includes output text content corresponding to the set of text feature units and output visual content corresponding to the set of visual feature units. For example, refer to Figure 8 The input content 801 of the target model is "xxx", and the output response content of the target model includes the output text content 802 "Step 1, aaa", the output visual content 803, the output text content 804 "Step 2, bbb", the output visual content 805, the output text content 806 "Step 3, ccc", and the output visual content 807.

[0064] In some embodiments, the input content of the target model can be not only text, but also a mixture of images and text. Furthermore, the output content of the target model can be not only text, but also images, or a mixture of images and text; there are no limitations on this.

[0065] In some embodiments, since the response content includes both output text content and output visual content, an indicator is needed to determine which modality of content the target model needs to generate. If the electronic device 110 detects a preset first indicator in the output of the target model, it can perform a noise reduction process using the target model in response to the detection of the indicator to generate a set of visual feature units. The first indicator <|vision_start|> is used to instruct the target model to generate an image. For example, if the model's output includes the first indicator, the target model can know that an image needs to be generated, and then it can apply the target model to perform a noise reduction process using a diffusion model to generate a set of visual feature units.

[0066] In some embodiments, the electronic device 110 may also output a preset second indicator in response to generating a set of visual units based on the noise reduction process. The second indicator <|visition_end|> is used to instruct the target model to generate text. For example, if the image generation task has been completed and a set of visual units has been generated, the second indicator can be output to instruct the target model to output text next, thereby generating a reference. Figure 8 The format alternates between Chinese text and images.

[0067] In some embodiments, the electronic device 110 may further utilize a first visual coding model to determine a third visual feature of the output visual content, and utilize a second visual coding model to determine a fourth visual feature of the output visual content. Then, based on the third and fourth visual features, the input feature unit sequence is updated to generate subsequent output feature units by the target model. Since the target model comprises multiple processing blocks, the feature unit sequence output by the current processing block can be used as the input feature unit sequence for the next processing block.

[0068] In some embodiments, during the training of the target model, the electronic device 110 can acquire sample data and then construct a training feature unit sequence based on the sample data. Next, based on the task type of the training task associated with the target model, an attention mask corresponding to the task type is determined. Then, the target model is trained based on the training feature sequence and the attention mask. An attention mask is a method used to control the computational scope of the attention mechanism. It determines which input information can interact and which needs to be ignored through a mask matrix; its core function is to constrain the allocation of attention weights.

[0069] In some embodiments, reference Figure 3A For the image generation task, during training, reference visual content 314 can be obtained, and features of the reference visual content 314 can be extracted using the VAE model 312, with noise 313 added to it. During inference, however, the reference visual content 314 is not required; noise can be directly input.

[0070] In some embodiments, the target model needs to alternately generate response content of different modalities, such as image-text-image-text. During model training, since the images are generated by a diffusion model, noise is added during training, causing the training of subsequent modalities to depend on these noisy images. Furthermore, during training, the input to subsequent modalities is noisy images, while during inference, the model actually generates clean images. That is, the model learns the "noisy image-text" mapping during training, but needs to process the "clean image-text" mapping during inference. This inconsistency causes a neighborhood shift, leading to a degraded model performance. Therefore, an efficient full teacher coercion method can be used for model training, which can solve the problem of inconsistency between the training and inference processes in multimodal interleaved generation tasks.

[0071] In some embodiments, the sample data includes first sample text, first sample visual content, second sample visual content, and second sample text, and the training feature unit sequence includes: reference Figure 6 Text 601 is the first part corresponding to the first sample text; image 602 is the second part corresponding to the first trained visual feature of the first sample visual content; image 604 is the third part corresponding to the second trained visual feature of the first sample visual content; image 603 is the fourth part corresponding to the third trained visual feature of the second sample visual content; image 605 is the fifth part corresponding to the fourth trained visual feature of the second sample visual content; text 606 is the sixth part corresponding to the second sample text; noisy image 607 is the seventh part corresponding to the first noisy visual feature of the first sample visual content; and noisy image 608 is the eighth part corresponding to the second noisy visual feature of the second sample visual content.

[0072] The first training visual feature is determined using the first visual encoding model, the second training visual feature is determined using the second visual encoding model, the third training visual feature is determined using the first visual encoding model, and the second training visual feature is determined using the second visual encoding model.

[0073] In some embodiments, because all image modalities are subjected to noise perturbation by the diffusion model during training, the input of subsequent modalities is forced to rely on these noisy images, leading to inconsistencies between training and inference. Furthermore, if some tasks do not require noisy images, forcibly adding noise to all images introduces unnecessary computational overhead. Therefore, training data can be divided into clean data and noisy data. Clean data is the data without noise, while noisy data is the data with added noise. By separating clean and noisy data, the model can learn different tasks more efficiently, thereby avoiding interference from noise on non-diffusion tasks.

[0074] In some embodiments, determining an attention mask corresponding to a task type based on the task type of the training task associated with the target model includes at least one of the following: in response to a training task being a text generation task associated with the second sample text content, determining a first attention mask, the first attention mask indicating that the sixth part is associated with the first part, the third part, and the fifth part; in response to a training task being a visual content generation task associated with the first sample visual content, determining a second attention mask, the second attention mask indicating that the seventh part is associated with the first part; in response to a training task being a visual content generation task associated with the second sample visual content, determining a third attention mask, the second attention mask indicating that the eighth part is associated with the first part, the second part, and the third part.

[0075] For example, refer to Figure 7 Part 6 (Text 1), associated with Part 1 (Text 0), Part 3 (Image 0'), and Part 5 (Image 1'), is shown in black, indicating that the historical information corresponding to Part 6 includes Part 1 (Text 0), Part 3 (Image 0'), and Part 5 (Image 1'), while the rest are hidden. Part 7, associated with Part 1 (Text 0), is shown in black, indicating that the historical information corresponding to Part 7 includes Part 1 (Text 0), while the rest are hidden. Part 8, associated with Part 1 (Text 0), Part 2 (Image 0), and Part 3 (Image 0'), is shown in black, indicating that the historical information corresponding to Part 8 includes Part 1 (Text 0), Part 2 (Image 0), and Part 3 (Image 0'), while the rest are hidden.

[0076] pass Figure 7 The attention mask allows the target model to focus only on clean data when generating text, and on noisy data and associated text when generating images. This method eliminates the need to process noisy data during text generation, saving computational resources, and ensures the text decoder is always trained on clean data, aligning with inference conditions.

[0077] In some embodiments, classifier-free guidance can also be used to improve the quality of model generation. Classifier-free guidance is a commonly used technique in generative models to balance the determinism and randomness of the model when generating samples, thereby improving the quality and controllability of the generated samples. This technique combines conditional and unconditional prediction to generate results that better meet specified conditions. The formula is as follows:

[0078]

[0079] c represents a condition that can be decomposed into image modalities c. img and text modality c txtγ represents the classifier-free guidance coefficient. Images within the given conditions can provide a shortcut for the model to generate subsequent images, leading to a phenomenon similar to time standstill. Different classifier-free guidance coefficients γ and γ can also be introduced. img To control the influence of different modes, the formula is as follows:

[0080]

[0081] To effectively adjust the influence of different modes on image generation, classifier-free guidance coefficients γ = 7.5 and γ can be used respectively. img =1.5.

[0082] In some embodiments, a fixed resolution strategy can reduce the efficiency of both model training and inference, and limit the model's visual understanding capabilities. Therefore, a native resolution strategy can be adopted to utilize the original resolution information of the image as much as possible, thereby reducing unnecessary information loss or distortion. The input image can be converted into a variable number of visual feature units; that is, the number of visual feature units generated is not fixed and will vary depending on the content and characteristics of the image. Furthermore, since the original aspect ratio of the image is not changed during the conversion of the image into visual markers, the shape features of the image can be guaranteed not to be distorted.

[0083] In some embodiments, when training a complex multimodal model using multiple GPUs, the loss and weights of each data sample differ across GPUs. Therefore, it is necessary to find a suitable balance point so that the model learns well on different tasks. For example, the weights on each GPU can be summed to calculate the average weight, and then this average weight can be normalized to obtain an approximate but unbiased gradient. Since only a small amount of scalar data needs to be transferred between GPUs and asynchronous communication is used, the communication cost is very low and will not affect training efficiency. The formula can be as follows:

[0084]

[0085] The loss for the i-th feature unit, Let n be the weight, and n represent the nth GPU.

[0086] In some embodiments, the model can be trained using Next Token Prediction Loss (NTP), a metric used to measure a model's performance on the next token prediction task. Training the model by minimizing the NTP loss enables it to predict the next token in the sequence more accurately. For example, the model can be optimized by minimizing the cross-entropy between the probability distribution of the model's output and the data distribution. The NTP loss L... NTP The formula is as follows:

[0087]

[0088] z = (z0, ..., z) L ) represents the input text content, θ VLM The parameters of the target model are represented by x = (x0, ..., x). N Let φ represent image information, and x can be φ. The probability distribution output by the model is the probability distribution predicted by the model based on the input information, while the data distribution is the probability distribution of each event occurring in the real data. Cross-entropy is an indicator used to measure the degree of difference between two probability distributions; the smaller the cross-entropy, the closer the two distributions are. In this way, the probability distribution predicted by the model can be made closer to the distribution of the real data.

[0089] In some embodiments, rectified flow matching can also be used to model the generated distribution of images. Flow matching is a method that maps a simple distribution to a complex data distribution by learning a probability path. Its core idea is to define a continuous transformation path to progressively transform noisy data into target data, and then approximate this ideal path by optimizing the model.

[0090] For example, we can define an image distribution x ~ π and a simple prior distribution ε ∈ N(0,1) to construct a linear trajectory defined over time t, as shown in the following formula:

[0091] x t = t·x + (1-t)·ε, t∈[0,1]

[0092] The ordinary differential equation can be obtained from the above formula:

[0093]

[0094] The velocity field is v = x - ε. To generate the image, the ordinary differential equation in the above formula can be integrated in reverse time, starting from x0 to N(0,1). Therefore, for image generation tasks, the model can be trained by optimizing the flow matching loss, where the flow matching loss L... flow The formula is as follows:

[0095]

[0096] In some embodiments, the sequence distribution can be modeled causally at the modal level, as shown in the following formula:

[0097]

[0098] θ represents the parameters of the model, x t Represents visual modality ximg Or text modal x txt Instead of a single label, the generated x... t+1 This will be added to the historical information as a condition. In this way, the target model can facilitate the generation of any mode conditioned on arbitrary mixed modes, i.e., staggered multimodal generation.

[0099] In some embodiments, the loss function may include NTP loss. NTP Matching loss L flow Overall loss L total The formula is as follows:

[0100] L total =λ·L NTP +L flow

[0101] In some embodiments, during training, the target model can distinguish between visual generation mode and text generation mode using two indicators. A first indicator <|vision_start|> and a second indicator <|visition_end|> are inserted at the beginning and end of each image sequence, respectively. During inference, the model can decide whether to generate an image based on the context. For example, if the first indicator is generated, the model switches to image generation mode; if the second indicator is generated, the model switches to text generation mode.

[0102] The embodiments of this disclosure can apply various visual coding models to extract visual features, thereby enabling the target model to possess both understanding and image generation capabilities. Furthermore, the multi-dimensional position embedding method can enhance the model's positional awareness. By applying multiple attention units and multiple feedforward units to process information from different modalities, feature units from multiple modalities can be deeply fused in each processing block, thereby generating higher-quality responses. Additionally, during model training, the application of attention masks can mitigate problems caused by inconsistencies between the training and inference processes.

[0103] Example devices and equipment

[0104] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 9 A schematic structural block diagram of a device 900 for touch interaction according to certain embodiments of the present disclosure is shown. The device 900 may be implemented as or included in the electronic device 110 discussed above. The various modules / components in the device 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0105] like Figure 9As shown, the apparatus 900 includes an acquisition module 910 configured to acquire a processing request for a target model; a determination module 920 configured to, in response to the processing request including input visual content, determine a first visual feature of the input visual content using a first visual encoding model and determine a second visual feature of the input visual content using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; a construction module 930 configured to construct an input feature unit sequence corresponding to the processing request based at least on the first and second visual features; and a generation module 940 configured to acquire at least one output feature unit generated by the target model based on the input feature unit sequence to generate response content for the processing request.

[0106] In some embodiments, at least one output feature unit includes a set of text feature units and a set of visual feature units, and the response content includes output text content corresponding to the set of text feature units and output visual content corresponding to the set of visual feature units.

[0107] In some embodiments, the apparatus 900 is further configured to detect a preset first indicator from the output of the target model; and in response to detecting the indicator, to perform a noise reduction process using the target model to generate a set of visual feature units.

[0108] In some embodiments, the target model is further configured to output a preset second indicator in response to generating a set of visual units based on the noise reduction process.

[0109] In some embodiments, the apparatus 900 is further configured to determine a third visual feature of the output visual content using a first visual coding model and a fourth visual feature of the output visual content using a second visual coding model; and to update the input feature unit sequence based on the third and fourth visual features to generate subsequent output feature units by the target model.

[0110] In some embodiments, the construction module 930 is configured to determine the text features corresponding to the input text content; and to construct and process a sequence of input feature units corresponding to the request based on the first visual features, the second visual features, and the text content.

[0111] In some embodiments, the input feature unit sequence includes a visual sequence portion corresponding to a first visual feature and a second visual feature, and the visual sequence portion is associated with a preset indicator.

[0112] In some embodiments, the target model includes a plurality of processing blocks, each processing block including a first attention unit, a second attention unit and a fusion unit, wherein the first attention unit is configured to process a first visual feature and a text feature unit in an input feature unit sequence, the second attention unit is configured to process a second visual feature, and the fusion unit is configured to fuse the first output result of the first attention unit and the second output result of the second attention unit.

[0113] In some embodiments, the first set of parameters of the first attention unit is independent of the second set of parameters of the second attention unit.

[0114] In some embodiments, the target model is configured to: for a target feature unit in an input feature unit sequence, determine index information of the target feature unit based on the position of the target feature unit in the input feature unit sequence and the type of the target feature unit; and determine attention information associated with the target feature unit based on the index information.

[0115] In some embodiments, the apparatus 900 is configured to, in response to a target feature unit corresponding to text content, determine a first value of a first dimension of index information based on sequence position; and set other dimensions of the index information to preset values.

[0116] In some embodiments, the apparatus 900 is configured to, in response to a target feature unit corresponding to an image patch in an image, determine a second value of a first dimension of index information based on a sequence position, wherein multiple feature units corresponding to an image are associated with the same sequence position; and determine a third value of the second dimension and a fourth value of the third dimension of the index information based on the two-dimensional position of the image patch in the image.

[0117] In some embodiments, the target model is trained based on the following process: acquiring sample data; constructing a sequence of training feature units based on the sample data; determining an attention mask corresponding to the task type based on the task type of the training task associated with the target model; and training the target model based on the training feature sequence and the attention mask.

[0118] In some embodiments, the sample data includes first sample text, first sample visual content, second sample visual content, and second sample text, and the training feature unit sequence includes: a first part corresponding to the first sample text; a second part corresponding to a first training visual feature of the first sample visual content, the first training visual feature being determined using a first visual encoding model; a third part corresponding to a second training visual feature of the first sample visual content, the second training visual feature being determined using a second visual encoding model; a fourth part corresponding to a third training visual feature of the second sample visual content, the third training visual feature being determined using a first visual encoding model; a fifth part corresponding to a fourth training visual feature of the second sample visual content, the second training visual feature being determined using a second visual encoding model; a sixth part corresponding to the second sample text; a seventh part corresponding to a first noisy visual feature of the first sample visual content; and an eighth part corresponding to a second noisy visual feature of the second sample visual content.

[0119] In some embodiments, determining an attention mask corresponding to a task type based on the task type of the training task associated with the target model includes at least one of the following: in response to a training task being a text generation task associated with the second sample text content, determining a first attention mask, the first attention mask indicating that the sixth part is associated with the first part, the second part, and the fourth part; in response to a training task being a visual content generation task associated with the first sample visual content, determining a second attention mask, the second attention mask indicating that the seventh part is associated with the first part; in response to a training task being a visual content generation task associated with the second sample visual content, determining a third attention mask, the second attention mask indicating that the eighth part is associated with the first part, the second part, and the third part.

[0120] In some embodiments, the first visual coding model includes a coding model based on a visual transducer, and the second visual coding model includes a coding model based on a variational autoencoder.

[0121] The units included in device 900 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 900 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0122] Figure 10 A block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 10 The electronic device 1000 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 10 The electronic device 1000 shown can be used to achieve Figure 1 The electronic device 110 shown.

[0123] like Figure 10 As shown, electronic device 1000 is in the form of a general-purpose electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processing unit 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 1000.

[0124] Electronic device 1000 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1030 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 1000.

[0125] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 10 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0126] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0127] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 1000, or with any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0128] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0129] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0130] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0131] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Obtain the processing request for the target model; In response to the processing request including input visual content, a first visual feature of the input visual content is determined using a first visual encoding model, and a second visual feature of the input visual content is determined using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model. Based at least on the first visual feature and the second visual feature, construct an input feature unit sequence corresponding to the processing request; as well as Obtain at least one output feature unit generated by the target model based on the input feature unit sequence to generate response content for the processing request.

2. The method according to claim 1, wherein the at least one output feature unit comprises a set of text feature units and a set of visual feature units, and the response content comprises output text content corresponding to the set of text feature units and output visual content corresponding to the set of visual feature units.

3. The method according to claim 2, further comprising: Detect a preset first indicator from the output of the target model; as well as In response to the detection of the indicator, a noise reduction process is performed using the target model to generate the set of visual feature units.

4. The method according to claim 3, wherein the target model is further configured as follows: In response to generating the set of visual units based on the noise reduction process, a preset second indicator is output.

5. The method according to claim 2, further comprising: The third visual feature of the output visual content is determined using the first visual coding model, and the fourth visual feature of the output visual content is determined using the second visual coding model. as well as Based on the third visual feature and the fourth visual feature, the input feature unit sequence is updated so that the target model can generate subsequent output feature units.

6. The method of claim 1, wherein the processing request further includes input text content, and constructing an input feature unit sequence corresponding to the processing request based at least on the first visual feature and the second visual feature includes: Determine the text features corresponding to the input text content; as well as Based on the first visual feature, the second visual feature, and the text content, the input feature unit sequence corresponding to the processing request is constructed.

7. The method of claim 6, wherein the input feature unit sequence includes a visual sequence portion corresponding to the first visual feature and the second visual feature, and the visual sequence portion is associated with a preset indicator.

8. The method of claim 1, wherein the target model comprises a plurality of processing blocks, each processing block comprising a first attention unit, a second attention unit and a fusion unit, wherein the first attention unit is configured to process the first visual feature and the text feature unit in the input feature unit sequence, the second attention unit is configured to process the second visual feature, and the fusion unit is configured to fuse the first output result of the first attention unit and the second output result of the second attention unit.

9. The method of claim 8, wherein the first set of parameters of the first attention unit is independent of the second set of parameters of the second attention unit.

10. The method of claim 1, wherein the target model is configured as follows: For a target feature unit in the input feature unit sequence, based on the position and type of the target feature unit in the input feature unit sequence, the index information of the target feature unit is determined; and Based on the index information, attention information associated with the target feature unit is determined.

11. The method of claim 10, wherein determining the index information of the target feature unit based on the sequence position of the target feature unit in the input feature unit sequence and the type of the target feature unit includes: In response to the target feature unit corresponding to text content, a first value of the first dimension of the index information is determined based on the sequence position; as well as Set the other dimensions of the index information to preset values.

12. The method of claim 10, wherein determining the index information of the target feature unit based on the sequence position of the target feature unit in the input feature unit sequence and the type of the target feature unit includes: In response to the target feature unit corresponding to an image patch in an image, a second value of the first dimension of the index information is determined based on the sequence position, wherein multiple feature units corresponding to the image are associated with the same sequence position; as well as Based on the two-dimensional position of the image patch in the image, the third value of the second dimension and the fourth value of the third dimension of the index information are determined.

13. The method of claim 1, wherein the target model is trained based on the following process: Obtain sample data; Based on the sample data, a training feature unit sequence is constructed; Based on the task type of the training task associated with the target model, determine the attention mask corresponding to the task type; and The target model is trained based on the training feature sequence and the attention mask.

14. The method of claim 13, wherein the sample data includes first sample text, first sample visual content, second sample visual content, and second sample text, and the training feature unit sequence includes: The first part corresponding to the first sample text; The second part corresponds to the first training visual feature of the visual content of the first sample, wherein the first training visual feature is determined using the first visual coding model; The third part corresponds to the second training visual features of the visual content of the first sample, wherein the second training visual features are determined using the second visual coding model; The fourth part corresponds to the third training visual feature of the second sample visual content, wherein the third training visual feature is determined using the first visual coding model; The fifth part corresponds to the fourth training visual feature of the second sample visual content, wherein the second training visual feature is determined using the second visual coding model; The sixth part corresponding to the second sample text; The seventh part corresponding to the first noisy visual feature of the visual content of the first sample; as well as The eighth part corresponding to the second noisy visual feature of the second sample visual content.

15. The method of claim 14, wherein determining the attention mask corresponding to the task type based on the task type of the training task associated with the target model includes at least one of the following: In response to the training task being a text generation task associated with the content of the second sample text, a first attention mask is determined, the first attention mask indicating that the sixth part is associated with the first part, the third part, and the fifth part; In response to the training task being a visual content generation task associated with the visual content of the first sample, a second attention mask is determined, the second attention mask indicating that the seventh part is associated with the first part; In response to the training task being a visual content generation task associated with the second sample visual content, a third attention mask is determined, wherein the second attention mask indicates that the eighth part is associated with the first part, the second part, and the third part.

16. The method of claim 1, wherein the first visual coding model comprises a coding model based on a visual transducer, and the second visual coding model comprises a coding model based on a variational autoencoder.

17. An information processing apparatus, comprising: The acquisition module is configured to acquire processing requests for the target model; The determination module is configured to, in response to the processing request including input visual content, determine a first visual feature of the input visual content using a first visual encoding model and determine a second visual feature of the input visual content using a second visual encoding model, wherein the first visual encoding model is different from the second visual encoding model; The construction module is configured to construct an input feature unit sequence corresponding to the processing request based at least on the first visual feature and the second visual feature; as well as The generation module is configured to acquire at least one output feature unit generated by the target model based on the input feature unit sequence, in order to generate response content for the processing request.

18. An electronic device comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processing unit.

19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 16.

20. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 16.

Images