Multi-modal information processing method and device

Abstract

The embodiment of the invention provides a multi-modal information processing method and device.The multi-modal information processing method comprises the steps that multi-modal information is obtained and input into a language model, and the language model comprises a feature extraction layer, a linear projection layer and a generation sub-model; processing the multi-modal information by using the feature extraction layer to obtain a first information feature and a second information feature, the first information feature and the second information feature corresponding to different information types; projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multi-modal fusion feature according to a projection result; and inputting the multi-modal fusion features and the multi-modal information into the generation sub-model for processing, and obtaining response information corresponding to the multi-modal information.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of machine learning technology, and in particular, to a multimodal information processing method and device. Background Art

[0002] With the development of computer and Internet technology, the technology of large language models has become more mature. Multimodal large language models have become the focus of development today. Current multimodal large language models (MLLMs), such as LLaVA and BLIP-2, mainly rely on the input of a single image. These models usually include a visual encoder to convert images into visual tokens, combined with a language model to process and generate multimodal outputs. For example, the LLaVA model uses a pre-trained visual transformer (ViT) to extract features from images, and projects these features onto visual tokens, which are then input into the language model together with the language token for processing. In multimodal learning, the core issue is how to effectively process and integrate a large amount of visual information and text information. In the existing technology, multimodal large language models generate a large number of visual tokens when processing multi-image inputs, which not only increases memory overhead, but may also lead to a decline in model performance. In addition, existing multimodal large language models (MLLMs) are usually only trained on single image datasets and cannot learn from multimodal instances, which limits their application in complex scenarios. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a multimodal information processing method. One or more embodiments of this specification also relate to a multimodal information processing apparatus, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a multimodal information processing method is provided, including: Acquire multimodal information, and input the multimodal information into a language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; Processing the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; Projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result; The multimodal fusion features and the multimodal information are input into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

[0005] Optionally, the using the feature extraction layer to process the multimodal information to obtain a first information feature and a second information feature includes: Extracting image information and text information from the multimodal information, and determining an image feature extraction layer and a text feature extraction layer in the feature extraction layer; Processing the image information using the image feature extraction layer to obtain a first information feature; The text information is processed using the text feature extraction layer to obtain a second information feature.

[0006] Optionally, projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result includes: Projecting the first information feature and the second information feature to a fused virtual unit through the linear projection layer, and generating a fused virtual unit sequence according to the projection results; Horizontally splicing the target fused virtual units contained in the fused virtual unit sequence to obtain multimodal fusion features corresponding to the multimodal information.

[0007] Optionally, generating a fused virtual unit sequence according to the projection result includes: Generate multiple initial fusion virtual units according to the projection results; Performing image hidden state adjustment processing on the multiple initial fused virtual units, and obtaining multiple target fused virtual units according to the processing results; A fused virtual unit sequence is generated based on the plurality of target fused virtual units.

[0008] Optionally, the determination of the fused virtual unit includes: fusing the first information feature with the feature space corresponding to the second information feature through forward propagation processing; A fused virtual unit having the same feature length as the second information feature is generated according to the fusion result.

[0009] Optionally, the training of the language model includes: Determining a multimodal information data set, and extracting multimodal sample information from the multimodal information data set; Inputting the multimodal sample information into an initial language model for processing to obtain predicted response information corresponding to the multimodal sample information; Optimizing an initial linear projection layer in the initial language model based on the sample response information corresponding to the multimodal sample information and the predicted response information; The language model is generated according to the optimization result of the initial linear projection layer.

[0010] Optionally, before the step of generating the language model according to the optimization result of the initial linear projection layer is performed, the method further includes: According to a preset model learning rate and a model optimizer, detecting whether the initial language model optimized by the initial linear projection layer satisfies an optimization stop condition; If not, extracting candidate multimodal sample information from the multimodal information data set, using the candidate multimodal sample information as the multimodal sample information, and performing the step of inputting the multimodal sample information into the initial language model for processing; If so, the initial language model after the initial linear projection layer is optimized is used as the language model.

[0011] According to a second aspect of an embodiment of this specification, another multimodal information processing method is provided, which is applied to a server, including: Selecting a language model in response to multimodal information uploaded by a client, and inputting the multimodal information into the language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; Processing the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; Projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result; The multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, response information corresponding to the multimodal information is obtained, and the response information is fed back to the client.

[0012] According to a third aspect of an embodiment of this specification, a multimodal information processing device is provided, including: An acquisition module is configured to acquire multimodal information and input the multimodal information into a language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; a processing module, configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; A projection module, configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The generation module is configured to input the multimodal fusion features and the multimodal information into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

[0013] According to a fourth aspect of the embodiments of this specification, another multimodal information processing device is provided, which is applied to a server, including: An input model module is configured to select a language model in response to multimodal information uploaded by a client, and input the multimodal information into the language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model; an information processing module, configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; A projection feature module, configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The feedback information module is configured to input the multimodal fusion feature and the multimodal information into the generation sub-model for processing, obtain response information corresponding to the multimodal information, and feed back the response information to the client.

[0014] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned multimodal information processing method are implemented.

[0015] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned multimodal information processing method are implemented.

[0016] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned multimodal information processing method when executed by a processor.

[0017] The multimodal information processing method provided in this embodiment can improve the flexibility of the model and reduce memory overhead. After obtaining the multimodal information, the multimodal information can be input into the language model, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the multimodal information consumption in the processing of more memory overhead, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented by the fusion virtual unit at the same time, so as to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced through fusion, so that the model can output more accurate and higher quality response information for user convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a multimodal information processing method provided by an embodiment of this specification; Figure 2 is a flowchart of a multimodal information processing method provided by an embodiment of this specification; Figure 3 is a flowchart of another multimodal information processing method provided by an embodiment of this specification; Figure 4 is a processing flow chart of a multimodal information processing method provided by an embodiment of this specification; Figure 5 is a schematic diagram of the structure of a multimodal information processing device provided by an embodiment of this specification; Figure 6 is a schematic diagram of the structure of another multimodal information processing device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0019] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0020] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0022] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] First, the terms involved in one or more embodiments of this specification are explained.

[0024] Multimodal Large Language Models (MLLMs) is a technology that combines multi-modal information such as vision and hearing with large language models (LLMs). This model can process and understand data from different modalities, such as text, images, audio, etc., and generate natural language output based on it.

[0025] LLaVA (Large Language and Vision Assistant) is a large multimodal model trained end-to-end that connects a visual encoder and a large language model to achieve general vision and language understanding. LLaVA is designed to improve the performance of the model on a variety of tasks such as image understanding, OCR (optical character recognition), knowledge information extraction (KIE), etc. by combining visual and language information.

[0026] BLIP-2 is an innovative low-computational cost vision-language pre-training large model that reduces pre-training costs and improves performance on vision-language tasks by combining a frozen image encoder and a large language model.

[0027] Visual token usually refers to the process of dividing the image into small patches or feature areas when processing image data, and converting these small patches or feature areas into vector representations (embedding). These vector representations are called "visual tokens", which contain local information in the image and can be used for subsequent model processing and analysis.

[0028] In this specification, a multimodal information processing method is provided. This specification also relates to a multimodal information processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0029] See also Figure 1 As shown in the schematic diagram, the multimodal information processing method provided by this embodiment can, in order to improve the flexibility of the model and reduce the memory overhead, input the multimodal information into the language model after acquiring the multimodal information, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the multimodal information consumption in the processing of more memory overhead, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented by the fusion virtual unit at the same time, so as to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced through fusion, so that the model can output more accurate and higher quality response information for user convenience.

[0030] See also Figure 2 , Figure 2 A flowchart of a multimodal information processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0031] Step S202, obtaining multimodal information, and inputting the multimodal information into a language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model.

[0032] The multimodal information processing method provided in this embodiment can be applied to any visual language task, such as generating a title for an image, generating a question and answer result for an image, generating description information for an input image, etc. This embodiment takes the application of generating a title for an image as an example to illustrate the multimodal information processing method. The descriptions in other scenarios can refer to the same or corresponding descriptions in this embodiment, and this embodiment will not be repeated in detail here.

[0033] Specifically, multimodal information specifically refers to multimedia information containing at least two different types of information, and the information contained includes but is not limited to at least two of the following: image information, text information, video information, audio information, etc. This embodiment does not make any restrictions here. Correspondingly, the language model specifically refers to a multimodal large language model, which can be used to input information of multiple modalities to generate corresponding response information according to needs, such as inputting an image and a description text of the image, and outputting an image title; or inputting audio and a description of the audio, and outputting an audio content summary, etc. During specific implementation, the application scenario of the language model can be set according to actual needs, and the functions that can be implemented can also be trained according to actual needs. This embodiment does not make any restrictions here.

[0034] Correspondingly, the feature extraction layer specifically refers to a layer used to extract features from each modal information in the multimodal information. Correspondingly, the linear projection layer specifically refers to a layer that can cross-modally fuse information features corresponding to different modal information, and then can be used as the input of the generated sub-model. Correspondingly, the generated sub-model is a character model in the language model used to generate predicted answers for input questions, which can be a large language model LLM. In practical applications, the selection of the generated sub-model can be set according to actual needs, and this embodiment does not make any limitation here.

[0035] Based on this, in order to improve the flexibility of the model and reduce memory overhead, after obtaining the multimodal information, the multimodal information can be input into the language model, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the multimodal information consumption in the processing of more memory overhead, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented by the fusion virtual unit at the same time, so as to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced by fusion, so that the model can output more accurate and higher quality response information for user use.

[0036] Step S204: Process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types.

[0037] Specifically, after the multimodal information is obtained and input into the language model, in order to realize cross-modal fusion of different modal information, thereby improving the accuracy of the answer output by the generated sub-model and reducing the memory overhead of the fusion of different modal information, the multimodal information can be processed using the feature extraction layer in the language model, and then the first information feature and the second information feature corresponding to the different information types can be obtained according to the processing results. Subsequently, the first information feature and the second information feature are linearly projected to realize cross-modal information fusion, so as to facilitate the subsequent use of the generated sub-model.

[0038] The first information feature and the second information feature specifically refer to vector expressions corresponding to information of different information types in the multimodal information. For example, if the multimodal information includes image information and text information, the first information feature is the information feature corresponding to the image information, and the second information feature is the information feature corresponding to the text information.

[0039] Furthermore, considering that multimodal information contains different types of information, the feature extraction layer in the language model needs to set feature extraction layers corresponding to different modalities to complete information feature extraction. In this embodiment, the specific implementation method is as follows: Extracting image information and text information from the multimodal information, and determining an image feature extraction layer and a text feature extraction layer in the feature extraction layer; processing the image information using the image feature extraction layer to obtain a first information feature; and processing the text information using the text feature extraction layer to obtain a second information feature.

[0040] Specifically, the image feature extraction layer refers to a processing layer that can process image information separately to obtain the visual token corresponding to the image information. It can be implemented using the generator VPG; correspondingly, the text feature extraction layer refers to a processing layer that can process text information separately to obtain the hidden state of the corresponding text information.

[0041] Based on this, after the multimodal information is input into the language model, the image information and text information can be first extracted from the multimodal information, and the image feature extraction layer and the text feature extraction layer can be determined in the feature extraction layer; on this basis, the image information can be processed using the image feature extraction layer to obtain the first information feature; and the text information can be processed using the text feature extraction layer to obtain the second information feature, so as to subsequently perform multimodal information fusion and response information prediction based on the first information feature and the second information feature.

[0042] For example, a user uploads an image of red maple leaves blown down by the wind through the client interface, and enters a question text {Please generate an article describing the beauty of autumn based on the image} for the image. At this time, the image and question text can be input into the multimodal language model. Then the image can be input into the visual prompt generator VPG in the multimodal language model for processing, and the visual token of the corresponding image can be obtained. At the same time, the question text can be input into the MLLM built into the multimodal language model for processing, and the hidden state corresponding to the question text can be obtained. After that, the article generation process can be performed based on the visual token corresponding to the image and the hidden state corresponding to the question text.

[0043] In summary, by setting different feature extraction layers for different modal information, it is possible to accurately extract the feature expression corresponding to the same modal information, which can be facilitated for subsequent use.

[0044] Step S206: project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result.

[0045] Specifically, after obtaining the first information feature and the second information feature corresponding to different information types, further, considering that different information features correspond to different information types, and the information features corresponding to different information types are obtained by mapping the corresponding feature space, that is, the information features corresponding to the image and the information features corresponding to the text are vector expressions obtained by mapping the information in different feature spaces, and they are not interoperable with each other. In order to fully understand the image semantics and text semantics in the language model processing stage, the first information feature and the second information feature can be projected to the fused virtual unit through the linear projection layer, so that the semantic information of the associated image and text can be simultaneously mapped out through the fused virtual unit. In this way, multimodal fusion features are generated according to the projection results, so that the multimodal fusion features that fuse the multimodal information can be used to complete the prediction of the response information.

[0046] The fused virtual unit specifically refers to a virtual token that can fuse the information features corresponding to the image and the information features corresponding to the text, so that the image and the text can express their respective semantic information through the virtual token and can be fused with each other. Correspondingly, the multimodal fusion feature specifically refers to the vector expression obtained by fusing the information features corresponding to the image and the information features corresponding to the text.

[0047] Furthermore, when constructing multimodal fusion features, in order to reflect the semantic fusion results of different types of information through the feature, a sequence construction method can be used to obtain the overall expression corresponding to the multimodal information, and then splicing can be performed to obtain the multimodal fusion feature. In this embodiment, the specific implementation method is as follows: The first information feature and the second information feature are projected to a fused virtual unit through the linear projection layer, and a fused virtual unit sequence is generated according to the projection results; horizontal splicing is performed on the target fused virtual units contained in the fused virtual unit sequence to obtain a multimodal fusion feature corresponding to the multimodal information.

[0048] Specifically, the fused virtual unit sequence refers to a sequence of multiple target fused virtual units obtained after projecting the first information feature and the second information feature onto the fused virtual unit, and the target fused virtual unit contained therein is a vector expression containing text features and image features.

[0049] Based on this, after obtaining the first information feature and the second information feature, in order to realize the fusion of multimodal information features, the first information feature and the second information feature can be projected to the fused virtual unit through a linear projection layer, so as to generate a fused virtual unit sequence containing multiple target fused virtual units according to the projection result; all feature fusions of the first information feature and the second information feature can be completed through the fused virtual unit sequence, and then horizontal splicing is performed on the target fused virtual units contained in the fused virtual unit sequence to obtain the multimodal fusion features corresponding to the multimodal information, so as to facilitate the subsequent prediction of the response information.

[0050] In summary, by constructing a fused virtual unit sequence and horizontally splicing it, the multimodal information features can be effectively fused, which facilitates the subsequent response information prediction.

[0051] Furthermore, considering that information features of different modalities correspond to different feature spaces, in order to achieve the fusion of information of different modalities and reduce interference, the image hidden state adjustment method can be used to ensure the projection result. In this embodiment, the specific implementation method is as follows: A plurality of initial fused virtual units are generated according to the projection results; image hidden state adjustment processing is performed on the plurality of initial fused virtual units, and a plurality of target fused virtual units are obtained according to the processing results; and a fused virtual unit sequence is generated based on the plurality of target fused virtual units.

[0052] Specifically, the multiple initial fused virtual units specifically refer to the fused virtual tokens obtained by projecting the first information feature and the second information feature onto the fused virtual unit. Correspondingly, the image hidden state adjustment specifically refers to the processing operation of discarding the image hidden feature after fusing the first information feature and the second information feature.

[0053] Based on this, when performing information feature projection processing, multiple initial fused virtual units can be generated according to the projection results; on this basis, in order to achieve cross-modal information fusion, image hidden state adjustment processing can be performed on multiple initial fused virtual units, thereby discarding the image hidden state and obtaining multiple target fused virtual units according to the processing results; on this basis, a fused virtual unit sequence can be generated based on multiple target fused virtual units.

[0054] In addition, in order to support the fusion of different modal information by the fusion virtual unit, the image information features can be first mapped to the text feature space to obtain the image features represented by the text expression, which can be used as the fusion virtual unit, and then the text information features can be fused to obtain the fusion result of the multimodal information features. In this embodiment, the specific implementation method is as follows: The first information feature is fused with the feature space corresponding to the second information feature through forward propagation processing; and a fused virtual unit having the same feature length as the second information feature is generated according to the fusion result.

[0055] Specifically, the feature space corresponding to the second information feature specifically refers to a vector expression mapping space for constructing information features for text information.

[0056] Based on this, in order to achieve the fusion of image features and text features, a fused virtual unit can be constructed first. At this time, it is considered that the first information feature corresponding to the image information and the second information feature corresponding to the text information are vector expressions corresponding to different dimensions and have different lengths. Therefore, the first information feature can be fused with the feature space corresponding to the second information feature through forward propagation processing, so as to achieve the expression of the first information feature through the information feature space corresponding to the second information feature. After that, a fused virtual unit with the same feature length as the second information feature can be generated according to the fusion result; on this basis, the first information feature is projected to the fused virtual unit, and the image hidden state is discarded, so as to obtain a fused virtual unit sequence, which is convenient for the subsequent construction of multimodal fusion features.

[0057] In specific implementation, it can be understood that the information features corresponding to the image are forward propagated to the feature space corresponding to the text information. At this time, a fused virtual token can be obtained in which the image features are expressed through the feature space corresponding to the text information. After that, the information features corresponding to the text information are projected onto the fused virtual token to achieve the fusion of the image information features and the text information features, thereby obtaining a fused virtual unit sequence. And the fused virtual unit sequence has the same length as the information features corresponding to the text information, thereby reducing the memory space occupied. On this basis, the hidden state of the image in the fused virtual unit sequence is discarded, so that the multimodal fusion features can be obtained for subsequent use.

[0058] Continuing with the above example, after obtaining the visual token corresponding to the image and the hidden state corresponding to the question text, the visual token corresponding to the image can be combined with the feature space of the question text through forward propagation to generate a fused virtual token with the same length as the question text. After that, the hidden state corresponding to the question text can be re-projected to the fused virtual token using the linear layer in the multimodal language model, and the hidden state corresponding to the image can be discarded to obtain an instance sequence with the same length as the fused virtual token and the text. Furthermore, by horizontally splicing the fused virtual tokens in the instance sequence, an efficient instance sequence D can be obtained, instance sequence D=Ŷ1⊕Ŷ2⊕...⊕Ŷn. Among them, Ŷn represents the first to nth fused virtual tokens, and ⊕ represents a horizontal connection operation. After that, the instance sequence D, image and question text can be combined to perform article generation processing.

[0059] In summary, by constructing a fused virtual feature with the same length as the second information feature, the sequence length can be effectively shortened, and the integrity of the image information can be fully preserved, making it easier to construct the subsequent response information.

[0060] Step S208: input the multimodal fusion features and the multimodal information into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

[0061] Specifically, after obtaining the multimodal fusion features as described above, the multimodal fusion features and the multimodal information can be further input into the generation sub-model for processing, so as to realize the generation of response information corresponding to the multimodal information according to the processing results. The response information specifically refers to the answer to the multimodal information feedback, such as generating an image title, generating an article for an image, etc.

[0062] Continuing with the above example, after obtaining the instance sequence D, the instance sequence D, the image, and the question text can be input into the LLM in the multimodal language model for autoregressive processing, so that an article describing the beauty of autumn can be obtained and the article can be fed back to the user.

[0063] Furthermore, in order to provide accurate prediction capabilities through the language model, the linear projection layer can be optimized during the model training phase so that the layer can effectively integrate information features of different modalities. In this embodiment, the specific implementation method is as follows: Determine a multimodal information data set and extract multimodal sample information from the multimodal information data set; input the multimodal sample information into an initial language model for processing to obtain predicted response information corresponding to the multimodal sample information; optimize an initial linear projection layer in the initial language model based on sample response information corresponding to the multimodal sample information and the predicted response information; and generate the language model according to the optimization result of the initial linear projection layer.

[0064] Specifically, the multimodal information dataset specifically refers to a dataset used to train a language model, which may be an MMC4 multimodal dataset, which includes image and text pairs; it should be noted that in order to ensure the effectiveness of model training, the text labels in the multimodal dataset may not overlap. Accordingly, the initial language model specifically refers to a multimodal large language model that has not yet been trained. Accordingly, the predicted response information specifically refers to the predicted result output by the initial language model.

[0065] Based on this, in order to ensure the effectiveness of model training and improve the efficiency of model training, only the linear projection layer of the model can be trained during the model training phase, while the other parts remain frozen. Specifically, a multimodal information data set can be determined, and multimodal sample information can be extracted from the multimodal information data set; at this time, the multimodal sample information can be input into the initial language model for processing, and then the predicted response information corresponding to the multimodal sample information can be obtained; thereafter, the initial linear projection layer in the initial language model can be optimized based on the sample response information and predicted response information corresponding to the multimodal sample information; and a language model that meets the usage requirements can be trained by optimizing only the linear projection layer and freezing other layers.

[0066] In summary, by training the language model in the above way, the robustness and generalization ability of the model can be effectively improved, and stable prediction services can be provided after deployment.

[0067] In addition, in the model optimization stage, in order to improve the optimization efficiency and training accuracy, the learning rate and optimizer can be used. In this embodiment, the specific implementation method is as follows: According to the preset model learning rate and model optimizer, detect whether the initial language model after the initial linear projection layer is optimized meets the optimization stop condition; if not, extract candidate multimodal sample information from the multimodal information data set, use the candidate multimodal sample information as the multimodal sample information, and execute the step of inputting the multimodal sample information into the initial language model for processing; if so, use the initial language model after the initial linear projection layer is optimized as the language model.

[0068] Based on this, in the model optimization stage, the initial language model optimized by the initial linear projection layer can be detected according to the preset model learning rate and model optimizer to see whether it meets the optimization stop condition; if not, it means that the current model still needs to be trained, so the candidate multimodal sample information can be extracted from the multimodal information data set, and the candidate multimodal sample information can be used as the multimodal sample information, and the step of inputting the multimodal sample information into the initial language model for processing can be performed. If so, the initial language model optimized by the initial linear projection layer can be directly used as the language model.

[0069] The multimodal information processing method provided in this embodiment can improve the flexibility of the model and reduce memory overhead. After obtaining the multimodal information, the multimodal information can be input into the language model, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the multimodal information consumption in the processing of more memory overhead, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented by the fusion virtual unit at the same time, so as to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced through fusion, so that the model can output more accurate and higher quality response information for user convenience.

[0070] See also Figure 3 , Figure 3 A flowchart of another multimodal information processing method provided according to an embodiment of the present specification is shown, which is applied to a server and specifically includes the following steps.

[0071] Step S302, selecting a language model in response to the multimodal information uploaded by the client, and inputting the multimodal information into the language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model; Step S304, using the feature extraction layer to process the multimodal information to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; Step S306, projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result; Step S308: input the multimodal fusion feature and the multimodal information into the generation sub-model for processing, obtain response information corresponding to the multimodal information, and feed back the response information to the client.

[0072] The above is a schematic scheme of another multimodal information processing method of this embodiment. It should be noted that the technical scheme of this multimodal information processing method and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal information processing method can be found in the description of the technical scheme of the multimodal information processing method described above.

[0073] The following combination Figure 4 , taking the application of the multimodal information processing method provided in this specification in the image title generation scenario as an example, the multimodal information processing method is further described. Among them, Figure 4 A processing flow chart of a multimodal information processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0074] Step S402, obtaining multimodal information, and inputting the multimodal information into a language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model.

[0075] Step S404: extracting image information and text information from the multimodal information, and determining an image feature extraction layer and a text feature extraction layer in the feature extraction layer.

[0076] Step S406, using the image feature extraction layer to process the image information to obtain the first information feature, and using the text feature extraction layer to process the text information to obtain the second information feature.

[0077] Step S408: project the first information feature and the second information feature to a fused virtual unit through a linear projection layer, and generate a plurality of initial fused virtual units according to the projection results.

[0078] Step S410 , performing image hidden state adjustment processing on a plurality of initial fused virtual units, and obtaining a plurality of target fused virtual units according to the processing results.

[0079] Step S412: generating a fused virtual unit sequence based on multiple target fused virtual units, horizontally splicing the target fused virtual units included in the fused virtual unit sequence, and obtaining multimodal fusion features corresponding to the multimodal information.

[0080] Step S414: input the multimodal fusion features and the multimodal information into the generation sub-model for processing to obtain the title information corresponding to the image information in the multimodal information.

[0081] The multimodal information processing method provided in this embodiment can improve the flexibility of the model and reduce memory overhead. After obtaining the multimodal information, the multimodal information can be input into the language model, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the multimodal information consumption in the processing of more memory overhead, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented by the fusion virtual unit at the same time, so as to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced through fusion, so that the model can output more accurate and higher quality response information for user convenience.

[0082] Corresponding to the above method embodiment, this specification also provides a multimodal information processing device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of a multimodal information processing device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises: An acquisition module 502 is configured to acquire multimodal information and input the multimodal information into a language model, wherein the language model comprises a feature extraction layer, a linear projection layer and a generation sub-model; A processing module 504 is configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; A projection module 506 is configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The generation module 508 is configured to input the multimodal fusion features and the multimodal information into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

[0083] In an optional embodiment, the processing module 504 is further configured to: Extracting image information and text information from the multimodal information, and determining an image feature extraction layer and a text feature extraction layer in the feature extraction layer; processing the image information using the image feature extraction layer to obtain a first information feature; and processing the text information using the text feature extraction layer to obtain a second information feature.

[0084] In an optional embodiment, the projection module 506 is further configured to: The first information feature and the second information feature are projected to a fused virtual unit through the linear projection layer, and a fused virtual unit sequence is generated according to the projection results; horizontal splicing is performed on the target fused virtual units contained in the fused virtual unit sequence to obtain a multimodal fusion feature corresponding to the multimodal information.

[0085] In an optional embodiment, the projection module 506 is further configured to: A plurality of initial fused virtual units are generated according to the projection results; image hidden state adjustment processing is performed on the plurality of initial fused virtual units, and a plurality of target fused virtual units are obtained according to the processing results; and a fused virtual unit sequence is generated based on the plurality of target fused virtual units.

[0086] In an optional embodiment, the determination of the fused virtual unit includes: The first information feature is fused with the feature space corresponding to the second information feature through forward propagation processing; and a fused virtual unit having the same feature length as the second information feature is generated according to the fusion result.

[0087] In an optional embodiment, the training of the language model includes: Determine a multimodal information data set and extract multimodal sample information from the multimodal information data set; input the multimodal sample information into an initial language model for processing to obtain predicted response information corresponding to the multimodal sample information; optimize an initial linear projection layer in the initial language model based on sample response information corresponding to the multimodal sample information and the predicted response information; and generate the language model according to the optimization result of the initial linear projection layer.

[0088] In an optional embodiment, before the step of generating the language model according to the optimization result of the initial linear projection layer is executed, the method further includes: According to the preset model learning rate and model optimizer, detect whether the initial language model after the initial linear projection layer is optimized meets the optimization stop condition; if not, extract candidate multimodal sample information from the multimodal information data set, use the candidate multimodal sample information as the multimodal sample information, and execute the step of inputting the multimodal sample information into the initial language model for processing; if so, use the initial language model after the initial linear projection layer is optimized as the language model.

[0089] In order to improve the flexibility of the model and reduce the memory overhead, the multimodal information processing device provided in this embodiment can input the multimodal information into the language model after acquiring the multimodal information, wherein the language model includes a feature extraction layer, a linear projection layer and a generation sub-model; at this time, the multimodal information can be first processed by the feature extraction layer to obtain the first information feature and the second information feature, wherein the first information feature and the second information feature correspond to different information types; and then, in order to reduce the memory overhead consumed by the multimodal information in the processing, the first information feature and the second information feature can be projected to the fusion virtual unit through the linear projection layer, so that two different information features can be represented simultaneously through the fusion virtual unit to achieve the semantic fusion of each information in the multimodal information. Then, the multimodal fusion feature can be generated according to the projection result; on this basis, the multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, and the response information corresponding to the multimodal information can be generated according to the processing result, so that when processing the multimodal information, the memory overhead can be reduced by feature mapping. And the number of model processing features can be reduced through fusion, so that the model can output more accurate and higher quality response information for user convenience.

[0090] The above is a schematic scheme of a multimodal information processing device of this embodiment. It should be noted that the technical scheme of the multimodal information processing device and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal information processing device can be found in the description of the technical scheme of the multimodal information processing method described above.

[0091] Corresponding to the above method embodiment, this specification also provides another multimodal information processing device embodiment, Figure 6 FIG. 2 shows a schematic diagram of the structure of another multimodal information processing device provided by an embodiment of the present specification. Figure 6 As shown, the device is applied to the server, including: An input model module 602 is configured to select a language model in response to multimodal information uploaded by a client, and input the multimodal information into the language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model; An information processing module 604 is configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; The projection feature module 606 is configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The feedback information module 608 is configured to input the multimodal fusion feature and the multimodal information into the generation sub-model for processing, obtain response information corresponding to the multimodal information, and feed back the response information to the client.

[0092] The above is a schematic scheme of a multimodal information processing device of this embodiment. It should be noted that the technical scheme of the multimodal information processing device and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal information processing device can be found in the description of the technical scheme of the multimodal information processing method described above.

[0093] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0094] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0095] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0096] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0097] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned multimodal information processing method.

[0098] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the multimodal information processing method described above.

[0099] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned multimodal information processing method.

[0100] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the multimodal information processing method described above.

[0101] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned multimodal information processing method when executed by a processor.

[0102] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the multimodal information processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the multimodal information processing method described above.

[0103] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0105] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0106] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A multimodal information processing method, characterized in that: include: Acquire multimodal information, and input the multimodal information into a language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; Processing the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; Projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result; The multimodal fusion features and the multimodal information are input into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

2. The multimodal information processing method according to claim 1, characterized in that: The step of processing the multimodal information by using the feature extraction layer to obtain a first information feature and a second information feature includes: Extracting image information and text information from the multimodal information, and determining an image feature extraction layer and a text feature extraction layer in the feature extraction layer; Processing the image information using the image feature extraction layer to obtain a first information feature; The text information is processed using the text feature extraction layer to obtain a second information feature.

3. The multimodal information processing method according to claim 1, characterized in that: The projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result, includes: Projecting the first information feature and the second information feature to a fused virtual unit through the linear projection layer, and generating a fused virtual unit sequence according to the projection results; Horizontally splicing the target fused virtual units contained in the fused virtual unit sequence to obtain multimodal fusion features corresponding to the multimodal information.

4. The multimodal information processing method according to claim 3, characterized in that: The step of generating a fused virtual unit sequence according to the projection result includes: Generate multiple initial fusion virtual units according to the projection results; Performing image hidden state adjustment processing on the multiple initial fused virtual units, and obtaining multiple target fused virtual units according to the processing results; A fused virtual unit sequence is generated based on the plurality of target fused virtual units.

5. The multimodal information processing method according to any one of claims 1 to 4, characterized in that: The determination of the fused virtual unit includes: fusing the first information feature with the feature space corresponding to the second information feature through forward propagation processing; A fused virtual unit having the same feature length as the second information feature is generated according to the fusion result.

6. The multimodal information processing method according to any one of claims 1 to 4, characterized in that: The training of the language model includes: Determining a multimodal information data set, and extracting multimodal sample information from the multimodal information data set; Inputting the multimodal sample information into an initial language model for processing to obtain predicted response information corresponding to the multimodal sample information; Optimizing an initial linear projection layer in the initial language model based on the sample response information corresponding to the multimodal sample information and the predicted response information; The language model is generated according to the optimization result of the initial linear projection layer.

7. The multimodal information processing method according to claim 6, characterized in that: Before the step of generating the language model according to the optimization result of the initial linear projection layer is executed, the method further includes: According to a preset model learning rate and a model optimizer, detecting whether the initial language model optimized by the initial linear projection layer satisfies an optimization stop condition; If not, extracting candidate multimodal sample information from the multimodal information data set, using the candidate multimodal sample information as the multimodal sample information, and performing the step of inputting the multimodal sample information into the initial language model for processing; If so, the initial language model after the initial linear projection layer is optimized is used as the language model.

8. A multimodal information processing method, characterized in that: Applied to the server, including: Selecting a language model in response to multimodal information uploaded by a client, and inputting the multimodal information into the language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; Processing the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; Projecting the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generating a multimodal fusion feature according to the projection result; The multimodal fusion feature and the multimodal information are input into the generation sub-model for processing, response information corresponding to the multimodal information is obtained, and the response information is fed back to the client.

9. A multimodal information processing device, characterized in that: include: An acquisition module is configured to acquire multimodal information and input the multimodal information into a language model, wherein the language model comprises a feature extraction layer, a linear projection layer, and a generation sub-model; a processing module, configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; A projection module, configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The generation module is configured to input the multimodal fusion features and the multimodal information into the generation sub-model for processing to obtain response information corresponding to the multimodal information.

10. A multimodal information processing device, characterized in that: Applied to the server, including: An input model module is configured to select a language model in response to multimodal information uploaded by a client, and input the multimodal information into the language model, wherein the language model includes a feature extraction layer, a linear projection layer, and a generation sub-model; an information processing module, configured to process the multimodal information using the feature extraction layer to obtain a first information feature and a second information feature, wherein the first information feature and the second information feature correspond to different information types; A projection feature module, configured to project the first information feature and the second information feature to a fusion virtual unit through the linear projection layer, and generate a multimodal fusion feature according to the projection result; The feedback information module is configured to input the multimodal fusion feature and the multimodal information into the generation sub-model for processing, obtain response information corresponding to the multimodal information, and feed back the response information to the client.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images