A data processing method, system, device, computer program product and storage medium

Abstract

The embodiments of the present application provide a data processing method, system, device, computer program product and storage medium. In the embodiments of the present application, a retrieval enhancement device is designed, which can be connected to any multimodal large model to add a retrieval enhancement generation function to the multimodal large model. Moreover, the retrieval results are no longer directly input into the multimodal large model, but after the retrieval results are subjected to multimodal feature extraction and denoising by the retrieval enhancement device, the fused feature representation is provided to the multimodal large model. In this way, the contextual understanding ability of different industry knowledge can be quickly injected into the external retrieval enhancement device, so that the retrieval noise can be better removed. Therefore, it is no longer necessary to rely on the contextual understanding ability of the multimodal large model itself, so that the interference of the retrieval noise on the input results can be avoided, and the input effect of the multimodal large model can be effectively improved in the retrieval enhancement generation process.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, system, device, computer program product and storage medium. Background Art

[0002] Retrieval Augmented Generation (RAG) technology aims to provide more reliable and fact-based information to help solve the inherent defects of generative models, such as hallucination tendency and lack of professional ability.

[0003] At present, retrieval augmented generation technology has been applied to multimodal large models to assist them in outputting more accurate results. The basic principle of the existing multimodal retrieval augmented generation model (MMRAG) is that the multimodal large model receives the prompt input by the retriever, and the multimodal large model uses the prompt based on its own context understanding ability to perform retrieval augmented generation (RAG).

[0004] However, there is a lot of noise in these prompts, and the large multimodal model has limited ability to understand the context of different industry knowledge. Therefore, the process of using prompts is often disturbed by noise, resulting in unsatisfactory accuracy of the output results. Summary of the invention

[0005] Multiple aspects of the present application provide a data processing method, system, device, computer program product and storage medium for improving the output effect of a multimodal large model during the retrieval enhancement generation process.

[0006] The present application provides a data processing method applicable to a search enhancement device, the method comprising:

[0007] Retrieving target reference information adapted to the input data received by the multimodal large model from the knowledge base;

[0008] Extracting multimodal features from the multimodal reference data included in the target reference information;

[0009] Performing denoising processing on the extracted multimodal features to generate a first feature representation corresponding to the target reference information;

[0010] The first feature representation is input into the multimodal large model to assist the multimodal large model in generating an output result for the input data.

[0011] The embodiment of the present application also provides a retrieval enhancement device, including a multimodal retriever and a feature fusion module;

[0012] The multimodal retriever is used to retrieve target reference information that is compatible with input data received by the multimodal large model from the knowledge base;

[0013] The feature fusion module is used to extract multimodal features from the multimodal reference data included in the target reference information; perform denoising on the extracted multimodal features to generate a first feature representation corresponding to the target reference information;

[0014] The multimodal retriever is further used to input the first feature representation into the multimodal large model to assist the multimodal large model in generating output results for the input data.

[0015] An embodiment of the present application also provides a data processing system, including a retrieval enhancement device and a multimodal large model, wherein the retrieval enhancement device is connected to the multimodal large model, and the retrieval enhancement device executes the aforementioned data processing method to assist the multimodal large model in generating output results.

[0016] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the aforementioned data processing method.

[0017] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the aforementioned data processing method.

[0018] In an embodiment of the present application, a retrieval enhancement device is designed, which can be connected to any multimodal large model and can retrieve target reference information that is compatible with the input data received by the multimodal large model from a knowledge base; on this basis, the retrieval enhancement device can extract multimodal features from the multimodal reference data contained in the target reference information and perform denoising processing to generate a first feature representation corresponding to the target reference information; thereafter, the first feature representation can be input into the multimodal large model to assist the multimodal large model in generating output results. Accordingly, by connecting to the retrieval enhancement device, a retrieval enhancement generation function can be added to any multimodal large model. Moreover, the retrieval results are no longer directly input into the multimodal large model. Instead, the retrieval results are subjected to multimodal feature extraction and denoising processing by the retrieval enhancement device, and the fused feature representation is provided to the multimodal large model. In this way, the contextual understanding capabilities of different industry knowledge can be quickly injected into the external retrieval enhancement device, thereby better removing retrieval noise. Therefore, it is no longer necessary to rely on the contextual understanding capabilities of the multimodal large model itself, thereby avoiding the interference of retrieval noise on the input results, and further effectively improving the input effect of the multimodal large model during the retrieval enhancement generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A schematic diagram of the structure of a data processing system provided for an exemplary embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of a search enhancement device provided by an exemplary embodiment of the present application;

[0022] Figure 3 A schematic diagram of an exemplary network structure of a feature fusion module provided for an exemplary embodiment of the present application;

[0023] Figure 4 A schematic diagram of an exemplary application scenario provided for an exemplary embodiment of the present application;

[0024] Figure 5 A flowchart of a data processing method provided for another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] Before starting to explain in detail the technical solutions provided by the embodiments of the present application, several technical concepts involved in the present application are briefly explained as follows.

[0027] Retrieval-Augmented Generation (RAG) can be understood as a system that combines retrieval and generation techniques to improve the prediction quality and accuracy of natural language processing tasks. The basic principle is that the RAG system first retrieves information related to the current task from a huge knowledge base, and then uses the retrieved information to guide the text generation process. This combination of retrieval and generation enhances the model's prediction ability and accuracy.

[0028] A large model can refer to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large language models (LLM), multi-modal pre-training models, etc. It should be noted that in actual applications, the pre-trained model can be fine-tuned with a small number of samples so that the large model can be applied to different tasks. For example, large models can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc. They can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation.

[0029] The multimodal big model refers to the ability to process multimodal information inputs such as images, audio, and video, based on the aforementioned big model.

[0030] Multimodality can be understood as a way of using two or more senses to interact with information at the same time. In the field of artificial intelligence, multimodal technology improves the ability of artificial intelligence models to understand and process complex information by integrating data and information from different senses. Multimodal data can be understood as data and information that uses multiple senses, such as images, audio, video, text and other diverse data types. Based on multimodal technology, multimodal large models have the ability to directly process multimodal data (such as images and text).

[0031] During the research, the inventors found that there are some multimodal large models that support retrieval-enhanced generation. However, these multimodal large models usually integrate retrieval mechanisms. In the process of generating results, relevant knowledge is retrieved from external knowledge bases. These retrieval results will be used as prompts by the multimodal large models. The multimodal large models need to understand the background knowledge from the prompts based on their own contextual understanding capabilities to guide the generator to generate results. As introduced in the background technology, these prompts contain a lot of retrieval noise, and because the multimodal large models have limited contextual understanding capabilities for knowledge in different industries, this leads to frequent interference from retrieval noise during the text generation process. Therefore, the output effect of the multimodal large models that currently support retrieval-enhanced generation is not ideal.

[0032] To this end, this embodiment proposes a retrieval enhancement generation scheme and designs a retrieval enhancement device, which can be connected to any multimodal large model. Without the need to modify and fine-tune the multimodal large model, the retrieval enhancement generation function can be given to the multimodal large model that originally does not have the retrieval enhancement function. In other words, the retrieval enhancement device provided by this embodiment can be combined with any multimodal large model to form a data processing system. In practical applications, the retrieval enhancement device designed in this embodiment can be implemented as a plug-in. Based on this, the retrieval enhancement device can be connected to any multimodal large model in a pluggable manner, has good universality, and is compatible with various multimodal large model products.

[0033] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0034] Figure 1 The structural diagram of a data processing system provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the system includes: a retrieval enhancement device and a multimodal large model. As mentioned above, Figure 1 The multimodal large model in can be any multimodal large model that has the need for retrieval enhancement generation.

[0035] In this embodiment, a knowledge base is set up for the retrieval enhancement device, and the knowledge base contains reference information. It is worth emphasizing that the reference information in this embodiment contains multimodal reference data. In this embodiment, the modalities supported in the knowledge base may include but are not limited to images, videos, audios, 3D files, and texts, etc., and may also support web pages, heat maps, and charts, etc. No further examples of the modalities supported in the knowledge base are given here. It should be understood that from the perspective of a single reference information, this embodiment does not require that the reference information contain data of all the above modalities, but may contain data of some of the modalities according to actual needs. For example, a reference information may contain reference data of both image and text modalities.

[0036] In this embodiment, the knowledge base of the retrieval enhancement device can be flexibly filled with reference information of the required industry according to actual needs. Since the connected multimodal big model does not directly use the knowledge base, this flexible filling will not cause industry bias in the multimodal big model, that is, it will not affect the original industry versatility of the multimodal big model.

[0037] On this basis, reference Figure 1 , the user can submit input data to the multimodal large model. In this embodiment, the application scenario of the multimodal large model is not limited. For example, it can be an application scenario such as visual question answering or image understanding. In different application scenarios, the input data submitted by the user may be different. For example, in the visual question answering scenario, the input data submitted by the user may include data in two modes: image and question text. It should be understood that the input data in this embodiment supports multimodality.

[0038] refer to Figure 1 After the search enhancement device in this embodiment is connected to the multimodal large model, the input data received by the multimodal large model can be obtained. In practical applications, the search enhancement device can access the interface (Application Programming Interface, API) for receiving input data in the multimodal large model, from which the input data received by the multimodal large model can be obtained. Of course, when implementing software engineering, the acquisition method can be flexibly selected. This embodiment is not limited to this, and no further examples are given here.

[0039] The following will explain the functional logic inside the search enhancement device in detail.

[0040] refer to Figure 1, after acquiring the input data, the retrieval enhancement device can retrieve the target reference information that matches the input data from the knowledge base. In this link, the number of target reference information retrieved from the knowledge base can be one or more. Preferably, in this embodiment, the number of target reference information retrieved from the knowledge base can be set to multiple. In addition, the retrieved target reference information contains multimodal reference data. For example, an exemplary retrieval result in this embodiment may be [(i1:qa1), (i2:qa2)…(ik:qak)], which contains k target reference information, and each target reference information contains reference data of two modalities, i represents image, and qa represents text.

[0041] For example, the input data received by the multimodal large model may include an image and a question text, and the target reference information retrieved by the retrieval enhancement device from the knowledge base may include an image, a question text and an answer text. The answer text can provide background knowledge for the question text in the input data.

[0042] In this embodiment, the retrieval enhancement device does not directly provide the above retrieval results (that is, the target reference information) to the multimodal large model, but continues to process the retrieval results.

[0043] refer to Figure 1 For each retrieved target reference information, the retrieval enhancement device can extract multimodal features from the multimodal reference data contained in the target reference information. Here, the modalities involved in the multimodal features are consistent with the modalities involved in the multimodal reference data. For example, if the multimodal reference data contains images and texts, the multimodal features include visual features corresponding to the images and text features corresponding to the texts.

[0044] Preferably, in the feature extraction link, the encoding function of the multimodal large model can be reused to realize the extraction of multimodal features from the multimodal reference data. During the research process, the inventor found that the multimodal large model usually includes an encoding layer. Since the multimodal large model itself supports multimodality, the encoding layer of the multimodal large model usually includes encoders of multiple modes, such as visual encoders and word vector encoders. This embodiment proposes that these encoders in the multimodal large model can be reused to extract features from the multimodal reference data.

[0045] To this end, it is proposed in the preferred implementation scheme that: the retrieval enhancement device can input the multimodal reference data into the encoding layer of the multimodal large model; use the encoder contained in the encoding layer of the multimodal large model to encode the different modal data in the multimodal reference data to generate encoding features corresponding to the different modal data; and retrieve the generated encoding features from the encoding layer of the multimodal large model as the multimodal features corresponding to the multimodal reference data.

[0046] For example, if the multimodal reference data includes images and text (questions + answers), the visual encoder in the encoding layer of the multimodal large model can be used to encode the image to extract visual features; the word vector encoder in the encoding layer of the multimodal large model can also be used to encode the text to extract text features. The retrieval enhancement device can retrieve the visual features and text features extracted for the multimodal reference data from the encoding layer of the multimodal large model, which can be used as the multimodal features corresponding to the multimodal reference data.

[0047] In this preferred implementation, the encoding function of the multimodal large model is reused to extract multimodal features for the multimodal reference data, which can effectively save computing resources such as GPU memory. Of course, this is only exemplary. In this embodiment, encoders of multiple modes can also be set inside the retrieval enhancement device without reusing the encoder of the multimodal large model, which is not limited here.

[0048] In this way, multimodal features can be extracted to retrieve target reference information.

[0049] refer to Figure 1 After extracting multimodal features for the retrieved target reference information, the retrieval enhancement device may perform denoising on the extracted multimodal features to generate a first feature representation corresponding to the target reference information.

[0050] It can be understood that in this embodiment, the contextual understanding capability of industry knowledge can be flexibly injected into the retrieval enhancement device, so that the retrieval enhancement device can perform denoising on the extracted multimodal features through its own contextual understanding capability of industry knowledge. Here, denoising can be understood as removing redundant features in the multimodal features that are insufficiently correlated with the input data. Since the retrieval enhancement device is externally connected to the multimodal large model, in this embodiment, while keeping the multimodal large model unchanged, it is possible to support fine-turning of the retrieval enhancement device to inject the contextual understanding capability of industry knowledge into the retrieval enhancement device. Through fine-tuning, the retrieval enhancement device in this embodiment can obtain a sufficiently high contextual understanding capability, thereby ensuring the denoising effect. The implementation process of fine-tuning will be described in detail later.

[0051] Thus, the search enhancement device in this embodiment can extract multimodal features from the search results to more fully apply the reference data of various modalities in the search results to the search enhancement generation process, and through denoising, the search noise can be effectively reduced. That is, the search enhancement device can process the search results into feature representations from which the search noise has been removed.

[0052] refer to Figure 1 , the retrieval enhancement device may input the first feature representation into the multimodal large model to assist the multimodal large model in generating output results for the aforementioned input data. The feature representation space of the retrieval enhancement device in this embodiment may be consistent with the feature representation space of the multimodal large model, which can be achieved by pre-training the retrieval enhancement device, so that the first feature representation generated by the retrieval enhancement device can be directly used by the multimodal large model without any obstacles.

[0053] As mentioned above, the first feature representation has removed the retrieval noise, which allows the first feature representation to provide high-quality background knowledge with sufficient relevance for the multimodal large model, and therefore can effectively enhance the output effect of the multimodal large model. It can be understood that in this embodiment, the multimodal large model no longer needs to rely on its own contextual understanding ability to directly understand the context of the retrieval results, but the retrieval enhancement device in this embodiment can be responsible for the contextual understanding link and provide the multimodal large model with a denoised feature representation. In this way, without modifying or fine-tuning the multimodal large model, it is possible to avoid retrieval noise interference and optimize the output effect.

[0054] In this embodiment, the input method for inputting the first feature representation into the multimodal large model may be determined according to the input format supported by the multimodal large model. An optional input method may be:

[0055] From the multimodal large model, a second feature representation constructed for the input data is captured;

[0056] The concatenated features generated by concatenating the first feature representation and the second feature representation are input into the multimodal large model.

[0057] This input method does not require changing the input format of the multimodal large model and can be applied to various existing multimodal large model products, thereby making the retrieval enhancement device of this embodiment have better universality and compatibility.

[0058] It should be understood that this input method is only exemplary and the present embodiment is not limited thereto. After the search results have been denoised and characterized, the input method of the first feature representation can be flexibly designed to ensure that the first feature representation can be smoothly input into the generator in the multimodal large model. No further examples are given here.

[0059] In summary, a retrieval enhancement device is designed in this embodiment, which can be connected to any multimodal large model, and can retrieve target reference information that is compatible with the input data received by the multimodal large model from the knowledge base; on this basis, the retrieval enhancement device can extract multimodal features from the multimodal reference data contained in the target reference information and perform denoising processing to generate a first feature representation corresponding to the target reference information; thereafter, the first feature representation can be input into the multimodal large model to assist the multimodal large model in generating output results. Accordingly, by connecting to the retrieval enhancement device, a retrieval enhancement generation function can be added to any multimodal large model. Moreover, the retrieval results are no longer directly input into the multimodal large model. Instead, the retrieval results are subjected to multimodal feature extraction and denoising processing by the retrieval enhancement device, and the fused feature representation is provided to the multimodal large model. In this way, the contextual understanding capabilities of different industry knowledge can be quickly injected into the external retrieval enhancement device, thereby better removing retrieval noise. Therefore, it is no longer necessary to rely on the contextual understanding capabilities of the multimodal large model itself, thereby avoiding the interference of retrieval noise on the input results, and further effectively improving the input effect of the multimodal large model during the retrieval enhancement generation process.

[0060] In the above or below embodiments, the retrieval enhancement device may use a variety of implementation methods to perform denoising on the extracted multimodal features.

[0061] Figure 2 A schematic diagram of the structure of a search enhancement device provided by an exemplary embodiment of the present application. Figure 2 In the retrieval enhancement device provided in this embodiment, a multimodal retriever, a feature extraction module and a feature fusion module may be provided. Among them, the multimodal retriever may be responsible for the work logic related to the retrieval, the feature extraction module may be responsible for the multimodal feature extraction work on the retrieval results, and the feature fusion module may be responsible for the denoising of the multimodal features. It is worth noting that the multimodal retriever, feature extraction module and feature fusion module in this embodiment are module names defined according to the functional division of the retrieval enhancement device, and such module names should not limit the internal functional logic of the retrieval enhancement device.

[0062] Based on this, in an optional implementation, the retrieval enhancement device can input the extracted multimodal features into a feature fusion module, which has the context understanding capability required for denoising; the multimodal features can be denoised through the feature fusion module to output a first feature representation.

[0063] In this optional implementation, the feature fusion module may be pre-trained to inject the context understanding capability required for denoising into the feature fusion module. An exemplary pre-training scheme may be:

[0064] After the retrieval enhancement device and the multimodal large model are connected, an end-to-end training method is adopted to adjust the parameters of the feature fusion module included in the retrieval enhancement device while keeping the model parameters of the multimodal large model unchanged until the multimodal large model converges.

[0065] In this exemplary pre-training scheme, the retrieval enhancement device and the multimodal large model are combined to form a data processing system. The input end and the output end of the multimodal large model are the two ends of the data processing system. Figure 1 , and the retrieval enhancement device is a branch in the system. Here, the end-to-end training method can be understood as: inputting the training sample to the input end of the multimodal large model; using the retrieval enhancement device in this embodiment to generate the aforementioned first feature representation for the training sample; after providing the first feature representation generated for the training sample to the multimodal large model, the output end of the multimodal large model generates the output result corresponding to the training sample; by comparing the output result with the true value result corresponding to the training sample to calculate the loss function such as the cross entropy loss, and based on the calculated loss function, performing back propagation on the data processing system to adjust the parameters of the feature fusion module while keeping the model parameters of the multimodal large model unchanged until the multimodal large model converges. It is worth emphasizing here that in the back propagation process, the model parameters of the multimodal large model can be frozen, and only the relevant parameters of the feature fusion module in the retrieval enhancement device are adjusted, which can achieve fine-tuning of the feature fusion module so that the feature fusion module can obtain the contextual understanding ability of industry knowledge. After appropriate fine-tuning, when the multi-modal large model converges, it means that the feature fusion module has good context understanding ability and can accurately remove the retrieval noise in the retrieval results, and the pre-training can be ended. The pre-trained feature fusion module can be used to perform denoising in the inference stage.

[0066] This end-to-end pre-training method can more efficiently fine-tune the feature fusion module without affecting the original capabilities of the multimodal large model. Of course, this is only exemplary. In this embodiment, other pre-training methods can also be used to inject the required context understanding capabilities into the feature fusion module without affecting the multimodal large model. No further examples are given here.

[0067] In this embodiment, the feature fusion module can adopt a variety of network structures to support denoising of multimodal features. Figure 3 This is a schematic diagram of an exemplary network structure of a feature fusion module provided by an exemplary embodiment of the present application. Figure 3 In this exemplary network structure, the feature fusion module may include a first fusion network and a second fusion network. The first fusion network may be used to denoise a single piece of reference information, and the second fusion network may be used to denoise multiple pieces of reference information.

[0068] Based on this, the feature fusion module performs denoising on the multimodal features to output the first feature representation, which may include:

[0069] Inputting the multimodal features corresponding to the target reference information into the first fusion network, the number of the target reference information being multiple;

[0070] By using the first fusion network, a denoising process is performed on the multimodal features corresponding to a single target reference information as a unit to generate fusion features corresponding to each of the multiple target reference information;

[0071] Through the second fusion network, denoising is performed on a feature set composed of fused features of multiple target reference information to generate a first feature representation.

[0072] In this process, the second fusion network is connected after the first fusion network, and the denoising result of the first fusion network is further denoised to generate a first feature representation.

[0073] The denoising logic in the first fusion network and the second fusion network is described below.

[0074] refer to Figure 3 , the multimodal features extracted from the multiple target reference information can be input into the first fusion network. The first fusion network can perform denoising on the multimodal features of each target reference information. For ease of description, the first reference information is taken as an example to illustrate the denoising logic in the first fusion network. The first reference information can be any target reference information.

[0075] In the first fusion network, the text features included in the target multimodal features corresponding to the first reference information can be set as query information; according to the attention mechanism, the features that do not match the text features are screened out from the features of other modalities included in the target multimodal features to generate the fusion features corresponding to the first reference information.

[0076] It is understandable that the first fusion network can use the attention mechanism to denoise the multimodal features of a single reference information. In the attention mechanism, there are three core concepts: query, key and value. Among them, query represents the information to be queried, that is, the information or data that the model wants to obtain at the current moment. This process involves calculating the similarity or correlation between the query and all keys, and obtaining the attention weight distribution through the softmax function. This distribution is then used as a weight to weight the sum of all value vectors to obtain the context vector for each query position. This process allows the model to dynamically adjust the degree of importance attached to information at different positions according to the focus of the current query, so as to achieve effective extraction and integration of information.

[0077] Optionally, the first fusion network may adopt a structure such as Q-former, which is not limited here and no further examples are given.

[0078] In the first fusion network, the text features contained in the target multimodal features are used as queries in the attention mechanism, while the features of other modalities in the target multimodal features are used as key / value. Therefore, for the features of any other modality, features with sufficiently high correlation with the text features can be screened out, thereby filtering out features with insufficient correlation, which can achieve denoising at the level of single target reference information.

[0079] For example, if the multimodal features extracted for the first reference information include text features and visual features, the text features can be used as the query in the attention, and the feature parts with a sufficiently high correlation with the text features can be screened out from the visual features to achieve text-guided visual feature screening, thereby screening out visual features that are insufficiently relevant to the text.

[0080] In this way, the first fusion network can perform denoising for the multiple target reference information respectively. The first fusion network can generate fusion features for the multiple target reference information respectively.

[0081] refer to Figure 3 The second fusion network can obtain the fusion features corresponding to the multiple target reference information from the first fusion network, and these fusion features form a feature set; in the second fusion network, the retrieval features generated for the input data during the retrieval process can be set as query information; according to the attention mechanism, the fusion features that do not match the retrieval features are screened out from the feature set to generate a first feature representation.

[0082] It is understandable that the second fusion network can use the attention mechanism to continue to denoise the fused features of multiple target reference information. In the second fusion network, the retrieval features generated for the input data during the retrieval process can be used as the query in the attention mechanism, and the fused features of multiple target reference information can be used as the key / value. Therefore, from the fused features of multiple target reference information, fused features with a high enough correlation with the retrieval features can be screened out, thereby removing fused features with insufficient correlation, which can achieve denoising from the contextual relationship level between multiple target reference information.

[0083] Similarly, optionally, the second fusion network may also adopt a structure such as Q-former, which is not limited here and no further examples are given.

[0084] Based on this, the feature fusion module can be based on a two-layer fusion network. In the first layer of the fusion network, text-guided feature screening is implemented for a single target reference information. In the second layer of the fusion network, feature screening guided by the retrieval features of the input data is implemented for multiple target reference information. This cascaded network structure can not only remove the retrieval noise in each target reference information, but also reduce the interference that may be caused by abnormal target reference information, thereby effectively improving the robustness to retrieval noise and ensuring the output effect of the multimodal large model. In addition, after feature screening in the two-layer fusion network, the amount of data represented by the generated first feature can be effectively compressed, thereby providing more lightweight background knowledge to the multimodal large model, so that the multimodal large model can generate output results more efficiently.

[0085] It should be understood that the above network structure is only exemplary, and the feature fusion module may also adopt other network structures, for example, it may only include the above first fusion network, and use the output result of the first fusion network as the first feature representation, or it may only include the above second fusion network, and directly input the multimodal features of the target reference information into the second fusion network to generate the first feature representation. These network structures can also provide a certain denoising effect, and the network structure of the feature fusion module is not limited here, and no more examples are given.

[0086] In summary, in this embodiment, while keeping the multimodal large model unchanged, the feature fusion model in the retrieval enhancement device can be flexibly fine-tuned to inject the required contextual understanding capability into the feature fusion module. The contextual understanding capability that the feature fusion module can obtain is much higher than the contextual understanding capability of the multimodal large model itself. Therefore, the retrieval results can be better denoised and characterized, avoiding the retrieval noise from being passed to the multimodal large model, thereby effectively ensuring the output effect of the multimodal large model.

[0087] In the above or below embodiments, the search enhancement device may use a variety of implementation methods to retrieve suitable target reference information for input data.

[0088] In an optional implementation: the retrieval enhancement device can use the aforementioned multimodal retriever to encode the input data to obtain input features corresponding to different modal data in the input data; splice the obtained input features to generate retrieval features corresponding to the input data; and retrieve target reference information that matches the retrieval features from the knowledge base using the retrieval features as indexes.

[0089] The knowledge base in this embodiment may adopt a vector database, and may store the retrieval features (in vector format) corresponding to each reference information. Here, the retrieval features in the knowledge base comply with the feature representation space of the multimodal retriever to support the multimodal retriever to perform retrieval without obstacles. The retrieval features corresponding to the reference information are also generated by splicing. It should be understood that the feature representation space of the multimodal retriever may be unrelated to the feature representation space of the multimodal large model.

[0090] Furthermore, in this optional implementation, the multimodal retriever may select N reference information with the highest matching degree between the retrieval features and the retrieval features from the knowledge base based on the retrieval features of the input data; if there is special reference information containing the input data among the N reference information, the special reference information is deleted from the N reference information to obtain the target reference information.

[0091] Wherein, N is a positive integer, and the value of N can be flexibly set according to actual needs, for example, N can be set to 3, etc. It is proposed here to delete the special reference information containing the input data, which can effectively prevent the multimodal large model from taking a shortcut by directly using the retrieved target reference information containing the input data as the output result.

[0092] It should be understood that the above implementation is only exemplary. In this embodiment, the search enhancement device can also use other implementations to complete the search. For example, the search features in the knowledge base can follow the feature representation space of the multimodal large model. In this way, the multimodal searcher can use the second feature representation corresponding to the input feature mentioned above as the search basis to perform the search in the knowledge base. This embodiment is not limited to this, and no further examples are given here.

[0093] In summary, in this embodiment, the retrieval features can be used as indexes in the knowledge base, and in the reasoning stage, a multimodal retriever can be used to efficiently recall target reference information that matches the input data from the knowledge base.

[0094] Figure 4 A schematic diagram of an exemplary application scenario provided for an exemplary embodiment of the present application. Figure 4 In this exemplary application scenario, the retrieval enhancement device provided in this embodiment is implemented as a plug-in and pluggably connected to a multimodal large model to form a RAG system. The input data submitted by the user includes an image and a question "What urban governance problem is described in the picture?" That is, the input data submitted by the user contains data in two modalities: image and text. Figure 4 The multimodal models exemplified in the examples can be various multimodal large models such as Qwen-VL, LLaVA, InternVL, etc., which can be divided into three parts: encoding layer, multimodal mapping layer, and large language model. Figure 4Also shown are the multimodal retriever, knowledge base and feature fusion module set in the retrieval enhancement device.

[0095] The functional logic of the retrieval enhancement device in this exemplary application scenario is explained below from the fine-tuning stage and the reasoning stage respectively.

[0096] Fine-tuning stage

[0097] (1) For the sample images I and questions Q sampled in the training set, the visual feature representation is extracted through a multimodal model.

[0098] Input the image I into the "visual encoder" in the encoding layer of the multimodal large model to extract the visual feature Eimg, where the visual encoder can be a clip model. Input the visual feature Eimg into the multimodal mapping layer, map the visual feature Eimg to the feature representation space of the large language model, and obtain the visual feature representation Token Tproj. Encode the question Q through the "word vector encoding model" in the encoding layer to obtain the text feature representation Token Ttext. Concatenate the visual feature representation Token Tproj and the text feature representation Token Ttext to obtain the multimodal feature representation Token Tvlm (corresponding to the second feature representation mentioned above).

[0099] (2) For the above sample image I and question Q, retrieval features are extracted through a multimodal retriever.

[0100] The image I and the question Q are encoded by the multimodal retriever to obtain the visual feature Eimg_r and the text feature Etext_r, and then the visual feature Eimg_r and the text feature Etext_r are concatenated to obtain the retrieval feature Einput_r. The retrieval feature Einput_r is used as the input of the multimodal retriever to retrieve the Top-K related image-text pairs (image, question text and answer text) from the knowledge base as the retrieval result R = [(i1:qa1), (i2:qa2)…(ik:qak)]. In particular, k can be set to 3.

[0101] (3) For the retrieval result R, during the fine-tuning stage, the rule filtering module is used to filter out the image-text pairs containing the image I and the question Q to obtain the retrieval data R1.

[0102] (4) Input the images in the retrieval data R1 into the visual encoder mentioned in step (1) to extract the visual features Erag_i = [erag_i1,…,erag_ik], and input the texts (question-answer pairs) in the retrieval data R1 into the word vector encoding model mentioned in step (1) to encode and obtain the text features Erag_t = [erag_t1, …,erag_tk]. Among them, the visual feature erag_ii and the text feature erag_ti are paired, that is, they belong to the same image-text pair (corresponding to a target reference information in the previous text). In particular, the visual encoder and word vector encoding model used in this step can share parameter weights with the visual encoder and word vector encoding model of the multimodal large model.

[0103] (5) The visual feature Erag_i and the text feature Erag_t are input into the feature fusion module for feature fusion to achieve denoising.

[0104] The feature fusion module may include a cascade fusion network consisting of a single sample fusion network (first fusion network) and a multi-sample fusion network (second fusion network). Both the single sample fusion network and the multi-sample fusion network may adopt the Q-former architecture. Based on this, each paired visual feature erag_ii and text feature erag_ti may be input into the first fusion network to extract text-guided visual features. The multiple text-guided visual features generated by the first fusion network are input into the second fusion network for context feature fusion to obtain the output feature representation Token_Trag (corresponding to the first feature representation mentioned above);

[0105] (6) The output feature representation Token_Trag is concatenated in front of the multimodal feature representation Token_Tvlm, and then input into the large language model for generative output, and the cross-entropy loss with the true value is calculated. Finally, the loss is back-propagated to optimize the parameters of the feature fusion module until the multimodal large model converges. In particular, during the fine-tuning stage, except for the parameters of the feature fusion module that need to be updated with gradients, other parameters (such as the model parameters of the multimodal large model and the weight parameters in the multimodal retriever, etc.) are frozen, which can effectively prevent the catastrophic forgetting of the already learned capabilities of the multimodal large model during the fine-tuning process.

[0106] (7) After the model training converges, the trained RAG system is loaded to enter the inference phase.

[0107] Reasoning Phase

[0108] The process of the inference phase can repeat the aforementioned steps (1) to (6), but the inference phase does not update parameters or calculate losses, and only returns the output results generated by the multimodal large model.

[0109] In this exemplary application scenario, the search enhancement device provided in this embodiment can achieve the following technical effects:

[0110] 1. We designed a lightweight search enhancement plug-in. By fine-tuning the feature fusion module in the plug-in, we can adapt the search results to the feature representation space of the multimodal large model, which is more conducive to the multimodal large model to understand the search results, provide background knowledge for subsequent generation results, and effectively improve the utilization rate of the search results. By fine-tuning the feature fusion module, we can efficiently adapt and optimize the new domain knowledge, thereby indirectly improving the context understanding ability of the multimodal large model and the efficient injection of new knowledge.

[0111] 2. Based on the multimodal retrieval features in the knowledge base, it can support the retrieval of multimodal reference data to improve the similarity between the retrieval results and the input data. Through the feature fusion module, the retrieval noise in a single image can be screened out, and the interference of a single abnormal sample can be reduced, thereby effectively improving the robustness of the multimodal large model to retrieval noise, and then effectively reducing the interference of retrieval noise on the output results, ensuring the output effect of the multimodal large model.

[0112] 3. The feature fusion module adopts a cascade network structure based on the Q-former structure. Through two-layer cascade interaction compression, it can effectively reduce the amount of data represented by the first feature input into the multimodal large model, thereby effectively improving the efficiency of the multimodal large model in generating output results.

[0113] Figure 5 A flowchart of a data processing method provided by another exemplary embodiment of the present application is provided. The method can be executed by a search enhancement device. The search enhancement device can be implemented as software, hardware, or a combination of software and hardware. The product form of the search enhancement device can be a plug-in, etc. Figure 5 , the method may include:

[0114] Step 500: Retrieve target reference information that matches the input data received by the multimodal large model from the knowledge base;

[0115] Step 501: extracting multimodal features from multimodal reference data included in the target reference information;

[0116] Step 502: performing denoising processing on the extracted multimodal features to generate a first feature representation corresponding to the target reference information;

[0117] Step 503: Input the first feature representation into the multimodal large model to assist the multimodal large model in generating an output result for the input data.

[0118] In an optional embodiment, the retrieval enhancement device includes a feature fusion module, and the denoising process is performed on the extracted multimodal features to generate a first feature representation corresponding to the target reference information, including:

[0119] Inputting the multimodal features into the feature fusion module, wherein the feature fusion module has the context understanding capability required for denoising;

[0120] The multimodal features are denoised by the feature fusion module to output the first feature representation.

[0121] In an optional embodiment, the feature fusion module includes a first fusion network for denoising a single piece of reference information and a second fusion network for denoising multiple pieces of reference information, and denoising the multimodal features to output the first feature representation, including:

[0122] Inputting the multimodal features corresponding to the target reference information into the first fusion network, wherein the number of the target reference information is multiple;

[0123] By using the first fusion network, a denoising process is performed on a multimodal feature corresponding to a single piece of target reference information as a unit, so as to generate fusion features corresponding to each of the plurality of pieces of target reference information;

[0124] The second fusion network performs denoising processing on a feature set composed of fused features of the multiple pieces of target reference information to generate the first feature representation.

[0125] In an optional embodiment, the first fusion network performs denoising processing on a multimodal feature corresponding to a single target reference information as a unit to generate fusion features corresponding to each of the multiple target reference information, including:

[0126] For a target multimodal feature corresponding to the first reference information, in the first fusion network, setting a text feature included in the target multimodal feature as query information;

[0127] According to the attention mechanism, from the features of other modes included in the target multimodal feature, the features that do not match the text feature are screened out to generate a fusion feature corresponding to the first reference information;

[0128] The first reference information is any retrieved target reference information.

[0129] In an optional embodiment, performing denoising processing on a feature set composed of fused features of the plurality of pieces of target reference information through the second fusion network to generate the first feature representation includes:

[0130] In the second fusion network, setting the retrieval features generated for the input data during the retrieval process as query information;

[0131] According to the attention mechanism, from the feature set composed of the fused features of the multiple target reference information, the fused features that do not match the retrieval features are screened out to generate the first feature representation.

[0132] In an optional embodiment, the pre-training process of the feature fusion module includes:

[0133] After the retrieval enhancement device and the multimodal large model are connected, generating the first feature representation for the training sample submitted to the multimodal large model;

[0134] Inputting the first feature representation corresponding to the training sample into the multimodal large model to assist the multimodal large model in generating an output result for the training sample;

[0135] Based on the output results, construct a loss function;

[0136] While keeping the model parameters of the multimodal large model unchanged, the parameters of the feature fusion module are adjusted according to the loss function until the multimodal large model converges.

[0137] In an optional embodiment, the retrieval enhancement device includes a multimodal retriever, which retrieves target reference information that matches the input data received by the multimodal large model from the knowledge base, including:

[0138] Encoding the input data using the multimodal retriever to obtain input features corresponding to different modal data in the input data;

[0139] The obtained input features are concatenated to generate search features corresponding to the input data;

[0140] From the knowledge base, target reference information matching the search feature is retrieved using the search feature as an index.

[0141] In an optional embodiment, retrieving target reference information matching the search feature from the knowledge base using the search feature as an index includes:

[0142] Based on the search feature of the input data, selecting N reference information with the highest matching degree with the search feature from the knowledge base;

[0143] If there is special reference information including the input data in the N reference information, the special reference information is deleted from the N reference information to obtain the target reference information.

[0144] In an optional embodiment, extracting multimodal features from the multimodal reference data included in the target reference information includes:

[0145] Inputting the multimodal reference data into the encoding layer of the multimodal large model;

[0146] Encoding different modal data in the multimodal reference data using an encoder included in the encoding layer of the multimodal large model to generate encoding features corresponding to the different modal data;

[0147] The generated encoding features are retrieved from the encoding layer of the multimodal large model as multimodal features corresponding to the multimodal reference data.

[0148] In an optional embodiment, inputting the first feature representation into the multimodal large model includes:

[0149] intercepting a second feature representation constructed for the input data from the multimodal large model;

[0150] The spliced ​​features generated by splicing the first feature representation and the second feature representation are input into the multimodal large model.

[0151] It is worth noting that the technical details in the above method embodiment can refer to the relevant description of the retrieval enhancement device in the aforementioned system embodiment. In order to save space, they will not be repeated here, but this should not cause a loss in the scope of protection of this application.

[0152] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The operation serial numbers such as 501, 502, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different fusion networks or reference information, etc., and do not represent the order of precedence, nor do they limit the "first" and "second" to be different types.

[0153] Another exemplary embodiment of the present application may also provide a retrieval enhancement device, referring to Figure 2 , the retrieval enhancement device may include a multimodal retriever, a feature extraction module and a feature fusion module;

[0154] The multimodal retriever is used to retrieve target reference information that is compatible with input data received by the multimodal large model from the knowledge base;

[0155] The feature extraction module is used to extract multimodal features from the multimodal reference data included in the target reference information;

[0156] The feature fusion module is used to perform denoising on the extracted multimodal features to generate a first feature representation corresponding to the target reference information;

[0157] The multimodal retriever is further used to input the first feature representation into the multimodal large model to assist the multimodal large model in generating output results for the input data.

[0158] It is worth noting that the technical details in the above-mentioned embodiments of the retrieval enhancement device can be referred to the relevant description of the retrieval enhancement device in the aforementioned system embodiment. In order to save space, they will not be repeated here, but this should not cause a loss in the scope of protection of this application.

[0159] Accordingly, an embodiment of the present application also provides a computing device, which may include: a memory, a processor, and a communication component, etc. The processor can be used to execute a computer program in the memory to implement each step in the above method embodiment.

[0160] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can implement the steps in the above method embodiment when the computer program is executed.

[0161] Accordingly, an embodiment of the present application also provides a computer program product, which can implement the steps in the above method embodiment when the computer program included in the product is executed.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0166] The various multimodal large models involved in this application are in compliance with relevant laws and standards.

[0167] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0168] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applicable to a retrieval enhancement device, the retrieval enhancement device is externally connected to a multimodal large model, and the method includes: Retrieving target reference information adapted to the input data received by the multimodal large model from a knowledge base; Extracting multimodal features from the multimodal reference data included in the target reference information; Based on the contextual understanding capability of different industry knowledge injected into the retrieval enhancement device, the extracted multimodal features are subjected to denoising processing to generate a first feature representation corresponding to the target reference information; the denoising processing is to remove redundant features in the multimodal features that are insufficiently relevant to the input data; The first feature representation is input into the multimodal large model to assist the multimodal large model in generating an output result for the input data.

2. The method according to claim 1, characterized in that The retrieval enhancement device includes a feature fusion module, and the denoising process is performed on the extracted multimodal features to generate a first feature representation corresponding to the target reference information, including: Inputting the multimodal features into the feature fusion module, wherein the feature fusion module has the context understanding capability required for denoising; The multimodal features are denoised by the feature fusion module to output the first feature representation.

3. The method according to claim 2, characterized in that The feature fusion module includes a first fusion network for denoising a single piece of reference information and a second fusion network for denoising multiple pieces of reference information, performs denoising on the multimodal features to output the first feature representation, including: Inputting the multimodal features corresponding to the target reference information into the first fusion network, wherein the number of the target reference information is multiple; By using the first fusion network, a denoising process is performed on a multimodal feature corresponding to a single piece of target reference information as a unit, so as to generate fusion features corresponding to each of the plurality of pieces of target reference information; The second fusion network performs denoising processing on a feature set composed of fused features of the multiple pieces of target reference information to generate the first feature representation.

4. The method according to claim 3, characterized in that The first fusion network performs denoising processing on the multimodal features corresponding to a single target reference information as a unit to generate fusion features corresponding to the multiple target reference information, including: For a target multimodal feature corresponding to the first reference information, in the first fusion network, setting a text feature included in the target multimodal feature as query information; According to the attention mechanism, from the features of other modes included in the target multimodal feature, the features that do not match the text feature are screened out to generate a fusion feature corresponding to the first reference information; The first reference information is any retrieved target reference information.

5. The method according to claim 3, characterized in that: The method further comprises: performing a denoising process on a feature set composed of fused features of the plurality of target reference information through the second fusion network to generate the first feature representation, including: In the second fusion network, setting the retrieval features generated for the input data during the retrieval process as query information; According to the attention mechanism, from the feature set composed of the fused features of the multiple target reference information, the fused features that do not match the retrieval features are screened out to generate the first feature representation.

6. The method according to claim 2, characterized in that The pre-training process of the feature fusion module includes: After the retrieval enhancement device and the multimodal large model are connected, generating the first feature representation for the training sample submitted to the multimodal large model; Inputting the first feature representation corresponding to the training sample into the multimodal large model to assist the multimodal large model in generating an output result for the training sample; Based on the output results, construct a loss function; While keeping the model parameters of the multimodal large model unchanged, the parameters of the feature fusion module are adjusted according to the loss function until the multimodal large model converges.

7. The method according to any one of claims 1 to 6, characterized in that: The retrieval enhancement device includes a multimodal retriever, which retrieves target reference information that is compatible with input data received by the multimodal large model from a knowledge base, including: Encoding the input data using the multimodal retriever to obtain input features corresponding to different modal data in the input data; The obtained input features are concatenated to generate search features corresponding to the input data; From the knowledge base, target reference information matching the search feature is retrieved using the search feature as an index.

8. The method according to claim 7, characterized in that Retrieving target reference information matching the search feature from the knowledge base using the search feature as an index includes: Based on the search feature of the input data, selecting N reference information with the highest matching degree with the search feature from the knowledge base; If there is special reference information including the input data in the N reference information, the special reference information is deleted from the N reference information to obtain the target reference information.

9. The method according to claim 1, characterized in that: Extracting multimodal features from the multimodal reference data included in the target reference information includes: Inputting the multimodal reference data into the encoding layer of the multimodal large model; Encoding different modal data in the multimodal reference data using an encoder included in the encoding layer of the multimodal large model to generate encoding features corresponding to the different modal data; The generated encoding features are retrieved from the encoding layer of the multimodal large model as multimodal features corresponding to the multimodal reference data.

10. The method according to claim 1, characterized in that Inputting the first feature representation into the multimodal large model comprises: intercepting a second feature representation constructed for the input data from the multimodal large model; The spliced ​​features generated by splicing the first feature representation and the second feature representation are input into the multimodal large model.

11. A search enhancement device, characterized in that: The retrieval enhancement device is externally connected to the multimodal large model, and includes a multimodal retriever, a feature extraction module and a feature fusion module; The multimodal retriever is used to retrieve target reference information that is compatible with the input data received by the multimodal large model from a knowledge base; The feature extraction module is used to extract multimodal features from the multimodal reference data included in the target reference information; The feature fusion module is used to perform denoising on the extracted multimodal features based on the contextual understanding capability of different industry knowledge injected into the retrieval enhancement device to generate a first feature representation corresponding to the target reference information; The denoising process is to remove redundant features in the multimodal features that are insufficiently correlated with the input data; The multimodal retriever is further used to input the first feature representation into the multimodal large model to assist the multimodal large model in generating output results for the input data.

12. A data processing system, characterized in that: It comprises a retrieval enhancement device and a multimodal large model, wherein the retrieval enhancement device is connected to the multimodal large model, and the retrieval enhancement device executes the data processing method described in any one of claims 1 to 10 to assist the multimodal large model in generating output results.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to execute the data processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by one or more processors, causes the one or more processors to execute the data processing method according to any one of claims 1 to 10.

Images