Data processing method and device, medium, electronic equipment and computer program product

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a data processing method and device, a medium, electronic equipment and a computer program product. The method comprises the steps that multi-modal data are obtained, the multi-modal data are data of at least two modals, and the data of the at least two modals at least comprise text data; inputting the multi-modal data into a multi-modal large model to obtain a multi-modal reasoning result; the text data are input into a large reasoning model to obtain a text reasoning result, and the large reasoning model is a large language model with the thinking chain generation capacity; and determining a target reasoning result according to the multi-modal reasoning result and the text reasoning result. On the basis of reasoning the multi-modal data by using the multi-modal large model, the large reasoning model is introduced to guide the reasoning result, the strong text reasoning ability can be migrated to the multi-modal task without additional training and data labeling, the resource investment is reduced, the comprehensive reasoning performance of the model under multi-modal input is improved, and the reasoning efficiency is improved. The application range is widened.

Description

Technical Field

[0001] This disclosure belongs to the field of artificial intelligence technology, and in particular relates to a data processing method, a data processing device, a computer-readable storage medium, an electronic device, and a computer program product. Background Technology

[0002] Current full-modal models still lag behind the latest large-scale text reasoning models in terms of complex reasoning capabilities.

[0003] Mainstream improvement methods primarily include fine-tuning the model using large-scale labeled data or optimizing the model's inference chain using reinforcement learning. While these methods are effective, they generally suffer from problems such as large data requirements, high training costs, and difficulty in quickly adapting to new tasks. Furthermore, existing optimization methods for the decoding stage, such as contrastive decoding and surrogate tuning, while improving inference capabilities to some extent, are mostly limited to single-modal or input-modal scenarios, making it difficult to achieve true cross-modal collaborative inference. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides a data processing method, a data processing apparatus, a computer-readable storage medium, an electronic device, and a computer program product.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, comprising: Acquire multimodal data, wherein the multimodal data comprises data in at least two modalities, and the data in at least two modalities includes at least text data; The multimodal data is input into a large multimodal model to obtain multimodal inference results; The text data is input into a large-scale reasoning model to obtain text reasoning results. The large-scale reasoning model is a large language model with the ability to generate thought chains. The target reasoning result is determined based on the multimodal reasoning result and the text reasoning result.

[0006] Optionally, the method further includes: The text data is segmented into words to obtain text vectors.

[0007] Optionally, the method further includes: The audio data and / or the graphic data are encoded to obtain an encoded vector.

[0008] Optionally, the multimodal large model includes a first large model and a second large model, and the multimodal inference result includes a first inference result and a second inference result. The step of inputting the multimodal data into a large multimodal model to obtain multimodal inference results includes: The text vector and the encoded vector are input into the first large model to obtain the first inference result; The text vector is input into the second large model to obtain the second inference result.

[0009] Optionally, determining the target reasoning result based on the multimodal reasoning result and the text reasoning result includes: The reasoning difference result is determined based on the second reasoning result and the text reasoning result, and the influence factor is determined based on the reasoning difference result and the preset target coefficient; The target inference result is obtained by normalizing the first inference result and the influencing factor.

[0010] Optionally, determining the influencing factor based on the inference difference result and the preset target coefficient includes: The first reasoning result, the second reasoning result, and the text reasoning result are normalized to obtain the corresponding probability distributions, and the probability distributions are processed to obtain the probability distribution differences. The probability distribution differences are normalized to obtain the target coefficient, and the influencing factor is determined based on the inference difference results and the target coefficient.

[0011] Optionally, the probability distribution includes a first probability distribution, a second probability distribution, and a third probability distribution. The step of normalizing the first inference result, the second inference result, and the text inference result to obtain the corresponding probability distributions includes: The first probability distribution is obtained by normalizing the first inference result; The second inference result is normalized to obtain the second probability distribution; The text reasoning results are normalized to obtain a third probability distribution.

[0012] Optionally, the probability distribution difference includes a first distribution difference and a second distribution difference. The step of calculating the probability distribution difference by performing distribution difference calculation on the probability distribution includes: The difference between the first probability distribution and the second probability distribution is obtained by calculating the similarity between the first probability distribution and the second probability distribution. The difference between the second probability distribution and the third probability distribution is obtained by calculating the similarity between them.

[0013] Optionally, the target coefficient includes a first coefficient and a second coefficient. The process of normalizing the probability distribution differences to obtain the target coefficient includes: The sum of differences is determined based on the first distribution difference and the second distribution difference; A first coefficient is determined based on the first distribution difference and the sum of the differences, and a second coefficient is determined based on the second distribution difference and the sum of the differences.

[0014] Optionally, the impact factor includes a first impact factor and a second impact factor. The step of determining the influencing factor based on the inference difference result and the target coefficient includes: The first influence factor is determined based on the first coefficient and the inference difference result; The second influence factor is determined based on the second coefficient and the inference difference result.

[0015] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising: The data acquisition module is configured to acquire multimodal data, wherein the multimodal data is data in at least two modalities, and the data in at least two modalities includes at least text data; The first inference module is configured to input the multimodal data into a large multimodal model to obtain multimodal inference results; The second reasoning module is configured to input the text data into a large-scale reasoning model to obtain text reasoning results. The large-scale reasoning model is a large language model with the ability to generate thought chains. The result determination module is configured to determine the target reasoning result based on the multimodal reasoning result and the text reasoning result.

[0016] Optionally, the data processing device is further configured to: The text data is segmented into words to obtain text vectors.

[0017] Optionally, the data processing device is further configured to: The audio data and / or the graphic data are encoded to obtain an encoded vector.

[0018] Optionally, the multimodal large model includes a first large model and a second large model, and the multimodal inference result includes a first inference result and a second inference result. The first inference module is configured as follows: The text vector and the encoded vector are input into the first large model to obtain the first inference result; The text vector is input into the second large model to obtain the second inference result.

[0019] Optionally, the result determination module is configured as follows: The reasoning difference result is determined based on the second reasoning result and the text reasoning result, and the influence factor is determined based on the reasoning difference result and the preset target coefficient; The target inference result is obtained by normalizing the first inference result and the influencing factor.

[0020] According to a third aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the data processing method provided in any of the first aspects of the present disclosure.

[0021] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the executable instructions to implement the data processing method provided in any of the first aspects of this disclosure.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of a data processing method provided in any of the first aspects of the present disclosure.

[0023] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the methods and apparatus provided by the exemplary embodiments of this disclosure, based on the multimodal reasoning results obtained by using a large multimodal model to reason about multimodal data, a large-scale reasoning model is introduced to continue reasoning about text data to obtain text reasoning results. Then, the text reasoning results are used to guide the multimodal reasoning results to obtain the target reasoning results. Strong text reasoning capabilities can be transferred to multimodal tasks without additional training and data annotation. This not only reduces resource investment but also improves the comprehensive reasoning performance of the model under multimodal inputs and broadens the application scope.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 The schematic diagram illustrates a flow chart of a data processing method according to an exemplary embodiment of the present disclosure; Figure 2The schematic diagram illustrates a flowchart of a method for reasoning using a multimodal large model in an exemplary embodiment of the present disclosure; Figure 3 The schematic diagram illustrates a flowchart of a method for determining a target inference result in an exemplary embodiment of the present disclosure; Figure 4 The schematic diagram illustrates a flowchart of a method for determining influence factors in an exemplary embodiment of this disclosure; Figure 5 The schematic diagram illustrates a flowchart of a method for determining a probability distribution in an exemplary embodiment of this disclosure; Figure 6 The schematic diagram illustrates a flowchart of a method for calculating distribution differences in an exemplary embodiment of this disclosure; Figure 7 The schematic diagram illustrates a flow chart of a normalization process in an exemplary embodiment of the present disclosure; Figure 8 The schematic diagram illustrates a flowchart of a method for determining influence factors in an exemplary embodiment of this disclosure; Figure 9 This schematic diagram illustrates the structure of a data processing system in an application scenario according to an exemplary embodiment of this disclosure; Figure 10 The illustration schematically shows a flowchart of a method for sampling the next lexical unit in an application scenario according to an exemplary embodiment of the present disclosure; Figure 11 The schematic diagram illustrates an interface diagram for reasoning in an application scenario of an exemplary embodiment of this disclosure; Figure 12 This schematic diagram illustrates the structure of a data processing apparatus according to an exemplary embodiment of the present disclosure; Figure 13 This schematic diagram illustrates the structure of another data processing apparatus in an exemplary embodiment of the present disclosure; Figure 14 The schematic diagram illustrates the structure of another data processing apparatus in an exemplary embodiment of the present disclosure. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0029] With the rapid development of large-scale model technology, both industry and academia have invested significant resources in promoting the research and application of multimodal models. Mainstream multimodal large-scale models can achieve the fusion and understanding of information from multiple modalities, including text, images, audio, and video, and demonstrate excellent performance in tasks such as open-domain dialogue and complex question answering. Meanwhile, many open-source projects are continuously narrowing the gap with commercial closed-source models.

[0030] Existing advanced large language models are developing towards broader perception capabilities and deeper reasoning capabilities, gradually forming two types: reasoning language models and full-modal language models.

[0031] However, there is currently a lack of models that combine broad perception with deep thinking capabilities. Therefore, there is an urgent need to develop and design an intelligent system that combines full-modal perception and deep thinking capabilities. At present, the mainstream methods to improve the reasoning ability of multimodal models are mainly to supervise and fine-tune the model through large-scale labeled data or to improve the model's reasoning ability using reinforcement learning methods.

[0032] Supervised fine-tuning and reinforcement learning schemes typically require collecting large amounts of multimodal reasoning data and repeatedly training the model. While this approach can improve the model's reasoning ability, it has extremely high demands on data and computing power, long training cycles, and difficulty adapting to new domains or tasks. Decoding optimization methods, on the other hand, guide the decoding process of the base model by comparing the outputs of the expert model and the base model, thereby artificially changing the output habits of the original model, such as introducing long thought chains to improve the model's reasoning ability. However, most of these decoding optimization methods are only applicable to models with consistent input modalities (e.g., pure text or pure vision) and cannot achieve effective transfer of reasoning ability in multimodal tasks.

[0033] To address the problems existing in related technologies, this disclosure provides a data processing method. Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment, such as... Figure 1 As shown, the method may include at least the following steps: Step S110. Obtain multimodal data, which consists of data in at least two modalities, including at least text data.

[0034] Step S120. Input the multimodal data into the multimodal large model to obtain the multimodal inference results.

[0035] Step S130. Input the text data into a large-scale reasoning model to obtain text reasoning results. The large-scale reasoning model is a large language model with the ability to generate thought chains.

[0036] Step S140. Determine the target reasoning result based on the multimodal reasoning result and the text reasoning result.

[0037] In the exemplary embodiments of this disclosure, based on the multimodal reasoning results obtained by using a large multimodal model to reason about multimodal data, a large-scale reasoning model is introduced to continue reasoning about text data to obtain text reasoning results. Then, the text reasoning results are used to guide the multimodal reasoning results to obtain the target reasoning results. Strong text reasoning capabilities can be transferred to multimodal tasks without additional training and data annotation. This not only reduces resource investment but also improves the model's comprehensive reasoning performance under multimodal inputs and broadens the application scope.

[0038] The following section provides a detailed explanation of each step in the data processing method.

[0039] In step S110, multimodal data is acquired. The multimodal data consists of data in at least two modalities, and the data in at least two modalities includes at least text data.

[0040] In exemplary embodiments of this disclosure, modality refers to different types of information perception or expression methods, which may include text modality, graphics modality, video modality, audio modality, etc. Different modalities indicate that data can be perceived or expressed in different ways.

[0041] In step S120, multimodal data is input into a multimodal large model to obtain multimodal inference results.

[0042] In an exemplary embodiment of this disclosure, after obtaining multimodal data, the multimodal data can be preprocessed first.

[0043] In an optional embodiment, the text data is segmented to obtain a text vector.

[0044] Specifically, the text data is input into a text tokenizer so that the text tokenizer can extract features from the text data to obtain a text vector.

[0045] In optional embodiments, the multimodal data further includes audio data and / or graphical data. Encode the audio data and / or graphics data to obtain the encoded vector.

[0046] The graphic data can be in the form of images or videos, and this exemplary embodiment does not impose any special limitations on it.

[0047] Specifically, the audio data can be input into an audio encoder to extract features and obtain the corresponding encoded vector; the graphic data can be input into a visual encoder to extract features and obtain the corresponding encoded vector.

[0048] In an optional embodiment, Figure 2 A flowchart illustrating a method for inference using a multimodal large model is shown, such as... Figure 2 As shown, the method may include at least the following steps: In step S210, the text vector and the encoding vector are input into the first large model to obtain the first inference result.

[0049] When using a multimodal large model for inference, two large models can be set up: a first large model and a second large model. Therefore, the text vector and the encoded vector can be input into the first large model to obtain the first inference result, which can then be used... express.

[0050] In step S220, the text vector is input into the second large model to obtain the second inference result.

[0051] Correspondingly, this text vector can be input into the second large model to obtain the second inference result, which can be used... It is worth noting that the first and second largest models are identical, and their parameter settings are also the same.

[0052] In step S130, text data is input into a large-scale reasoning model to obtain text reasoning results. The large-scale reasoning model is a large language model with the ability to generate thought chains.

[0053] In an exemplary embodiment of this disclosure, while inputting the text vector and encoding vector into the multimodal large model, the text vector corresponding to the text data can also be input into the large reasoning model (LRM), so that the large reasoning model outputs the corresponding text reasoning result, which can be used... express.

[0054] In step S140, the target reasoning result is determined based on the multimodal reasoning result and the text reasoning result.

[0055] In an exemplary embodiment of this disclosure, after obtaining the multimodal reasoning result and the text reasoning result, the final target reasoning result can be determined based on the multimodal reasoning result and the text reasoning result.

[0056] In an optional embodiment, Figure 3 A flowchart illustrating the method for determining the target inference result is shown, such as... Figure 3As shown, the method may include at least the following steps: in step S310, determining the reasoning difference result based on the second reasoning result and the text reasoning result, and determining the influence factor based on the reasoning difference result and the preset target coefficient.

[0057] pass This method can determine the difference between the second reasoning result and the text reasoning result.

[0058] Furthermore, based on the difference in the reasoning results and the corresponding target coefficient, an influencing molecule can be determined to represent the impact of the difference in reasoning results between the second reasoning result and the text reasoning result on the first reasoning result.

[0059] Generally, the target coefficient is a preset value.

[0060] One way to determine the target coefficient is to set a preset value. Determined, at this point the target coefficient The influence of LRM on OLLM (Omni-modal Large Language Model) is controlled. Therefore, the size of the target coefficient can be set according to the situation and needs. This exemplary embodiment does not impose any special limitations on this.

[0061] In this case, according to This method can identify the corresponding influencing molecules.

[0062] Another way to determine the target coefficient is to calculate it based on the first inference result, the second inference result, and the text inference result, thereby further determining the influence factor.

[0063] In an optional embodiment, Figure 4 A flowchart illustrating the method for determining impact factors is shown, such as... Figure 4 As shown, the method may include at least the following steps: in step S410, the first reasoning result, the second reasoning result and the text reasoning result are normalized to obtain the corresponding probability distributions, and the probability distributions are processed to obtain the probability distribution differences.

[0064] In an optional embodiment, the probability distribution includes a first probability distribution, a second probability distribution, and a third probability distribution. Figure 5 A flowchart illustrating the method for determining the probability distribution is shown, such as... Figure 5 As shown, the method may include at least the following steps: In step S510, the first inference result is normalized to obtain a first probability distribution.

[0065] The first inference result is obtained by using Softmax (normalized exponential function). Converting to a probability distribution yields the first probability distribution. .

[0066] In step S520, the second inference result is normalized to obtain the second probability distribution.

[0067] The second inference result is obtained through Softmax. Converting to a probability distribution yields the second probability distribution. .

[0068] In step S530, the text reasoning results are normalized to obtain the third probability distribution.

[0069] The text reasoning results are processed using Softmax. Converting to a probability distribution yields the third probability distribution. .

[0070] In an optional embodiment, the probability distribution difference includes a first distribution difference and a second distribution difference. Figure 6 A flowchart illustrating the method for calculating distribution differences is shown, as follows: Figure 6 As shown, the method may include at least the following steps: in step S610, the similarity between the first probability distribution and the second probability distribution is calculated to obtain the first distribution difference.

[0071] The difference in the first probability distribution can be obtained by calculating the Jensen-Shannon divergence between the first and second probability distributions. .

[0072] In step S620, the similarity between the second probability distribution and the third probability distribution is calculated to obtain the difference in the second distribution.

[0073] The difference between the second and third probability distributions can be obtained by calculating the Jensen-Shannon divergence between them. .

[0074] In step S420, the probability distribution differences are normalized to obtain the target coefficient, and the influencing factor is determined based on the inference difference results and the target coefficient.

[0075] In an optional embodiment, the target coefficient includes a first coefficient and a second coefficient. Figure 7 A flowchart illustrating the normalization process is shown, such as... Figure 7 As shown, the method may include at least the following steps: in step S710, the sum of differences is determined based on the first distribution difference and the second distribution difference.

[0076] The total difference can be determined by summing the differences between the first and second distributions, that is... .

[0077] In step S720, a first coefficient is determined based on the first distribution difference and the sum of the differences, and a second coefficient is determined based on the second distribution difference and the sum of the differences.

[0078] Furthermore, the first coefficient can be determined based on the first distribution difference and the sum of the differences. The second coefficient is determined based on the difference in the second distribution and the sum of the differences. .

[0079] In an optional embodiment, the impact factor includes a first impact factor and a second impact factor. Figure 8 A flowchart illustrating the method for determining impact factors is shown, such as... Figure 8 As shown, the method may include the following steps: in step S810, a first influence factor is determined based on the first coefficient and the inference difference result.

[0080] Based on the first coefficient and the inference difference results Determine the first impact factor.

[0081] In step S820, the second influence factor is determined based on the second coefficient and the inference difference result.

[0082] Based on the difference between the second coefficient and the inference results Determine the second impact factor.

[0083] In step S320, the first inference result and the influence factor are normalized to obtain the target inference result.

[0084] When the target coefficient is a preset value When the result is obtained, normalization can be performed according to formula (1) to obtain the target reasoning result: (1) Among them, scalar The influence of LRM on OLLM is controlled. After obtaining the mixed logits, they are normalized to probabilities, and the next term is sampled in the usual way.

[0085] When the two target coefficients are and When the result is obtained, normalization can be performed according to formula (2) to obtain the target reasoning result: (2) The data processing method in this embodiment of the present disclosure will be described in detail below with reference to an application scenario.

[0086] Figure 9 A schematic diagram of the data processing system in the application scenario is shown, such as... Figure 9 As shown, to address the technical challenge of improving the inference capability of multimodal large models, a novel technical solution based on decoding stage guidance and dynamic fusion is proposed. This solution mainly includes two core innovative modules: an inference model guidance mechanism and a progressive comparison scaling module.

[0087] Inference Model Guidance: This approach separates the input modalities of a multimodal large language model. For multimodal inputs, such as text, images, and audio, they are fed into their respective perceptual models for feature extraction.

[0088] Meanwhile, the system introduces a readily available large-scale reasoning model, focusing on logical reasoning capabilities for textual or structured information. During the decoding phase, OLLM is responsible for understanding and representing multimodal information, while LRM serves as a guiding component for reasoning capabilities, generating reasoning guidance.

[0089] Specifically, It means OLLM. This represents LRM. As shown in Figure (a), the basic logits (the model's output, used for predicting the next word) are first calculated under the complete full-modal input. Subsequently, all modal content is discarded, and only content from all modalities is sent to the target modal. Provide text prefixes The result is used as negative logits. On the same prefix, positive logits are generated by LRM. The modified probability distribution is shown in formula (1).

[0090] By integrating the inference output of LRM during the decoding process, the logical reasoning and complex problem-solving capabilities of multimodal models can be effectively improved. This mechanism eliminates the need for retraining or fine-tuning the original model; signal fusion is only required during the decoding phase, significantly reducing the complexity and resource costs of technology implementation.

[0091] Progressive Comparison Scaling: Considering the significant differences in the requirements for inference and perception signals across different tasks and input modalities, signal fusion using fixed hyperparameters is insufficient to cover all scenarios. Therefore, this solution further incorporates a progressive comparison scaling module.

[0092] This module can analyze the model's prediction results in real time during the decoding process and dynamically adjust the weight allocation of the LRM and OLLM output signals based on the inference difficulty and input characteristics of the current task. Specifically, in each decoding step... First, the logits output by each model are converted into a probability distribution using Softmax, thus obtaining the data from... of From of and from of Then the Jensen–Shannon divergence between the distributions is calculated, denoted as . and Therefore, the mixing coefficients are obtained by normalizing the two divergences. and The logits are linearly combined according to the coefficients, and then Softmax is used to obtain the final output distribution as shown in formula (2).

[0093] when and At the same time, it is relatively small. Degenerate into the primitive When both are large, normalization distributes more weight to the component with greater divergence, thereby adaptively balancing the influence of the two signals while ensuring numerical stability.

[0094] Figure 10 The flowchart illustrates a method for sampling the next lexical unit in an application scenario, as shown below. Figure 10 As shown, the system automatically optimizes the fusion ratio of inference signals and perception signals based on the confidence level, prediction distribution, and task type of each decoding step, enabling the model to flexibly adapt to inference needs in different scenarios and improve overall inference accuracy and generalization ability.

[0095] Figure 11 The diagram illustrates the interface for reasoning in an application scenario, such as... Figure 11 As shown, this solution is applicable to various artificial intelligence scenarios that require multimodal reasoning capabilities.

[0096] Taking virtual assistants as an example, users can input text, images, and audio information simultaneously. The cloud-based big data model system uses this method to perform collaborative reasoning on multimodal inputs, providing answers that are more logical and in line with actual needs.

[0097] In the field of intelligent robots, robots can make decisions in complex environments by combining multimodal information such as vision and voice.

[0098] In medical diagnosis scenarios, doctors can input patients' text descriptions, medical images, and voice consultation data together. The system then uses this method to perform joint reasoning and provide more scientific suggestions for diagnosis.

[0099] In addition, this solution can also be applied to scenarios that require multi-source information fusion and high-order reasoning, such as autonomous driving and intelligent security.

[0100] The exemplary embodiments disclosed herein effectively address several core shortcomings of existing methods for enhancing full-modal reasoning capabilities. Firstly, traditional approaches such as supervised fine-tuning and reinforcement learning heavily rely on large amounts of high-quality labeled data and complex training processes, resulting in high R&D and deployment costs and difficulty in quickly adapting to new scenarios. This approach employs a decoding-stage guidance strategy, eliminating the need for additional training and data labeling, significantly reducing resource investment, and achieving plug-and-play improvement in reasoning capabilities.

[0101] Secondly, existing decoding optimization methods are typically only applicable to models with consistent input modalities, making it difficult to achieve cross-modal collaborative reasoning. This solution, through a reasoning model guidance mechanism, transfers strong text reasoning capabilities to multimodal tasks, effectively improving the model's comprehensive reasoning performance under multimodal inputs such as text, images, and audio, thus broadening its application scope.

[0102] Furthermore, traditional methods often employ fixed weights or manual parameter tuning, making it difficult to flexibly adapt to different task requirements and resulting in limited model generalization ability. To address this issue, this solution innovatively introduces a progressive contrast scaling module, which dynamically adjusts the fusion ratio of inference and perception signals based on real-time prediction results, improving the model's adaptability and performance under diverse tasks. Experimental results show that this solution achieves significant performance improvements on multiple public multimodal inference benchmarks while maintaining high inference efficiency, demonstrating good engineering integration and scalability.

[0103] In summary, this solution not only addresses the shortcomings of existing technologies in terms of data requirements, training costs, modal applicability, and dynamic fusion, but also provides an efficient and universal new paradigm for improving the reasoning capabilities of multimodal intelligent systems.

[0104] Furthermore, in an exemplary embodiment of this disclosure, a data processing apparatus is also provided. Figure 12 A schematic diagram of the data processing device is shown, such as... Figure 12 As shown, the data processing device 1200 may include: a data acquisition module 1210, a first inference module 1220, a second inference module 1230, and a result determination module 1240. Wherein: The data acquisition module 1210 is configured to acquire multimodal data, wherein the multimodal data is data in at least two modalities, and the data in at least two modalities includes at least text data; The first inference module 1220 is configured to input the multimodal data into a multimodal large model to obtain multimodal inference results; The second reasoning module 1230 is configured to input the text data into a large-scale reasoning model to obtain text reasoning results. The large-scale reasoning model is a large language model with the ability to generate thought chains. The result determination module 1240 is configured to determine the target reasoning result based on the multimodal reasoning result and the text reasoning result.

[0105] In some embodiments of this disclosure, the data processing apparatus 1200 is further configured to: The text data is segmented into words to obtain text vectors.

[0106] In some embodiments of this disclosure, the multimodal data further includes audio data and / or graphical data, and the data processing device 1200 is further configured to: The audio data and / or the graphic data are encoded to obtain an encoded vector.

[0107] In some embodiments of this disclosure, the multimodal large model includes a first large model and a second large model, and the multimodal inference result includes a first inference result and a second inference result. The first inference module 1220 is configured as follows: The text vector and the encoded vector are input into the first large model to obtain the first inference result; The text vector is input into the second large model to obtain the second inference result.

[0108] In some embodiments of this disclosure, the result determination module 1240 is configured to: The reasoning difference result is determined based on the second reasoning result and the text reasoning result, and the influence factor is determined based on the reasoning difference result and the preset target coefficient; The target inference result is obtained by normalizing the first inference result and the influencing factor.

[0109] In some embodiments of this disclosure, the result determination module 1240 is configured to: The first reasoning result, the second reasoning result, and the text reasoning result are normalized to obtain the corresponding probability distributions, and the probability distributions are processed to obtain the probability distribution differences. The probability distribution differences are normalized to obtain the target coefficient, and the influencing factor is determined based on the inference difference results and the target coefficient.

[0110] In some embodiments of this disclosure, the probability distribution includes a first probability distribution, a second probability distribution, and a third probability distribution. The result determination module 1240 is configured as follows: The first probability distribution is obtained by normalizing the first inference result; The second inference result is normalized to obtain the second probability distribution; The text reasoning results are normalized to obtain a third probability distribution.

[0111] In some embodiments of this disclosure, the probability distribution difference includes a first distribution difference and a second distribution difference. The result determination module 1240 is configured as follows: The difference between the first probability distribution and the second probability distribution is obtained by calculating the similarity between the first probability distribution and the second probability distribution. The difference between the second probability distribution and the third probability distribution is obtained by calculating the similarity between them.

[0112] In some embodiments of this disclosure, the target coefficient includes a first coefficient and a second coefficient. The result determination module 1240 is configured as follows: The sum of differences is determined based on the first distribution difference and the second distribution difference; A first coefficient is determined based on the first distribution difference and the sum of the differences, and a second coefficient is determined based on the second distribution difference and the sum of the differences.

[0113] In some embodiments of this disclosure, the impact factor includes a first impact factor and a second impact factor. The result determination module 1240 is configured as follows: The first influence factor is determined based on the first coefficient and the inference difference result; The second influence factor is determined based on the second coefficient and the inference difference result.

[0114] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0115] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the data processing method provided in this disclosure.

[0116] Figure 13 This is a block diagram illustrating another data processing apparatus 1300 according to an exemplary embodiment. For example, apparatus 1300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0117] Reference Figure 13The device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power supply component 1306, a multimedia component 1308, an audio component 1310, an input / output interface 1312, a sensor component 1314, and a communication component 1316.

[0118] Processing component 1302 typically controls the overall operation of device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0119] Memory 1304 is configured to store various types of data to support the operation of device 1300. Examples of such data include instructions for any application or method operating on device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0120] Power supply component 1306 provides power to various components of device 1300. Power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1300.

[0121] Multimedia component 1308 includes a screen that provides an output interface between the device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When the device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0122] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0123] Input / output interface 1312 provides an interface between processing component 1302 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0124] Sensor assembly 1314 includes one or more sensors for providing status assessments of various aspects of device 1300. For example, sensor assembly 1314 may detect the on / off state of device 1300, the relative positioning of components such as the display and keypad of device 1300, changes in the position of device 1300 or a component of device 1300, the presence or absence of user contact with device 1300, the orientation or acceleration / deceleration of device 1300, and temperature changes of device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0125] Communication component 1316 is configured to facilitate wired or wireless communication between device 1300 and other devices. Device 1300 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0126] In an exemplary embodiment, the apparatus 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0127] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 1320 of the device 1300 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0128] The aforementioned device can be a standalone electronic device or a part of a standalone electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be a single IC or a collection of multiple ICs. The chip can include, but is not limited to, the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and SoC (System on Chip). The aforementioned integrated circuit or chip can be used to execute executable instructions (or code) to implement the aforementioned data processing method. The executable instructions can be stored in the integrated circuit or chip or obtained from other devices or equipment. For example, the integrated circuit or chip includes a processor, memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the above-described data processing method can be implemented; alternatively, the integrated circuit or chip can receive the executable instructions through the interface and transmit them to the processor for execution to implement the above-described method.

[0129] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the data processing method described above when executed by the programmable device.

[0130] Figure 14 This is a block diagram illustrating another data processing apparatus 1400 according to an exemplary embodiment. For example, apparatus 1400 may be provided as a server. (Refer to...) Figure 14 The apparatus 1400 includes a processing component 1422, which further includes one or more processors, and memory resources represented by memory 1432 for storing instructions, such as application programs, that can be executed by the processing component 1422. The application programs stored in memory 1432 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1422 is configured to execute instructions to perform the data processing methods described above.

[0131] Device 1400 may also include a power supply component 1426 configured to perform power management of device 1400, a wired or wireless network interface 1450 configured to connect device 1400 to a network, and an input / output interface 1458. Device 1400 can operate on an operating system stored in memory 1432.

[0132] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0133] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: acquiring multi-modal data, the multi-modal data being data of at least two modalities, the data of at least two modalities comprising at least text data; inputting the multi-modal data into a multi-modal large model to obtain a multi-modal inference result; inputting the text data into a large inference model to obtain a text inference result, the large inference model being a large language model with the ability to generate a thinking chain; determining a target inference result according to the multi-modal inference result and the text inference result.

2. The data processing method according to claim 1, characterized in that, The method further comprises: performing word segmentation processing on the text data to obtain a text vector.

3. The data processing method according to claim 2, characterized in that, The multi-modal data further comprises audio data and / or graphical data, and the method further comprises: performing encoding processing on the audio data and / or the graphical data to obtain an encoded vector.

4. The data processing method according to claim 3, characterized in that, The multi-modal large model comprises a first large model and a second large model, and the multi-modal inference result comprises a first inference result and a second inference result, The inputting the multi-modal data into a multi-modal large model to obtain a multi-modal inference result comprises: inputting the text vector and the encoded vector into the first large model to obtain a first inference result; inputting the text vector into the second large model to obtain a second inference result.

5. The data processing method of claim 1, wherein, The determining a target inference result according to the multi-modal inference result and the text inference result comprises: determining an inference difference result according to the second inference result and the text inference result, and determining an influence factor according to the inference difference result and a preset target coefficient; performing normalization processing on the first inference result and the influence factor to obtain a target inference result.

6. The data processing method according to claim 5, characterized in that, The determining an influence factor according to the inference difference result and a preset target coefficient comprises: performing normalization processing on the first inference result, the second inference result and the text inference result respectively to obtain corresponding probability distributions, and performing distribution difference processing on the probability distributions to obtain a probability distribution difference; performing normalization processing on the probability distribution difference to obtain a target coefficient, and determining an influence factor according to the inference difference result and the target coefficient.

7. The data processing method according to claim 6, characterized in that, The probability distributions comprise a first probability distribution, a second probability distribution and a third probability distribution, The performing normalization processing on the first inference result, the second inference result and the text inference result respectively to obtain corresponding probability distributions comprises: performing normalization processing on the first inference result to obtain a first probability distribution; performing normalization processing on the second inference result to obtain a second probability distribution; performing normalization processing on the text inference result to obtain a third probability distribution.

8. The data processing method according to claim 7, characterized in that, The probability distribution difference comprises a first distribution difference and a second distribution difference, The performing distribution difference calculation on the probability distributions to obtain a probability distribution difference comprises: performing similarity calculation on the first probability distribution and the second probability distribution to obtain the first distribution difference; performing similarity calculation on the second probability distribution and the third probability distribution to obtain the second distribution difference.

9. The data processing method according to claim 7, characterized in that, The target coefficient comprises a first coefficient and a second coefficient, The performing normalization processing on the probability distribution difference to obtain a target coefficient comprises: determine a difference sum according to the first distribution difference and the second distribution difference; determine a first coefficient according to the first distribution difference and the difference sum, and determine a second coefficient according to the second distribution difference and the difference sum.

10. The data processing method according to claim 9, characterized in that, The influence factor includes a first influence factor and a second influence factor, The determining of the influence factor according to the reasoning difference result and the target coefficient includes: determining the first influence factor according to the first coefficient and the reasoning difference result; determining the second influence factor according to the second coefficient and the reasoning difference result.

11. A data processing apparatus, characterized by including: a data acquisition module configured to acquire multi-modal data, the multi-modal data being data of at least two modalities, and the data of at least two modalities including at least text data; a first reasoning module configured to input the multi-modal data into a multi-modal large model to obtain a multi-modal reasoning result; a second reasoning module configured to input the text data into a large reasoning model to obtain a text reasoning result, the large reasoning model being a large language model with the ability to generate a thinking chain; a result determination module configured to determine a target reasoning result according to the multi-modal reasoning result and the text reasoning result.

12. The data processing apparatus according to claim 11, characterized by The data processing apparatus is further configured to: perform word segmentation processing on the text data to obtain a text vector.

13. The data processing apparatus according to claim 12, characterized in that, The data processing apparatus is further configured to: perform encoding processing on the audio data and / or the graphic data to obtain an encoding vector.

14. The data processing apparatus according to claim 13, characterized by The multi-modal large model includes a first large model and a second large model, and the multi-modal reasoning result includes a first reasoning result and a second reasoning result, The first reasoning module is configured to: input the text vector and the encoding vector into the first large model to obtain the first reasoning result; input the text vector into the second large model to obtain the second reasoning result.

15. The data processing apparatus according to claim 11, wherein, The result determination module is configured to: determine a reasoning difference result according to the second reasoning result and the text reasoning result, and determine an influence factor according to the reasoning difference result and a preset target coefficient; perform normalization processing on the first reasoning result and the influence factor to obtain a target reasoning result.

16. A computer-readable storage medium having stored thereon computer program instructions, wherein, The program instructions are executed by the processor to implement the steps of the method of any one of claims 1-10.

17. A computer program product, characterised in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-10.

18. An electronic device, comprising: including: a memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-10.

Images