Intelligent task matching method based on multi-modal data and RAG technology

Abstract

The invention discloses an intelligent task matching method based on multi-modal data and an RAG technology, and relates to the technical field of intelligent management, and the method comprises the steps: collecting multi-modal data, building modal data feature representation, and carrying out feature fusion based on a cross-modal collaborative attention mechanism; reading an intelligent task, performing data evaluation of a fusion result, establishing an RAG retrieval requirement, executing retrieval matching of an external database, and establishing a generation database; and constructing and optimizing a multi-learning framework for executing the intelligent task, and executing task processing of the corresponding intelligent task according to the converged multi-learning framework. The technical problems that in the prior art, a traditional single-modal processing method cannot meet the multi-modal data requirement in a complex scene, and the task processing adaptability and accuracy are low are solved, and the multi-modal data fusion, RAG retrieval enhancement and cross-task cooperation mechanism are achieved. And the adaptability and accuracy of intelligent task matching are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent management technology, and in particular to an intelligent task matching method based on multimodal data and RAG technology. Background Art

[0002] With the rapid development of multimedia technology, multimodal data (such as text, images, audio, etc.) is increasingly used in various fields. How to efficiently process and fuse different types of data has become an important challenge in the current field of artificial intelligence. Traditional single-modality processing methods can no longer meet the needs of multimodal data in complex scenarios, especially when dealing with intelligent systems involving multiple tasks. There are problems such as low model adaptation and low task processing accuracy. How to perform cross-modal feature fusion and task collaborative optimization has become the key. Summary of the invention

[0003] This application provides an intelligent task matching method based on multimodal data and RAG technology, which is used to solve the technical problems that the traditional single-modal processing method in the prior art cannot meet the multimodal data requirements in complex scenarios and the adaptability and accuracy of task processing are low. RAG (Retrieval-Augmented Generation) technology is a technical method that combines retrieval and generation, aiming to enhance the ability of large language models (LLMs) in processing knowledge-intensive tasks.

[0004] The present application provides an intelligent task matching method based on multimodal data and RAG technology, the method comprising: collecting multimodal data, establishing a multimodal data set, and converting the multimodal data set into a unified format through encoding technology to establish a multimodal data conversion result; establishing a modal data feature representation based on the multimodal data conversion result, and performing feature fusion based on the modal data feature representation based on a cross-modal collaborative attention mechanism to generate a fusion result of the multimodal data conversion result; reading an intelligent task, performing data evaluation of the fusion result based on the intelligent task, and establishing a data evaluation result, the data evaluation result including a data trust evaluation result and a data richness evaluation result, and the intelligent task is at least 2 tasks; establishing a RAG retrieval requirement based on the data evaluation result, performing retrieval matching of an external database, and establishing a generated database; using the fusion result and the generated database as basic data, synchronously executing the construction of a multi-learning framework for intelligent tasks, the multi-learning framework being provided with a cross-task collaborative module, and optimizing the multi-learning framework based on a loss function after feature sharing according to the cross-task collaborative module; and performing task processing of the corresponding intelligent task according to the converged multi-learning framework.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages: The intelligent task matching method based on multimodal data and RAG technology provided in the present application relates to the field of intelligent management technology. It collects and converts multimodal data, uses a cross-modal collaborative attention mechanism to fuse features, builds RAG retrieval requirements through intelligent task data evaluation, retrieves external databases to generate new databases, builds a multi-task learning framework based on the fusion results and the generated database, optimizes feature sharing, and finally efficiently performs intelligent task matching. It solves the technical problems in the prior art that traditional single-modal processing methods cannot meet the multimodal data requirements in complex scenarios and have low adaptability and accuracy in task processing, and achieves the technical effect of improving the adaptability and accuracy of intelligent task matching through multimodal data fusion, RAG retrieval enhancement and cross-task collaborative mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0007] Figure 1 A schematic diagram of a process flow of an intelligent task matching method based on multimodal data and RAG technology provided in an embodiment of the present application; Figure 2 A schematic diagram of a process for performing feature fusion based on feature representation of modal data in an intelligent task matching method based on multimodal data and RAG technology provided in an embodiment of the present application; Figure 3 A schematic diagram of a process for activating a cross-task collaboration module for feature sharing in an intelligent task matching method based on multimodal data and RAG technology provided in an embodiment of the present application. DETAILED DESCRIPTION

[0008] The present application provides an intelligent task matching method based on multimodal data and RAG technology, which is used to solve the technical problems in the prior art that the traditional single-modal processing method cannot meet the multimodal data requirements in complex scenarios and the adaptability and accuracy of task processing are low.

[0009] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0010] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0011] Examples, such as Figure 1 As shown, the present application provides an intelligent task matching method based on multimodal data and RAG technology, the method comprising: P10: Collect multimodal data, establish a multimodal dataset, and convert the multimodal dataset into a unified format through encoding technology to establish multimodal data conversion results.

[0012] Specifically, we build a comprehensive and standardized multimodal dataset to support subsequent intelligent task matching. First, we collect a wide range of multimodal data from different sources, including but not limited to text, images, audio, and possibly video and sensor data. This process relies on efficient data collection techniques and strategies to ensure the diversity and representativeness of the data.

[0013] Subsequently, the collected multimodal data is organized into a structured dataset, namely a multimodal dataset. A multimodal dataset refers to a collection of multiple data modalities, ensuring that the system can handle different types of information input. This step involves not only the physical storage of the data, but also the logical organization and indexing of the data so that it can be quickly accessed and processed later.

[0014] Since the data structure and characteristics of each modality are different (for example, text is a linear structure, while images are two-dimensional pixel matrices), specific encoding techniques are needed to convert data of different modalities into a unified format. Encoding techniques can represent data such as text, images, and audio as embedded vectors that can be used for calculations based on neural networks (such as Transformer encoders) or other deep learning models. For text data, word embedding or sentence embedding techniques may be used to convert natural language text into vector representations in a high-dimensional space. These vectors can capture the semantic information in the text. For image data, feature maps extracted by convolutional neural networks (CNNs) may be used as the encoded representation of the image. These feature maps can reflect the visual information in the image. Audio data may be converted into spectral features or time series features through audio processing algorithms (such as MFCC, Mel Spectrogram, etc.).

[0015] Through the above encoding technology, the multimodal data set is successfully converted into a unified format that can be understood by the model, that is, the multimodal data conversion result. This process not only achieves data standardization, but also lays a solid foundation for subsequent operations such as feature fusion, data evaluation, and RAG retrieval. It is worth noting that the choice of encoding technology should be optimized according to the specific task and data characteristics to ensure the accuracy and efficiency of data conversion.

[0016] P20: Establish a modal data feature representation based on the multimodal data conversion result, and perform feature fusion based on the modal data feature representation based on a cross-modal collaborative attention mechanism to generate a fusion result of the multimodal data conversion result.

[0017] Further, such as Figure 2 As shown, step P20 of the embodiment of the present application also includes: P21: Place the feature representation of the modal data into a shared space, perform feature alignment loss analysis, and establish feature alignment results; P22: Perform granular segmentation of its own features, use the granular segmentation results as the minimum matching unit, perform association matching based on the feature alignment results, perform granular fusion of the association matching results, and establish a cross-modal collaborative attention mechanism; P23: Perform feature fusion of the modal data feature representation according to the cross-modal collaborative attention mechanism.

[0018] It should be understood that first, a data feature representation of each modality is established based on the multimodal data conversion result. The modal data feature representation is a high-dimensional representation obtained by extracting features from data of different modalities (such as text, images, audio, etc.) through encoding technology, ensuring that the information of each modality can be effectively used in subsequent processing. For each modality, the system uses a special feature extraction method to generate a data feature representation of the modality. Exemplarily, for text data, a pre-trained natural language processing model (such as BERT, GPT, etc.) or word embedding technology (such as Word2Vec, GloVe) is usually used to encode the text. Through these models, the text can be represented as a high-dimensional vector that captures the semantic and contextual information in the text. For example, the sentence "I like apples" is encoded into an embedded vector through the BERT model, which represents the semantic information of the sentence. For image data, a convolutional neural network (CNN) can be used to extract image features. CNN extracts low-level features (such as edges, textures, etc.) and high-level features (such as shapes, objects, etc.) of images through multiple convolutional layers, pooling layers, and fully connected layers, and finally represents the image as a feature vector. For example, a picture containing an apple can be extracted through CNN models such as ResNet and converted into a vector that represents the object contained in the image and its features.

[0019] Next, these modal data feature representations are placed in a shared space, which can be achieved through a series of feature transformations and mapping operations, aiming to eliminate the differences in representation of different modal data so that they can be compared and analyzed under the same framework. The shared space is a common representation space for collaborative analysis of different modal data features. In this space, feature alignment loss analysis is performed, and the alignment effect is analyzed by calculating the similarity of different modal features, and feature alignment results are generated based on these results to ensure that data from different modalities can be reasonably compared and fused in the same semantics or context for subsequent matching and fusion.

[0020] After that, the granular segmentation of its own features is performed. Granular segmentation is to divide complex modal features into smaller feature units to ensure that each minimum unit can be compared and matched more carefully. These granular segmentation results are used as the minimum matching unit to perform association matching based on the feature alignment results. The goal of association matching is to find mutually related or complementary information by aligning the features between different modalities. Subsequently, the association matching results are granularly fused to form a consistent representation, and then a cross-modal collaborative attention mechanism is established.

[0021] The cross-modal collaborative attention mechanism is one of the core technologies of this step. By paying attention to the important information in different modal data, feature fusion between modalities is achieved. The cross-modal collaborative attention mechanism is used to perform feature fusion on the feature representation of modal data. This process deeply interacts and fuses the features of different modalities in a shared space to generate a fusion result of the multimodal data conversion result. Ensure that data from different modalities can work together in a shared feature space to generate more accurate multimodal data fusion results. The fusion result not only contains the original information of each modality, but also produces new and richer feature representations through cross-modal synergy. These feature representations can more comprehensively reflect the essential characteristics of the data and provide strong support for subsequent intelligent task matching. In this way, the model can simultaneously utilize information from multiple modalities such as text, images, and audio to improve the accuracy and efficiency of intelligent task matching.

[0022] Furthermore, step P22 of the embodiment of the present application also includes: P22-1: Perform granularity backtracking search based on the association matching results, and the granularity backtracking search is a search from the minimum granularity segmentation result to the maximum granularity segmentation result; P22-2: Configure the association threshold, and determine the termination granularity based on the granularity backtracking search and the association threshold; P22-3: Establish an association interval based on the minimum granularity segmentation result and the termination granularity, and perform granularity fusion based on the association matching results within the association interval.

[0023] Optionally, after completing the association matching, a granularity backtracking search is performed. This process starts from the smallest granularity segmentation result and gradually searches for larger granularity segmentation results. Granularity here refers to the coarseness of the information unit in the feature representation. The smallest granularity usually corresponds to the finest and most basic feature unit, while the largest granularity may contain broader and more comprehensive information. The purpose of granularity backtracking search is to find the feature granularity combination that best suits the current task requirements, so that the association information between data can be more accurately captured and expressed when the features are fused. At the same time, the implementation of granularity backtracking search depends on efficient search algorithms, such as binary search, heuristic search, or graph-based search algorithms. These algorithms can minimize the computational complexity and improve the search efficiency while ensuring the accuracy of the search.

[0024] In the process of granular backtracking search, it is necessary to configure an association threshold. This threshold is used to evaluate the strength of association between features at different granularities. When a certain granularity is searched, if its association strength is lower than the set threshold, it is considered that continuing to increase the granularity can no longer significantly improve the effect of feature fusion. At this time, the search is terminated, and this granularity is the termination granularity. Among them, the setting of the association threshold is usually based on task requirements, data characteristics and experimental experience. In practical applications, it may be necessary to find the most appropriate threshold through multiple experiments and adjustments. In addition, some statistical methods (such as significance tests) can also be used to assist in determining the threshold.

[0025] After determining the minimum granularity segmentation result and the termination granularity, an association interval is established between the two granularities. All granularity segmentation results within the association interval will be regarded as valid matching units for subsequent granularity fusion operations. Furthermore, granularity fusion is performed based on the associated matching results within the association interval. The granularity fusion refers to integrating and fusing the different granularity segmentation results within the association interval to form the final cross-modal feature representation. Various methods can be used, such as weighted averaging, maximum pooling, attention mechanism, etc. These methods can flexibly adjust the contribution of different granularity features in the fusion results according to different task requirements and data characteristics. In addition, in order to ensure the stability and reliability of the fusion results, the fusion process needs to be fully experimentally verified and optimized. Thereby improving the accuracy and richness of the feature representation, providing a more solid foundation for subsequent intelligent task matching.

[0026] P30: Read the intelligent task, perform data evaluation on the fusion result based on the intelligent task, and establish a data evaluation result, wherein the data evaluation result includes a data trust evaluation result and a data richness evaluation result, and the intelligent task is at least two tasks.

[0027] In a possible embodiment of the present application, this step focuses on applying the fused multimodal data to specific intelligent tasks, and evaluating the effectiveness and quality of these data for task completion through data evaluation. First, the system reads and parses one or more intelligent tasks, which may involve multiple fields such as text classification, image recognition, sentiment analysis, etc., and each task has its specific goals and requirements.

[0028] Next, based on the specific needs of these intelligent tasks, the fusion results are evaluated. Data evaluation is a comprehensive process that aims to evaluate the degree of data support for task completion from multiple dimensions and establish data evaluation results, which include data trust evaluation results and data richness evaluation results. Among them, the data trust evaluation results focus on the reliability and accuracy of the data, that is, to what extent the data can truly reflect the actual situation or meet the task requirements. In order to evaluate the trustworthiness of data, a variety of technical means can be used, such as data cleaning, outlier detection, data consistency verification, etc., to ensure the authenticity and accuracy of the data.

[0029] On the other hand, the data richness evaluation results focus on evaluating the completeness and diversity of the data. In intelligent task matching, rich data can often provide more comprehensive information support, thereby improving the accuracy and efficiency of task completion. In order to evaluate data richness, the source, type, quantity, and correlation and complementarity of the data can be analyzed. Through these analyses, we can understand the coverage and diversity of the data set, and then judge whether it can meet the task requirements.

[0030] It is worth noting that since intelligent tasks may involve multiple fields and aspects, the data evaluation results also need to take into account the differences between different tasks. In practical applications, different data evaluation indicators and methods may need to be customized for different tasks to ensure the accuracy and pertinence of the evaluation. By establishing data evaluation results, important data support and guarantee can be provided for subsequent intelligent task matching. This process not only helps to improve the accuracy and efficiency of task completion, but also provides feedback and guidance for the continuous optimization of data quality.

[0031] P40: Establish RAG search requirements based on the data evaluation results, perform search and matching of external databases, and establish a generated database.

[0032] Furthermore, step P40 of the embodiment of the present application also includes: P41: Call the data trust evaluation result, and establish a trust verification retrieval requirement based on the data trust evaluation result; P42: Call the data richness evaluation result, and establish a data supplement retrieval requirement; P43: Establish a RAG retrieval requirement based on the trust verification retrieval requirement and the data supplement retrieval requirement.

[0033] It should be understood that constructing and executing targeted RAG retrieval requirements based on data evaluation results aims to enhance and optimize the current dataset through retrieval matching of external databases, thereby generating a richer and more reliable generative database to enhance the processing capabilities of intelligent tasks.

[0034] First, call the data trust evaluation results and build trust verification retrieval requirements based on the evaluation results. The purpose of establishing trust verification retrieval requirements is to ensure that the data retrieved from the external database has a high degree of credibility with the existing fusion results to ensure the reliability and consistency of the information. These requirements are particularly targeted at those data parts that are marked as having low trust. By designing precise retrieval strategies, corresponding and more reliable data sources are retrieved from external databases to verify or replace existing data, thereby improving the trust of the entire data set. During retrieval, the retrieval strategy will be adjusted according to the level of data credibility in the evaluation to ensure that the returned results meet the task's requirements for data trust.

[0035] Next, the data richness evaluation results are called to establish data supplementation retrieval requirements. This requirement aims to further supplement the insufficient information in the fusion results through external databases to improve the richness and diversity of the data. For example, if the current task requires more background knowledge or contextual information, the system will obtain relevant content from the external database through this retrieval process.

[0036] Finally, by combining the trust verification retrieval requirements and the data supplementation retrieval requirements, a complete RAG retrieval requirement is finally constructed. The construction of this retrieval requirement enables the system to simultaneously meet the requirements for data credibility and the supplementary requirements for data richness when performing external database matching. Through this dynamic retrieval and generation process, the system can obtain useful information from the external database and ultimately generate a richer, more reliable and diverse generation database, further improving the effect of intelligent task matching.

[0037] In terms of technical support, in order to achieve efficient execution of RAG retrieval requirements, it is necessary to rely on advanced retrieval technologies and algorithms, such as natural language processing (NLP) technology for understanding and parsing query requirements, distributed storage and computing technology for processing large-scale data, and efficient indexing and sorting algorithms for improving retrieval speed and accuracy. In addition, a sound data management system needs to be built to ensure accurate storage and fast access to retrieval results.

[0038] P50: Using the fusion result and the generated database as basic data, a multi-learning framework for synchronously executing intelligent tasks is built. The multi-learning framework is provided with a cross-task collaboration module. After feature sharing is performed according to the cross-task collaboration module, the multi-learning framework is optimized based on the loss function.

[0039] Further, such as Figure 3 As shown, step P50 of the embodiment of the present application also includes: P51: Configure the task collaborative authentication channel, and after performing subtask collaborative authentication based on the task collaborative authentication channel, activate the cross-task collaborative module to perform feature sharing. The subtask collaborative authentication includes: a: performing task relevance authentication of the subtask, and calculating the task relevance authentication result; b: performing feature sharing degree calculation on the subtask, and generating a feature sharing degree calculation result; c: configuring the initial collaborative weight based on the task relevance authentication result and the feature sharing degree calculation result, and completing the subtask collaborative authentication; P52: Activate the cross-task collaborative module to perform feature sharing based on the initial collaborative weight.

[0040] Optionally, the previously generated fusion results and generated database are used as basic data to build a multi-learning framework for intelligent tasks. The core of the multi-learning framework is that it can handle multiple tasks at the same time and improve task processing efficiency by sharing underlying features. The framework also has a cross-task collaboration module, which is used for feature sharing between different tasks. During the task execution process, the system will continuously adjust and improve the multi-learning framework by optimizing the loss function to ensure that each task achieves the best effect when processed collaboratively.

[0041] Specifically, based on the fusion results and generated database, a multi-learning framework for intelligent tasks is built. First, the system needs to clearly define the types and goals of intelligent tasks. For example, different intelligent tasks may include multimodal tasks such as text classification, image recognition, and speech processing. Then, the system further decomposes the task into multiple subtasks. Especially in the multi-task learning framework, the tasks may have different focuses, such as classification, regression, generation, etc.

[0042] In order to support the construction of a multi-learning framework, the system needs to provide basic data by fusing results and generating databases. During the framework construction process, a network architecture suitable for handling multiple tasks needs to be adopted. Common multi-task learning network architectures include a combination of shared underlying networks and task-specific branches. First, a shared underlying feature extraction network is established to extract common features from the fused multimodal data. This layer may be based on a deep learning model, such as a convolutional neural network (CNN) for image processing, or a Transformer network for text processing. These networks will be shared among multiple tasks to extract common basic features and ensure the interoperability of information flow between tasks. Furthermore, based on the shared feature layer, each intelligent task will have its own task-specific network layer. These dedicated layers further process the shared features to generate output results for specific tasks. The dedicated layer for each task will be designed according to the goals and requirements of the task. For example, a fully connected layer may be used for a classification task, and an RNN or Transformer decoder may be used for a generation task.

[0043] Furthermore, in order to ensure the collaborative processing between multiple tasks, the system introduces a cross-task collaboration module. First, a task collaboration authentication channel is configured, through which the collaborative authentication of subtasks is performed. The subtask collaborative authentication is to ensure that there is sufficient correlation and reasonable sharing between tasks when sharing features. The subtask collaborative authentication includes three key steps: first, the task relevance authentication of subtasks is performed, and the relevance between them is evaluated by calculating the similarity or dependency between tasks, and the task relevance authentication result is generated; second, the feature sharing degree of subtasks is calculated, that is, the sharing potential and value of different tasks at the feature level are evaluated, and the feature sharing degree calculation result is generated; finally, based on the task relevance authentication result and the feature sharing degree calculation result, the initial collaborative weight is configured. For example, first, the system will assign a basic weight to each task group according to the task relevance result. Tasks with high correlation will get higher weights because the synergy between them is larger. Then, the system will refer to the feature sharing degree and further adjust the weight. If the sharing degree of two tasks is high, it means that their features can be effectively utilized, and the weight will be further improved; if the sharing degree is low, the weight will be reduced accordingly. These weights reflect the importance and contribution of different tasks in the collaborative process, thus completing the subtask collaborative certification.

[0044] After completing the collaborative authentication of the subtasks, the cross-task collaborative module is activated and starts to share features based on the initial collaborative weights. In this process, feature information between different tasks can be circulated and integrated, so that each task can draw useful information from other tasks, thereby improving its own execution effect. The realization of feature sharing relies on advanced feature extraction and fusion technologies, which can ensure that while retaining key information, redundancy and noise are removed, and the utilization and effectiveness of features are improved.

[0045] In order to further optimize the performance of the multi-learning framework, it is also necessary to train and optimize the entire framework based on the loss function. The loss function is an important indicator to measure the difference between the model's predicted results and the actual results. By constantly adjusting the model parameters to minimize the loss function value, the model can be closer to the actual situation and the execution accuracy and efficiency of intelligent tasks can be improved.

[0046] Furthermore, step P52 of the embodiment of the present application also includes: P52-1: Establish an adaptive weight update network for subtasks as follows: ; ;in, For subtask The collaborative weight of For subtask The loss function is are model parameters, Representation subtask The loss function is relative to the model parameters The gradient of To adjust the parameters, used to control the impact of gradients on weights, is the normalization constant, Represents the total number of subtasks, is the subtask index; P52-2: monitor the feature sharing results according to the adaptive weight update network, update the collaborative weights based on the monitoring results, and continue feature sharing with the updated collaborative weights.

[0047] It should be understood that further refinement of the working mechanism of the cross-task collaboration module, especially by introducing an adaptive weight update network to dynamically adjust the collaboration weights between subtasks, can improve the flexibility and adaptability of the multi-learning framework and ensure efficient feature sharing in different task scenarios.

[0048] Specifically, we first establish an adaptive weight update network for subtasks. The core of this network is to use the loss function of the subtask and its gradient information relative to the model parameters to dynamically adjust the collaborative weight of each subtask. The collaborative weight (denoted as ω) is calculated by the following formula: ; ;in, For subtask The collaborative weight of For subtask The loss function is are model parameters, Representation subtask The loss function is relative to the model parameters The gradient of To adjust the parameters, used to control the impact of gradients on weights, is the normalization constant, Represents the total number of subtasks, is the subtask index. When β is larger, the subtask with smaller gradient will get higher weight, and vice versa. Finally, the normalization constant (i.e., the denominator of the sum of all subtask weights) is used to ensure that the sum of all subtask weights is 1, maintaining the relativity and comparability of the weights.

[0049] Furthermore, the above-mentioned adaptive weight update network is used to monitor the results of feature sharing. This monitoring process aims to evaluate the effect of feature sharing in real time and dynamically adjust the collaborative weight according to the monitoring results. Specifically, if a subtask has achieved significant performance improvement through feature sharing, its collaborative weight may be increased accordingly to encourage more information sharing; conversely, if feature sharing is not helpful or even has a negative impact on the subtask, its collaborative weight may be reduced to reduce unnecessary interference. In this way, it can be ensured that each subtask in the multi-learning framework can participate in the feature sharing process in the best way, thereby maximizing the overall performance. It not only improves the adaptability and robustness of the multi-learning framework, but also provides more reliable and accurate data support for subsequent intelligent task matching.

[0050] P60: Perform task processing of corresponding intelligent tasks according to the converged multi-learning framework.

[0051] Specifically, when the multi-learning framework reaches a convergence state through multiple iterations of training, the system can formally execute the corresponding intelligent task. Convergence means that the loss function of the model gradually stabilizes during the training process, indicating that the model parameters have been adjusted to a better state and can better process the input data in the task. In the multi-learning framework, this means that the collaborative work between the various subtasks has reached a relatively balanced state, and the effect of feature sharing has been fully reflected. The converged multi-learning framework has a strong task processing capability. It can not only efficiently process a single task, but also work effectively in a multi-task environment.

[0052] Subsequently, the corresponding intelligent tasks are performed according to the converged multi-learning framework. These intelligent tasks may include but are not limited to image recognition, natural language processing, speech recognition and other fields. During the task processing, the multi-learning framework can make full use of its internally constructed complex network structure and rich feature representation capabilities to deeply analyze and process the input data, thereby extracting useful information and generating corresponding outputs. The key to this process is that through previous multi-task learning and feature sharing, the multi-learning framework has accumulated rich experience and knowledge in multiple tasks. Therefore, when executing specific tasks, it can rely on this knowledge to complete task processing more quickly and accurately, thereby significantly improving the processing efficiency and accuracy of the overall intelligent system.

[0053] Furthermore, the embodiment of the present application further includes step P70, and step P70 further includes: P71: Establish a feedback database, which is used to receive task processing results and extract processing feedback; P72: Establish a cycle optimization strategy based on the processing feedback, and use the cycle optimization strategy to perform multi-learning framework adaptive optimization.

[0054] Optionally, in order to enhance the practicality and maintainability of the multi-learning framework, the adaptive optimization of the framework can be achieved by introducing a feedback database and a periodic optimization strategy. First, a feedback database is established. The main function of the feedback database is to receive the results after each task processing and record the processing feedback related to the task. The processing feedback can include the execution efficiency of the task, the accuracy of the processing, error analysis, and other key performance indicators. Through these feedbacks, the system can gain an in-depth understanding of the performance of the model in the actual task execution, thereby providing a basis for subsequent optimization.

[0055] Next, a periodic optimization strategy is formulated based on the data in the feedback database. The periodic optimization strategy is a regularly executed optimization mechanism that aims to discover the problems and deficiencies in the multi-learning framework during task processing by analyzing the information in the feedback database, and propose targeted improvement measures accordingly. Specifically, the strategy may include adjusting the model's parameter settings, optimizing the feature extraction algorithm, improving the collaborative weight update mechanism, etc. By periodically executing these optimization measures, it can be ensured that the multi-learning framework is always in a good working state, and its ability and efficiency in processing intelligent tasks can be continuously improved.

[0056] In addition, in order to achieve the effective execution of cycle optimization strategies, data mining technology can be used to extract valuable information and patterns from the feedback database, and machine learning algorithms can be used to automatically adjust and optimize model parameters based on this information. Automated testing tools can also be used to ensure that no new errors or problems are introduced during the optimization process.

[0057] Furthermore, the embodiment of the present application further includes step P80, and step P80 further includes: P81: Establish a domain database, configure incremental data through the domain database, and perform incremental learning of associated multi-learning frameworks; P82: Perform intelligent task matching in the corresponding domain based on the incremental learning results.

[0058] In a possible embodiment of the present application, deep optimization and precise processing of specific fields can be achieved by introducing a domain database and an incremental learning mechanism, so that the multi-learning framework can better adapt to the needs of different fields and improve the pertinence and accuracy of task processing.

[0059] First, a domain database is established. The domain database is a database specifically used to store and manage knowledge in a specific domain, which contains various data samples, feature representations, label information, and domain-specific knowledge rules in the domain. By configuring incremental data to the domain database, new learning materials and challenges can be continuously provided to the multi-learning framework, promoting its continuous optimization and progress in the field. Incremental data usually refers to data samples or information newly added to the original data set, which can reflect the latest changes or trends in the field.

[0060] Next, the incremental data in the domain database is used to perform incremental learning of the multi-learning framework. Incremental learning is a machine learning technique that allows the model to update and improve itself by learning new data samples without retraining the entire dataset. In the multi-learning framework, incremental learning can be used for vertical optimization of specific fields, that is, by learning the incremental data in the field, the processing power and effect of the framework in the field are significantly improved. In order to achieve incremental learning, a series of advanced algorithms and technologies, such as online learning and transfer learning, need to be adopted to ensure that the model can efficiently absorb new knowledge and adapt to the new environment.

[0061] Finally, intelligent task matching for the corresponding domain is performed based on the results of incremental learning. Since the multi-learning framework has been optimized for a specific domain through incremental learning, it can now more accurately understand and process tasks and data in that domain. During the intelligent task matching process, the multi-learning framework will make full use of the knowledge and rules it has learned in the domain database, deeply analyze and process the input data, and generate output results that meet the domain requirements. This process not only improves the accuracy and efficiency of task processing, but also enhances the adaptability and flexibility of the multi-learning framework.

[0062] In summary, the embodiments of the present application have at least the following technical effects: This application collects and converts multimodal data, uses the cross-modal collaborative attention mechanism for feature fusion, builds RAG retrieval requirements through intelligent task data evaluation, retrieves external databases to generate new databases, builds a multi-task learning framework based on the fusion results and generated database, optimizes feature sharing, and finally efficiently performs intelligent task matching.

[0063] The technical effect of improving the adaptability and accuracy of intelligent task matching has been achieved through multimodal data fusion, RAG retrieval enhancement and cross-task collaboration mechanism.

[0064] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0066] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. An intelligent task matching method based on multimodal data and RAG technology, characterized in that: The method comprises: Collect multimodal data, establish a multimodal data set, and convert the multimodal data set into a unified format through encoding technology to establish a multimodal data conversion result; Establishing a modal data feature representation based on the multimodal data conversion result, and performing feature fusion based on the modal data feature representation based on a cross-modal collaborative attention mechanism to generate a fusion result of the multimodal data conversion result; Reading an intelligent task, performing data evaluation on a fusion result based on the intelligent task, and establishing a data evaluation result, wherein the data evaluation result includes a data trust evaluation result and a data richness evaluation result, and the intelligent task is at least two tasks; Establish RAG search requirements based on the data evaluation results, perform search matching of external databases, and establish a generated database; The fusion result and the generated database are used as basic data to build a multi-learning framework for synchronously executing intelligent tasks, wherein the multi-learning framework is provided with a cross-task collaboration module, and after feature sharing is performed according to the cross-task collaboration module, the multi-learning framework is optimized based on a loss function; The task processing of the corresponding intelligent task is performed according to the converged multi-learning framework.

2. The intelligent task matching method based on multimodal data and RAG technology according to claim 1, characterized in that: After the feature sharing is performed according to the cross-task collaborative module, the multi-learning framework is optimized based on the loss function, and further includes: A task collaborative authentication channel is configured, and after performing subtask collaborative authentication based on the task collaborative authentication channel, the cross-task collaborative module is activated for feature sharing. The subtask collaborative authentication includes: a: Perform task relevance authentication of subtasks and calculate task relevance authentication results; b: Calculate the feature sharing degree of the subtasks and generate the feature sharing degree calculation results; c: configuring the initial collaborative weight based on the task relevance authentication result and the feature sharing degree calculation result, and completing the subtask collaborative authentication; The cross-task collaboration module is activated to perform feature sharing based on the initial collaboration weight.

3. The intelligent task matching method based on multimodal data and RAG technology as claimed in claim 2, characterized in that: The activating the cross-task collaboration module to perform feature sharing based on the initial collaboration weight also includes: Establish the adaptive weight update network of the subtask as follows: ； ； in, For subtask The collaborative weight of For subtask The loss function is are model parameters, Representation subtask The loss function is relative to the model parameters The gradient of To adjust the parameters, used to control the impact of gradients on weights, is the normalization constant, Represents the total number of subtasks, Index for subtasks; The feature sharing result is monitored according to the adaptive weight updating network, and the collaborative weight is updated according to the monitoring result, and the feature sharing is continued with the updated collaborative weight.

4. The intelligent task matching method based on multimodal data and RAG technology according to claim 1, characterized in that: The method of performing feature fusion based on modal data feature representation based on the cross-modal collaborative attention mechanism to generate a fusion result of a multi-modal data conversion result also includes: Placing the feature representation of the modal data into a shared space, performing feature alignment loss analysis, and establishing a feature alignment result; Perform granular segmentation of its own features, use the granular segmentation results as the minimum matching unit, perform association matching based on feature alignment results, perform granular fusion of the association matching results, and establish a cross-modal collaborative attention mechanism; Feature fusion of modal data feature representation is performed according to the cross-modal collaborative attention mechanism.

5. The intelligent task matching method based on multimodal data and RAG technology as claimed in claim 4, characterized in that: The granular fusion of the association matching results also includes: Performing a granularity backtracking search based on the associated matching results, wherein the granularity backtracking search is a search from the smallest granularity segmentation result to the largest granularity segmentation result; Configure an associated threshold, and determine a termination granularity based on the granularity backtracking search and the associated threshold; An associated interval is established according to the minimum granularity segmentation result and the termination granularity, and granularity fusion is performed based on the associated matching result within the associated interval.

6. The intelligent task matching method based on multimodal data and RAG technology according to claim 1, characterized in that: The establishing of the RAG search requirement according to the data evaluation result also includes: Calling the data trust evaluation result, and establishing a trust verification retrieval requirement according to the data trust evaluation result; Calling the data richness evaluation results to establish data supplementary retrieval requirements; The RAG retrieval requirement is established according to the trust verification retrieval requirement and the data supplement retrieval requirement.

7. The intelligent task matching method based on multimodal data and RAG technology according to claim 1, characterized in that: The method further comprises: Establishing a feedback database, wherein the feedback database is used to receive task processing results and extract processing feedback; A cycle optimization strategy is established according to the processing feedback, and the cycle optimization strategy is used to perform multi-learning framework adaptive optimization.

8. The intelligent task matching method based on multimodal data and RAG technology according to claim 1, characterized in that: The method further comprises: Establishing a domain database, configuring incremental data through the domain database, and performing incremental learning of associated multiple learning frameworks; Perform intelligent task matching in the corresponding field based on the incremental learning results.

Images