Multi-modal data processing method and system, terminal and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a multi-modal data processing method and system, a terminal and a storage medium, and the method comprises the steps: obtaining and preprocessing original multi-modal data to obtain standardized data; mining cross-modal semantic association based on standardized data, and extracting and constructing structured key knowledge; jointly inputting the key knowledge and the standardized data into a deep neural network model for multi-task joint training, so that the model learns the representation of deep fusion; and providing intelligent knowledge service by using the trained model, and collecting user feedback data to optimize model parameters to form a closed-loop self-optimization system. According to the method, accurate understanding and intelligent application of multi-modal data are realized through deep fusion and knowledge extraction, system performance is continuously improved by means of closed-loop feedback, and accuracy, adaptability and initiative of knowledge service are remarkably enhanced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data processing method, system, terminal, and computer-readable storage medium. Background Technology

[0002] With the rapid development of information technology, especially the popularization of big data and artificial intelligence, enterprises and organizations are increasingly reliant on deep insights into massive, heterogeneous, and multi-source data for their daily operations and decision-making. This data includes not only traditional structured text and reports, but also a wider range of unstructured images, videos, audio, sensor logs, time-series data, and other multimodal information. Traditional single-modal data processing technologies, such as standalone text analysis tools or image recognition systems, can only interpret information from a single dimension and are no longer sufficient to meet the urgent needs of comprehensive, interconnected, and accurate knowledge understanding and complex scenario applications in real-world business applications such as smart finance, intelligent manufacturing, precision medicine, and digital content management.

[0003] Against this backdrop, multimodal data fusion and knowledge acquisition are considered key directions for breaking through existing bottlenecks in intelligent systems. While current research and practice in industry and academia have made some progress, several serious challenges remain: Challenges in fusion depth and accuracy: Existing methods mostly remain at a shallow stage of "feature stitching" or "post-fusion of results," failing to achieve deep alignment and interaction at the intrinsic semantic level of the data. For example, when fusing image and text data, the model struggles to accurately understand the fine-grained correspondence between specific regions in the image and the text description, resulting in noise and ambiguity in the fused representation, directly affecting the accuracy of downstream tasks.

[0004] Challenges in intelligent knowledge extraction and application: Most systems focus on data association and presentation, lacking the ability to proactively mine and condense structured knowledge (such as entities, relationships, events, and rules) from multimodal data to form a reasonable and evolving knowledge system. The knowledge acquisition process relies heavily on manual annotation and domain customization, making it difficult to achieve the leap from "data fusion" to "intelligent knowledge services" and unable to proactively support complex business decisions and question answering.

[0005] Challenges to system versatility and scalability: Existing solutions are mostly siloed systems designed for specific scenarios (such as security monitoring and medical image reporting), with highly customized model architectures and data processing workflows. When faced with new business domains or new modal data, they often require expensive redesign and training costs. The lack of a unified, flexible, and adaptable general framework that can accommodate different task and modal combinations severely restricts the large-scale deployment and application effectiveness of the technology.

[0006] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0007] The main objective of this invention is to provide a multimodal data processing method, system, terminal, and computer-readable storage medium, aiming to solve the problems of inaccurate multimodal semantic alignment, weak knowledge extraction and application capabilities, and rigid systems that are difficult to adapt to different business scenarios in the prior art.

[0008] To achieve the above objectives, the present invention provides a multimodal data processing method, which includes the following steps: The system obtains raw multimodal data generated by the business system through a data interface, and preprocesses the raw multimodal data to obtain standardized multimodal data. Based on the standardized multimodal data, semantic relationships between different modalities are mined to extract key knowledge. The key knowledge and the standardized multimodal data are input into a deep neural network model, and the deep neural network model is jointly trained on multiple tasks so that the deep neural network model learns the fusion representation of the multimodal data. The system provides intelligent knowledge services to users based on the trained deep neural network model, collects feedback data generated by users when using the services, and uses the feedback data to update the parameters of the deep neural network model in order to optimize model performance.

[0009] Optionally, the multimodal data processing method, wherein obtaining raw multimodal data generated by the business system through a data interface, and preprocessing the raw multimodal data to obtain standardized multimodal data, specifically includes: Obtain text, image, and voice data generated by the business system through data interfaces; The text data is segmented and part-of-speech tagged to obtain standardized text data; The image data is subjected to denoising, resolution adjustment and feature extraction to obtain standardized image data; The speech data is processed using speech-to-text technology to convert it into text format, resulting in standardized speech data.

[0010] Optionally, the multimodal data processing method, wherein the step of mining semantic relationships between different modalities based on the standardized multimodal data and extracting key knowledge specifically includes: A cross-modal attention mechanism is used to analyze standardized text data, standardized image data, and standardized speech data to establish semantic relationships between text, image, and speech modalities. Based on the semantic association, key knowledge is constructed using entity recognition and relation extraction techniques, with entity-relationship-entity as the basic unit, and the key knowledge is stored in the fusion knowledge base.

[0011] Optionally, in the multimodal data processing method, the step of inputting the key knowledge and the standardized multimodal data into a deep neural network model, and performing multi-task joint training on the deep neural network model so that the deep neural network model learns the fused representation of the multimodal data, specifically includes: The standardized multimodal data is input into the encoder part of the deep neural network model to obtain multimodal feature vectors; The multimodal feature vectors and the knowledge constraints obtained from the key knowledge are input together into the decoder or prediction head of the deep neural network model; The deep neural network model is trained by jointly optimizing the cross-modal matching loss and the knowledge graph embedding loss using a multi-task learning approach.

[0012] Optionally, the multimodal data processing method, wherein providing intelligent knowledge services to users based on the trained deep neural network model, collecting feedback data generated by users when using the service, and updating the parameters of the deep neural network model using the feedback data to optimize model performance, specifically includes: In the intelligent question answering scenario, the system receives a query from the user that contains at least one modality, and based on the user's needs and behavior, calls the trained deep neural network model and the key knowledge to generate an answer and feed it back to the user. In the precise recommendation scenario, based on the multimodal data of the user's historical interactions, the trained deep neural network model predicts the user's preferences and generates a personalized recommendation list from the multimodal resources; The user's interaction with the answer or the personalized recommendation list is recorded as feedback data. After cleaning and labeling, the feedback data is used for incremental training or parameter fine-tuning of the deep neural network model.

[0013] Optionally, in the multimodal data processing method, the deep neural network model adopts a multimodal encoder-decoder model based on the Transformer architecture; The multimodal encoder is used to perform cross-modal joint encoding on the standardized multimodal data to generate a unified fusion representation; The decoder model is used to perform the generation or prediction task corresponding to the intelligent knowledge service based on the fused representation and the knowledge constraints.

[0014] Optionally, in the multimodal data processing method, recording the user's interaction with the answer or the personalized recommendation list as feedback data specifically includes: Collect user actions such as accepting, rejecting, or modifying the answers, as well as data on the duration of clicks, ignoring, or dwelling on the personalized recommendation list; The interactive behavior is transformed into reinforcement learning reward signals or supervised learning sample labels, which constitute the feedback data.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a multimodal data processing system, wherein the multimodal data processing system comprises: The data preprocessing module is used to obtain raw multimodal data generated by the business system through a data interface, and preprocess the raw multimodal data to obtain standardized multimodal data. The knowledge extraction module is used to mine the semantic relationships between different modal data based on the standardized multimodal data and extract key knowledge. The model training module is used to input the key knowledge and the standardized multimodal data into the deep neural network model, and perform multi-task joint training on the deep neural network model so that the deep neural network model learns the fusion representation of the multimodal data. The intelligent service and optimization module is used to provide intelligent knowledge services to users based on the trained deep neural network model, collect feedback data generated by users when using the service, and use the feedback data to update the parameters of the deep neural network model in order to optimize the model performance.

[0016] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal data processing program stored in the memory and executable on the processor, wherein when the multimodal data processing program is executed by the processor, it implements the steps of the multimodal data processing method as described above.

[0017] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described above.

[0018] In this invention, raw multimodal data generated by a business system is acquired through a data interface. This raw multimodal data is preprocessed to obtain standardized multimodal data. Based on this standardized multimodal data, semantic relationships between different modalities are mined to extract key knowledge. This key knowledge, along with the standardized multimodal data, is input into a deep neural network model. The deep neural network model undergoes multi-task joint training, enabling it to learn the fusion representation of the multimodal data. Based on the trained deep neural network model, intelligent knowledge services are provided to users. Feedback data generated by users during service usage is collected, and the parameters of the deep neural network model are updated using this feedback data to optimize model performance. This invention achieves accurate understanding and intelligent application of multimodal data through deep cross-modal semantic fusion and proactive extraction of structured knowledge. Furthermore, the closed-loop feedback mechanism enables the system to continuously self-optimize, significantly improving the accuracy, proactivity, and adaptability of knowledge services. Attached Figure Description

[0019] Figure 1 This is a flowchart of a preferred embodiment of the multimodal data processing method of the present invention; Figure 2 This is a schematic diagram of the entire process of the multimodal knowledge service scheme in a preferred embodiment of the multimodal data processing method of the present invention; Figure 3 This is a structural diagram of a preferred embodiment of the multimodal data processing system of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0021] The multimodal data processing method described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the multimodal data processing method includes the following steps: Step S10: Obtain the raw multimodal data generated by the business system through the data interface, and preprocess the raw multimodal data to obtain standardized multimodal data.

[0022] Specifically, data is acquired through data interfaces (such as APIs, database connections, file uploads, etc.) from business systems (such as CRM, ERP, medical information systems), including text data, image data, and voice data (such as large amounts of multimodal data such as text contracts, product images, and meeting recordings generated by the business systems). Data preprocessing involves using preprocessing algorithms to perform initial processing on the data, including word segmentation (e.g., using Jieba segmentation) and part-of-speech tagging of the text data to obtain standardized text data; noise reduction, resolution adjustment, and feature extraction processing are performed on the image data to obtain standardized image data; and speech-to-text technology is used to perform speech recognition processing on the voice data, converting it into text format to obtain standardized voice data.

[0023] For example, in a medical setting, the system may simultaneously acquire a CT image (image), a doctor's voice diagnosis record (voice), and an electronic medical record summary (text). The preprocessing module will enhance and extract features from the CT image, convert the voice diagnosis into text, and segment the medical record text, making all three standardized data that can be analyzed uniformly.

[0024] Step S20: Based on the standardized multimodal data, mine the semantic relationships between different modal data and extract key knowledge.

[0025] Specifically, cross-modal attention mechanisms (such as variants of the cross-modal attention network ViLBERT or CLIP) are used to analyze standardized text data, standardized image data, and standardized speech data to establish semantic associations between text, image, and speech modalities. Based on these semantic associations, key knowledge is constructed using entity recognition and relation extraction techniques, with entity-relationship-entity as the basic unit, and the key knowledge is stored in a fusion knowledge base.

[0026] For example, in e-commerce, product description text and product image data are fused to extract comprehensive knowledge about product attributes, appearance characteristics, etc., and stored in a fused knowledge base (i.e., a knowledge graph). For instance, when analyzing a news article with an accompanying image, the model calculates the feature association strength between the word "champion" in the text and the "podium" area in the image. Then, based on the strongly associated regions, Named Entity Recognition (NER) technology is used to identify entities such as "athlete" and "event," and a relation extraction model is used to determine the relationships between entities (e.g., "won"). Finally, these (entity, relation, entity) triples are structured and stored in a graph database (such as Neo4j) to form a "fused knowledge base," which is continuously updated and evolved with new data.

[0027] Step S30: Input the key knowledge and the standardized multimodal data into the deep neural network model, and perform multi-task joint training on the deep neural network model so that the deep neural network model learns the fusion representation of the multimodal data.

[0028] Specifically, the standardized multimodal data is input into the encoder part of the deep neural network model to obtain multimodal feature vectors; the multimodal feature vectors and the knowledge constraints obtained from the key knowledge are jointly input into the decoder or prediction head of the deep neural network model; in a multi-task learning manner, the cross-modal matching loss (e.g., determining whether a given text description matches an image) and the knowledge graph embedding loss (e.g., TransE loss, ensuring the rationality of entities and relations in the knowledge base in the vector space) are jointly optimized to complete the training of the deep neural network model.

[0029] The deep neural network model employs a multimodal encoder-decoder model based on the Transformer architecture (e.g., UNITER or VisualBERT architecture); the multimodal encoder is used to perform cross-modal joint encoding on the standardized multimodal data to generate a unified fusion representation; the decoder model is used to perform the generation or prediction task corresponding to the intelligent knowledge service based on the fusion representation and the knowledge constraints.

[0030] Step S40: Provide intelligent knowledge services to users based on the trained deep neural network model, collect feedback data generated by users when using the services, and use the feedback data to update the parameters of the deep neural network model to optimize model performance.

[0031] Specifically, in the intelligent question-answering scenario, the system receives a query from a user containing at least one modality. Based on the user's needs and behavior, it invokes the trained deep neural network model and the key knowledge to generate an answer and provides it back to the user. In the precise recommendation scenario, based on the user's historical multimodal interaction data, the system predicts user preferences using the trained deep neural network model and generates a personalized recommendation list from multimodal resources. It collects data on the user's acceptance, rejection, or modification of the answer, as well as click, ignore, or dwell time data on the personalized recommendation list. The interactive behavior is converted into reinforcement learning reward signals or supervised learning sample labels to constitute the feedback data. After cleaning and labeling, the feedback data is used for incremental training or parameter fine-tuning of the deep neural network model.

[0032] In intelligent question answering scenarios, the system can understand multimodal questions input by users (such as text questions paired with relevant images) and quickly provide accurate answers from a fusion knowledge base. For example, a user uploads a product image and asks, "What is this material?" The model combines image features with material knowledge about the product in the knowledge base to generate the answer, "The main material is oak."

[0033] In recommendation systems, multimodal knowledge resources that match a user's interests are recommended based on the user's historical browsing data (such as documents viewed and videos watched). Precise recommendations: For example, for online education platforms, the system predicts a user's interests based on their recent viewing of several programming tutorial videos (images + audio) and reading of related technical documents (text), and recommends a course called "Python Data Analysis in Practice".

[0034] After users use services such as intelligent question answering and accurate recommendations, the system collects feedback information during their use (the feedback data includes the user's acceptance or rejection of answers, clicks on recommended items, dwell time, etc., which are converted into reinforcement learning rewards or supervised learning samples for incremental training of the model). After cleaning, the data is used to optimize model parameters, thereby improving the closed loop of knowledge fusion and extraction.

[0035] The technical effects that this invention can bring are as follows: (1) Deep fusion and precision: Through cross-modal attention mechanism and joint training, deep alignment and complementarity of text, image and speech at the semantic level are achieved, generating high-quality unified fusion representation and solving the problem of shallow fusion.

[0036] (2) Active knowledge structuring: It can automatically mine and construct structured key knowledge (knowledge graph) from multimodal data, transforming raw data into reasonable and applicable intelligent assets, thereby improving the system's cognitive and decision support capabilities.

[0037] (3) The system is universal and self-evolving: It adopts a modular design and a general model architecture based on Transformer, which can flexibly adapt to different business scenarios; by introducing a closed-loop optimization mechanism based on user feedback, the system can continuously learn from practical applications, and its performance can continuously improve itself, thus possessing good universality and scalability.

[0038] Furthermore, such as Figure 3 As shown, based on the above-described multimodal data processing method, the present invention also provides a multimodal data processing system, wherein the multimodal data processing system includes: The data preprocessing module 51 is used to obtain the raw multimodal data generated by the business system through the data interface, and preprocess the raw multimodal data to obtain standardized multimodal data. The knowledge extraction module 52 is used to mine the semantic relationships between different modal data based on the standardized multimodal data and extract key knowledge. The model training module 53 is used to input the key knowledge and the standardized multimodal data into the deep neural network model, and perform multi-task joint training on the deep neural network model so that the deep neural network model learns the fusion representation of the multimodal data. The intelligent service and optimization module 54 is used to provide intelligent knowledge services to users based on the trained deep neural network model, collect feedback data generated when users use the service, and use the feedback data to update the parameters of the deep neural network model in order to optimize the model performance.

[0039] Furthermore, such as Figure 4 As shown, based on the above-mentioned multimodal data processing method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0040] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multimodal data processing program 40, which can be executed by the processor 10 to implement the multimodal data processing method of this application.

[0041] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multimodal data processing method.

[0042] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminal's processor 10, memory 20, and display 30 communicate with each other via a system bus.

[0043] In one embodiment, when the processor 10 executes the multimodal data processing program 40 in the memory 20, the following steps are performed: The system obtains raw multimodal data generated by the business system through a data interface, and preprocesses the raw multimodal data to obtain standardized multimodal data. Based on the standardized multimodal data, semantic relationships between different modalities are mined to extract key knowledge. The key knowledge and the standardized multimodal data are input into a deep neural network model, and the deep neural network model is jointly trained on multiple tasks so that the deep neural network model learns the fusion representation of the multimodal data. The system provides intelligent knowledge services to users based on the trained deep neural network model, collects feedback data generated by users when using the services, and uses the feedback data to update the parameters of the deep neural network model in order to optimize model performance.

[0044] Specifically, the step of acquiring raw multimodal data generated by the business system through a data interface and preprocessing the raw multimodal data to obtain standardized multimodal data includes: Obtain text, image, and voice data generated by the business system through data interfaces; The text data is segmented and part-of-speech tagged to obtain standardized text data; The image data is subjected to denoising, resolution adjustment and feature extraction to obtain standardized image data; The speech data is processed using speech-to-text technology to convert it into text format, resulting in standardized speech data.

[0045] Specifically, the step of mining semantic relationships between different modalities based on the standardized multimodal data and extracting key knowledge includes: A cross-modal attention mechanism is used to analyze standardized text data, standardized image data, and standardized speech data to establish semantic relationships between text, image, and speech modalities. Based on the semantic association, key knowledge is constructed using entity recognition and relation extraction techniques, with entity-relationship-entity as the basic unit, and the key knowledge is stored in the fusion knowledge base.

[0046] Specifically, the step of inputting the key knowledge and the standardized multimodal data into a deep neural network model, and performing multi-task joint training on the deep neural network model so that the deep neural network model learns the fused representation of the multimodal data, includes: The standardized multimodal data is input into the encoder part of the deep neural network model to obtain multimodal feature vectors; The multimodal feature vectors and the knowledge constraints obtained from the key knowledge are input together into the decoder or prediction head of the deep neural network model; The deep neural network model is trained by jointly optimizing the cross-modal matching loss and the knowledge graph embedding loss using a multi-task learning approach.

[0047] Specifically, the provision of intelligent knowledge services to users based on the trained deep neural network model, collecting feedback data generated by users when using the service, and updating the parameters of the deep neural network model using the feedback data to optimize model performance includes: In the intelligent question answering scenario, the system receives a query from the user that contains at least one modality, and based on the user's needs and behavior, calls the trained deep neural network model and the key knowledge to generate an answer and feed it back to the user. In the precise recommendation scenario, based on the multimodal data of the user's historical interactions, the trained deep neural network model predicts the user's preferences and generates a personalized recommendation list from the multimodal resources; The user's interaction with the answer or the personalized recommendation list is recorded as feedback data. After cleaning and labeling, the feedback data is used for incremental training or parameter fine-tuning of the deep neural network model.

[0048] The deep neural network model adopts a multimodal encoder-decoder model based on the Transformer architecture; The multimodal encoder is used to perform cross-modal joint encoding on the standardized multimodal data to generate a unified fusion representation; The decoder model is used to perform the generation or prediction task corresponding to the intelligent knowledge service based on the fused representation and the knowledge constraints.

[0049] Specifically, recording user interactions with the answers or the personalized recommendation list as feedback data includes: Collect user actions such as accepting, rejecting, or modifying the answers, as well as data on the duration of clicks, ignoring, or dwelling on the personalized recommendation list; The interactive behavior is transformed into reinforcement learning reward signals or supervised learning sample labels, which constitute the feedback data.

[0050] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described above.

[0051] In summary, this invention provides a multimodal data processing method, system, terminal, and computer-readable storage medium. The method includes: acquiring raw multimodal data generated by a business system through a data interface; preprocessing the raw multimodal data to obtain standardized multimodal data; mining semantic relationships between different modalities based on the standardized multimodal data to extract key knowledge; inputting the key knowledge and the standardized multimodal data into a deep neural network model; performing multi-task joint training on the deep neural network model to enable the deep neural network model to learn the fusion representation of multimodal data; providing intelligent knowledge services to users based on the trained deep neural network model; collecting feedback data generated by users when using the service; and updating the parameters of the deep neural network model using the feedback data to optimize model performance. This invention achieves accurate understanding and intelligent application of multimodal data through deep cross-modal semantic fusion and proactive extraction of structured knowledge, and enables the system to have continuous self-optimization capabilities through a closed-loop feedback mechanism, thereby significantly improving the accuracy, proactivity, and adaptability of knowledge services.

[0052] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0053] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0054] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal data processing method, characterized in that, The multimodal data processing method includes: The system obtains raw multimodal data generated by the business system through a data interface, and preprocesses the raw multimodal data to obtain standardized multimodal data. Based on the standardized multimodal data, semantic relationships between different modalities are mined to extract key knowledge. The key knowledge and the standardized multimodal data are input into a deep neural network model, and the deep neural network model is jointly trained on multiple tasks so that the deep neural network model learns the fusion representation of the multimodal data. The system provides intelligent knowledge services to users based on the trained deep neural network model, collects feedback data generated by users when using the services, and uses the feedback data to update the parameters of the deep neural network model in order to optimize model performance.

2. The multimodal data processing method according to claim 1, characterized in that, The process of acquiring raw multimodal data generated by the business system through a data interface and preprocessing the raw multimodal data to obtain standardized multimodal data specifically includes: Obtain text, image, and voice data generated by the business system through data interfaces; The text data is segmented and part-of-speech tagged to obtain standardized text data; The image data is subjected to denoising, resolution adjustment and feature extraction to obtain standardized image data; The speech data is processed using speech-to-text technology to convert it into text format, resulting in standardized speech data.

3. The multimodal data processing method according to claim 2, characterized in that, The process of mining semantic relationships between different modalities based on the standardized multimodal data and extracting key knowledge specifically includes: A cross-modal attention mechanism is used to analyze standardized text data, standardized image data, and standardized speech data to establish semantic relationships between text, image, and speech modalities. Based on the semantic association, key knowledge is constructed using entity recognition and relation extraction techniques, with entity-relationship-entity as the basic unit, and the key knowledge is stored in the fusion knowledge base.

4. The multimodal data processing method according to claim 1, characterized in that, The step of inputting the key knowledge and the standardized multimodal data into a deep neural network model, and performing multi-task joint training on the deep neural network model so that the deep neural network model learns the fused representation of the multimodal data, specifically includes: The standardized multimodal data is input into the encoder part of the deep neural network model to obtain multimodal feature vectors; The multimodal feature vectors and the knowledge constraints obtained from the key knowledge are input together into the decoder or prediction head of the deep neural network model; The deep neural network model is trained by jointly optimizing the cross-modal matching loss and the knowledge graph embedding loss using a multi-task learning approach.

5. The multimodal data processing method according to claim 4, characterized in that, The method of providing intelligent knowledge services to users based on the trained deep neural network model, collecting feedback data generated by users when using the service, and using the feedback data to update the parameters of the deep neural network model to optimize model performance, specifically includes: In the intelligent question answering scenario, the system receives a query from the user that contains at least one modality, and based on the user's needs and behavior, calls the trained deep neural network model and the key knowledge to generate an answer and feed it back to the user. In the precise recommendation scenario, based on the multimodal data of the user's historical interactions, the trained deep neural network model predicts the user's preferences and generates a personalized recommendation list from the multimodal resources; The user's interaction with the answer or the personalized recommendation list is recorded as feedback data. After cleaning and labeling, the feedback data is used for incremental training or parameter fine-tuning of the deep neural network model.

6. The multimodal data processing method according to claim 4, characterized in that, The deep neural network model adopts a multimodal encoder-decoder model based on the Transformer architecture; The multimodal encoder is used to perform cross-modal joint encoding on the standardized multimodal data to generate a unified fusion representation; The decoder model is used to perform the generation or prediction task corresponding to the intelligent knowledge service based on the fused representation and the knowledge constraints.

7. The multimodal data processing method according to claim 5, characterized in that, The recording of user interactions with the answers or the personalized recommendation list as feedback data specifically includes: Collect user actions such as accepting, rejecting, or modifying the answers, as well as data on the duration of clicks, ignoring, or dwelling on the personalized recommendation list; The interactive behavior is transformed into reinforcement learning reward signals or supervised learning sample labels, which constitute the feedback data.

8. A multimodal data processing system, characterized in that, The multimodal data processing system includes: The data preprocessing module is used to obtain raw multimodal data generated by the business system through a data interface, and preprocess the raw multimodal data to obtain standardized multimodal data. The knowledge extraction module is used to mine the semantic relationships between different modal data based on the standardized multimodal data and extract key knowledge. The model training module is used to input the key knowledge and the standardized multimodal data into the deep neural network model, and perform multi-task joint training on the deep neural network model so that the deep neural network model learns the fusion representation of the multimodal data. The intelligent service and optimization module is used to provide intelligent knowledge services to users based on the trained deep neural network model, collect feedback data generated by users when using the service, and use the feedback data to update the parameters of the deep neural network model in order to optimize the model performance.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multimodal data processing program stored in the memory and executable on the processor. When the multimodal data processing program is executed by the processor, it implements the steps of the multimodal data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described in any one of claims 1-7.

Images