Multimodal data processing method and apparatus, electronic device, and storage medium

Abstract

The application relates to a multi-modal data processing method and device, electronic equipment and a storage medium. The method comprises: collecting multi-modal data of multiple platforms; the multi-modal data has a data identifier; the multi-modal data is analyzed respectively to obtain formatted analysis data, and standardized data is generated based on the analysis data and the data identifier; a reprocessing process of a custom large model is defined based on a business scenario; when a user task is obtained, the standardized data is subjected to multi-modal understanding according to the user task to obtain multi-modal understanding results; and the multi-modal understanding results are processed by using the reprocessing process. The scheme provided by the application can realize integrated collection, analysis, understanding and structured processing of multi-platform multi-modal data, simultaneously supports a custom AI processing process, adapts to differentiated business scenarios, improves data processing efficiency and reusability, and reduces research and development and labor costs.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to multimodal data processing methods, apparatus, electronic devices and storage media. Background Technology

[0002] In the current process of internet business operations, data analysis, and content mining, a large amount of multimodal unstructured data, including text, images, videos, and audio, is collected from multiple content platforms. This type of data is a core asset for business decision-making, content production, and precision marketing. However, multimodal data is heterogeneous in format and fragmented in content, making it impossible to directly connect to downstream business systems (such as product recommendation, knowledge graph construction, and AIGC (Artificial Intelligence Generated Content) production). It needs to be understood and transformed into standardized and structured data before its value can be utilized. At the same time, different business departments have different needs for processing multimodal data, such as content review, information extraction, and trend analysis, requiring the support of generalized and configurable intelligent data processing capabilities.

[0003] In related technologies, a fixed AI (Artificial Intelligence) processing flow is usually adopted, which cannot be customized according to different business scenarios (such as product data augmentation, AIGC content production, and knowledge graph construction), and cannot meet the differentiated business data processing needs. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a multimodal data processing method, apparatus, electronic device, and storage medium, which can realize integrated acquisition, parsing, understanding, and structured processing of multimodal data across multiple platforms. It also supports customized AI processing workflows, adapts to differentiated business scenarios, improves data processing efficiency and reusability, and reduces R&D and labor costs.

[0005] The first aspect of this application provides a multimodal data processing method, including: Collect multimodal data from multiple platforms; the multimodal data has data identification. The multimodal data is parsed to obtain formatted parsed data, and standardized data is generated based on the parsed data and the data identifier. Customize the reprocessing flow of large models based on business scenarios; Upon obtaining a user task, multimodal understanding is performed on the standardized data based on the user task to obtain the multimodal understanding result; The multimodal understanding results are processed using the aforementioned reprocessing procedure.

[0006] In a preferred embodiment of this application, the data types of the multimodal data include at least document data, image data, video data, and audio data; the parsing of the multimodal data includes: The multimodal data is parsed according to its data type to obtain document data, image data, video data, and audio data. Formatted parsed data is obtained from document data, image data, video data, and audio data.

[0007] In a preferred embodiment of this application, the reprocessing flow of the customized large model based on the business scenario includes: Encapsulate the functionality of a large model into functional nodes; Based on the business scenario, the functional nodes are configured with the reprocessing process in sequence, and the reprocessing process is saved as a task template.

[0008] In a preferred embodiment of this application, the step of performing multimodal understanding on the standardized data according to the user task to obtain a multimodal understanding result includes: The user task is written into a distributed message queue, and the user task is consumed from the distributed message queue; A preset understanding model is invoked, which performs multimodal understanding on the standardized data according to the user task to obtain a multimodal understanding result, and the multimodal understanding result is written into the distributed message queue.

[0009] In a preferred embodiment of this application, the reprocessing procedure for processing the multimodal understanding result includes: The functional nodes of the reprocessing flow are called sequentially; The functional node consumes the multimodal understanding results from the distributed message queue and processes the multimodal processing results.

[0010] In a preferred embodiment of this application, the method further includes: After the reprocessing process is completed, a structured field is defined according to the business scenario, and the processing result of the reprocessing process is structured according to the structured field to obtain a structured processing result. The structured processing result is associated with the data identifier, and the association result is saved to a preset knowledge base.

[0011] In a preferred embodiment of this application, the method further includes: Set the original link of the multimodal data as a key; Standardized structured data is generated based on the association results between the key values ​​and the multimodal data, and a structured dataset is generated based on the structured data; the structured dataset is used to interface with downstream business systems.

[0012] A second aspect of this application provides a multimodal data processing apparatus, comprising: The acquisition module is used to acquire multimodal data from multiple platforms; the multimodal data has data identifiers. The parsing module is used to parse the multimodal data separately to obtain formatted parsed data, and generate standardized data based on the parsed data and the data identifier; Custom modules are used to customize the reprocessing flow of large models based on business scenarios; The multimodal understanding module is used to perform multimodal understanding on the standardized data based on the user task when a user task is obtained, and to obtain the multimodal understanding result. The reprocessing module is used to process the multimodal understanding results using the reprocessing process.

[0013] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0014] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0015] The technical solution provided in this application may include the following beneficial effects: This application provides a multimodal data processing method, including: collecting multimodal data from multiple platforms; assigning data identifiers to the multimodal data; parsing the multimodal data separately to obtain formatted parsed data; generating standardized data based on the parsed data and data identifiers; customizing a large-scale model reprocessing flow based on business scenarios; upon receiving a user task, performing multimodal understanding on the standardized data according to the user task to obtain multimodal understanding results; and processing the multimodal understanding results using the reprocessing flow. This method enables integrated collection, parsing, understanding, and structured processing of multimodal data from multiple platforms, while also supporting custom AI processing flows to adapt to differentiated business scenarios, improve data processing efficiency and reusability, and reduce R&D and labor costs.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0018] Figure 1 This is a schematic flowchart illustrating a multimodal data processing method according to an embodiment of this application; Figure 2 This is another schematic flowchart illustrating the multimodal data processing method in the embodiments of this application; Figure 3 This is a system architecture diagram of a multimodal data processing system shown in an embodiment of this application; Figure 4 This is a user flowchart illustrated in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a multimodal data processing device shown in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0019] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] In current internet business operations, data analysis, and content mining, a large amount of multimodal unstructured data, including text, images, videos, and audio, is collected from multiple content platforms. This type of data is a core asset for business decision-making, content production, and precision marketing. However, multimodal data is heterogeneous in format and fragmented in content, making it impossible to directly interface with downstream business systems. It needs to be understood and transformed into standardized, structured data before its value can be realized. At the same time, different business departments have different needs for processing multimodal data, such as content review, information extraction, and trend analysis, requiring the support of generalized and configurable intelligent data processing capabilities.

[0023] In related technologies, a fixed AI processing flow is usually adopted, which cannot be customized according to different business scenarios and cannot meet the differentiated business data processing needs.

[0024] To address the aforementioned issues, this application provides a multimodal data processing method that enables integrated acquisition, parsing, understanding, and structured processing of multimodal data across multiple platforms. It also supports custom AI processing workflows, adapts to diverse business scenarios, improves data processing efficiency and reusability, and reduces R&D and labor costs.

[0025] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 This is a flowchart illustrating a multimodal data processing method according to an embodiment of this application.

[0027] See Figure 1 The method includes: Step 110: Collect multimodal data from multiple platforms; the multimodal data has data identification.

[0028] In this application embodiment, the collection protocols of multiple mainstream content platforms can be abstracted to construct a unified collection interface for multiple mainstream content platforms. The collection interface supports URL (Uniform Resource Locator) / link as input parameters and can automatically identify the platform type and call the corresponding collection logic.

[0029] After a user enters a batch or single link to platform content in the task management section, the interface receives the pull request sent by the user and pulls the multimodal data corresponding to the link with one click. At the same time, it obtains the basic metadata of the multimodal data, such as publication time, source platform, author information, etc.

[0030] In practical use, users can build an enterprise-level unified resource pool using a distributed storage architecture. After each collection of multimodal data, the multimodal data is stored in the resource pool. The retrieved multimodal data is then standardized and categorized for storage within the resource pool. Multimodal data is stored according to link dimensions, with each link's content categorized into title, body, video, and image types. For example, the link table stores the task ID (Identification) + batch ID + link MD5 (Message-Digest Algorithm 5, data encryption MD5). The content table is associated based on the task ID + batch ID + link MD5 from the link table. A unique data identifier is assigned to each data type, and standardized resource download and access capabilities are provided to avoid duplicate data storage and improve data reuse efficiency. In one example, the data identifier in the link table is: task ID + batch ID + link MD5; the data identifier in the content table is: task ID + batch ID + link MD5 + content type + content link MD5 (used to distinguish that multiple videos or images may exist under the same link).

[0031] In this embodiment of the application, a multi-platform content acquisition SaaS (Software as a Service) service can also be provided to external parties, supporting third parties to achieve batch acquisition of multimodal data through interface calls.

[0032] This application embodiment can collect multimodal data from multiple platforms, breaking down the barriers between the collection interfaces of various content platforms, realizing one-stop aggregation of multimodal data from multiple content platforms, and solving the problem of scattered data storage through an enterprise-level unified resource pool, improving data management and reuse efficiency, and avoiding duplicate data collection by various departments.

[0033] Step 120: Parse the multimodal data separately to obtain formatted parsed data, and generate standardized data based on the parsed data and data identifiers.

[0034] This application also encapsulates a standardized multimodal data parsing toolset, which includes OCR (Optical Character Recognition) tools, video frame extraction tools, audio-to-text tools, and document parsing tools such as PDF / Word documents. All tools output parsing results that conform to a unified data format specification. When a parsing command is triggered for multimodal data in the unified resource pool, the data type can be automatically identified and the corresponding parsing tool can be invoked to perform integrated parsing of the multimodal data in the unified resource pool, resulting in formatted parsed data.

[0035] After parsing, the preliminary parsed data in a unified format is associated with the data identifiers of the original data to generate standardized data, which is then stored in a standardized data intermediate library to provide standardized input for subsequent processing. The following is a code example for standardized data: { "title": "xxx", "content": "xxxxx", "media_list": [ { "type": 1, "url": "xxxxx", "content": "Guangzhou craft beer route: try 6 beers in one go! All within walking distance! Save this!" "complete": true } ] } In addition, the multimodal data parsing toolset can also provide a data format conversion API (Application Programming Interface) to support third-party calls to implement single / multimodal data format parsing.

[0036] This application embodiment realizes integrated parsing of multimodal data including images, videos, audio, and documents, transforming unstructured data into standardized data, breaking down barriers to the utilization of multimodal data, providing a high-quality data foundation for subsequent processing, and the multimodal data parsing toolset can be flexibly expanded, with new parsing capabilities (such as short video special effects recognition, image scene recognition, etc.) to be added quickly in the future.

[0037] Step 130: Customize the reprocessing flow of the large model based on the business scenario.

[0038] This application provides a drag-and-drop visual AI capability orchestration platform. Users (including business personnel without technical backgrounds) can customize the reprocessing workflow of large models according to their own business scenarios (such as product data augmentation, AIGC article production, knowledge graph construction, etc.), for example, configuring a reprocessing workflow of "OCR recognition → large model summary → keyword extraction → information structuring". This solution enables business personnel without technical backgrounds to independently configure AI processing workflows without relying on developers, thus lowering the technical threshold for business exploration.

[0039] Step 140: When the user task is obtained, perform multimodal understanding on the standardized data according to the user task to obtain the multimodal understanding result.

[0040] This application adopts an asynchronous and decoupled task scheduling framework, which divides the task into two sub-processes: "multimodal understanding" and "reprocessing", and deploys them to different processing nodes respectively.

[0041] User tasks may include links to or identifiers of multimodal data to be processed, as well as information such as task name, processing flow, data volume, and priority. Upon receiving a user task, multimodal understanding is performed first. Based on the various types of content in the standardized data, multimodal understanding is executed to obtain the multimodal understanding results. If the multimodal understanding sub-process fails, it will automatically retry (the number of retries is configurable), and the reason for failure will be recorded.

[0042] Step 150: The multimodal understanding results are processed using a reprocessing workflow.

[0043] The multimodal understanding results undergo in-depth data processing according to a pre-configured custom large model reprocessing workflow. For example, if the configured reprocessing workflow includes "OCR recognition → large model summary → keyword extraction → information structuring," then the multimodal results are processed according to the workflow of OCR recognition, large model summary, keyword extraction, and information structuring. The reprocessing workflow in this embodiment is not a fixed workflow, but can be customized according to different business scenarios to adapt to differentiated business scenarios, such as product data augmentation, AIGC article production, and knowledge graph construction.

[0044] This application provides a multimodal data processing method, including: collecting multimodal data from multiple platforms; assigning data identifiers to the multimodal data; parsing the multimodal data separately to obtain formatted parsed data; generating standardized data based on the parsed data and data identifiers; customizing a large-scale model reprocessing flow based on business scenarios; upon receiving a user task, performing multimodal understanding on the standardized data according to the user task to obtain multimodal understanding results; and processing the multimodal understanding results using the reprocessing flow. This method enables integrated collection, parsing, understanding, and structured processing of multimodal data from multiple platforms, while also supporting custom AI processing flows to adapt to differentiated business scenarios, improve data processing efficiency and reusability, and reduce R&D and labor costs.

[0045] Figure 2 This is another schematic flowchart illustrating the multimodal data processing method in the embodiments of this application.

[0046] See Figure 2 The method includes: Step 210: Collect multimodal data from multiple platforms; the multimodal data has data identification.

[0047] In this application embodiment, the collection protocols of multiple mainstream content platforms can be abstracted to construct the same collection interface for multiple mainstream content platforms. The collection interface supports URL / link as input parameters and can automatically identify the platform type and call the corresponding collection logic.

[0048] After a user enters a batch or single link to platform content in the task management section, the interface receives the pull request sent by the user and pulls the multimodal data corresponding to the link with one click. At the same time, it obtains the basic metadata of the multimodal data, such as publication time, source platform, author information, etc.

[0049] In practical use, users can build an enterprise-level unified resource pool using a distributed storage architecture. After each collection of multimodal data, the multimodal data is stored in the resource pool. The retrieved multimodal data is then standardized and categorized for storage within the resource pool. Multimodal data is stored according to link dimensions, with content under each link categorized into title, body, video, and image types. For example, the link table stores task ID + batch ID + link MD5, and the content table is associated based on the task ID + batch ID + link MD5 from the link table. A unique data identifier is assigned to each data type, and standardized resource download and access capabilities are provided to avoid duplicate data storage and improve data reuse efficiency. In one example, the data identifier in the link table is: task ID + batch ID + link MD5; the data identifier in the content table is: task ID + batch ID + link MD5 + content type + content link MD5.

[0050] Step 220: Parse the multimodal data separately to obtain formatted parsed data, and generate standardized data based on the parsed data and data identifiers.

[0051] This application also encapsulates a standardized multimodal data parsing toolset, which includes OCR recognition tools, video frame extraction tools, audio-to-text tools, and document parsing tools such as PDF / Word. All tools output parsing results that conform to a unified data format specification. When a parsing instruction is triggered for multimodal data in the unified resource pool, the data type can be automatically identified and the corresponding parsing tool can be invoked to perform integrated parsing of the multimodal data in the unified resource pool, resulting in formatted parsed data.

[0052] After parsing, the preliminary parsed data in a unified format is associated with the data identifiers of the original data to generate standardized data, which is then stored in a standardized data intermediate library to provide standardized input for subsequent processing.

[0053] In an optional embodiment of this application, the data types of multimodal data include at least document data, image data, video data, and audio data; step 220 includes: The multimodal data is parsed according to its data type to obtain document data, image data, video data, and audio data. Formatted parsed data is obtained from document data, image data, video data, and audio data.

[0054] In practical implementation, multimodal data analysis can be performed according to data type. Document parsing tools such as PDF / Word can be used to parse the document data to obtain document data. OCR recognition tools can be used to parse the image data to obtain image data. Video frame extraction tools can be used to parse the video data to obtain video data. Audio-to-text tools can be used to parse the multimodal data to obtain audio data. The output of the parsing tools follows a unified data format specification, thus obtaining formatted parsed data.

[0055] Step 230: Encapsulate the functionality of the large model into functional nodes.

[0056] The embodiments of this application can be applied to multimodal data processing systems, see [link / reference]. Figure 3 This is a system architecture diagram of a multimodal data processing system. The multimodal data processing system adopts a four-layer architecture design (access layer → application layer → domain layer → infrastructure layer), with each layer decoupled and interacting, fully covering the entire multimodal data processing process. (1) The access layer can be regarded as a task management platform, including task management, AI capability configuration, result acquisition and user management. It provides users with a visual operation entry point, supports user task creation, reprocessing process orchestration, result viewing and permission control, and realizes no-code business configuration.

[0057] (2) The scheduling system in the application layer is responsible for task distribution, task scheduling, user system monitoring and evaluation management, and supports asynchronous high-concurrency task execution. The AI ​​capability system encapsulates AI parsing capability, AI understanding capability, AI insight capability, and AI capability orchestration into atomic services, supporting flexible orchestration of different reprocessing processes to adapt to different business scenarios.

[0058] (3) Task services in the domain layer include task creation, task execution, task result update, and task result review, performing full lifecycle management of user tasks, and completing data collection, parsing, understanding, and structured output, as well as task result update and review. User services include user permissions and user task records, providing permission control and task recording to ensure data security and traceability. AI engine services provide underlying AI capabilities such as multimodal understanding, text / image generation, content generation, and voice-to-text conversion.

[0059] (4) The K8S (Kubernetes, an open-source container orchestration system) in the infrastructure layer includes service scheduling, service management, and dynamic scaling, realizing containerized service scheduling and elastic scaling to support high-concurrency scenarios. External AI capabilities include underlying models and dynamic scaling, connecting to the underlying large models and providing elastic computing power. General-purpose storage ensures efficient data storage and task flow through MySQL (persistence), Redis (caching), and MQ (asynchronous decoupling).

[0060] In a multimodal data processing system, data is initiated from the access layer, scheduled by the application layer, executed by the domain layer, supported by the infrastructure layer, and finally flows back to the access layer for output. The layered and decoupled design ensures that the system is scalable, easy to maintain, and has the advantages of generalization, configurability, and high concurrency.

[0061] This application embodiment develops a drag-and-drop visual AI capability orchestration platform (i.e., a task management platform and AI capability system for a multimodal data processing system), which abstracts the general logic of "process configuration - node invocation - result aggregation" and encapsulates the functions of large models (such as large model summarization, keyword extraction, information extraction, sentiment analysis, and trend analysis) into draggable functional nodes.

[0062] Step 240: Configure the reprocessing process for the functional nodes in sequence based on the business scenario, and save the reprocessing process as a task template.

[0063] Users (including business personnel without technical backgrounds) can configure the reprocessing flow of functional nodes in sequence according to their own business scenarios (such as product data augmentation, AIGC article production, knowledge graph construction, etc.), for example, configuring a reprocessing flow of "OCR recognition → large model summary → keyword extraction → information structuring".

[0064] The multimodal data processing system supports visual configuration of prompt words for large models. Users can configure exclusive prompt words for each functional node of the large model to adapt to different processing needs. For example, in the scenario of product data augmentation, the prompt words can be configured to allow the large model to extract product functions, features, and applicable scenarios. An example of configuring prompt words is as follows: #Role You are a senior e-commerce data analyst and content mining expert, skilled at accurately extracting the core business elements of products from various product recommendation articles, reviews, press releases, or advertising copy.

[0065] After the reprocessing workflow is configured, users can save the reprocessing workflow as a task template on the task management platform. The template is persistently stored and can be called multiple times later. The task template can also be edited, modified, and deleted.

[0066] This application's embodiments enable business personnel without technical backgrounds to independently configure AI reprocessing workflows without relying on developers, thus lowering the technical threshold for business exploration; the templated workflow design allows for direct reuse of the same business scenarios, shortening the configuration cycle for business data processing; and the support for custom prompts further improves the matching degree between AI reprocessing results and business needs.

[0067] Step 250: When a user task is obtained, the user task is written to a distributed message queue, and the user task is consumed from the distributed message queue.

[0068] Based on the links / identifiers of the multimodal data to be processed, users can create user tasks on the task management platform. The task management platform records basic information about the user tasks, such as task name, processing flow, data volume, and priority. The task management platform writes the created user tasks to a distributed message queue and uses an asynchronous, decoupled task scheduling framework to divide the user tasks into two sub-processes: "multimodal understanding" and "reprocessing," which are deployed to different processing nodes respectively. The scheduling system consumes user tasks from the distributed message queue.

[0069] Step 260: Invoke the preset understanding model. The preset understanding model performs multimodal understanding on the standardized data according to the user task, obtains the multimodal understanding result, and writes the multimodal understanding result into the distributed message queue.

[0070] The process begins by invoking a pre-defined understanding model. This model performs multimodal understanding on various types of content in the standardized data according to the user's task, yielding multimodal understanding results. For example, it understands the content of document data to obtain document understanding results; extracts text information from image data or performs semantic understanding on image data to obtain image understanding results; extracts keyframes from video data and performs understanding similar to that on image data; extracts audio from video data and converts the audio into text, combining the understanding results of video frames and the converted text to obtain video understanding results; and directly converts audio data into text to obtain audio understanding results. After obtaining the multimodal understanding results, they can be written to a distributed message queue. If the multimodal understanding sub-process fails, it will automatically retry (the number of retries is configurable) and record the reason for failure.

[0071] Step 270: Invoke the function nodes of the reprocessing flow in sequence.

[0072] Users can select a saved task template on the task management platform, choose the corresponding reprocessing workflow, and call up each functional node of the reprocessing workflow in sequence.

[0073] Step 280: The functional node consumes the multimodal understanding results from the distributed message queue and processes the multimodal processing results.

[0074] Functional nodes consume multimodal understanding results from a distributed message queue and perform in-depth data processing on these results. For example, if the configured reprocessing workflow includes "OCR recognition → large model summary → keyword extraction → information structuring," then the multimodal results will be processed according to the workflow of OCR recognition, large model summary, keyword extraction, and information structuring. The reprocessing workflow also uses asynchronous modules, supporting batch parallel processing of user tasks and prioritizing user tasks, with higher-priority user tasks being consumed first. The task management platform provides real-time user task progress tracking, allowing users to view the overall execution progress of user tasks, the execution status of each sub-process, and details of failed user tasks. It also supports pausing, terminating, and re-executing user tasks.

[0075] The asynchronous user task and message queue decoupling design solve the problem of low efficiency of synchronous processing modules, supporting high concurrency and large-scale multimodal data processing to meet the needs of business expansion; failure retry and priority sorting functions improve the stability and flexibility of task processing; real-time progress tracking allows users to keep track of task status in a timely manner, improving the user experience.

[0076] In an optional embodiment of this application, the method further includes: After the reprocessing process is completed, structured fields are defined according to the business scenario. The processing results of the reprocessing process are then processed in a structured manner based on the structured fields to obtain the structured processing results. The structured processing results are associated with data identifiers, and the association results are saved to a preset knowledge base.

[0077] After the reprocessing process is complete, structured fields can be defined according to the business scenario. The processing results of the reprocessing process are then structured based on these structured fields to obtain the structured processing result. In one example, in a product data augmentation scenario, the structured fields include product name, function, features, applicable scenarios, and price range; in an AIGC article production scenario, the structured fields include the core theme of the material, keywords, core viewpoints, and content framework.

[0078] The structured processing results are associated with data identifiers, and the associated results are saved to a pre-defined knowledge base, such as a product database, content database, or knowledge graph database, thereby forming a queryable, reasonable, and reusable data asset. At the same time, it provides high-precision data retrieval and search capabilities, supporting businesses to quickly obtain relevant data insights.

[0079] In an optional embodiment of this application, the method further includes: Set the original links of the multimodal data as key-value pairs; Standardized structured data is generated based on the association results between key values ​​and multimodal data. Structured datasets are then generated based on the structured data. These structured datasets are used to interface with downstream business systems.

[0080] The original links of multimodal data can be set as key values. Standard structured data can be generated based on the key and the corresponding association results of the multimodal data. Structured datasets can be generated based on the structured data. The structured datasets can be connected to downstream business systems, such as product recommendation systems, precision marketing systems, AIGC content production systems, and intelligent search systems.

[0081] For different business scenarios, this application's embodiments support the output of structured data in multiple formats, such as JSON (a lightweight data exchange format), Excel (spreadsheet), and API interfaces, to meet the different integration needs of downstream systems.

[0082] The processed multimodal data is transformed into standardized structured data assets, enabling the data to be queryable, inferable, and reusable, thus fully unlocking the core business value of the data. The unified output format allows the data to be directly connected to downstream business systems, improving data flow efficiency and shortening the business cycle from data collection to application. The construction of an enterprise-level knowledge base provides a solid data foundation for subsequent business data analysis and trend mining.

[0083] For further explanation of the technical solutions in the embodiments of this application, please refer to... Figure 4 The system provides a user flow diagram, dividing the user process into two phases: configuration and usage. In the configuration phase, the user first logs into the task management platform, configures an initial task template, saves the initial task template in the task management system, configures the AI ​​reprocessing workflow, queries the AI ​​capability system to obtain AI capabilities based on the reprocessing workflow, orchestrates the reprocessing workflow based on the AI ​​capabilities, saves the reprocessing workflow in the AI ​​capability system, binds the reprocessing workflow to the initial task template to obtain the final task template, and saves the task template in the task management platform, thus completing the configuration. In the usage phase, the user logs into the task management platform, queries the task template in the task management system, selects a task template, uploads the multimodal data to be processed to create a user task, writes the user task to a distributed message queue, waits for multimodal understanding to be executed, the AI ​​capability system performs multimodal understanding, obtains the multimodal understanding result, then executes the reprocessing workflow on the multimodal understanding result, performs reprocessing according to the configured reprocessing workflow, saves the user task result after the reprocessing workflow is completed, and the user can query the user task result.

[0084] This application provides a multimodal data processing method that enables integrated acquisition, parsing, understanding, and structured processing of multimodal data across multiple platforms. It also supports custom AI processing workflows, adapts to differentiated business scenarios, improves data processing efficiency and reusability, reduces R&D and manpower costs, achieves distributed parallel processing of tasks, supports high-concurrency and large-batch multimodal data processing, and improves processing efficiency by several times.

[0085] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a multimodal data processing device, electronic device, and corresponding embodiments.

[0086] Figure 5 This is a schematic diagram of the structure of a multimodal data processing device shown in an embodiment of this application.

[0087] See Figure 5 The device includes: The acquisition module 510 is used to acquire multimodal data from multiple platforms; the multimodal data has data identification. The parsing module 520 is used to parse the multimodal data separately to obtain formatted parsed data, and generate standardized data based on the parsed data and data identifiers; Custom module 530 is used to customize the reprocessing flow of large models based on business scenarios; The multimodal understanding module 540 is used to perform multimodal understanding on standardized data based on the user task when the user task is obtained, and to obtain the multimodal understanding result. The reprocessing module 550 is used to process the multimodal understanding results using a reprocessing flow.

[0088] In an optional embodiment of this application, the data types of multimodal data include at least document data, image data, video data, and audio data; the parsing module 520 includes: The classification and parsing submodule is used to parse the multimodal data according to data type to obtain document data, image data, video data, and audio data. The data parsing submodule is used to obtain formatted parsed data from document data, image data, video data, and audio data.

[0089] In an optional embodiment of this application, the custom module 530 includes: Encapsulation submodules are used to encapsulate the functionality of a large model into functional nodes; The configuration submodule is used to configure the reprocessing process of functional nodes in sequence based on business scenarios, and save the reprocessing process as a task template.

[0090] In an optional embodiment of this application, the multimodal understanding module 540 includes: The write submodule is used to write user tasks to a distributed message queue and consume user tasks from the distributed message queue. The Understanding submodule is used to call a preset understanding model. The preset understanding model performs multimodal understanding on standardized data according to the user's task, obtains the multimodal understanding results, and writes the multimodal understanding results into a distributed message queue.

[0091] In an optional embodiment of this application, the reprocessing module 550 includes: Calling submodules is used to sequentially call the functional nodes of the reprocessing flow; The processing submodule is used by functional nodes to consume multimodal understanding results from the distributed message queue and process the multimodal processing results.

[0092] In an optional embodiment of this application, the device further includes: The structured processing module is used to define structured fields according to the business scenario after the reprocessing process is completed, and to perform structured processing on the processing results of the reprocessing process based on the structured fields to obtain the structured processing result. The association module is used to associate the structured processing results with data identifiers and save the association results to a preset knowledge base.

[0093] In an optional embodiment of this application, the device further includes: The configuration module is used to set the raw links of multimodal data as key-value pairs; The structured data module is used to generate standard structured data based on the association results between key values ​​and multimodal data, and to generate structured datasets based on the structured data; the structured datasets are used to interface with downstream business systems.

[0094] This application provides a multimodal data processing device that can achieve integrated acquisition, parsing, understanding and structured processing of multimodal data from multiple platforms. It also supports customized AI processing workflows, adapts to different business scenarios, improves data processing efficiency and reusability, and reduces R&D and labor costs.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0096] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0097] See Figure 6 The electronic device 600 includes a memory 610 and a processor 620.

[0098] The processor 620 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0099] Memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 620 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 610 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0100] The memory 610 stores executable code, which, when processed by the processor 620, can cause the processor 620 to execute part or all of the methods described above.

[0101] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0102] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) that, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0103] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the method described above.

[0104] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal data processing method, characterized in that, The method includes: Collect multimodal data from multiple platforms; the multimodal data has data identification. The multimodal data is parsed to obtain formatted parsed data, and standardized data is generated based on the parsed data and the data identifier. Customize the reprocessing flow of large models based on business scenarios; Upon obtaining a user task, multimodal understanding is performed on the standardized data based on the user task to obtain the multimodal understanding result; The multimodal understanding results are processed using the aforementioned reprocessing procedure.

2. The method according to claim 1, characterized in that, The data types of the multimodal data include at least document data, image data, video data, and audio data; the parsing of the multimodal data includes: The multimodal data is parsed according to its data type to obtain document data, image data, video data, and audio data. Formatted parsed data is obtained from document data, image data, video data, and audio data.

3. The method according to claim 1, characterized in that, The reprocessing flow of the customized large model based on business scenarios includes: Encapsulate the functionality of a large model into functional nodes; Based on the business scenario, the functional nodes are configured with the reprocessing process in sequence, and the reprocessing process is saved as a task template.

4. The method according to claim 3, characterized in that, The step of performing multimodal understanding on the standardized data according to the user task to obtain multimodal understanding results includes: The user task is written into a distributed message queue, and the user task is consumed from the distributed message queue; A preset understanding model is invoked, which performs multimodal understanding on the standardized data according to the user task to obtain a multimodal understanding result, and the multimodal understanding result is written into the distributed message queue.

5. The method according to claim 4, characterized in that, The reprocessing procedure for the multimodal understanding results includes: The functional nodes of the reprocessing flow are called sequentially; The functional node consumes the multimodal understanding results from the distributed message queue and processes the multimodal processing results.

6. The method according to claim 1 or 5, characterized in that, The method further includes: After the reprocessing process is completed, a structured field is defined according to the business scenario, and the processing result of the reprocessing process is structured according to the structured field to obtain a structured processing result. The structured processing result is associated with the data identifier, and the association result is saved to a preset knowledge base.

7. The method according to claim 6, characterized in that, The method further includes: Set the original link of the multimodal data as a key; Standardized structured data is generated based on the association results between the key values ​​and the multimodal data, and a structured dataset is generated based on the structured data; the structured dataset is used to interface with downstream business systems.

8. A multimodal data processing device, characterized in that, The device includes: The acquisition module is used to acquire multimodal data from multiple platforms; the multimodal data has data identifiers. The parsing module is used to parse the multimodal data separately to obtain formatted parsed data, and generate standardized data based on the parsed data and the data identifier; Custom modules are used to customize the reprocessing flow of large models based on business scenarios; The multimodal understanding module is used to perform multimodal understanding on the standardized data based on the user task when a user task is obtained, and to obtain the multimodal understanding result. The reprocessing module is used to process the multimodal understanding results using the reprocessing process.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images