Multi-modal data conversion method and device and computer equipment

Abstract

The invention discloses a multi-modal data conversion method and device and computer equipment. The method comprises the steps that multi-modal data to be converted are acquired, and the category of the multi-modal data comprises at least one of pictures, videos, audios and texts; performing multi-level attribute extraction on the multi-modal data to obtain a plurality of attributes of the multi-modal data; and determining structured data corresponding to the multi-modal data according to the multiple attributes. The multi-modal data processing method and device at least solve the technical problem that the multi-modal data processing efficiency is low in the prior art.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data conversion method, apparatus, and computer equipment. Background Art

[0002] Many companies face many challenges when processing complex multimodal unstructured data, such as documents, images, videos, and audio, to extract valuable information. Specifically, they include: 1) Cumbersome task orchestration: Similar technologies require developers to manually orchestrate complex tasks such as document classification, extraction, normalization, and verification, which is a cumbersome process and inefficient. 2) Poor performance: Existing technologies have difficulty processing large-scale data efficiently, with slow processing speeds and poor accuracy. 3) Complex management and coordination: Traditional technologies rely on multiple independent AI models and services, which not only increases the complexity of management and coordination, but also makes it difficult to ensure the consistency and accuracy of data processing. Therefore, the current multimodal data processing process is relatively complex, resulting in low multimodal data processing efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a multimodal data conversion method, apparatus, and computer device to at least solve the technical problem of low multimodal data processing efficiency in related technologies.

[0004] According to one aspect of an embodiment of the present application, a multimodal data conversion method is provided, comprising: obtaining multimodal data to be converted, wherein the categories of the multimodal data include at least one of the following: pictures, videos, audio, and text; performing attribute extraction on the multimodal data at multiple levels to obtain multiple attributes of the multimodal data; and determining structured data corresponding to the multimodal data based on the multiple attributes.

[0005] Optionally, multiple levels of attribute extraction are performed on the multimodal data to obtain multiple attributes of the multimodal data, including: using the first level of the multiple levels to extract common attributes of the multimodal data, wherein the common attributes include: information that each category of multimodal data has; obtaining the category to which the multimodal data belongs, and determining the extraction process of sub-modal attributes of the multimodal data based on the category to which the multimodal data belongs, wherein the sub-modal attributes include: information unique to each category of multimodal data; using the second level of the multiple levels to extract the sub-modal attributes of the multimodal data according to the extraction process of the sub-modal attributes.

[0006] Optionally, the structured data corresponding to the multimodal data is determined based on the multiple attributes, including: when no extension instruction is received, determining the structured data based on the general attributes of the multimodal data and the sub-modal attributes of the multimodal data; when an extension instruction is received, obtaining the category of the custom attribute from the extension instruction; and using the third level among multiple levels to extract the custom attributes of the multimodal data based on the category of the custom attribute.

[0007] Optionally, the first level among the multiple levels is used to extract the common attributes of the multimodal data, including: obtaining a preset format of the common attributes of the multimodal data, the common attributes including at least one of the following: a unique identifier for marking each data file in the multimodal data, a unique identifier for marking the multimodal data, a storage path of the multimodal data, a category of the multimodal data, a format of the multimodal data, a size of the storage space occupied by the multimodal data, a file name of the multimodal data, and a uniform resource locator of the multimodal data; and generating the common attributes of the multimodal data according to the preset format of the common attributes of the multimodal data.

[0008] Optionally, the second level among the multiple levels is used to extract the sub-modal attributes of the multimodal data according to the sub-modal attribute extraction process, including: when the category to which the multimodal data belongs is video, extracting the first category of sub-modal attributes of the multimodal data, wherein the first category of sub-modal attributes includes at least one of the following: metadata of the video, video summary and key frame information of the video; when the category to which the multimodal data belongs is text, extracting the second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary and information in the text that does not meet the preset requirements; when the category to which the multimodal data belongs is text, extracting the second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary and information in the text that does not meet the preset requirements; When the category is picture, extract the third type of sub-modal attributes of the multimodal data, wherein the third type of sub-modal attributes includes at least one of the following: metadata of the picture, picture summary, information in the picture that does not meet preset requirements, and text information contained in the picture; when the category to which the multimodal data belongs is audio, extract the fourth type of sub-modal attributes of the multimodal data, wherein the fourth type of sub-modal attributes includes at least one of the following: metadata of the audio, audio summary, text corresponding to the audio, information in the audio that does not meet preset requirements, multiple segments corresponding to the audio, and statistical information of the audio, and the statistical information of the audio includes at least: the number of words, the number of speakers, and the number of audio clips.

[0009] Optionally, the third level among the multiple levels is used to extract the custom attributes of the multimodal data according to the category of the custom attributes, including: obtaining the data structure of the custom attributes, wherein the data structure of the custom attributes contains information required for the custom attributes; and generating the custom attributes according to the data structure of the custom attributes.

[0010] Optionally, determining the structured data corresponding to the multimodal data based on the multiple attributes includes: storing the multiple attributes in a target format, wherein the target format is a JSON format; and integrating the stored multiple attributes into the structured data.

[0011] According to another aspect of an embodiment of the present application, a multimodal data conversion device is also provided, including: an acquisition module for acquiring multimodal data to be converted, wherein the category of the multimodal data includes at least one of the following: pictures, videos, audio and text; an extraction module for performing attribute extraction of multiple levels on the multimodal data to obtain multiple attributes of the multimodal data; and a determination module for determining structured data corresponding to the multimodal data based on the multiple attributes.

[0012] According to another aspect of the embodiments of the present application, a computer device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned multimodal data conversion method.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned multimodal data conversion method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned multimodal data conversion method when executed by a processor.

[0015] In an embodiment of the present application, multimodal data to be converted is obtained, wherein the categories of the multimodal data include at least one of the following: pictures, videos, audio and text; multiple levels of attribute extraction are performed on the multimodal data to obtain multiple attributes of the multimodal data; structured data corresponding to the multimodal data is determined based on the multiple attributes, thereby achieving the purpose of extracting various attribute information of the multimodal data at multiple levels and converting the multimodal data quickly and accurately, thereby achieving the technical effect of improving the efficiency of multimodal data conversion, and further solving the technical problem of low multimodal data processing efficiency in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a multimodal data conversion method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of a multimodal data conversion method according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of another multimodal data conversion method according to an embodiment of the present application;

[0020] Figure 4 is a flow chart of a scalable multimodal data conversion method according to an embodiment of the present application;

[0021] Figure 5 is a structural diagram of a multimodal data conversion system according to an embodiment of the present application;

[0022] Figure 6 is a structural diagram of a multimodal data conversion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered.

[0026] In order to solve the problems existing in the related art, the embodiment of the present application provides a multimodal data conversion method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0027] The multimodal data conversion method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a multimodal data conversion method. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0028] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimodal data conversion method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned multimodal data conversion method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0032] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0033] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a multimodal data conversion method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 2 is a flow chart of a multimodal data conversion method according to an embodiment of the present application, such as Figure 2As shown, the method includes the following steps:

[0035] Step S202: acquiring multimodal data to be converted, wherein the multimodal data may be in the form of at least one of the following: pictures, videos, audio, and text;

[0036] Step S204: extracting attributes of multiple levels from the multimodal data to obtain multiple attributes of the multimodal data;

[0037] Step S206: determining structured data corresponding to the multimodal data according to the multiple attributes.

[0038] Through the above steps S202 to S206, multimodal data to be converted is obtained, wherein the category of the multimodal data includes at least one of the following: pictures, videos, audio, and text; multiple levels of attribute extraction are performed on the multimodal data to obtain multiple attributes of the multimodal data; and structured data corresponding to the multimodal data is determined based on the multiple attributes, thereby achieving the purpose of extracting various attribute information of the multimodal data at multiple levels and achieving the purpose of quickly and accurately converting the multimodal data, thereby achieving the technical effect of improving the efficiency of multimodal data conversion and solving the technical problem of low multimodal data processing efficiency in related technologies. Detailed description is given below.

[0039] In some embodiments of the present application, multiple levels of attribute extraction are performed on the multimodal data to obtain multiple attributes of the multimodal data. The specific steps are as follows: using the first level of the multiple levels to extract common attributes of the multimodal data, wherein the common attributes include: information that each category of multimodal data has; obtaining the category to which the multimodal data belongs, and determining the extraction process of sub-modal attributes of the multimodal data based on the category to which the multimodal data belongs, wherein the sub-modal attributes include: information unique to each category of multimodal data; using the second level of the multiple levels according to the extraction process of the sub-modal attributes to extract the sub-modal attributes of the multimodal data.

[0040] It should be noted that the common attributes are mainly the meta-information of multimodal data, while the sub-modal attributes include: information unique to each category of multimodal data, such as: content characteristics of multimodal data.

[0041] Among them, in order to better adapt to the needs of different application scenarios, the embodiment of the present application also provides an extension function. The specific steps of determining the structured data corresponding to the multimodal data according to the multiple attributes are as follows: when no extension instruction is received, the structured data is determined according to the common attributes of the multimodal data and the sub-modal attributes of the multimodal data; when an extension instruction is received, the category of the custom attribute is obtained from the extension instruction; and the third level among multiple levels is used to extract the custom attributes of the multimodal data according to the category of the custom attribute.

[0042] In some embodiments of the present application, the specific steps of extracting the common attributes of the multimodal data using the first level of the multiple levels are as follows: obtaining a preset format of the common attributes of the multimodal data, the common attributes including at least one of the following: a unique identifier for marking each data file in the multimodal data, a unique identifier for marking the multimodal data, a storage path of the multimodal data, a category of the multimodal data, a format of the multimodal data, a size of the storage space occupied by the multimodal data, a file name of the multimodal data, and a uniform resource locator of the multimodal data; and generating the common attributes of the multimodal data according to the preset format of the common attributes of the multimodal data.

[0043] The common base attributes contain the following information, which is designed to be in a flat structured format due to its fixed structure, for example:

[0044] Id (a unique identifier for marking each data file in the multimodal data), ds_id (a unique identifier for marking the multimodal data), ds_path (the storage path of the multimodal data), modality (the category of the multimodal data), media_type (the format of the multimodal data), file_size (the size of the storage space occupied by the multimodal data), file_name (the file name of the multimodal data), url (the uniform resource locator of the multimodal data).

[0045] In some embodiments of the present application, the specific steps of extracting the sub-modal attributes of the multimodal data according to the sub-modal attribute extraction process using the second level among the multiple levels are as follows: when the category to which the multimodal data belongs is video, extracting a first category of sub-modal attributes of the multimodal data, wherein the first category of sub-modal attributes includes at least one of the following: metadata of the video, video summary, and key frame information of the video; when the category to which the multimodal data belongs is text, extracting a second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary, and information in the text that does not meet preset requirements; when the category to which the multimodal data belongs is text, extracting a second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary, and information in the text that does not meet preset requirements; When the category to which the multimodal data belongs is picture, a third category of sub-modal attributes of the multimodal data is extracted, wherein the third category of sub-modal attributes includes at least one of the following: metadata of the picture, a summary of the picture, information in the picture that does not meet preset requirements, and text information contained in the picture; when the category to which the multimodal data belongs is audio, a fourth category of sub-modal attributes of the multimodal data is extracted, wherein the fourth category of sub-modal attributes includes at least one of the following: metadata of the audio, a summary of the audio, text corresponding to the audio, information in the audio that does not meet preset requirements, multiple segments corresponding to the audio, and statistical information of the audio, wherein the statistical information of the audio includes at least: the number of words, the number of speakers, and the number of audio clips.

[0046] The structured data generated by the sub-modal attributes is as follows:

[0047] Id, ds_id, ds_path, modality, media_type, file_size, file_name, url, modal_arrtibutes (JSON structure data of modal attributes).

[0048] In an optional manner, the sub-modal attribute content of multimodal data of video type contains multiple parts of information, presented in JSON format, and the overall structure mainly includes video metadata, video content analysis results, key frame information, and statistical data. Metadata (metadata): records basic information of the video, such as frame rate (frame_rate), encoding format (codec), duration (duration), resolution (frame_width and frame_height). Video (video content analysis results): provides a summary of the entire video, summarizing the key themes, events, and information of the video. Frames (frame information): is an array, each element represents a frame in the video. Contains time code (timecode), timestamp (timestamp), frame index (index), and content review information (content_moderation).

[0049] The sub-processes corresponding to extracting multimodal attributes include video metadata information extraction, video key frame extraction, video summarization, video toxicity identification and other video recognition algorithms.

[0050] The sub-modal attributes of text-type multimodal attributes contain multiple pieces of information, presented in JSON format. The overall structure primarily includes document metadata and content information. Metadata includes information such as the document language and encoding format. Document content information includes a summary that summarizes key document information. Content moderation summarizes document violations, etc.

[0051] The sub-processes corresponding to extracting multimodal attributes include text metadata information extraction, text summarization, text toxicity detection and other text recognition algorithms.

[0052] The sub-modal attribute content of the multimodal attribute of the image type contains multiple parts of information, presented in JSON format. The overall structure mainly includes metadata, image analysis results, content review results, text information, etc., presented in JSON format. The overall structure mainly includes metadata: the width (image_width_pixels), height (image_height_pixels"), and color depth (color_depth") of the image, which reflect the size and color information of the image; image encoding format (image_encoding), such as JPEG, PNG, etc. image (image analysis result): includes summary (image summary): generates descriptive text for the image content, summarizing the main scenes or objects in the image. Image content review (content_moderation): includes review ID (id), review type (type), review category (category), such as the category of detected inappropriate content; confidence (confidence), which reflects the reliability of the review result. text_words (text word information): is an array, each element represents a word detected in the image, including word ID (id), word text (text), confidence (confidence), and location information (locations).

[0053] The sub-processes corresponding to extracting multimodal attributes include image metadata extraction, image summarization, image toxicity detection, image OCR and other image recognition and detection algorithms.

[0054] The sub-modal attributes of the multimodal attributes of the audio type contain multiple pieces of information, presented in JSON format. The overall structure mainly includes metadata, audio content analysis results, content review status, audio clip information, and statistical data, providing a clear structural framework for the extraction and analysis of audio information. Metadata: audio format ("format"), such as MP3, WAV, etc.; sampling rate ("sample_rate"), reflecting the sampling accuracy of the audio; bit rate ("bit_rate"), reflecting the data transmission rate of the audio; duration ("duration"), indicating the duration of the audio; number of channels ("channels"), indicating the channel settings of the audio.

[0055] Audio (audio content analysis results) includes a summary: This extracts key themes, events, and information from the entire audio file into a concise summary. Transcript: This fully presents all spoken content in the audio as text. Advanced speech recognition technology accurately transcribes audio elements such as dialogue and narration, and includes speaker identification information for easy speaker-based navigation and search of the audio content.

[0056] content_moderation (content moderation status): This is an array, each element of which represents the moderation result. It includes the moderation ID (id), moderation type (type), moderation category (category), such as the category of inappropriate content detected; confidence (confidence), which reflects the reliability of the moderation result; and start and end timestamps (start_timestamp_millis and end_timestamp_millis).

[0057] audio_segments (audio segment information): This is an array, each element of which represents an audio segment. It contains the segment ID (id), start and end timestamps (start_timestamp_millis and end_timestamp_millis), audio text content (text), and speaker information (speaker). The speaker information includes the speaker ID (speakerid), which is used to identify audio segments of different speakers.

[0058] Statistics: Provides statistical information about the audio content, including word count (word_count), speaker count (speaker_count), and segment count (segment_count). This data helps understand the richness and structure of the audio content. The corresponding subprocesses include audio recognition algorithms such as audio metadata extraction, audio summarization, audio toxicity detection, and audio text information statistics.

[0059] In some embodiments of the present application, the specific steps of extracting the custom attributes of the multimodal data according to the category of the custom attributes using the third level among multiple levels are as follows: obtaining the data structure of the custom attributes, wherein the data structure of the custom attributes contains the information required for the custom attributes; and generating the custom attributes according to the data structure of the custom attributes.

[0060] In the method provided in the embodiment of the present application, on top of the existing general basic attributes and modal basic attributes presented in JSON structure, this solution provides flexible custom extended attribute structure support, and the extended attributes also adopt JSON structure.

[0061] After defining extended attributes, you must accurately select the corresponding extended attribute structure during structured data storage. At the same time, you'll add a corresponding node to the currently generated default task flow. This node can be flexibly placed anywhere in the flow based on actual needs.

[0062] There are various options for processing nodes. They can be existing algorithm model nodes built into the system, which can achieve efficient processing with mature algorithms; they can also be customized algorithm model nodes for personalized in-depth configuration to meet unique business needs.

[0063] After the node is configured, the process will sequentially accumulate the output results of each algorithm model into the extended information to achieve deep integration and enrichment of data.

[0064] To ensure the accuracy and consistency of data, when storing structured data, it is necessary to perform field mapping and matching operations on extended attributes to accurately correspond to each extended attribute information.

[0065] The following is an example of the structured data finally generated:

[0066] id,ds_id,ds_path,modality,media_type,file_size,file_name,url,modal_arrtibutes,extend_attributes

[0067] Where extend_attributes is the JSON structure data of the custom multimodal attributes. Taking the invoice photo information scanning scenario as an example, the possible custom attribute structure design is as follows:

[0068]

[0069]

[0070]

[0071] Through the above series of designs and operations, we can finally realize the complete generation logic of customized processing flow and customized multimodal attributes, which gives high flexibility and scalability to multimodal data processing and meets the needs of diverse application scenarios.

[0072] In some embodiments of the present application, determining the structured data corresponding to the multimodal data based on the multiple attributes includes: storing the multiple attributes in a target format, wherein the target format is a JSON format; and integrating the stored multiple attributes into the structured data.

[0073] Figure 3 Another multimodal data conversion method is shown, such as Figure 3 Shown, including:

[0074] Step 1, dataset selection: Select multimodal datasets containing different forms of data such as pictures, videos, audio, and text from various sources to provide a data foundation for subsequent processing.

[0075] Step 2, general attribute generation: Perform preliminary processing on the selected multimodal dataset to extract general basic attributes applicable to all modal data, such as data creation time and storage location, to provide basic information for overall data management.

[0076] Step 3, automatic modal analysis: Use specific automatic analysis algorithms to identify and classify data modalities in the dataset, accurately distinguishing different types of data such as images, videos, audio, and text.

[0077] Step 4: Modal attribute processing: Based on the modal categories automatically analyzed, the corresponding basic attribute processing processes are initiated. For example, for images, attributes such as resolution and color mode are extracted; for videos, attributes such as frame rate and duration are obtained; for audio, attributes such as sampling rate and number of channels are analyzed; and for text, attributes such as word count and character encoding are extracted.

[0078] Step 5: Algorithm Adjustment and Optimization: Dynamically adjust the algorithm version and attributes used by each node in each modal processing flow. Based on data characteristics and processing requirements, select the most appropriate algorithm version and set optimal attribute parameters to improve processing effectiveness and quality.

[0079] Step 6: Structured storage of attributes: The basic attributes of each modality obtained through the above processing are stored according to the JSON structure to achieve structured organization of data.

[0080] Step 7, structured data set generation: Integrate the structured basic attributes of all modalities to finally generate a complete structured multimodal data set to facilitate subsequent data analysis, mining and application.

[0081] Figure 4 A scalable multimodal data conversion method is shown, such as Figure 4 Shown, including:

[0082] Step 1. Adding custom processing nodes: Users can add custom processing nodes to existing processing flows based on their specific business needs and data processing objectives. These custom nodes can perform specific data processing tasks, such as feature extraction, data transformation, or filtering.

[0083] Step 2. Generate custom extended attributes: Generate custom extended attributes that meet the user's personalized needs through the execution of custom processing nodes, and also store them in a JSON structure to further enrich the attribute dimensions of the data.

[0084] This application also proposes a multimodal data conversion system, such as Figure 5As shown, it includes: Algorithm model management module: This module is responsible for unified management of multimodal AI algorithm models. It integrates multiple different types of algorithm models and monitors the running status of the model in real time. Through version control of the model, it ensures that the most appropriate model can be called in different scenarios. Dataset management module: This module manages the input multimodal dataset and the output structured dataset, facilitates data query and call, and makes data management more standardized and orderly. Multimodal structured attribute design and management module: This module is constructed based on the three-level multimodal attribute design concept. In terms of design, it can flexibly define and manage general basic attributes, sub-modal basic attributes, and custom extended attributes to achieve structured conversion of multimodal data. It is the core part of the entire technology. Through scientific attribute design, it converts unstructured multimodal data into a structured format, provides convenience for subsequent data analysis and application, and greatly improves the accuracy of data processing. Processing task orchestration and management module: The design focus of this module is to automatically generate the processing task flow of the corresponding modality based on the data set to be processed to generate the basic attributes of the corresponding modality. For custom extended attributes, it supports adding custom nodes in the automatically generated process to generate the corresponding custom extended attributes, thereby realizing a more scalable automated processing process. Multimodal structured processing engine: It can integrate the functions of the above four modules to extract and process structured attributes of multimodal data. According to the task definition in the processing task orchestration and management module, the processing flow DAG graph that the final engine can run is generated and the processing task is executed; the reading and writing of the data set contained in the processing task flow requires the corresponding data set information to be obtained from the data set management module; in the process of multimodal data attribute extraction and processing, according to the process definition and configuration information, the corresponding algorithm model is loaded for calling, and the extracted attribute information is integrated after processing the call result, and finally matched with the data structure in the multimodal structured attribute design module to complete the structured conversion of multimodal data.

[0085] Figure 6 A multimodal data conversion device according to an embodiment of the present application includes:

[0086] An acquisition module 60 is configured to acquire multimodal data to be converted, wherein the multimodal data may be in the form of at least one of the following: images, videos, audio, and text;

[0087] An extraction module 62 is configured to extract attributes of the multimodal data at multiple levels to obtain multiple attributes of the multimodal data;

[0088] The determination module 64 is configured to determine structured data corresponding to the multimodal data according to the multiple attributes.

[0089] Through the above-mentioned multimodal data conversion device, multimodal data to be converted is obtained, wherein the category of the multimodal data includes at least one of the following: pictures, videos, audio and text; multiple levels of attribute extraction are performed on the multimodal data to obtain multiple attributes of the multimodal data; structured data corresponding to the multimodal data is determined based on the multiple attributes, thereby achieving the purpose of extracting various attribute information of the multimodal data at multiple levels and converting the multimodal data quickly and accurately, thereby achieving the technical effect of improving the efficiency of multimodal data conversion, and thus solving the technical problem of low multimodal data processing efficiency in related technologies.

[0090] The extraction module 62 includes: an extraction submodule, which is used to perform multiple-level attribute extraction on the multimodal data to obtain multiple attributes of the multimodal data, including: using the first level of the multiple levels to extract common attributes of the multimodal data, wherein the common attributes include: information that each category of multimodal data has; obtaining the category to which the multimodal data belongs, and determining the extraction process of sub-modal attributes of the multimodal data according to the category to which the multimodal data belongs, wherein the sub-modal attributes include: information unique to each category of multimodal data; using the second level of the multiple levels according to the extraction process of the sub-modal attributes to extract the sub-modal attributes of the multimodal data.

[0091] The extraction submodule includes: a determination unit, a first extraction unit, a second extraction unit, and a third extraction unit, which are used to determine the structured data corresponding to the multimodal data based on the multiple attributes, including: when no extension instruction is received, determining the structured data based on the common attributes of the multimodal data and the sub-modal attributes of the multimodal data; when an extension instruction is received, obtaining the category of the custom attribute from the extension instruction; and using the third level among multiple levels to extract the custom attribute of the multimodal data based on the category of the custom attribute.

[0092] A first extraction unit is configured to extract common attributes of the multimodal data using a first level among the multiple levels, including: obtaining a preset format of the common attributes of the multimodal data, the common attributes including at least one of the following: a unique identifier for marking each data file in the multimodal data, a unique identifier for marking the multimodal data, a storage path of the multimodal data, a category of the multimodal data, a format of the multimodal data, a size of the storage space occupied by the multimodal data, a file name of the multimodal data, and a uniform resource locator of the multimodal data; and generating the common attributes of the multimodal data according to the preset format of the common attributes of the multimodal data.

[0093] The second extraction unit is configured to extract the sub-modal attributes of the multimodal data according to the sub-modal attribute extraction process using the second level among the multiple levels, including: when the category to which the multimodal data belongs is video, extracting a first category of sub-modal attributes of the multimodal data, wherein the first category of sub-modal attributes includes at least one of the following: metadata of the video, video summary, and key frame information of the video; when the category to which the multimodal data belongs is text, extracting a second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary, and information in the text that does not meet preset requirements; when the category to which the multimodal data belongs is text, extracting a second category of sub-modal attributes of the multimodal data, wherein the second category of sub-modal attributes includes at least one of the following: metadata of the text, text summary, and information in the text that does not meet preset requirements; When the category to which the multimodal data belongs is picture, a third type of sub-modal attributes of the multimodal data is extracted, wherein the third type of sub-modal attributes includes at least one of the following: metadata of the picture, picture summary, information in the picture that does not meet preset requirements, and text information contained in the picture; when the category to which the multimodal data belongs is audio, a fourth type of sub-modal attributes of the multimodal data is extracted, wherein the fourth type of sub-modal attributes includes at least one of the following: metadata of the audio, audio summary, text corresponding to the audio, information in the audio that does not meet preset requirements, multiple segments corresponding to the audio, and statistical information of the audio, wherein the statistical information of the audio includes at least: the number of words, the number of speakers, and the number of audio clips.

[0094] A third extraction unit is configured to extract the custom attribute of the multimodal data according to the category of the custom attribute using a third level among the multiple levels, including: obtaining a data structure of the custom attribute, wherein the data structure of the custom attribute includes information required for the custom attribute; and generating the custom attribute according to the data structure of the custom attribute.

[0095] The determination module 64 includes: determining the structured data corresponding to the multimodal data based on the multiple attributes, including: storing the multiple attributes in a target format, wherein the target format is JSON format; and integrating the stored multiple attributes into the structured data.

[0096] It should be noted that Figure 6 The multimodal data conversion device shown is used to perform Figure 2 The multimodal data conversion method shown in FIG. 4 is a multimodal data conversion method, so the relevant explanations in the above multimodal data conversion method are also applicable to the multimodal data conversion device, and will not be repeated here.

[0097] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned multimodal data conversion method.

[0098] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned multimodal data conversion method by running the computer program.

[0099] An embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the multimodal data conversion method in the present application.

[0100] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0106] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal data conversion method, characterized in that: include: Acquire multimodal data to be converted, wherein the categories of the multimodal data include at least one of the following: picture, video, audio and text; Extracting attributes of the multimodal data at multiple levels to obtain multiple attributes of the multimodal data; Determine structured data corresponding to the multimodal data according to the multiple attributes.

2. The method according to claim 1, characterized in that Performing attribute extraction on the multimodal data at multiple levels to obtain multiple attributes of the multimodal data, including: Extracting common attributes of the multimodal data using a first level of the multiple levels, wherein the common attributes include: information common to each category of multimodal data; Obtaining the category to which the multimodal data belongs, and determining a process for extracting sub-modal attributes of the multimodal data according to the category to which the multimodal data belongs, wherein the sub-modal attributes include: information unique to each category of multimodal data; The sub-modal attributes of the multimodal data are extracted using a second level among the multiple levels according to the sub-modal attribute extraction process.

3. The method according to claim 2, characterized in that Determining structured data corresponding to the multimodal data according to the multiple attributes includes: In the case where no extension instruction is received, determining the structured data according to the general attributes of the multimodal data and the sub-modal attributes of the multimodal data; When receiving the extension instruction, obtaining the category of the custom attribute from the extension instruction; A third level among the multiple levels is used to extract custom attributes of the multimodal data according to categories of the custom attributes.

4. The method according to claim 2, characterized in that Extracting common attributes of the multimodal data using a first level of the multiple levels includes: Obtaining a preset format for common attributes of the multimodal data, the common attributes comprising at least one of the following: a unique identifier for marking each data file in the multimodal data, a unique identifier for marking the multimodal data, a storage path of the multimodal data, a category of the multimodal data, a format of the multimodal data, a size of storage space occupied by the multimodal data, a file name of the multimodal data, and a uniform resource locator of the multimodal data; Generate the common attributes of the multimodal data according to a preset format of the common attributes of the multimodal data.

5. The method according to claim 2, characterized in that Extracting the sub-modal attributes of the multimodal data according to the sub-modal attribute extraction process using the second level of the multiple levels includes: When the category to which the multimodal data belongs is video, extracting a first type of sub-modal attributes of the multimodal data, wherein the first type of sub-modal attributes includes at least one of the following: metadata of the video, a video summary, and key frame information of the video; When the category to which the multimodal data belongs is text, extracting a second type of sub-modal attribute of the multimodal data, wherein the second type of sub-modal attribute includes at least one of the following: metadata of the text, a text summary, and information in the text that does not meet preset requirements; When the category to which the multimodal data belongs is picture, extracting a third type of sub-modal attribute of the multimodal data, wherein the third type of sub-modal attribute includes at least one of the following: metadata of the picture, a picture summary, information in the picture that does not meet preset requirements, and text information contained in the picture; In a case where the multimodal data belongs to an audio category, a fourth category of sub-modal attributes of the multimodal data is extracted, wherein the fourth category of sub-modal attributes includes at least one of the following: metadata of the audio, an audio summary, text corresponding to the audio, information in the audio that does not meet preset requirements, multiple segments corresponding to the audio, and statistical information of the audio, wherein the statistical information of the audio includes at least: the number of words, the number of speakers, and the number of audio clips.

6. The method according to claim 3, characterized in that Extracting the custom attributes of the multimodal data according to the categories of the custom attributes using a third level among the multiple levels includes: Acquire a data structure of the custom attribute, wherein the data structure of the custom attribute includes information required by the custom attribute; The custom attribute is generated according to the data structure of the custom attribute.

7. The method according to claim 1, characterized in that Determining structured data corresponding to the multimodal data according to the multiple attributes includes: Storing the multiple attributes in a target format, wherein the target format is a JSON format; The stored multiple attributes are integrated into the structured data.

8. A multimodal data conversion device, characterized in that: include: An acquisition module, configured to acquire multimodal data to be converted, wherein the multimodal data may be in the form of at least one of the following: pictures, videos, audio, and text; An extraction module, configured to extract attributes of the multimodal data at multiple levels to obtain multiple attributes of the multimodal data; A determination module is used to determine structured data corresponding to the multimodal data based on the multiple attributes.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the multimodal data conversion method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the multimodal data conversion method according to any one of claims 1 to 7 is implemented.

Images