Multi-modal data management system, method, equipment and medium

Abstract

The invention provides a multi-modal data management system and method, equipment and a medium, and the system comprises a multi-modal data quality index creation module which is used for creating at least one multi-modal data quality index for multi-modal data; the multi-modal data quality detection task creation module is used for creating a multi-modal data quality detection task for the multi-modal data according to the at least one multi-modal data quality index and distributing the multi-modal data quality detection task to the multi-modal data detection module; the multi-modal data detection module is used for executing a multi-modal data quality detection task so as to carry out quality detection on the multi-modal data; and the multi-modal data detection result generation module is used for generating and displaying a multi-modal data quality detection report according to the execution result of the multi-modal data quality detection task, and the quality and management efficiency of the multi-modal data can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a multimodal data management system, method, device and medium. Background Art

[0002] With the rapid development of information technology, enterprises and organizations have accumulated a large amount of multimodal data in their daily operations, including text, images, audio and video. The richness of these multimodal data provides a more comprehensive information basis for data analysis and decision-making, but it also brings challenges to data quality management. Traditional data management methods often cannot effectively cope with the complexity and heterogeneity of multimodal data, resulting in the following technical problems: 1. Inconsistent data quality: The diversity of multimodal data sources results in uneven data quality and a lack of unified quality assessment standards, leading to problems with data accuracy, completeness, and consistency.

[0003] 2. Lack of effective quality detection mechanism: Existing data quality detection methods often target single-modal data and are difficult to comprehensively evaluate the overall quality of multi-modal data, resulting in the inability to promptly discover and correct potential problems.

[0004] 3. Inefficient data processing: When faced with massive multimodal data, traditional data quality management processes usually rely on manual review, which is inefficient and error-prone and cannot meet the needs of real-time data processing.

[0005] 4. Lack of visual quality feedback: Current data quality test results are often presented in the form of text or tables, lacking intuitive visual display, making it difficult to help users quickly understand the data quality status. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a multimodal data management system, method, device and medium, which aims to achieve comprehensive quality management of multimodal data by creating multimodal data quality indicators, automatically creating multimodal data quality detection tasks, performing quality detection and generating quality detection reports. The system can effectively improve the accuracy and consistency of data quality, optimize the data processing process, and improve the efficiency of data management, so as to overcome the above problems or at least partially solve the above problems.

[0007] Metadata: Metadata is information about the organization, domain and relationship of data. In short, metadata is data about data. It reflects the structure, characteristics, relationship and management of data. Metadata is data that describes information resources or data objects. Its purpose is to identify resources, describe resources, evaluate resources, track resources, etc., and to achieve effective discovery, understanding, organization and management of data resources.

[0008] Data quality: Data quality reflects the "suitability" of data, that is, the degree to which data meets usage requirements. It involves multiple dimensions of data, such as correctness, consistency, completeness, reliability, timeliness, etc. By comprehensively examining these dimensions, we can ensure that data plays the greatest value in business applications.

[0009] Data assets: Data assets are valuable, measurable, and readable data sets in cyberspace with data ownership (such as exploration rights, usage rights, and ownership).

[0010] Multimodal data governance refers to the process of integrating, managing, and analyzing multiple types of data. Artificial intelligence applications driven by big models require multimodal data for large-scale training. Data needs to continuously enhance data quality and richness to assist big models in achieving accurate cognition. At the same time, there are multiple ways of interaction and association between different modal data. For example, constructing a positive feedback method to input pictures, texts, audio and video, knowledge bases, etc. of the same specific business scenario into the big model can enhance the understanding of the big model. Constructing a negative feedback method, when deviations occur in text data training, other modal data can assist the big model in error correction, thereby reducing the "illusion" of the big model from the source. High-quality, multi-dimensional, and large-scale data values ​​are the key foundation for supporting the efficient application of big models. Therefore, multimodal data governance has gradually evolved into an important data processing method.

[0011] At present, the status quo of multimodal data governance can be analyzed from the following aspects: Rich data types: Multimodal data is diverse, including structured data and unstructured data. These data types come from different channels and systems, with different formats, making integration difficult.

[0012] Insufficient technical means: Multimodal data governance mainly relies on manual or semi-automatic processing, with relatively simple technical means and a lack of efficient and accurate data cleaning, integration and analysis tools.

[0013] Data security and privacy issues are prominent: Multimodal data involves data from multiple sources and different fields, and data security and privacy protection have become important issues. In the process of multimodal data governance, it is necessary to strengthen technical means such as data encryption and access control to ensure data security and privacy.

[0014] Insufficient data value mining: The core of multimodal data governance is to integrate, analyze and mine data to tap its potential value. However, at present, there are still certain limitations in the value mining of multimodal data, which needs further exploration and improvement.

[0015] Strong demand for industry applications: Multimodal data governance has broad application prospects in finance, medical care, education and other fields. However, in actual applications, different industries have different needs for data governance, and corresponding solutions need to be formulated according to specific circumstances.

[0016] A first aspect of an embodiment of the present application provides a multimodal data management system, the system comprising: A multimodal data quality indicator creation module, used to create at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator includes: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; A multimodal data quality detection task creation module, configured to create a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator, and assign the multimodal data quality detection task to a multimodal data detection module; A multimodal data detection module, used to execute the multimodal data quality detection task to perform quality detection on the multimodal data; The multimodal data detection result generation module is used to generate and display a multimodal data quality detection report according to the execution results of the multimodal data quality detection task. The multimodal data quality detection report includes: the overall quality of the multimodal data and the detection results of various multimodal data quality indicators.

[0017] Optionally, the system further comprises: A multimodal metadata extraction module, used to analyze and process the multimodal data through an AI trusted framework, and extract metadata from the multimodal data to obtain multimodal metadata; A multimodal metadata graph construction module, used to construct a metadata knowledge graph based on the multimodal metadata; A data feature space construction module is used to extract multimodal data features from the multimodal data, and create a first data feature space of the multimodal data based on the multimodal data features. The first data feature space is used to be combined with the metadata knowledge graph to manage the multimodal data in the first data feature space through the combined metadata knowledge graph.

[0018] Optionally, the multimodal metadata extraction module includes: An image metadata extraction unit, configured to identify visual elements contained in the image data in the multimodal data through a computer vision model, and convert the visual elements into first metadata, wherein the first metadata is: searchable and annotable keywords and attributes in the visual elements; a text metadata extraction unit, configured to obtain sentence structures, language patterns, and contextual meanings contained in the text data in the multimodal data through a natural language processing model, and extract second metadata from the sentence structures, language patterns, and contextual meanings contained in the text data, wherein the second metadata includes: characters, places, events, concepts, and emotional colors in the text data; The audio and video metadata extraction unit is used to obtain third metadata in the audio and video data in the multimodal data through a speech recognition model and an audio and video analysis model, wherein the third metadata includes: background music, sound type, voice emotion, key timestamp and summary information contained in the audio and video data.

[0019] Optionally, the system further comprises: A multimodal metadata tag building module, used to convert the multimodal data features into multiple different types of tags through knowledge graph technology, and build the multiple different types of tags into a standardized tag system; A multimodal data labeling module, used to create a labeling task and label the multimodal data through the labeling system; A multimodal data label correction module is used to correct the labeling results of the multimodal data whose similarity is greater than the similarity threshold and whose labeling results are different among the labeled multimodal data; The multimodal data display module is used to visualize the annotated multimodal data through label portraits.

[0020] Optionally, the system further comprises: A multimodal data asset management module is used to extract multimodal data features from the multimodal data, construct a second data feature space through a large language model, and combine it with the metadata knowledge graph to perform lineage query on the multimodal data and establish a multimodal data asset catalog through the combined metadata knowledge graph.

[0021] Optionally, the system further comprises: A multimodal data standard formulation module, used to formulate a multimodal data standard corresponding to the multimodal data according to the scope and type of the multimodal data; A standard file configuration module, used to configure a corresponding standard file according to the multimodal data standard, wherein the standard file is used for a user terminal to upload and download the multimodal data standard; The multimodal data standard detection module is used to detect the multimodal data standard and update the multimodal data standard.

[0022] Optionally, the multimodal data detection module is used to perform the multimodal data quality detection task to perform quality detection on the multimodal data, specifically: Performing quality detection of the richness index on the multimodal data through a data non-empty rate and a data non-repetitive rate, wherein the data non-empty rate is determined according to the number of non-empty files and the total number of files of the multimodal data, and the data non-repetitive rate is determined according to the number of non-repetitive files and the total number of files of the multimodal data; Performing logic verification on the multimodal data to achieve quality detection of the logic indicators; The content of the multimodal data is identified to achieve quality detection of the toxicity of the file.

[0023] A second aspect of an embodiment of the present application provides a multimodal data management method, the method comprising: Creating at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator comprises: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; Creating a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator; Executing the multimodal data quality detection task to perform quality detection on the multimodal data; According to the execution result of the multimodal data quality detection task, a multimodal data quality detection report is generated and displayed, and the multimodal data quality detection report includes: the overall quality status of the multimodal data and the detection results of various multimodal data quality indicators.

[0024] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the second aspect.

[0025] In a fourth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method described in the second aspect.

[0026] According to a fifth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method described in the second aspect is implemented.

[0027] Beneficial effects of this application: The embodiments of the present application provide a multimodal data management system, method, device and medium, including: a multimodal data quality indicator creation module, used to create at least one multimodal data quality indicator for multimodal data; a multimodal data quality detection task creation module, used to create a multimodal data quality detection task for multimodal data according to at least one multimodal data quality indicator, and assign the multimodal data quality detection task to the multimodal data detection module; a multimodal data detection module, used to execute the multimodal data quality detection task to perform quality detection on the multimodal data; a multimodal data detection result generation module, used to generate and display a multimodal data quality detection report according to the execution result of the multimodal data quality detection task.

[0028] Through the technical solution of this application, by creating multi-dimensional quality indicators such as richness, logic, and file toxicity, the quality of multimodal data can be comprehensively evaluated, ensuring that multimodal data meets the expected standards in many aspects, thereby improving the reliability of the data. In addition, by automatically creating quality inspection tasks and assigning them to corresponding inspection modules, the automation of data quality inspection is achieved. This reduces manual intervention, improves inspection efficiency, reduces the possibility of human error, and can also perform multimodal data quality inspection tasks in real time and generate corresponding multimodal data quality inspection reports, so that managers can quickly identify and deal with data quality issues, thereby taking timely measures, and significantly improving the quality and management efficiency of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0030] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 It is a schematic diagram of a framework of a multimodal data management system provided in one embodiment of the present application; Figure 2 It is a schematic diagram of a multimodal data governance capability framework of a multimodal data management system provided by an embodiment of the present application; Figure 3 It is a flowchart of a multimodal data management method provided in one embodiment of the present application; Figure 4 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] Figure 1 is a schematic diagram of a framework of a multimodal data management system provided by an embodiment of the present application, Figure 2 is a schematic diagram of a multimodal data governance capability framework of a multimodal data management system provided in one embodiment of the present application; Figure 1 As shown, an embodiment of the present application provides a multimodal data management system, the system comprising: A multimodal data quality indicator creation module 11, used to create at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator includes: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; A multimodal data quality detection task creation module 12, configured to create a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator, and assign the multimodal data quality detection task to a multimodal data detection module; A multimodal data detection module 13, used to execute the multimodal data quality detection task to perform quality detection on the multimodal data; The multimodal data detection result generation module 14 is used to generate and display a multimodal data quality detection report according to the execution result of the multimodal data quality detection task, and the multimodal data quality detection report includes: the overall quality of the multimodal data and the detection results of various multimodal data quality indicators.

[0035] Specifically, in one embodiment, a multimodal data management system is provided, which includes four main modules, each of which has its specific function: Multimodal Data Quality Indicator Creation Module 11: This module is responsible for defining and creating multimodal data quality indicators, which are used to measure the quality of multimodal data. These multimodal data quality indicators include but are not limited to: Richness metrics: measure the diversity and coverage of multimodal data, for example, whether customer service records contain all possible types of customer issues.

[0036] Logical indicators: Evaluate the consistency and accuracy of multimodal data, such as whether network performance data conforms to expected logical relationships, such as whether peak traffic periods correspond to higher network latency.

[0037] File toxicity index: Detects whether multimodal data contains inappropriate content, such as whether customer voice call records contain sensitive words or offensive language.

[0038] Multimodal data quality detection task creation module 12: This module is used to generate specific multimodal data quality detection tasks based on the multimodal data quality indicators created by the multimodal data quality indicator creation module 11. These multimodal data quality detection tasks will specify the data set to be detected and the multimodal data quality indicators of the application, and then assign the multimodal data quality detection tasks to the corresponding multimodal data detection modules 13.

[0039] Multimodal data detection module 13: This module is used to perform actual quality detection on multimodal data according to the assigned multimodal data quality detection task, for example, using a machine learning model to analyze the sentiment tendency of text data, or using image recognition technology to evaluate the quality of surveillance videos.

[0040] Multimodal data detection result generation module 14: After the multimodal data quality detection task is completed, the multimodal data detection result generation module 14 will collect the detection results corresponding to all multimodal data quality detection tasks and generate a multimodal data quality detection report. The multimodal data quality detection report not only shows the overall quality of the multimodal data, but also lists in detail the detection results of each multimodal data quality indicator.

[0041] In addition, the quality task management module can also be used to create, schedule, and monitor multimodal data quality detection tasks. Multimodal data quality detection tasks are scheduled according to their priorities and resource conditions, and assigned to appropriate multimodal data detection modules for execution. The execution progress and results of multimodal data quality detection tasks are monitored, and timely feedback is provided to users.

[0042] For example, when you need to evaluate the quality of multimodal data collected in its customer service system, including customer service records (text), communication network monitoring videos (video), network performance data (structured data), and customer voice call records (audio).

[0043] First, you need to create multimodal data quality indicators: Richness indicator: Check whether customer service records cover all service types, such as installation, maintenance, consultation, etc.

[0044] Logical metrics: Analyze network performance data to ensure that network latency during high traffic periods is consistent with expectations.

[0045] File toxicity indicators: Scan voice call logs to ensure they do not contain any offensive or inappropriate language.

[0046] Furthermore, a multimodal data quality detection task is created. In this process, the system creates a multimodal data quality detection task for each type of multimodal data for each quality indicator. For example, text analysis is performed on customer service records to check whether keywords of all service types are included. Statistical analysis is performed on network performance data to ensure that the data conforms to the expected logical pattern. Voice recognition and content review are performed on voice call records to ensure that there is no inappropriate content.

[0047] Further, the multimodal data detection module 13 starts working, using NLP technology to perform sentiment analysis and keyword extraction on text data; using video analysis technology to check the clarity and integrity of surveillance videos; using data analysis technology to check the statistical characteristics of network performance data; and using speech recognition and content filtering technology to check voice call records.

[0048] Finally, the multimodal data detection result generation module 14 will collect all the detection results and generate a multimodal data quality detection report, such as: the richness of customer service records reached 95%, which means that almost all service types are covered; the logical indicators of network performance data show that the network delay during high traffic periods is consistent with expectations, and the logical index is 98%; the file toxicity indicator of voice call records shows that no inappropriate content was found in 99% of call records.

[0049] Through the above embodiments, by creating multi-dimensional quality indicators such as richness, logic, and file toxicity, the quality of multimodal data can be comprehensively evaluated to ensure that the multimodal data meets the expected standards in many aspects, thereby improving the reliability of the data. In addition, by automatically creating quality inspection tasks and assigning them to corresponding inspection modules, the automation of data quality inspection is achieved. This reduces manual intervention, improves inspection efficiency, reduces the possibility of human error, and can also perform multimodal data quality inspection tasks in real time and generate corresponding multimodal data quality inspection reports, so that managers can quickly identify and handle data quality issues, thereby taking timely measures, and significantly improving the quality and management efficiency of multimodal data.

[0050] Optionally, the system further comprises: A multimodal metadata extraction module, used to analyze and process the multimodal data through an AI trusted framework, and extract metadata from the multimodal data to obtain multimodal metadata; A multimodal metadata graph construction module, used to construct a metadata knowledge graph based on the multimodal metadata; A data feature space construction module is used to extract multimodal data features from the multimodal data, and create a first data feature space of the multimodal data based on the multimodal data features. The first data feature space is used to be combined with the metadata knowledge graph to manage the multimodal data in the first data feature space through the combined metadata knowledge graph.

[0051] Specifically, in one embodiment, the multimodal data management system adds three optional modules on the basis of the original multiple modules: Multimodal metadata extraction module: This module can analyze and process multimodal data through the AI ​​trusted framework, extract metadata from it, and obtain multimodal metadata. Among them, the AI ​​trusted framework can extract descriptive metadata from digital content by comprehensively analyzing and processing various data forms, including but not limited to text, images, audio and video. The AI ​​trusted framework includes multiple technical fields, including computer vision, natural language processing (NLP), speech recognition and machine learning, forming a comprehensive analysis framework. Metadata is data about data, which can describe the characteristics, sources, creation time and other information of multimodal data.

[0052] Multimodal metadata graph construction module: This module constructs a metadata knowledge graph based on the metadata extracted from multimodal data. Among them, the knowledge graph is a structured semantic knowledge base that stores the relationship between entities in the form of a graph, making the association between multimodal data more intuitive and easy to understand. At the same time, the metadata knowledge graph can also realize the management of all metadata in one graph, support dynamic metadata relationship analysis, and can also perform large-scale graph computing: using GraphScope technology, support large-scale graph computing, and perform full-graph analysis including metadata events.

[0053] Data feature space construction module: This module is responsible for extracting data features from multimodal data and creating a first data feature space based on these data features. A first data feature space is created for each type of multimodal data, such as image data feature space, text data feature space, audio data feature space, video data feature space, etc., and the first data feature space is combined with the metadata knowledge graph, so that the multimodal data in the first data feature space can be effectively managed through the combined metadata knowledge graph. Among them, multimodal data features are extracted, the first data feature space is constructed, and it is organically combined with the metadata knowledge graph, which can support metadata extraction and fusion of multimodal data and break down the barriers between different data modalities.

[0054] Wherein, the multimodal metadata extraction module includes: The image metadata extraction unit is used to identify visual elements contained in the image data in the multimodal data through a computer vision model, and convert the visual elements into first metadata, wherein the first metadata is: searchable and annotable keywords and attributes in the visual elements. For example, computer vision technology is used to identify objects, scenes, and activities in images, extract visual elements such as color patterns, shapes, and textures, and convert them into searchable and annotable keywords and attributes.

[0055] A text metadata extraction unit is used to obtain sentence structure, language pattern and contextual meaning contained in the text data in the multimodal data through a natural language processing model, and extract second metadata from the sentence structure, language pattern and contextual meaning contained in the text data, wherein the second metadata includes: characters, places, events, concepts and emotional colors in the text data, for example, by parsing sentence structure, language pattern and contextual meaning through natural language processing technology, extracting key information about characters, places, events and concepts, and performing sentiment analysis to determine the emotional color of the text.

[0056] The audio and video metadata extraction unit is used to obtain third metadata in the audio and video data in the multimodal data through a speech recognition model and an audio and video analysis model, wherein the third metadata includes: background music, sound type, voice emotion, key timestamp and summary information contained in the audio and video data.

[0057] The multimodal metadata extraction process not only greatly improves the discoverability and accessibility of content, but also greatly enhances the richness and accuracy of information through cross-validation and fusion analysis of metadata. Combined with machine learning algorithms, the system can continuously learn and optimize the accuracy of metadata tags to support more complex data analysis and customer needs.

[0058] For example, for an intelligent medical system, it is necessary to process and analyze medical data from different sources and types, namely multimodal data, including patients' medical imaging materials (images), electronic health records (text), voice diagnosis records (audio), and surgical videos (video).

[0059] First, the intelligent medical system uses the AI ​​trusted framework to extract metadata for different types of medical data. For example, it uses the natural language processing model to extract key information such as patient information, symptoms, and diagnosis results from electronic health records as metadata. For medical imaging data, the system uses computer vision models to extract features in the image, such as the size and shape of the tumor, as metadata. Voice diagnosis records are converted into text through a voice recognition model, and key diagnostic information is extracted as metadata. Surgical videos use video analysis models to extract key surgical steps and timestamps as metadata.

[0060] Furthermore, the intelligent medical system constructs a metadata knowledge graph based on the extracted metadata. The metadata knowledge graph contains entities such as patients, symptoms, diagnoses, treatments, and the relationships between entities, such as "patient Zhang San suffers from high blood pressure."

[0061] Furthermore, the intelligent medical system needs to extract image features from medical imaging data, such as texture features of tumors, to create an image data feature space; extract text features from electronic health records, such as the frequency and correlation of symptoms, to create a text data feature space; extract audio features from voice diagnosis records, such as the pitch and intensity of voice, to create an audio data feature space; extract video features from surgical videos, such as the key steps and duration of the operation, to create a video data feature space, and combine these feature spaces with the metadata knowledge graph, so that the intelligent medical system can manage and retrieve multimodal data in various types of first data feature spaces through the graph.

[0062] Through the above embodiments, by constructing a metadata knowledge graph, multimodal data of different sources and types can be integrated together to form a unified data view, thereby improving the ability to integrate multimodal data. At the same time, the metadata knowledge graph provides an intuitive way to retrieve and manage multimodal data, making multimodal data retrieval more efficient. The construction of the first data feature space can perform a more in-depth analysis of multimodal data, explore potential connections and patterns between multimodal data, and enhance the depth and breadth of multimodal data analysis. Finally, by combining the metadata knowledge graph and the first data feature space, multimodal data can be more effectively managed and governed, thereby improving the level of automation and intelligence in multimodal data governance.

[0063] Optionally, the system further comprises: A multimodal metadata tag building module, used to convert the multimodal data features into multiple different types of tags through knowledge graph technology, and build the multiple different types of tags into a standardized tag system; A multimodal data labeling module, used to create a labeling task and label the multimodal data through the labeling system; A multimodal data label correction module is used to correct the labeling results of the multimodal data whose similarity is greater than the similarity threshold and whose labeling results are different among the labeled multimodal data; The multimodal data display module is used to visualize the annotated multimodal data through label portraits.

[0064] Specifically, in one embodiment, the multimodal data management system further adds four optional modules based on the original modules: Multimodal metadata label building module: This module uses knowledge graph technology to convert multimodal data features extracted from multimodal data into multiple different types of labels, and builds these labels into a standardized label system. This standardized label system helps to uniformly manage and retrieve multimodal data.

[0065] Multimodal data labeling module: This module is responsible for labeling tasks, which are used to label multimodal data by building a standardized labeling system. By labeling multimodal data with business, classification and other labels, multimodal data can be more easily identified and retrieved. The labeling of multimodal metadata can solve the problem of converting the data features of structured and unstructured multimodal data into a structured labeling system to facilitate user understanding and governance of multimodal data. It uses knowledge graph technology to develop, manage and classify labels.

[0066] By constructing label annotation tasks, it is possible to construct rules, modify, and delete business labels that need to be paid attention to in the asset field of multimodal data and labels such as reliability, security, and privacy that need to be paid attention to in the trusted field. At the same time, multimodal data annotation tasks can also be managed, including the creation, tracking, and updating of annotation tasks to ensure that the label production process is controllable.

[0067] Multimodal data label correction module: This module is used to check the annotated multimodal data, especially to correct the annotated results of multimodal data whose similarity is greater than the preset similarity threshold but different annotated results, so as to improve the accuracy and consistency of the annotation.

[0068] Multimodal data display module: This module is responsible for visualizing the labeled multimodal data through label portraits. This display method allows users to intuitively understand the labels and features of the data, facilitating further analysis and application.

[0069] For example, for an e-commerce platform system, the system needs to process and analyze product data from different sources and types, including product pictures (images), product descriptions (text), user reviews (audio), and product videos (video).

[0070] First, computer vision technology is used to extract visual features such as color and shape from product images and convert them into labels, such as "red" and "round". For product descriptions, the system uses natural language processing technology to extract keywords and convert them into labels, such as "durable" and "lightweight". User reviews are converted into text through voice recognition technology, and emotional features are extracted and converted into labels, such as "satisfied" and "dissatisfied". Product videos use video analysis technology to extract action features and convert them into labels, such as "easy to assemble" and "complex operation".

[0071] Furthermore, by creating labeling tasks and using a standardized labeling system to label all product data, for example, labeling a red round toy as "red" and "round".

[0072] Furthermore, when checking the labeling results, it is found that two similar objects, one is labeled as "durable" and the other is incorrectly labeled as "fragile". Since the similarity of the two objects exceeds the set similarity threshold, it will trigger automatic or manual label correction to ensure the consistency of the labeling of the two objects.

[0073] Finally, the labeled product data is visualized through label portraits. For example, in the product search results, users can see the label of each product, such as "red", "durable", etc., so that users can quickly understand the product characteristics and make choices.

[0074] Through the above-mentioned embodiments, the automated label construction and annotation module, the system can quickly annotate a large amount of multimodal data, improving the efficiency of data annotation. At the same time, the use of a standardized label system allows multimodal data to be more accurately retrieved and classified, improving the accuracy and efficiency of data retrieval. After annotation, the multimodal data label correction module ensures the consistency of annotations between similar multimodal data, reduces erroneous annotations, and improves the accuracy of the annotation results corresponding to the labels of the multimodal data. Finally, the multimodal data display module provides an intuitive display of the annotation results of multimodal data through label portraits, optimizing the user's retrieval and browsing experience.

[0075] Optionally, the system further comprises: A multimodal data asset management module is used to extract multimodal data features from the multimodal data, construct a second data feature space through a large language model, and combine it with the metadata knowledge graph to perform lineage query on the multimodal data and establish a multimodal data asset catalog through the combined metadata knowledge graph.

[0076] Specifically, in one embodiment, the multimodal data management system adds an optional module, a multimodal data asset management module, as follows: Multimodal data asset management module: This module is used to extract multimodal data features from multimodal data and construct a second data feature space through a large language model. The second data feature space is used to store and manage multimodal data. At the same time, the constructed second data feature space will be combined with the metadata knowledge graph, so that the entity and relationship information stored in the knowledge graph can be used to perform lineage query and asset cataloging on multimodal data. Lineage query refers to tracking the source and flow of multimodal data, and asset cataloging is to mark, classify and organize multimodal data assets to support data governance activities in complex business scenarios with multi-source heterogeneous multimodal data.

[0077] Among them, the full-link data lineage supports the visualization of the lineage graph of multimodal data, intuitively showing the relationship and impact path between multimodal data. It provides a data lineage query interface, allowing users to query the lineage relationship of specific data.

[0078] In addition, it is also possible to use large language model (LLM) technology to realize the intelligent generation of metadata for multimodal data, automatically extract key information from multimodal data, generate metadata tags and descriptions, and improve the efficiency and accuracy of data management.

[0079] Through the above-mentioned embodiments, by extracting multimodal data features from multimodal data and constructing a second data feature space, multimodal data can be managed and utilized more effectively, and the integration and availability of data can be improved. In addition, combined with the metadata knowledge graph, more comprehensive data background and related information can be provided to support intelligent decision-making based on data. Through the lineage query function, the source and flow of multimodal data can be quickly tracked, the traceability of multimodal data can be enhanced, and the governance and compliance management of multimodal data can be facilitated. Through the asset cataloging function, multimodal data assets can be clearly classified and organized, and the efficiency and accuracy of asset management of multimodal data can be improved.

[0080] Optionally, the system further comprises: A multimodal data standard formulation module, used to formulate a multimodal data standard corresponding to the multimodal data according to the scope and type of the multimodal data; A standard file configuration module, used to configure a corresponding standard file according to the multimodal data standard, wherein the standard file is used for a user terminal to upload and download the multimodal data standard; The multimodal data standard detection module is used to detect the multimodal data standard and update the multimodal data standard.

[0081] Specifically, in one embodiment, the multimodal data management system adds three optional modules on the basis of the original modules, which can apply the multimodal knowledge graph technology to realize the standard formulation, detection and analysis of multimodal data, and build a multi-level and scalable multimodal data standard system to ensure the availability, accuracy and consistency of multimodal data standards, as follows: Multimodal data standard formulation module: This module is used to formulate multimodal data standards corresponding to these data according to the scope and type of multimodal data. These standards can include data format, data structure, data quality requirements, etc., to ensure that different types of multimodal data can be managed and used consistently.

[0082] For example, determine the scope and type of multimodal data, including images, videos, text, audio, etc., and set unified standards and conventions for different types, including standards for data text, pictures, audio, and video.

[0083] Standard file configuration module: This module is responsible for configuring the corresponding standard files according to the established multimodal data standards. These standard files will be used for user terminals to upload and download multimodal data standards, ensuring that users can easily obtain and use the latest standard information of multimodal data standards. For example, standard files such as MPEG-7, DCMI, W3CRDF, Schema.org, etc. can be built-in according to user needs for relevant personnel to retrieve and learn, and support users to upload and download standards independently.

[0084] Multimodal data standard detection module: This module is used to detect the validity and applicability of multimodal data standards and regularly update these multimodal data standards, such as regularly detecting the validity of these standards. For example, with the development of technology, new data formats or new data quality requirements may emerge. The standard detection module will evaluate whether the existing multimodal data standards are still applicable and update them when necessary to ensure the timeliness and effectiveness of multimodal data standards. This ensures that multimodal data standards can adapt to changing technical and business needs, by detecting the use of multimodal standards, regularly evaluating and reviewing the standards, discovering and solving existing problems, and continuously optimizing and improving the standards.

[0085] Through the above embodiments, the formulation of multimodal data standards can provide unified management standards for different types of data to ensure data consistency and comparability. The configuration and provision of standard files enable users to easily obtain and use data standards, promote data upload and download, and improve the efficiency of data exchange. In addition, regular testing and updating of data standards ensures continuous control of data quality, allowing the system to adapt to new technologies and business requirements and maintain high quality of multimodal data.

[0086] Optionally, the multimodal data detection module is used to perform the multimodal data quality detection task to perform quality detection on the multimodal data, specifically: Performing quality detection of the richness index on the multimodal data through a data non-empty rate and a data non-repetitive rate, wherein the data non-empty rate is determined according to the number of non-empty files and the total number of files of the multimodal data, and the data non-repetitive rate is determined according to the number of non-repetitive files and the total number of files of the multimodal data; Performing logic verification on the multimodal data to achieve quality detection of the logic indicators; The content of the multimodal data is identified to achieve quality detection of the toxicity of the file.

[0087] Specifically, in one embodiment, the multimodal data detection module specifically performs multimodal data quality detection tasks, and can combine knowledge graph technology with a distributed computing engine, based on multimodal metadata and multimodal data standards, to formulate multimodal data quality detection indicators, and implement corresponding quality indicator detection algorithms for each data set and for specific data types. And build detection tasks based on detection indicators. Provide users with a comprehensive and efficient multimodal quality detection system to ensure that the quality of multimodal data meets specific standards and requirements. The overall functions of the system include multimodal data quality detection task management, multimodal data quality detection module, multimodal data quality indicator construction and evaluation, and multimodal data quality report.

[0088] It covers the quality detection of various multimodal data types such as images, videos, texts and audios. Each type of data covers multiple quality indicators to ensure comprehensive detection of multimodal data quality. It supports automated detection and manual intervention modes, has a high degree of automation and flexibility, and can adapt to the needs of different fields and scenarios. The generated multimodal data quality report supports visual display, and intuitively displays the quality of the data in the form of charts, statistical data, etc., which is convenient for users to understand and analyze.

[0089] Specifically include the following aspects: The quality detection of richness indicators refers to evaluating the richness indicators of multimodal data through data non-emptiness rate, data non-repetition rate, data richness, etc. Before this process, multimodal data needs to be divided into multiple data sets according to different types.

[0090] Among them, multimodal data used for large model training requires rich data resources. In addition to different types of data, higher requirements are placed on the richness of each type of data. The richness rule detection is the superposition of data non-emptiness rate and non-repetition rate. The data non-emptiness rate is determined based on the number of non-empty files and the total number of files in the multimodal data. Check how many proportions of files in the multimodal data in the data set are non-empty to ensure the integrity of the data set. The data non-repetition rate is determined based on the number of non-repetitive files and the total number of files in the multimodal data, which helps to evaluate how many proportions of files in the data set are unique, thereby measuring the diversity and coverage of the data.

[0091] For example, data non-empty rate = number of non-empty files / total number of files; Data non-repetitive rate = number of non-repetitive files / total number of files; Data enrichment rate = (number of file types / total number of types)*0.3+(number of non-empty files / total number of files)*0.35+(number of non-duplicate files / total number of files)*0.35.

[0092] The quality inspection of logical indicators refers to the logical verification of multimodal data to achieve the quality inspection of logical indicators. It can analyze industry expertise, data characteristics, data standards, etc. through intelligent algorithms, and perform logical verification on data files to meet general data requirements. Logical verification may include checking whether the relationship between data is reasonable and whether it conforms to the expected business logic. For example, in an order system, the order date should be before the date of shipment of the goods.

[0093] Quality detection of file toxicity refers to identifying the content of multimodal data to achieve quality detection of file toxicity. This may involve identifying and filtering out inappropriate content, such as sensitive content such as violence or hate speech. By identifying files involving drugs, pornography, and violence, avoiding data poisoning and complying with data ethics rules, the compliance and security of multimodal data in the dataset can be ensured.

[0094] Through the above-mentioned embodiments, the non-empty rate and non-duplication rate of multimodal data are detected, which can ensure the integrity and diversity of the multimodal data in the data set, avoid the data set being too single or containing a large amount of redundant data, and at the same time, logical verification helps to discover and correct errors and inconsistencies in the data, improve the accuracy and reliability of the data. In addition, file toxicity detection helps to identify and filter out inappropriate content, protect users from harmful information, and ensure that the platform content complies with legal and regulatory requirements, ultimately improving the data quality of multimodal data, thereby achieving more efficient and accurate data governance.

[0095] Figure 3 is a flowchart of a multimodal data management method provided by an embodiment of the present application, such as Figure 3 As shown, based on the same inventive concept, an embodiment of the present application further provides a multimodal data management method, the method comprising: Step S101, creating at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator comprises: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; Step S102, creating a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator; Step S103, executing the multimodal data quality detection task to perform quality detection on the multimodal data; Step S104: Generate and display a multimodal data quality detection report based on the execution result of the multimodal data quality detection task, wherein the multimodal data quality detection report includes: the overall quality status of the multimodal data and the detection results of various multimodal data quality indicators.

[0096] Optionally, the method further comprises: By using an AI trusted framework, the multimodal data is analyzed and processed, and metadata in the multimodal data is extracted to obtain multimodal metadata; Constructing a metadata knowledge graph based on the multimodal metadata; Multimodal data features are extracted from the multimodal data, and based on the multimodal data features, a first data feature space of the multimodal data is created, wherein the first data feature space is used to be combined with the metadata knowledge graph so as to manage the multimodal data in the first data feature space through the combined metadata knowledge graph.

[0097] Optionally, the analyzing and processing the multimodal data through the AI ​​trusted framework, and extracting metadata from the multimodal data to obtain multimodal metadata, includes: Identifying visual elements contained in the image data in the multimodal data through a computer vision model, and converting the visual elements into first metadata, wherein the first metadata is: searchable and annotable keywords and attributes in the visual elements; Acquiring sentence structures, language patterns, and contextual meanings contained in text data in the multimodal data through a natural language processing model, and extracting second metadata from the sentence structures, language patterns, and contextual meanings contained in the text data, wherein the second metadata includes: characters, places, events, concepts, and emotional colors in the text data; The third metadata in the audio and video data in the multimodal data is obtained through the speech recognition model and the audio and video analysis model, and the third metadata includes: background music, sound type, voice emotion, key timestamp and summary information contained in the audio and video data.

[0098] Optionally, the method further comprises: By using knowledge graph technology, the multimodal data features are converted into multiple different types of tags, and the multiple different types of tags are constructed into a standardized tag system; Creating a labeling task, and labeling the multimodal data using the labeling system; Correcting the labeling results of the multimodal data whose similarity among the labeled multimodal data is greater than the similarity threshold and whose labeling results are different; The labeled multimodal data is visualized through label profiling.

[0099] Optionally, the method further comprises: Multimodal data features are extracted from the multimodal data, and a second data feature space is constructed through a large language model, and combined with the metadata knowledge graph to perform lineage query on the multimodal data and establish a multimodal data asset catalog through the combined metadata knowledge graph.

[0100] Optionally, the method further comprises: Formulate a multimodal data standard corresponding to the multimodal data according to the scope and type of the multimodal data; A standard file corresponding to the multimodal data standard is configured according to the multimodal data standard, wherein the standard file is used for uploading and downloading the multimodal data standard by a user terminal; The multimodal data standard is detected and updated.

[0101] Optionally, the executing the multimodal data quality detection task to perform quality detection on the multimodal data includes: Performing quality detection of the richness index on the multimodal data through a data non-empty rate and a data non-repetitive rate, wherein the data non-empty rate is determined according to the number of non-empty files and the total number of files of the multimodal data, and the data non-repetitive rate is determined according to the number of non-repetitive files and the total number of files of the multimodal data; Performing logic verification on the multimodal data to achieve quality detection of the logic indicators; The content of the multimodal data is identified to achieve quality detection of the toxicity of the file.

[0102] The purpose of this application is to build a multimodal data governance system, introduce AI technologies such as NLP, dynamic knowledge graphs, and intelligent interactive analysis into data governance, and achieve a high degree of integration between AI and data governance. We strive to build a structured and unstructured integrated data governance system, carry out governance work such as quality management and asset management based on AI, and realize the automation and intelligence of multimodal data governance processes. We provide metadata processing and analysis capabilities for unstructured data, and combine knowledge graph data to realize intelligent governance of structured data, gradually reduce or replace manual combing and rule-making, and improve governance efficiency and quality.

[0103] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the multimodal data management method as described in any of the above embodiments.

[0104] Among them, electronic equipment refers to Figure 4 , Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be run on the processor 520 to implement the steps in the space recovery method disclosed in the above embodiment of the present application.

[0105] Based on the same inventive concept, another embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor to perform the multimodal data management method as described in any of the above embodiments.

[0106] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the multimodal data management method as described in any of the above embodiments is implemented.

[0107] As for the method, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0108] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0109] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0113] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0115] The above is a detailed introduction to a multimodal data management system, method, device and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal data management system, characterized in that: The system comprises: A multimodal data quality indicator creation module, used to create at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator includes: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; A multimodal data quality detection task creation module, configured to create a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator, and assign the multimodal data quality detection task to a multimodal data detection module; A multimodal data detection module, used to execute the multimodal data quality detection task to perform quality detection on the multimodal data; The multimodal data detection result generation module is used to generate and display a multimodal data quality detection report according to the execution results of the multimodal data quality detection task. The multimodal data quality detection report includes: the overall quality of the multimodal data and the detection results of various multimodal data quality indicators.

2. The multimodal data management system according to claim 1, characterized in that: The system further comprises: A multimodal metadata extraction module, used to analyze and process the multimodal data through an AI trusted framework, and extract metadata from the multimodal data to obtain multimodal metadata; A multimodal metadata graph construction module, used to construct a metadata knowledge graph based on the multimodal metadata; A data feature space construction module is used to extract multimodal data features from the multimodal data, and create a first data feature space of the multimodal data based on the multimodal data features. The first data feature space is used to be combined with the metadata knowledge graph to manage the multimodal data in the first data feature space through the combined metadata knowledge graph.

3. The multimodal data management system according to claim 2, characterized in that: The multimodal metadata extraction module comprises: An image metadata extraction unit, configured to identify visual elements contained in the image data in the multimodal data through a computer vision model, and convert the visual elements into first metadata, wherein the first metadata is: searchable and annotable keywords and attributes in the visual elements; a text metadata extraction unit, configured to obtain sentence structures, language patterns, and contextual meanings contained in the text data in the multimodal data through a natural language processing model, and extract second metadata from the sentence structures, language patterns, and contextual meanings contained in the text data, wherein the second metadata includes: characters, places, events, concepts, and emotional colors in the text data; The audio and video metadata extraction unit is used to obtain third metadata in the audio and video data in the multimodal data through a speech recognition model and an audio and video analysis model, wherein the third metadata includes: background music, sound type, voice emotion, key timestamp and summary information contained in the audio and video data.

4. The multimodal data management system according to claim 2, characterized in that: The system further comprises: A multimodal metadata tag building module, used to convert the multimodal data features into multiple different types of tags through knowledge graph technology, and build the multiple different types of tags into a standardized tag system; A multimodal data labeling module, used to create a labeling task and label the multimodal data through the labeling system; A multimodal data label correction module is used to correct the labeling results of the multimodal data whose similarity is greater than the similarity threshold and whose labeling results are different among the labeled multimodal data; The multimodal data display module is used to visualize the annotated multimodal data through label portraits.

5. The multimodal data management system according to claim 2, characterized in that: The system further comprises: A multimodal data asset management module is used to extract multimodal data features from the multimodal data, construct a second data feature space through a large language model, and combine it with the metadata knowledge graph to perform lineage query on the multimodal data and establish a multimodal data asset catalog through the combined metadata knowledge graph.

6. The multimodal data management system according to claim 1, characterized in that: The system further comprises: A multimodal data standard formulation module, used to formulate a multimodal data standard corresponding to the multimodal data according to the scope and type of the multimodal data; A standard file configuration module, used to configure a corresponding standard file according to the multimodal data standard, wherein the standard file is used for a user terminal to upload and download the multimodal data standard; The multimodal data standard detection module is used to detect the multimodal data standard and update the multimodal data standard.

7. The multimodal data management system according to claim 1, characterized in that: The multimodal data detection module is used to perform the multimodal data quality detection task to perform quality detection on the multimodal data, specifically: Performing quality detection of the richness index on the multimodal data through a data non-empty rate and a data non-repetitive rate, wherein the data non-empty rate is determined according to the number of non-empty files and the total number of files of the multimodal data, and the data non-repetitive rate is determined according to the number of non-repetitive files and the total number of files of the multimodal data; Performing logic verification on the multimodal data to achieve quality detection of the logic indicators; The content of the multimodal data is identified to achieve quality detection of the toxicity of the file.

8. A multimodal data management method, characterized in that: The method comprises: Creating at least one multimodal data quality indicator for the multimodal data, wherein the at least one multimodal data quality indicator comprises: at least one of a richness indicator, a logic indicator, and a file toxicity indicator; Creating a multimodal data quality detection task for the multimodal data according to the at least one multimodal data quality indicator; Executing the multimodal data quality detection task to perform quality detection on the multimodal data; According to the execution result of the multimodal data quality detection task, a multimodal data quality detection report is generated and displayed, and the multimodal data quality detection report includes: the overall quality status of the multimodal data and the detection results of various multimodal data quality indicators.

9. An electronic device, characterized in that: The system comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the multimodal data management method according to claim 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the multimodal data management method as claimed in claim 8 is implemented.

Images