Multi-language data management method

Abstract

The invention relates to a multi-language data management method which comprises the following steps: S1, performing language detection on collected data samples through a fine-tuned InternVL multi-modal large model, automatically generating classification tags, and storing different types of data in corresponding databases; s2, metadata information is recorded for each piece of collected data; s3, a corresponding processing flow is automatically configured, a character set corresponding to the classification label is selected to carry out data processing on the collected data, the processed data is put on a shelf, and the put-on-shelf data supports sharing; s4, updating the racked data in real time; s5, according to the authority level and language preference of the user, displaying the data on the shelf in the step S3; each of the steps corresponds to one module, and the modules realize multi-language intercommunication through related codes and a multi-language support library. Compared with the prior art, the method has the advantages that data interoperation under the multi-language environment is achieved, and the multi-language processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a multi-language data management method. Background Art

[0002] With the rapid development of artificial intelligence technology, large model technologies (such as large language models and large multimodal models) have made breakthroughs in their respective fields. These models are pre-trained on massive amounts of text, images, videos and other data, thus demonstrating powerful multimodal understanding and generation capabilities that go beyond language. However, the success of these models depends not only on their complex model architecture, but more importantly on the high-quality, large-scale datasets they rely on.

[0003] At present, how to effectively input various high-quality data sets into the model in an appropriate manner has become the core technology and key to differentiated competition for each model manufacturer. These input strategies are often not public, which makes it difficult for researchers to understand their logic and principles, and they face the confusion of choosing appropriate strategies in practice. Even with a large amount of high-quality data, inappropriate data input may still cause problems such as bias, prejudice, hallucination or overfitting in the model. Therefore, it is particularly important to develop an efficient data management method and system.

[0004] However, data management currently has problems with single language support or insufficient support for multiple languages, which cannot meet the data processing requirements in a multilingual environment. It cannot meet the needs of data retrieval, classification and processing in a multilingual environment. At the same time, the existing data management system lacks fine-grained management of different language characteristics, resulting in low efficiency in multilingual data fusion and analysis, and prone to data redundancy and consistency problems. Summary of the invention

[0005] The purpose of the present invention is to provide a multilingual data management method in order to achieve data interoperability in a multilingual environment and improve multilingual processing efficiency.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A multilingual data management method, the method comprising the following steps:

[0008] S1. Collect data according to the data collection objectives, and classify them according to data types and language types. Perform language detection on the collected data samples through the fine-tuned InternVL multimodal large model, automatically generate classification labels for language, language family, region, and dialect, store different types of data in the corresponding database, and associate specific service objects with each data to meet the needs of different application scenarios.

[0009] S2. storing metadata corresponding to the collected data, and recording metadata information for each piece of collected data;

[0010] S3. According to the language classification label, the corresponding processing flow is automatically configured, the character set corresponding to the classification label is selected to process the collected data, and the corresponding quality inspection pipeline is automatically triggered to score and rate the processed data, and the processed data is put on the shelves, and the put on the shelves supports sharing;

[0011] S4. Establish a data update mechanism to monitor changes in data sources and update the data that has been put on the shelves in real time;

[0012] S5, set up a permission control mechanism based on language preference recommendation, and display the data listed in S3 according to the user's permission level and language preference;

[0013] Each of the above steps corresponds to a module, and the module realizes multi-language intercommunication through relevant codes and multi-language support libraries.

[0014] Furthermore, the specific steps of storing different types of data in corresponding databases are:

[0015] Relational databases are used to store structured data, and object storage is used to store unstructured data.

[0016] Furthermore, storing different types of data in corresponding databases also includes storing frequently accessed and modified traditional files and directory structures using a file system storage layer.

[0017] Furthermore, storing different types of data in corresponding databases also includes using Elasticsearch+Kudu database to store real-time data.

[0018] Furthermore, the corresponding processing flow is marked with the person responsible for each flow node.

[0019] Furthermore, the data processing is performed in batches, and computing resources are allocated based on the priority of the batches during the data processing, with computing resources preferentially allocated to data with higher priority.

[0020] Furthermore, the listed data supports sharing specifically:

[0021] The listed data is interconnected with the open data platform Opendatalab, and supports CLI and SDK download access.

[0022] Furthermore, the listed data is also displayed based on the sensitivity category of the data itself.

[0023] Furthermore, the data's own sensitivity categories are divided into four categories: public data, internal use data, core sensitive data, and most sensitive data.

[0024] Furthermore, the metadata information includes various automatically determined tags, as well as encoding format, source, collection time and data quality indicators.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] First, the present invention realizes the refined metadata management of multilingual data. The present invention first proposes an automatic language tag generation method based on the fine-tuned InternVL multimodal large model based on the automatic generation of metadata of the multimodal large model, which can automatically distinguish data with different language characteristics and classify them for management. The present invention establishes a multilingual tag system: a unified multilingual metadata tag system is established to facilitate cross-language data retrieval and association analysis. The present invention improves the dynamic adaptability of data processing, and metadata management can be dynamically adjusted according to language characteristics, thereby improving the flexibility and accuracy of multilingual data processing.

[0027] Then, the present invention also realizes multi-language intercommunication through relevant coding and multi-language support library. Through unified coding and multi-language support library, it ensures that users of different languages ​​can access and understand data, realize data interoperability in multi-language environment, and meet the data processing requirements in multi-language environment. Existing coding standards and support libraries are often only for specific languages, lack a unified multi-language processing mechanism, resulting in complex cross-language data exchange and poor compatibility. Data conversion and interoperation between different languages ​​require a lot of customized development, which is costly and error-prone. Therefore, the present invention unifies the coding scheme: designs a unified coding standard compatible with multiple languages ​​to solve the problem of coding differences between different languages, and adopts a multi-language support library at the same time: develops a universal multi-language support library, provides a standardized interface, and realizes seamless intercommunication of data in different languages. The present invention realizes cross-language data processing, supports data reading, writing, conversion and analysis in a multi-language environment, and improves the multi-language processing capability and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a data management flow chart of the present invention;

[0029] Figure 2 It is a data classification schematic diagram of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0031] The present invention provides a multilingual data management method, which comprises the following steps:

[0032] S1. Collect data according to the data collection objectives, and classify them according to data types and language types. Perform language detection on the collected data samples through the fine-tuned InternVL multimodal large model, automatically generate classification labels for language, language family, region, and dialect, store different types of data in the corresponding database, and associate specific service objects with each data to meet the needs of different application scenarios.

[0033] S2. storing metadata corresponding to the collected data, and recording metadata information for each piece of collected data;

[0034] S3. According to the language classification label, the corresponding processing flow is automatically configured, the character set corresponding to the classification label is selected to process the collected data, and the corresponding quality inspection pipeline is automatically triggered to score and rate the processed data, and the processed data is put on the shelves, and the put on the shelves supports sharing;

[0035] S4. Establish a data update mechanism to monitor changes in data sources and update the data that has been put on the shelves in real time;

[0036] S5, set up a permission control mechanism based on language preference recommendation, and display the data listed in S3 according to the user's permission level and language preference;

[0037] Each of the above steps corresponds to a module, and the module realizes multi-language intercommunication through relevant codes and multi-language support libraries.

[0038] In view of the shortcomings of the prior art, the present invention aims to provide a more efficient, comprehensive and flexible data management solution through the following improvements:

[0039] Multi-language support: This invention focuses on multi-language data management and proposes a unified multi-language data set configuration and management method, which can efficiently process and manage data sets in different languages ​​and meet the multi-language data processing needs in a global environment.

[0040] Modular management: The present invention includes four core modules: data management, batch management, process management and data sharing, providing comprehensive functional support. This modular design can flexibly respond to complex data management requirements and improve the functionality and scalability of the system.

[0041] Data security and privacy protection: This invention introduces access control and authority management mechanisms to achieve different access rights allocation for data from different user perspectives. Through fine-grained authority management, customized data security and privacy protection are ensured, improving the security and flexibility of the system.

[0042] Data storage and scalability: The present invention implements a continuous large-scale data accumulation plan, and ensures the efficiency and availability of data through data classification and detailed metadata system. This not only meets the long-term and highly competitive data storage needs, but also ensures the scalability and long-term stable operation of the system.

[0043] Through the above improvements, the present invention significantly improves the performance and scope of application of the existing technology in terms of multilingual data management, modular function support, data security and privacy protection, data storage and scalability, and provides users with a more efficient, comprehensive and flexible data management solution.

[0044] The technical solution of the present invention mainly includes three parts:

[0045] 1. Establish a data management mechanism: systematically sort out data sources, data pools, management requirements and application goals, build a full life cycle management process from data planning, collection, processing to release and use, clarify the operating specifications and data management standards for each process link, and ensure the systematic and standardized data management;

[0046] 2. Build a data management platform: With data management, batch management, process management, and data sharing as the core, the system supports the data management process to ensure that the data management plan can be effectively implemented;

[0047] 3. Efficiently manage data reserves: implement a continuous, large-scale data accumulation plan to meet long-term and highly competitive reserve needs; ensure the efficiency and availability of data storage through data classification and detailed metadata systems.

[0048] First, the present invention systematically sorts out data sources, data pools, management requirements and application goals. Data sources involve multiple dimensions, and high-quality data is obtained through legal and compliant channels, including internal collection by the team, outsourced collection, data procurement, supply by other teams in the company, contribution of corpus alliance, download of public data sets, cooperation with other organizations, outsourced annotation, internal annotation, etc. In order to make full use of the advantages of different storage methods and provide the best solution for different types of data, the present invention adopts a variety of storage methods. For large-scale unstructured data (such as media files and data backup), object storage is a better choice; the file system storage layer is suitable for traditional files and directory structures that need to be frequently accessed and modified. For large-scale text data that requires fast search and real-time analysis, the present invention uses Elasticsearch+Kudu for real-time data analysis with fast insertion and efficient query. In addition, the present invention uses relational databases to process structured data and transaction processing applications, and uses non-relational databases for applications with high scalability and flexible data models. By comprehensively using these storage technologies, the present invention achieves efficient and reliable data management and storage in different application scenarios. In terms of data pool management, the present invention has established a variety of management mechanisms based on actual needs, including data classification at various granularities, detailed descriptive metadata, macro visualization analysis, data security and operational configuration, etc. This data management mechanism mainly serves large model applications in various vertical fields, including large models such as text to image, video, theoretical science, and three-dimensional scenes.

[0049] After clarifying the source, path, and service object of the data management mechanism, the present invention further constructs a full life cycle management process from data planning, collection, processing to release and use, and clarifies the operating specifications and data management standards of each process link. For example, during the warehousing process of the data asset platform, the present invention strictly assigns the person in charge of each process node, clarifies the field information that needs to be updated or retained at each node, and distinguishes the security authority level by individual. The data management mechanism of standardization, normalization, and process-based management ensures the efficiency and reliability of data management.

[0050] Based on the established data management mechanism, the present invention establishes a comprehensive data management platform with four modules of data management, batch management, process management and data sharing as the core. The system supports the data management process and ensures that the data management plan can be effectively implemented.

[0051] The data management module is responsible for the storage, processing and maintenance of data to ensure the integrity and consistency of data. The module supports the storage of multiple data formats, including structured data and unstructured data, as well as data in various languages. Through technologies such as relational database, object storage and file system storage, the high availability and accessibility of data are guaranteed. In addition, the module supports highly flexible and customizable visual analysis, and through data encryption and access control mechanisms, the module can ensure the security and privacy of data, and present different application perspectives to users according to different permission levels. The data management flow chart is as follows: Figure 1 shown.

[0052] Manage batch processing of data to ensure efficient and accurate data processing. This module can define and manage batches of data processing tasks, schedule and monitor the execution of each batch. Through the batch management module, users can set processing priorities, consider allocating computing resources, and monitor processing progress and results in real time.

[0053] Manage all aspects of data flow to ensure smooth and automated data flow. This module uses a workflow engine to define and execute data processing processes, including data collection, cleaning, conversion, storage, publishing and sharing. Through the process management module, users can customize data processing processes, view the execution of each process in real time, analyze and troubleshoot potential problems, making the entire processing process traceable and analyzable, greatly improving data processing efficiency and accuracy.

[0054] Provides data sharing and access functions to ensure efficient use of data and collaboration between users. The module is interconnected with the open data platform Opendatalab, supports CLI and SDK download access, and meets the needs of different users and applications. At the same time, the module introduces access control and permission management mechanisms to ensure that users can easily access and use data resources while ensuring data security.

[0055] All the above steps can be modularized, support internationalization and localization, and realize seamless sharing and collaborative work in multi-language links through technologies such as relevant coding and multi-language support libraries. In summary, through the organic combination of the four modules of data management, batch management, process management and data sharing, the data management platform can provide efficient and reliable data management services in a multi-language environment, support complex data processing requirements, and ensure the security, integrity and consistency of data.

[0056] Ensure the efficiency and availability of data storage through data classification and detailed metadata system.

[0057] According to the importance of data and its openness and shareability, data is managed in a hierarchical manner to ensure efficient and safe use of data. Specifically, data is divided into four categories: public data, internal use data, core sensitive data, and most sensitive data. Figure 2 shown.

[0058] The present invention establishes a detailed metadata system to record the source, processing process and usage of data, ensuring the traceability of data and the efficiency of management. The metadata system includes descriptive metadata, processing metadata and knowledge metadata. Descriptive metadata records the basic attributes of data, such as data modality, format, subject, etc.; processing metadata describes data quality information, processing process records and retention rate, etc.; knowledge metadata describes the relevant attributes of each data batch classified by content, such as book knowledge base, paper knowledge base, test question knowledge base, etc. In addition, by realizing the automatic collection and update mechanism of metadata, the present invention ensures the real-time and accuracy of metadata.

[0059] In summary, through the above technical means, the present invention realizes efficient management of data reserves, which not only meets the multilingual data management system's demand for large-scale data reserves, but also improves data utilization efficiency and management efficiency. Combined with advanced data collection, hierarchical management and metadata management technologies, the scalability and maintainability of the system are ensured, laying a solid foundation for the long-term development of the system.

[0060] The present invention solves many problems existing in the original data management, including:

[0061] Difficulty in language classification: It is difficult to classify large-scale, multimodal, and diverse data. Manual judgment or labeling is costly and prone to subjective errors.

[0062] Difficulty in data selection: It is difficult to find data and determine which data is suitable for use, especially when it comes to the management and selection of multilingual data. Data sets in different languages ​​have different characteristics, and a fine-grained labeling system is needed for data management and selection.

[0063] Data security risks: Data comes from various sources. During the circulation process through different channels (such as purchasing, downloading, and collection), data may be illegally obtained and used due to information leakage; data classification management is manually controlled, with multiple storage levels and confusing classification;

[0064] Lack of data standards: Lack of standard data naming and expression makes it difficult for data from different systems to communicate and exchange;

[0065] Difficulty in data perspective: It is difficult to fully grasp the overall storage status of data, which makes it difficult to organize and use data and plan the next step;

[0066] Difficulty in work evaluation: It is difficult to evaluate the contribution of data preparation work to the overall large model business and assist in further business decision-making.

[0067] The present invention is aimed at different users (data users, data producers, data managers), and meets the needs of users in different dimensions:

[0068] Data users: The present invention enables data users to quickly find target data by providing asset classification and powerful search functions, and confirm the availability of data through complete metadata information. This not only solves the problem of data users not knowing the location of data, but also ensures that they can easily browse and confirm the validity of the required data.

[0069] Data producers: The present invention provides a convenient metadata maintenance function, supports batch operations, and provides comprehensive data profile analysis tools to facilitate the maintenance, modification, and auditing of data assets. At the same time, the present invention also has a data governance function, which can offline useless data, archive cold data, and support data tracing, helping to quickly troubleshoot data quality issues and ensure the real-time availability and compliance of data.

[0070] Data managers: This invention provides data managers with comprehensive asset health evaluation indicators, such as model coverage, reuse, metadata completeness and data quality, to help managers evaluate the construction of data assets. In addition, by quantifying the quantity and quality of work of data developers, managers can more effectively evaluate the workload of data personnel and achieve the goal of reducing costs and increasing efficiency through platform health management and control.

[0071] The present invention solves the problem of unified configuration of multilingual data sets, constructs a unified descriptive language for multimodal minority language data, and realizes the mutual integration and information complementarity of minority language multimodal data.

[0072] The present invention has the following advantages:

[0073] 1. Multi-language support: The present invention can effectively solve the problem of unified configuration of multi-language data sets, and realize the standardized description and storage of data in different languages, modalities and formats. This multi-language support is particularly important in the current context of global data demand.

[0074] 2. Comprehensive modular management: The present invention realizes full process coverage and efficient collaboration of data management through four core modules: data management, batch management, process management and data sharing. The modular design makes the system more flexible and can be adjusted and expanded according to actual needs.

[0075] 3. Advanced data security and privacy protection: This invention introduces strict access control and authority management mechanisms to ensure data security and privacy protection in a multi-language, multi-channel environment. This is particularly critical for industries with high data security requirements.

[0076] 4. Efficient data storage and management: Through a detailed metadata system and a sustainable large-scale data accumulation plan, the present invention enables researchers in related fields to flexibly design experiments and adjust input strategies to optimize the model training and fine-tuning process. On the other hand, it enables the system to meet long-term data needs and support operators and managers to conduct efficient retrieval and management.

[0077] The present invention has completed the storage management and language classification of thousands of multilingual data sets, totaling more than 2,000 batches, with data size reaching PB level, and the total amount of data processed exceeding 10 billion. The types of data managed are diverse, covering multimodal data (interleaved), three-dimensional data, audio, video, images, and text, etc.; the supported languages ​​include Chinese, English, Russian, French, Vietnamese and dozens of other languages. In addition, the data managed by the present invention is relatively healthy, and the metadata completeness rate reaches 90% coverage.

[0078] In summary, the practical application of the present invention has proved its efficiency and reliability in multilingual and multimodal data management. Its wide language coverage and data type support fully demonstrate the applicability and excellent performance of the present invention in actual operation, and successfully cope with the complex challenges of large-scale data management.

[0079] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multilingual data management method, characterized in that: The method comprises the following steps: S1. Collect data according to the data collection objectives, and classify them according to data types and language types. Perform language detection on the collected data samples through the fine-tuned InternVL multimodal large model, automatically generate classification labels for language, language family, region, and dialect, store different types of data in the corresponding database, and associate specific service objects with each data to meet the needs of different application scenarios. S2. storing metadata corresponding to the collected data, and recording metadata information for each piece of collected data; S3. According to the language classification label, the corresponding processing flow is automatically configured, the character set corresponding to the classification label is selected to process the collected data, and the corresponding quality inspection pipeline is automatically triggered to score and rate the processed data, and the processed data is put on the shelves, and the put on the shelves supports sharing; S4. Establish a data update mechanism to monitor changes in data sources and update the data that has been put on the shelves in real time; S5, set up a permission control mechanism based on language preference recommendation, and display the data listed in S3 according to the user's permission level and language preference; Each of the above steps corresponds to a module, and the module realizes multi-language intercommunication through relevant codes and multi-language support libraries.

2. A multilingual data management method according to claim 1, characterized in that: The specific steps of storing different types of data in corresponding databases are: Relational databases are used to store structured data, and object storage is used to store unstructured data.

3. A multilingual data management method according to claim 2, characterized in that: Storing different types of data in corresponding databases also includes using a file system storage layer to store frequently accessed and modified traditional files and directory structures.

4. A multilingual data management method according to claim 2, characterized in that: Storing different types of data in corresponding databases also includes using Elasticsearch+Kudu database to store real-time data.

5. A multilingual data management method according to claim 1, characterized in that: The corresponding processing flow is marked with the person responsible for each process node.

6. A multilingual data management method according to claim 1, characterized in that: The data processing is performed in batches, and computing resources are allocated based on the priority of the batches during the data processing, with computing resources preferentially allocated to data with higher priority.

7. A multilingual data management method according to claim 1, characterized in that: The listed data supports sharing specifically: The listed data is interconnected with the open data platform Opendatalab, and supports CLI and SDK download access.

8. A multilingual data management method according to claim 1, characterized in that: The listed data is also displayed based on the sensitivity category of the data itself.

9. A multilingual data management method according to claim 8, characterized in that: The data’s own sensitivity categories are divided into four categories: public data, internal use data, core sensitive data, and most sensitive data.

10. A multilingual data management method according to claim 1, characterized in that: The metadata information includes various automatically determined tags, as well as encoding format, source, collection time and data quality indicators.

Images