Large-scale sample storage management method and system based on multi-dimensional label system

Abstract

The invention discloses a large-scale sample storage management method and system based on a multi-dimensional label system. The method comprises the following steps: acquiring large-scale sample data input, and establishing a multi-dimensional label system for each piece of sample data based on a sample source, sample quality and sample feature dimensions; performing automatic auxiliary identification and classification and quality evaluation on the sample data, generating a multi-dimensional label for the sample according to a multi-dimensional label system, selecting a proper storage layer according to the multi-dimensional label of the sample, and storing the sample in a specified storage layer after the storage space is optimized; and returning a retrieval result according to a retrieval condition input by a user, and distributing the retrieved sample to a required task or analysis scene according to a requirement. According to the method, by establishing the multi-dimensional metadata labels, retrieval, positioning and management of the samples can be completed quickly and efficiently. Meanwhile, automatic auxiliary classification and recognition of the samples are achieved, and the data management efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and specifically to a large-scale sample storage management method and system based on a multidimensional label system, which is mainly used for the storage, retrieval and management of large-scale data samples, and serves the data needs of machine learning and artificial intelligence in various production fields. Background Art

[0002] With the rapid development of artificial intelligence and machine learning, the number and scale of data samples are growing exponentially. The effective storage and management of large-scale data samples is the key to ensuring the efficiency and accuracy of model training. Traditional storage systems are difficult to adapt to the retrieval and classification needs of these large-scale data, especially in terms of multi-dimensional metadata management.

[0003] Currently, many storage systems lack fine-grained label management mechanisms for samples, which means that researchers and engineers often need to spend a lot of time locating and screening samples when using them. In addition, existing classification methods mainly rely on manual intervention, which is not only inefficient but also prone to misclassification, increasing the risk and cost of model training. Summary of the invention

[0004] Purpose of the invention: The present invention provides a large-scale sample storage management method and system based on a multi-dimensional label system. By establishing multi-dimensional metadata labels, sample retrieval, positioning and management can be completed quickly and efficiently.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, a large-scale sample storage management method based on a multi-dimensional label system comprises the following steps:

[0007] Obtain large-scale sample data input, and establish a multi-dimensional labeling system for each sample data based on sample source, sample quality, and sample feature dimensions;

[0008] Automatically assist in the identification and classification of sample data. For text samples, natural language processing algorithms are used to extract key features of the text and classify the text into specified categories. For image / video samples, computer vision technology is used to identify the main content or target objects in the image or video and classify them into the specified corresponding categories.

[0009] Based on the classified samples, the quality assessment of the samples is automatically performed using predefined quality standards, and a quality report is generated based on the assessment results. The quality assessment dimensions include completeness, accuracy, and consistency.

[0010] For samples that have undergone quality assessment, check the storage requirements of the samples, generate multidimensional labels for the samples according to the multidimensional label system, select a suitable storage layer based on the multidimensional labels of the samples, and store the samples in the specified storage layer after optimizing the storage space;

[0011] Obtain the search conditions entered by the user, find samples that meet the conditions based on the multi-dimensional label system and feature configuration of the samples, return the search results, and distribute the retrieved samples to the required tasks or analysis scenarios as needed.

[0012] Furthermore, the sample source dimension includes the data source; the sample quality dimension includes the accuracy, completeness, and consistency of the data; the sample feature dimension includes the data type, sample structure, and feature attributes. The options and / or ranges and thresholds related to these dimensions are configured through the interface tools provided by the system.

[0013] Furthermore, computer vision technology identifies the main content or target objects in an image or video, including:

[0014] For an image sample, classify the sample into a corresponding category according to visual features in the sample, wherein the visual features include one or more of color, texture, and shape;

[0015] For video samples, frame sequence analysis and target detection technology are used to identify moving targets and scenes in the video, and finally classify them into specific video categories.

[0016] Furthermore, quality assessment is performed periodically, which is achieved by setting up scheduled tasks according to business needs. After the assessment, a quality report is generated containing the assessment results, quality issues and possible improvement suggestions for each sample.

[0017] Furthermore, check the storage requirements of the sample, including:

[0018] Identify the type of sample;

[0019] Evaluate the data volume of the sample and calculate the required storage space;

[0020] Set the frequency of sample visits;

[0021] Evaluate and set storage security requirements based on the sensitivity of the sample.

[0022] Furthermore, the samples are stored in the specified storage layer, including:

[0023] Write sample data to the specified storage location; record sample metadata, including labels and storage location; update sample status and generate storage reports.

[0024] Furthermore, based on the multi-dimensional label system and feature configuration of the samples, samples that meet the conditions are found and the search results are returned, including:

[0025] Obtaining search conditions input by the user through the interface tool, the search conditions are based on multidimensional tags, sample features or other pre-set conditions, based on the multidimensional tag system and feature configuration of the sample, using indexing technology to search for samples that meet the conditions in a large-scale sample library, and returning the search results to the user;

[0026] Distribute the retrieved samples to the required tasks or analysis scenarios as needed, including:

[0027] Get the selected scheduling strategy, schedule and distribute samples according to task requirements, time or features, and transmit them to the specified storage layer, task pipeline or analysis tool.

[0028] In the second aspect, a large-scale sample storage management system based on a multi-dimensional label system includes:

[0029] Input layer, used to receive large-scale sample data input;

[0030] The label management layer is used to establish a multi-dimensional label system for each sample data based on the sample source, sample quality, and sample feature dimensions;

[0031] The storage management layer is used for automatic auxiliary identification and classification, quality assessment and storage of sample data. For text samples, natural language processing algorithms are used to process them, extract key features of the text and classify the text into specified categories; for image / video samples, computer vision technology is used to identify the main content or target objects in the image or video and classify them into the corresponding specified categories; based on the classified samples, the quality assessment of the samples is automatically performed according to the predefined quality standards, and a quality report is generated according to the assessment results, wherein the quality assessment dimensions include completeness, accuracy and consistency; for samples that have undergone quality assessment, the storage requirements of the samples are checked, multi-dimensional labels are generated for the samples according to the multi-dimensional label system, and the appropriate storage layer is selected according to the multi-dimensional labels of the samples, and the samples are stored in the specified storage layer after optimizing the storage space;

[0032] The retrieval scheduling layer is used to find samples that meet the search conditions based on the multi-dimensional label system and feature configuration of the samples according to the search conditions entered by the user, return the search results, and distribute the retrieved samples to the required tasks or analysis scenarios as needed.

[0033] Furthermore, at the label management layer, the system provides a variety of predefined label templates for users to choose from, and the label system supports dynamic expansion, which can re-evaluate existing samples based on new dimensional labels added by users and apply new labels.

[0034] Furthermore, at the storage management level, the system supports automatic auxiliary labeling for each sample and groups samples with similar labels. The similarity is measured based on the number and weight of labels. An interface is provided to support manual adjustment of classification, including manual modification of labels or adjustment of classification rules. Automatic classification supports hierarchical structure. Sample classification can be set to main categories and subcategories. The main categories are divided according to the most important label dimension, and the subcategories are more finely classified according to secondary dimensions, allowing users to freely define multi-level classification standards according to actual needs.

[0035] Furthermore, at the storage management level, the system provides a distributed storage architecture that supports distributed storage and management of large-scale sample data and distributes data on multiple nodes; it supports version management of sample data, and the system can record all historical versions of each sample, providing users with flexible options to choose to retain or delete historical versions. The differences between different versions are displayed visually, allowing users to quickly locate modifications; it has the ability to quickly retrieve large-scale data sets; it provides data expiration and automatic cleanup functions, and automatically archives or deletes sample data after the expiration date according to the user-configured retention period.

[0036] Furthermore, at the search scheduling layer, the system supports precise or fuzzy matching by single tag or multiple tag combinations to quickly locate the required samples; the system supports assigning different weights to tags, and adjusting the priority according to the weight during the search process, so that important tags have higher matching priority to meet complex business search requirements; the system provides sorting, filtering and aggregation operations for search results, optimizes search results according to user screening and operation of tags, and enhances the user's operation experience; the system uses a cache mechanism to cache the search results of commonly used tags during the search process, reducing repeated calculations and improving search speed;

[0037] The search results support batch download, batch editing, and batch deletion operations, allowing users to perform unified operations on multiple samples at one time, and edit or redefine batch tags through batch search results. The system automatically updates the tag information and applies it to the storage management layer to achieve efficient maintenance of sample data.

[0038] Furthermore, at the retrieval scheduling layer, the system provides two modes: batch scheduling and manual scheduling. Batch scheduling: the system supports batch selection of qualified samples and automatic distribution according to preset task requirements or model training requirements; manual scheduling: supports users to manually select samples and assign them to different tasks or analysis processes;

[0039] The system supports automatic adjustment of scheduling strategies according to task types, so that high-priority tasks will obtain samples first; it supports users to schedule and distribute samples based on time arrangements or sample characteristics to meet different business needs.

[0040] Furthermore, at the retrieval scheduling layer, the system supports dynamic task optimization: dynamically adjusting the scheduling strategy based on sample data volume, label update frequency, and system load; and supports task priority management: reasonably allocating resources based on the priority set by the user to ensure that critical tasks are executed first.

[0041] According to a third aspect, a computer device is provided, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the large-scale sample storage management method based on a multidimensional labeling system as described in the first aspect of the present invention are implemented.

[0042] In a fourth aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the large-scale sample storage management method based on a multidimensional label system as described in the first aspect of the present invention are implemented.

[0043] Compared with the prior art, the invention has the following beneficial effects:

[0044] The present invention achieves accurate management and rapid retrieval of samples by establishing a multi-dimensional metadata tag system for large-scale data samples. The system uses natural language processing technology and computer vision algorithms to assist in the identification and classification of samples, reducing manual participation and improving the overall efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the overall architecture of the system of the present invention.

[0046] Figure 2 This is a flow chart for constructing the multidimensional labeling system in the present invention.

[0047] Figure 3 This is a flow chart of sample auxiliary identification and classification in the system of the present invention.

[0048] Figure 4 This is a flow chart of sample classification and storage in the system of the present invention.

[0049] Figure 5 Schematic diagram of sample retrieval and scheduling in the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0051] The present invention provides a large-scale sample storage management method based on a multidimensional label system. The method is completed based on a large-scale sample storage management system, and realizes efficient storage and management of large-scale sample data through label system construction, sample auxiliary identification and classification, sample storage management, and intelligent retrieval and scheduling.

[0052] Step S1, label system construction.

[0053] The system first conducts a comprehensive analysis of the metadata of the data sample and establishes a multi-dimensional labeling system. The labeling system includes but is not limited to the following dimensions:

[0054] 1) Sample source: including data source (such as experimental data, public data sets, etc.);

[0055] 2) Sample quality: evaluated based on factors such as data accuracy and completeness;

[0056] 3) Sample characteristics: such as data type, sample structure, feature attributes, etc.

[0057] Each sample is labeled according to these dimensions to form a flexible and fine-grained label library. This label library supports dynamic expansion to meet future expansion needs.

[0058] Step S2: sample auxiliary identification and classification.

[0059] In order to further improve the automation of storage and management, this system integrates natural language processing technology and computer vision algorithms. The system can automatically extract features and match labels for uploaded sample data, and automatically classify and store samples based on the generated labels.

[0060] 1) Natural language processing technology: used for automatic recognition and labeling of text data.

[0061] 2) Computer vision algorithm: used for feature extraction and classification of image or video data.

[0062] In this step, label generation is closely related to the storage process. The results of algorithm identification and classification will be used for sample classification and storage. The system reduces manual intervention through automated classification and improves the accuracy and efficiency of sample storage.

[0063] Step S3: sample storage management.

[0064] After auxiliary identification and classification are completed, the system will store and manage samples in a hierarchical manner based on these labels. Each sample will be assigned to a corresponding storage area based on its label information, thereby ensuring efficient organization and storage of data. Storage management is closely integrated with the label system. The label system is the core of storage management, and labels are used to guide how samples are stored and retrieved.

[0065] Step S4: intelligent retrieval and scheduling.

[0066] Based on the establishment of the tag system, the system supports multi-dimensional combined queries under complex conditions. Users can enter tags of multiple dimensions to conduct precise searches and quickly find the samples they need.

[0067] 1) The retrieval module will parse the input tags and locate and schedule relevant samples based on the existing sample information in the tag library.

[0068] 2) Tag combinations can be flexibly defined. For example, by querying “sample source” and “sample quality” at the same time, the system can respond quickly and return samples that meet the conditions.

[0069] During this process, the retrieval function is closely linked with the label system and sample storage module to ensure that each retrieval can quickly locate the target sample, thereby improving the operating efficiency of the system.

[0070] Reference Figure 1 The present invention provides a large-scale sample intelligent storage and management system based on a multi-dimensional label system. The system consists of four main layers: input layer, label management layer, storage management layer, and retrieval scheduling layer. Different layers have different functions and roles. The system realizes efficient storage, classification, and retrieval of large-scale data samples through the collaborative work of each layer.

[0071] The input layer is used to receive large-scale sample data input. The samples can be images, texts, or other types of multimodal data. This layer is the starting point of the system. After entering the system, all sample data will first enter the input layer for preprocessing.

[0072] The label management layer is one of the core layers of the present invention, responsible for establishing a multi-dimensional label system for each sample data. The label system is constructed based on multi-dimensional information such as the source, quality, and characteristics of the sample. This layer provides the basis for sample classification and retrieval in the system.

[0073] The storage management layer realizes intelligent storage and classification management of samples with the support of the multi-dimensional label system constructed by the label management layer. The system automatically classifies samples into different storage areas by parsing the sample label information, and quickly locates them when they are used or retrieved later.

[0074] The retrieval scheduling layer helps users to realize intelligent retrieval and scheduling of samples based on the multi-dimensional characteristics of the tag system, and quickly find sample data that meets specific needs. The retrieval scheduling layer works closely with the storage management layer. By parsing the tag combination conditions entered by the user, the system can efficiently return relevant results and improve work efficiency.

[0075] Reference Figure 2 The multi-dimensional label system construction flow chart of the present invention shows the detailed process of label construction, and the specific steps are as follows:

[0076] Step S11, sample source:

[0077] The system automatically generates a "source" label based on the source of the data sample (such as experimental data, public data sets, etc.). This dimension enables researchers to quickly filter data based on the source when searching for samples in the future.

[0078] Step S12: Sample quality:

[0079] According to the multi-dimensional labeling system construction method of the present invention, sample quality assessment is an important step for the system to classify and manage samples, involving a series of custom configurations and automated processing.

[0080] Before the system starts to evaluate the quality of samples, users need to define appropriate quality evaluation rules based on business needs or research objectives. These rules usually include the following dimensions:

[0081] Integrity: Check whether the sample data is complete and whether important information is missing. For example, for image samples, the system can detect whether the image is damaged or partially missing.

[0082] Consistency: Evaluate whether the sample data conforms to the expected format or standard, for example, whether the data value is within a reasonable range, whether the text data conforms to grammatical rules, etc.

[0083] Accuracy: Verify the accuracy of sample data, for example, whether certain numerical data is reasonable, or whether the color or shape in the image sample matches the label description.

[0084] Users can configure these rules and select appropriate thresholds and standards through the interface provided by the system. The configured quality assessment rules are stored in the label management layer of the system as the basis for subsequent sample quality assessment.

[0085] Step S13: Sample features

[0086] In the multi-dimensional labeling system of the present invention, the construction of sample features is customized by the user according to specific needs and application scenarios. Sample features are an important basis for the system to classify and retrieve, usually manually specified and configured through the system's interface tools. The following are the specific steps for constructing sample features:

[0087] 1) Custom configuration of sample features

[0088] The system provides a customizable feature construction interface, and users or system administrators can set a variety of sample features according to business needs. For example, for image samples, users can define features such as "color", "shape", and "theme"; for text samples, features such as "theme", "keyword", and "emotion" can be configured.

[0089] Each feature can be defined through different classification dimensions and assigned corresponding values ​​or ranges. For example, users can define the "color" feature and assign values ​​to "red", "green", etc., or define the "emotion" feature as "positive" or "negative". These customized features will become an important basis for the system to classify and manage samples in the future.

[0090] 2) Management and application of sample characteristics

[0091] After defining the features, the system will automatically match the corresponding feature tags for each sample. For example, when a new sample enters the system, the user can manually or batch tag the automatically identified feature tags for the sample through the interface tool. The system will automatically parse these tags and classify the samples and store them in the corresponding sample library.

[0092] 3) Dynamic adjustment of feature configuration

[0093] Feature construction is not limited to the initial configuration. Users can adjust sample feature configuration at any time according to actual conditions and needs. For example, when the sample set is expanded or research needs change, users can add new feature dimensions or modify the value range of existing features. The system can automatically update feature labels based on these adjustments to ensure the accuracy and flexibility of sample classification.

[0094] According to an embodiment of the present invention, at the tag management layer, the construction of the tag system includes:

[0095] Define labels based on sample feature dimensions, including but not limited to color, shape, size, timestamp, source, business type, associated entity, related events, etc. These labels will be automatically generated or manually configured based on different sample features to mark the unique attributes of each sample;

[0096] Through predefined label templates or custom configurations, users can create or modify label dimensions according to specific business needs. The system provides a variety of templates for users to choose from, and users can also fully customize label configurations, making the label system flexible to adapt to different scenarios;

[0097] The label system supports dynamic expansion. Users can add new label dimensions at any time. The system will re-evaluate existing samples based on the new dimensions and apply new labels. In addition, the label system can be dynamically loaded and applied to ensure that the system will not be affected by restarts.

[0098] Reference Figure 3 The sample auxiliary identification and classification flow chart of the present invention shows the detailed process of sample classification. The specific steps are as follows:

[0099] After receiving the sample data, the system will start the process of automatic auxiliary recognition and classification. This step mainly relies on modern machine learning algorithms, such as natural language processing and computer vision, to automatically classify the characteristics of the sample.

[0100] Step S21: Identification and classification of text samples

[0101] The system uses relevant natural language processing algorithms to process text samples, analyze the text's semantics, content themes, keywords and other features, and automatically determine the type and classification of the text.

[0102] Natural language processing extracts important features from text (such as keywords, entity recognition, topic modeling, etc.) and classifies text samples into specific categories such as news, research documents, social media, etc. according to preset classification criteria.

[0103] Step S22: Identification and classification of image / video samples

[0104] For image and video samples, the system uses computer vision technology for auxiliary classification. Through trained deep learning models (such as convolutional neural networks, CNN), the system can automatically identify the main content or target objects in images or videos.

[0105] 1) Image classification usually classifies samples into corresponding categories based on the visual features in the samples (such as color, texture, shape, etc.), such as medical images, traffic monitoring, product pictures, etc.

[0106] 2) Video samples are identified through frame sequence analysis and target detection technology to identify moving targets and scenes in the video, and finally classified into specific video categories.

[0107] According to an embodiment of the present invention, at the storage management level, the implementation of the sample auxiliary identification and classification function includes:

[0108] The system automatically assists in labeling each sample. By analyzing the multi-dimensional features of the sample, the system can automatically generate a set of corresponding labels for each sample. This process combines machine learning algorithms and rule engines to ensure accurate matching of labels.

[0109] Sample classification is based on the similarity of labels. The system automatically groups samples with similar labels, thereby achieving automatic classification of samples. The similarity algorithm takes into account the number and weight of labels to ensure that the classification process takes into account both efficiency and accuracy;

[0110] Supports manual adjustment of classification. Users can manually optimize the classification results through the visual interface, including manually modifying labels or adjusting classification rules to ensure that the classification results are more in line with actual business scenarios.

[0111] Automatic classification supports hierarchical structure. Sample classification can be set to main categories and subcategories. The main categories are divided according to the most important label dimension, and the subcategories are more finely classified according to the secondary dimensions. This hierarchical structure allows users to manage sample data in a gradually refined manner from global to local.

[0112] Furthermore, the sample classification supports a multi-level classification structure, including:

[0113] The main categories are divided according to important tags, such as business lines or sample sources;

[0114] Subcategories are further divided based on detail tags, such as timestamp, geographic location, size, etc.

[0115] The custom classification hierarchy allows users to freely define multi-level classification standards according to actual needs, and flexibly adjust them in subsequent operations to ensure that the classification results always meet business needs.

[0116] After the sample is classified, the system will then conduct a quality assessment. The quality assessment judges the overall quality of the sample based on predefined criteria. The dimensions of the assessment include completeness, accuracy, and consistency set by the user in the labeling system.

[0117] Step S31: Automated evaluation task

[0118] The system automatically performs sample quality assessment tasks based on preset quality standards. Whenever a new batch of data comes in, these tasks will be automatically run to ensure continuous monitoring of sample data.

[0119] The periodicity of the evaluation can also be set as a scheduled task based on business needs, such as daily, weekly, or monthly evaluation to ensure that the data maintains high quality during continuous storage and processing.

[0120] Step S32: Generating quality assessment results

[0121] After the evaluation is completed, the system will summarize the results and generate a detailed quality report. The report will include the evaluation results of each sample, quality issues, and possible improvement suggestions.

[0122] These reports provide important references for subsequent data management decisions, and users can adjust evaluation criteria or take corrective measures based on the content in the reports.

[0123] Step S33: Feedback and adjustment

[0124] Based on the evaluation report, the system allows users to provide feedback and reconfigure the evaluation rules to more accurately reflect the actual data status. This closed-loop mechanism ensures continuous optimization of sample quality assessment, thereby maintaining data consistency and reliability in the sample library.

[0125] Reference Figure 4 The sample intelligent storage flow chart of the present invention shows the detailed process of sample storage, and the specific steps are as follows:

[0126] Step S41: Check the storage requirements of the sample

[0127] After the sample feature configuration, classification, and quality assessment are completed, the system will analyze the sample storage requirements. This step includes the following:

[0128] 1) Sample type identification: Determine the type of sample (such as text, image, video, etc.), which affects the choice of storage.

[0129] 2) Data volume calculation: Evaluate the data volume of the sample and calculate the required storage space.

[0130] 3) Access frequency setting: Set the access frequency of samples (such as real-time data, high-frequency access data, long-term archived data).

[0131] 4) Security requirements: Based on the sensitivity of the sample, evaluate and set storage security requirements (such as encrypted storage, access control, etc.).

[0132] Step S42: Configure sample multi-dimensional labels

[0133] After the auxiliary identification and classification process generates multi-dimensional labels for the sample data according to the established multi-dimensional label system, it is manually reviewed and configured, and the storage period and usage frequency of the sample data are configured. These labels can help the system accurately classify and manage samples:

[0134] Storage period: The storage time requirement for samples.

[0135] Frequency of use: How often the sample is accessed (e.g., real-time, frequent, occasional).

[0136] These labels generated according to the labeling system will be used to guide the storage strategy and location selection of the samples.

[0137] Step S43: Select a suitable storage layer

[0138] The system selects a suitable storage layer based on the multi-dimensional labels of the samples. The selection of the storage layer is based on the following aspects:

[0139] 1) Cold storage tier: used for long-term archiving and infrequently accessed data. This type of data is usually stored in lower-cost tape storage or low-cost cloud storage.

[0140] 2) Warm storage layer: used for medium-frequency access data, which requires faster access speed but does not require real-time processing. It is usually stored in SSD (Solid State Disk) or standard cloud storage.

[0141] 3) Hot storage layer: used for real-time processing and high-frequency access data. These data require the fastest access speed and are usually stored in memory or high-performance SSD.

[0142] Step S44: Intelligent algorithm optimizes storage space

[0143] After determining the storage tier, the system runs intelligent algorithms to optimize the use of storage space. These algorithms include:

[0144] 1) Data compression: compress compressible samples to reduce the required storage space.

[0145] 2) Dynamic expansion: When storage capacity approaches the upper limit, storage resources are automatically expanded to ensure storage sustainability.

[0146] Step S45: Store the sample in the specified storage layer

[0147] After the optimization is completed, the samples are stored in the corresponding storage layer. This step includes:

[0148] 1) Data writing: Write the sample data to the specified storage location.

[0149] 2) Metadata recording: Record the metadata of the sample (such as labels, storage location, etc.) for future retrieval and management.

[0150] 3) Status update: Update the status of the sample to "stored" and generate a storage report to ensure that all samples have gone through the complete storage process.

[0151] According to an embodiment of the present invention, at the sample storage layer, the sample storage management is implemented as follows:

[0152] Classified storage of sample data based on labels, samples are automatically stored in the corresponding storage pool according to their labels. The system supports the storage of multiple data types such as text, pictures, videos, audio, etc., and optimizes samples of different formats to ensure storage efficiency and consistency;

[0153] The system provides a distributed storage architecture that supports distributed storage and management of large-scale sample data. By distributing data on multiple nodes, it ensures that the system can handle massive amounts of data while improving the reliability and access efficiency of sample data.

[0154] The system supports efficient retrieval and high concurrent access. The system has the ability to quickly retrieve large-scale data sets and ensure stable operation of the system in high concurrent access scenarios. The optimized storage structure ensures a smooth operation experience even when the number of samples is extremely large.

[0155] Sample storage management also includes data archiving and version control, including:

[0156] Version control function supports version management of sample data. The system can record all historical versions of each sample and provide users with flexible options to choose to keep or delete historical versions. The differences between different versions are visualized to facilitate users to quickly locate modifications;

[0157] Data lifecycle management: The system provides data expiration and automatic cleanup functions. Users can configure the retention period of sample data, which will be automatically archived or deleted after expiration to save storage space.

[0158] Reference Figure 5 The sample retrieval and scheduling diagram of the present invention shows the detailed process of sample retrieval and scheduling.

[0159] After the sample data is stored, the system provides an intelligent retrieval function to help users quickly find the required data in massive samples. The main contents of this step are as follows:

[0160] 1) Input search conditions: Users input search conditions through the system interface. The conditions can be based on multi-dimensional tags, sample features or other pre-set conditions. Common search conditions include:

[0161] The type of sample (e.g., image, text, etc.);

[0162] Generation time or data source;

[0163] The quality assessment score of the sample (such as high-quality or low-quality sample);

[0164] 2) Multi-condition search: The system supports combined condition search. Users can set multiple search conditions and conduct cross-queries. For example, you can filter by data quality and sample characteristics at the same time to ensure more accurate results.

[0165] 3) Efficient matching: Based on the multi-dimensional label system and feature configuration of samples, the system uses indexes to accelerate the retrieval process and can quickly find qualified samples in a large-scale sample library. This efficient positioning mechanism relies on intelligent indexing and fast matching algorithms, such as inverted indexing and Boolean retrieval.

[0166] 4) Result feedback and adjustment: After the search results are returned, users can check whether the samples meet their expectations. If the search results are inaccurate, users can further adjust the search conditions, and the system will re-match until a suitable sample is found.

[0167] After successfully retrieving qualified samples, the system provides a sample scheduling function to quickly and effectively distribute the retrieved samples to the required tasks or analysis scenarios. The specific steps are as follows:

[0168] Step 1: Scheduling strategy selection

[0169] 1) Batch scheduling: The system supports batch scheduling, which selects and distributes appropriate samples in batches according to preset task requirements or model training requirements. For example, specific samples are pushed to the model training pipeline at regular intervals to ensure that tasks are completed on time.

[0170] 2) Manual Scheduling: Users can manually select samples and assign them to different tasks or analysis processes. This option is suitable for scenarios that require precise control, such as special data analysis tasks.

[0171] Step 2: Sample Distribution

[0172] 1) Scheduling by task requirements: The system automatically adjusts the scheduling strategy based on the type of task (such as model training or data analysis). High-priority tasks will be given priority in obtaining samples to ensure smooth task execution.

[0173] 2) Scheduling by time or features: In certain cases, users can schedule and distribute samples based on time arrangements (such as daily scheduling) or sample features.

[0174] 3) Sample delivery: After the sample is successfully scheduled, the system transfers it to the designated storage layer, task pipeline or analysis tool to ensure smooth subsequent data use.

[0175] According to the implementation mode of the present invention, at the retrieval scheduling layer, in order to realize the intelligent retrieval and scheduling functions, the system has the following processing capabilities:

[0176] Multi-dimensional tag search: users can intelligently search for samples by tags, and support precise or fuzzy matching by single tag or multiple tag combinations. Through multi-dimensional tag search, users can quickly locate the required samples;

[0177] Tag weight sorting: the system supports assigning different weights to tags and adjusting the priority according to the weight during the search process, so that some important tags have higher matching priority to meet complex business search requirements;

[0178] Search result optimization: the system provides sorting, filtering and aggregation operations for search results. Users can further filter and operate tags to optimize search results and enhance the user's operation experience.

[0179] Retrieval cache and optimization: During the retrieval process, the system uses a cache mechanism to cache the retrieval results of commonly used tags, reducing repeated calculations and improving retrieval speed.

[0180] Furthermore, the intelligent search mechanism supports batch operations, including:

[0181] Batch processing of search results: search results support batch download, batch editing, and batch deletion operations. Users can perform unified operations on multiple samples at one time, greatly improving operational efficiency.

[0182] Batch adjustment of labels: users can edit or redefine batch labels through batch retrieval results. The system automatically updates the label information and applies it to the storage management module to achieve efficient maintenance of sample data.

[0183] Furthermore, the intelligent scheduling function includes:

[0184] Scheduling strategy selection: the system provides two methods: batch scheduling and manual scheduling:

[0185] Batch Scheduling: The system supports batch selection of qualified samples and automatic distribution according to preset task requirements or model training requirements. For example, the system can push specific samples to the model training pipeline at a fixed time to ensure that tasks are completed on time and run efficiently.

[0186] Manual Scheduling: Users can manually select samples and assign them to different tasks or analysis workflows. This option is suitable for scenarios that require precise control, such as processing specific data analysis tasks.

[0187] Sample distribution:

[0188] 1) Scheduling by task requirements: The system automatically adjusts the scheduling strategy based on the task type (such as model training or data analysis). High-priority tasks will be given priority in obtaining samples to ensure smooth completion of the task.

[0189] 2) Scheduling by time or features: Users can schedule and distribute samples based on time arrangements (such as daily scheduling) or sample features to flexibly respond to different business needs.

[0190] 3) Sample delivery: After the sample is successfully scheduled, the system will distribute it to the specified storage layer, task pipeline or analysis tool to ensure the smooth progress of subsequent tasks or analysis operations.

[0191] Furthermore, intelligent scheduling tasks support dynamic adjustment, including:

[0192] Dynamic task optimization: The system dynamically adjusts the scheduling strategy based on the sample data volume, label update frequency, system load, etc. to ensure efficient execution of scheduling tasks while avoiding affecting system performance.

[0193] Task priority management: Users can set the priority of tasks, and the system will reasonably allocate resources based on the priority to ensure that key tasks are executed first and achieve optimal resource utilization.

[0194] The present invention also provides a computer device, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the large-scale sample storage management method based on a multidimensional labeling system as described above are implemented.

[0195] The present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large-scale sample storage management method based on a multi-dimensional label system as described above.

[0196] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems (devices), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0197] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A system that specifies functions within a process or multiple processes.

[0198] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A function specified in a process or multiple processes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A large-scale sample storage management method based on a multi-dimensional label system, characterized in that: The method is implemented based on a large-scale sample storage management system based on a multi-dimensional label system, and includes the following steps: Obtain large-scale sample data input, and establish a multi-dimensional labeling system for each sample data based on sample source, sample quality, and sample feature dimensions; Automatically assist in the identification and classification of sample data. For text samples, natural language processing algorithms are used to extract key features of the text and classify the text into specified categories. For image / video samples, computer vision technology is used to identify the main content or target objects in the image or video and classify them into the specified corresponding categories. Based on the classified samples, the quality assessment of the samples is automatically performed using predefined quality standards, and a quality report is generated based on the assessment results. The quality assessment dimensions include completeness, accuracy, and consistency. For samples that have undergone quality assessment, check the storage requirements of the samples, generate multidimensional labels for the samples according to the multidimensional label system, select a suitable storage layer based on the multidimensional labels of the samples, and store the samples in the specified storage layer after optimizing the storage space; Obtain the search conditions entered by the user, find samples that meet the conditions based on the multi-dimensional label system and feature configuration of the samples, return the search results, and distribute the retrieved samples to the required tasks or analysis scenarios as needed.

2. The method according to claim 1, characterized in that: The sample source dimension includes the data source; the sample quality dimension includes the accuracy, completeness, and consistency of the data; the sample feature dimension includes the data type, sample structure, and feature attributes. The options and / or ranges and thresholds related to these dimensions are configured through the interface tools provided by the system.

3. The method according to claim 1, characterized in that Computer vision technology identifies the main content or target objects in an image or video, including: For an image sample, classify the sample into a corresponding category according to visual features in the sample, wherein the visual features include one or more of color, texture, and shape; For video samples, frame sequence analysis and target detection technology are used to identify moving targets and scenes in the video, and finally classify them into specific video categories.

4. The method according to claim 1, characterized in that Quality assessment is carried out periodically, which is achieved by setting up scheduled tasks according to business needs. After the assessment, a quality report is generated containing the assessment results, quality issues and possible improvement suggestions for each sample.

5. The method according to claim 1, characterized in that Review the storage requirements for your samples, including: Identify the type of sample; Evaluate the data volume of the sample and calculate the required storage space; Set the frequency of sample access; Evaluate and set storage security requirements based on the sensitivity of the sample.

6. The method according to claim 1, characterized in that Store samples to the specified storage layer, including: Write sample data to the specified storage location; record sample metadata, including labels and storage location; update sample status and generate storage reports.

7. The method according to claim 1, characterized in that Based on the multi-dimensional label system and feature configuration of samples, find samples that meet the conditions and return search results, including: Obtaining search conditions input by the user through the interface tool, the search conditions are based on multidimensional tags, sample features or other pre-set conditions, based on the multidimensional tag system and feature configuration of the sample, using indexing technology to search for samples that meet the conditions in a large-scale sample library, and returning the search results to the user; Distribute the retrieved samples to the required tasks or analysis scenarios as needed, including: Get the selected scheduling strategy, schedule and distribute samples according to task requirements, time or features, and transmit them to the specified storage layer, task pipeline or analysis tool.

8. A large-scale sample storage management system based on a multi-dimensional label system, characterized in that: include: Input layer, used to receive large-scale sample data input; The label management layer is used to establish a multi-dimensional label system for each sample data based on the sample source, sample quality, and sample feature dimensions; The storage management layer is used for automatic auxiliary identification and classification, quality assessment and storage of sample data. For text samples, natural language processing algorithms are used to process them, extract key features of the text and classify the text into specified categories; for image / video samples, computer vision technology is used to identify the main content or target objects in the image or video and classify them into the corresponding specified categories; based on the classified samples, the quality assessment of the samples is automatically performed according to the predefined quality standards, and a quality report is generated according to the assessment results, wherein the quality assessment dimensions include completeness, accuracy and consistency; for samples that have undergone quality assessment, the storage requirements of the samples are checked, multi-dimensional labels are generated for the samples according to the multi-dimensional label system, and the appropriate storage layer is selected according to the multi-dimensional labels of the samples, and the samples are stored in the specified storage layer after optimizing the storage space; The retrieval scheduling layer is used to find samples that meet the search conditions based on the multi-dimensional label system and feature configuration of the samples according to the search conditions entered by the user, return the search results, and distribute the retrieved samples to the required tasks or analysis scenarios as needed.

9. A computer device, characterized in that: The device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the large-scale sample storage management method based on a multidimensional labeling system as described in any one of claims 1-7 are implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large-scale sample storage management method based on a multi-dimensional labeling system as described in any one of claims 1 to 7 are implemented.

Images