AI data set processing method

Abstract

The invention relates to an AI data set processing method which comprises the following steps: designing a DSDL (Data Set Description Language) framework which comprises a data set description file, a basic data type, an unstructured object class, a structural body class, a class domain, a data module and a link library; and performing corresponding description processing on the data set of the current task by utilizing the data set description language framework. Compared with the prior art, the method has the advantages that a uniform standard is provided to simplify data set processing, data of different modalities and structures are expressed by adopting a consistent format, the workload of data retrieval, preprocessing and use can be remarkably reduced, a user can conveniently and quickly describe the data set, and the user experience is improved. And meanwhile, the method can be conveniently applied to the steps of data set visualization, model training reasoning and the like, and has relatively good expansibility.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to an AI data set processing method. Background Art

[0002] In the era of artificial intelligence, large-scale, diverse, and high-quality datasets are essential factors in promoting technological progress. However, the diversity of modes and tasks in datasets, as well as different representations of the same content, lead to extremely complex data formats. When dealing with multi-modal and multi-task model training, researchers and developers usually need to perform tedious format conversions, which not only increases the preprocessing workload, but also may lead to data storage waste. For example, different research institutions, or even the same institution, may store multiple copies of data due to different format conversion methods.

[0003] At present, with the continuous development of deep learning technology, more and more large models and various data sets are emerging. Due to the diversity of data set formats, data modes and annotation formats, the data cannot be used directly. It needs to be understood and converted in format before it can be used by researchers or developers with different needs. The existing data set framework cannot be applied to all data sets at the same time. It usually does not have the universality to standardize all data sets. It can only be used in specific scenarios and requires developers to perform complicated conversions. For example, OpenBayes, or only focuses on the storage, loading and transmission of AI (Artificial Intelligence) data sets, but does not provide a framework to unify the description of all data sets, such as DeepLake and FFCV.

[0004] Among them, the OpenBayes data format is a set of data set formatting standards defined by OpenBayes, which is used in OpenBayes automatic modeling and other related products. The OpenBayes data format uses meta.csv as the main format file of the data set. The file is in csv format. The first line contains the field type and field name in the format of [type]_[name]. The second line and each subsequent line are data samples. Users need to process the data set by themselves to construct the meta.csv file.

[0005] The DeepLake dataset format provides a simple API for creating, storing, and collaboratively processing AI datasets of any size. DeepLake's data layout enables fast data conversion and streaming while training models at scale.

[0006] FFCV combines technologies such as efficient file storage formats, caching, data preloading, asynchronous data transfer, and just-in-time compilation to significantly improve data loading and transmission efficiency, ensure that the GPU can be fully utilized, and asynchronously offload as much data processing as possible to the CPU, freeing up GPU cycles for training.

[0007] However, none of the above existing methods can provide a unified dataset language concept to simplify the description and use of datasets. In practical applications, datasets are often defined in different forms. If a new dataset is to be incorporated into the workflow, a new customized tool or script needs to be redeveloped, which will inevitably incur huge overhead in terms of exchange, integration and use. Summary of the invention

[0008] The purpose of the present invention is to provide an AI data set processing method in order to overcome the defects of the above-mentioned prior art, which can provide a unified standard to simplify the processing of data sets and use a consistent format to express data of different modalities and structures.

[0009] The purpose of the present invention can be achieved by the following technical solution: A method for processing an AI data set, comprising the following steps:

[0010] Design the Data Structure Description Language (DSDL) framework, including data set description files, basic data types, unstructured object classes, structure classes, category domains, data modules, and link libraries;

[0011] The dataset description language framework is used to describe the dataset of the current task accordingly.

[0012] Furthermore, the data set description file is used to describe the data structure, including a header, a mate section, a defs section and a data section, and the header is used to clarify how the current data set description file needs to be parsed;

[0013] The mate section is used to provide meta information of the current data set;

[0014] The defs section is used to provide global definitions;

[0015] The data section is used to describe the sample data in the data set.

[0016] Furthermore, the dataset description file is defined using the data interaction language YAML or JSON.

[0017] Furthermore, the basic data types include general basic data types, special basic data types, category annotation data types, object location data types, usage type parameters and list data types, and the general basic data types include Bool Boolean type, Int integer type, Num general numeric type, and Str string type;

[0018] The special basic data types include:

[0019] Coord, that is, 2D coordinates, in the format of [x, y];

[0020] Coord3D, that is, 3D coordinates, in the format of [x, y, z];

[0021] Interval, that is, the sequence interval, the format is [begin, end];

[0022] BBox, that is, bounding box, the format is [x, y, w, h];

[0023] RotatedBBox, that is, rotated bounding box, the format is [x1,y1,x2,y2,x3,y3,x4,y4] or [x,y,w,h,angle];

[0024] Keypoint, that is, key point, the format is [[x1,y1,visable1],[x2,y2,visable2],...];

[0025] Polygon, that is, a polygon is represented by multiple 2D coordinates in the format of [[x1,y1],[x2,y2],...];

[0026] Date, that is, the date is represented by a string, and the format follows the rules of Python strftime;

[0027] Time, that is, time is represented by a string, and the format follows the rules of Python strftime;

[0028] The category annotation data types include:

[0029] Based on the class name, the format is " <class-domain> :: <class-name>";

[0030] Based on category index, the format is " <class-domain> :: <class-index>";

[0031] Enter the category name directly in the format of <class-name>;

[0032] Enter the category index directly in the format <class-index>;

[0033] The object location data type is the Loc data type, which is used to reference unstructured objects;

[0034] The usage type parameters include the Label data type of the usage type parameter, and the Date and Time data types of the usage type parameter;

[0035] The list data type includes two parameters, etype and ordered, where etype indicates the data type of a single element in the list, and ordered indicates whether there is an order relationship between the elements in the list. The default value is false. When the sequence is ordered, it is set to true.

[0036] Furthermore, the unstructured object class includes:

[0037] Image, each instance is an image, represented by a matrix of pixels;

[0038] Video, each instance is decoded into a sequence of video frames, each of which is an image;

[0039] Text, each instance is a sequence of words;

[0040] PointCloud, each instance is a collection of 3D points used to represent the shape of a 3D object;

[0041] LabelMap, each instance is a matrix of integer labels, each integer label corresponds to a category.

[0042] Furthermore, the structure class is used to represent mixed objects, including nested structures and structures with parameters.

[0043] Furthermore, the category field is used to define the dom parameter in the Label data type field.

[0044] Furthermore, the data module includes a sample data definition unit, a field missing processing unit, and a global information supplement unit, wherein the sample data definition unit is used to define data types, set sample storage paths, and save sample information of data sets;

[0045] The missing field processing unit is used to store the missing field names in a setting list;

[0046] The global information supplement unit is used to define the global information type, set the global information storage location and save the specific content of the global information.

[0047] Furthermore, the global information includes category information and data information.

[0048] Furthermore, the link library is used to store common classes in the same field, and the data set description process is simplified by directly importing the link library.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] The present invention relates to a framework of a dataset description language, including a dataset description file, basic data types, unstructured object classes, structure classes, category domains, data modules and link libraries. The framework of the dataset description language is used to simplify the description and use of datasets, and can provide a unified standard for AI datasets to express data of different modes and structures, significantly reducing the workload of data retrieval, preprocessing and use, facilitating users to quickly describe datasets, and can be easily applied to steps such as dataset visualization and model training and reasoning, with good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example

[0054] A dataset is essentially a data structure stored in permanent storage, usually consisting of unstructured objects (such as images, videos, audio, text, etc.) and annotations associated with them. These elements are combined into a dataset in a specific way. It is worth noting that the unstructured objects mentioned above usually contain a large amount of data. In order to facilitate the rapid distribution of data, the present invention is designed to separate the structured description of a dataset from the unstructured content.

[0055] The present invention proposes an AI data set processing method, such as Figure 1 As shown, the following steps are included:

[0056] Design the Dataset Description Language (DSDL) framework, including dataset description files, basic data types, unstructured object classes, structure classes, category domains, data modules, and link libraries;

[0057] The dataset description language framework is used to describe the dataset of the current task accordingly.

[0058] In the designed DSDL framework, the main contents include:

[0059] 1. Basic data model: DSDL uses a set of basic elements to describe a data set. The basic elements consist of containers such as structs, lists, and sets.

[0060] 1.1 Basic elements: Independent units in the data set description, including not only basic types such as numbers and strings, but also elements that can facilitate target positioning and expression of annotation results.

[0061] 1.2 Unstructured objects: Special basic elements such as images, videos, audio, and text, because they are invisible in the dataset description. Specifically, unstructured objects are represented by object locators, which are used to specify their storage location instead of putting the entire media file into the description. In addition, optional additional descriptors can be used to provide additional information about the object, such as the format or resolution of an image.

[0062] 1.3 Aggregates: Combine basic elements into a data structure. DSDL provides list and struct types to represent data structures. Specifically, a single sample is represented by a structure (aggregation of multiple fields), and a data set consists of a list of samples.

[0063] 2. Extended type system: In DSDL, all elements and structure units have types. DSDL uses a simple and easily extendable type system. Specifically, DSDL has three types.

[0064] 2.1 Primitive types: types that contain basic element values ​​such as Boolean, number, and string. DSDL provides a large unified set of primitive types.

[0065] 2.2 Unstructured object classes: abstractions of unstructured objects. For example, images, videos, audio, point clouds, text, etc. Although these objects have rich and diverse internal structures, they are treated as indivisible units in the dataset definition. DSDL provides a series of predefined unstructured object class sets that cover common application scenarios, while allowing open source forces to expand the set and achieve this goal by registering unstructured object classes through a minimum interface set.

[0066] 2.3 Struct classes: An abstraction of aggregate data structures in DSDL. Each instance in a structure class is called a struct, which contains multiple fields, each of which has its own type. An important application of structures is to represent data samples. DSDL provides a set of predefined structure classes for common tasks in the standard library, and allows users to customize structure classes for special tasks.

[0067] 2.4 Types must be defined before use (whether built-in types, third-party defined types, or user-defined types). Circular references are not allowed in the current DSDL.

[0068] 3. Object locators.

[0069] As mentioned above, unstructured targets are not fully embedded in the dataset description. Instead, they are represented by target locators. Specifically, a target locator is a string type with a special format. When the corresponding target is loaded, it will be converted into a real address by the DSDL parser.

[0070] The introduction of the object locator is the key to separating the structured description and unstructured media files in the dataset. This makes it unnecessary to move large amounts of media data, making the distribution of dataset descriptions more lightweight. At the same time, it allows for fast operations on the dataset, such as combining multiple sets, merging attributes, or taking a subset.

[0071] 4. Based on YAML or JSON.

[0072] DSDL is a domain-specific language that is defined based on the currently popular data exchange languages ​​​​YAML or JSON. It should be noted that the elements in YAML or JSON are not associated with specific meanings at the language level. DSDL can give these elements specific semantics to describe data sets in a meaningful way.

[0073] This design allows users to use a rich tool system corresponding to the YAML and JSON languages. Based on these tools, it is easy to build a complete system that supports parsing, validation, and query functions, and conveniently interacts based on the Internet ecosystem.

[0074] In specific applications, DSDL adopts a bottom-up overall architecture. This embodiment first defines the DSDL specification, which serves as the basis for grammatical rules. On this basis, this embodiment further defines DSDL data types and object locators. Using these building blocks, users can design dataset description files suitable for their AI tasks. The introduction of object locators greatly simplifies the process of reading and using data from multiple sources, whether the data is stored locally, in clusters, or in the cloud. To further facilitate users, this embodiment predefines data templates for mainstream tasks to simplify the use of datasets. A comprehensive grammar parser can convert DSDL descriptions into Python objects, thereby realizing the full functionality of the dataset description language. In addition, this embodiment also develops a set of DSDL toolkits that allow users to easily download mainstream datasets and their DSDL description files through the OpenDataLab CLI, further simplifying the data visualization and model training process.

[0075] The specific definition of DSDL includes:

[0076] 1. Dataset Description File

[0077] In DSDL, a dataset is represented by a dataset description file. The following example is a standard dataset description file. The dataset description file can be in JSON or YAML format.

[0078] JSON format:

[0079]

[0080]

[0081] YAML format:

[0082]

[0083]

[0084] Both JSON and YAML formatted dataset description files can accurately describe the same data structure. In DSDL, a dataset description file can be roughly divided into four parts:

[0085] Header: specifies how the current dataset description file needs to be parsed;

[0086] meta section: provides some meta information of the current dataset;

[0087] defs section: provides some global definitions, such as user-defined class domains

[0088] and structures (struct);

[0089] data section: describes the sample data in the dataset.

[0090] It should be noted that:

[0091] Attribute names prefixed with $ are reserved and have special meanings in DSDL.

[0092] The DSDL version (the attribute name in the header is $dsdl-version) must be explicitly specified because the DSDL interpreter needs to know the version information of the current file through this attribute to correctly parse the current dataset description file.

[0093] Usually, some common types are defined in the standard library or extension library. In most cases, users do not need to define categories themselves. In the above example, ImageClassificationSample is defined only for demonstration and to make the example independent.

[0094] 2. Basic Data Types

[0095] Basic data types are some data types abstracted from the basic elements in DSDL. Instances of these basic data types are the basic building blocks for constructing a dataset description file.

[0096] The JSON and YAML formats used by dataset description files also provide some syntax to represent some simple categories, such as Boolean type, number type, and string type. When using these syntaxes in DSDL to represent a value, DSDL will retain the original meaning of these values ​​in the JSON / YAML format. It is important to note that there is no one-to-one correspondence between DSDL basic data types and JSON data types. Different basic data types in DSDL may use the same JSON type to represent their respective values. For example, the target locator type (Loc) and the category label type (Label) are different data types in DSDL, but they both use the string type to represent their respective values.

[0097] 2.1 Common basic data types

[0098] DSDL defines four common basic data types. These four data types can be simply interpreted and have no special meaning.

[0099] Bool: Boolean type, the value can be true or false.

[0100] Int: Integer type, the value can be any integer, such as 12, -3 or 0. When a number is declared as an Int type, the DSDL interpreter verifies whether it is really an integer.

[0101] Num: A general numeric type. The value can be any number, such as 12.5, -13, 1.25e-6.

[0102] Str: string type, the value can be any string, such as "hello", "a", "".

[0103] 2.2 Special basic data types

[0104] DSDL defines some basic data types with special meanings. The values ​​of these data types are also represented by strings or other basic types in JSON format, but they have special semantics, and the DSDL parser will parse them accordingly according to their respective semantics.

[0105] Coord: 2D coordinates in the format of [x, y];

[0106] Coord3D: 3D coordinates in the format of [x, y, z];

[0107] Interval: sequence interval, the format is [begin,end];

[0108] BBox: bounding box, in the format of [x, y, w, h];

[0109] RotatedBBox: rotated bounding box, the format is [x1,y1,x2,y2,x3,y3,x4,y4] or [x,y,w,h,angle];

[0110] Keypoint: key point, in the format of [[x1,y1,visable1],[x2,y2,visable2],...];

[0111] Polygon: A polygon is represented by multiple 2D coordinates in the format of [[x1,y1],[x2,y2],...];

[0112] Date: The date is represented by a string, and the format follows the rules of Python strftime;

[0113] Time: The time is represented by a string, and the format follows the rules of Python strftime.

[0114] 2.3Label: Category label data type

[0115] The classification task aims to give semantic information to the target. Therefore, the present invention usually needs to use a category label to describe the category to which the target belongs. In DSDL, a category label is a string whose type is specified as Label.

[0116] In practice, the categories in different classification tasks should also be different. DSDL introduces the concept of class domain to describe different contextual information in different classification tasks. Each class domain provides a list of categories or a class hierarchy. Given a class domain, the class annotation can be described in one of the following four forms:

[0117] Based on the category name: the format is " <class-domain> :: <class-name>", for example, COCO::cat represents the category cat in the COCO category domain.

[0118] Based on category index: the format is " <class-domain> :: <class-index>", for example, COCO::3 represents the third category in the COCO category domain.

[0119] Directly enter the category name: the format is <class-name>, for example, cat represents the category of cat in the classdomain of the Label.

[0120] Directly enter the category index: the format is <class-index>, for example, 3 represents the category with index 3 in the classdomain of the Label.

[0121] For a category domain with a hierarchical relationship, it needs to be defined in class_dom. The present invention attempts to unify the category annotation systems of various task types. Therefore, the present invention will reserve a domain name std for describing the category domain of standard classification tasks in the future.

[0122] 2.4Loc: Object Location Data Type

[0123] The object locator is used as a reference to unstructured objects such as images, videos, and text. The data type of the object locator is the Loc data type, and the specific value of the object locator is represented by a special string. Specifically, DSDL supports the following methods to describe an object locator:

[0124] Relative path: The path relative to the root path of the dataset. The object locator uses a relative path by default. When the object locator string has no special prefix, it will be treated as a relative path. For example, "abc / 001.jpg" will be parsed as " <data-root> / abc / 001.jpg", where data-root is the root directory where all data objects are stored, and the root directory can be specified in the environment configuration file.

[0125] 2.5 Using type parameters

[0126] From the perspective of the DSDL compiler, the type of an element determines how the element is parsed and validated. In addition to the type name, DSDL also allows users to provide type parameters to customize how elements are represented, parsed, and validated.

[0127] 2.5.1 Label data type using type parameters

[0128] In the example in the dataset description file above, the data type of the label field in ImageClassificationSample is specified as Label[dom=MyClassDom]. Here, Label is a parametric data type that specifies a type parameter dom. The dom parameter here specifies the category domain to which the label belongs.

[0129] Once the class domain is explicitly specified (in this case, MyClassDom), there is no need to specify the class domain of the annotation class in the data object. For example, the annotation class MyClassDom::cat in the data object can be abbreviated to cat; the annotation class MyClassDom[2] in the data object can be abbreviated to 2.

[0130] 2.5.2 Date and Time Data Types Using Type Parameters

[0131] For Date and Time data types, when the type parameter is not explicitly specified, its string type value should follow the ISO 8601 format. The parser will use date.fromisoformat and time.fromisoformat to parse the string.

[0132] Users can also customize the format by specifying the type parameter fmt. For example, users can use the data type Time[fmt = "%H:%M"] to require the data value to follow the format of %H:%M, such as "15:32". When fmt is explicitly specified, the value of fmt is passed to the strptime function to parse the time string. The Data data type can also specify this parameter.

[0133] 2.6 List Data Type

[0134] DSDL provides the parameterized data type List to represent unordered or ordered lists. Specifically, an instance of the List data type is a list containing multiple elements of the same data type.

[0135] The parameterized data type List has two parameters:

[0136] etype: The data type of a single element in the list. This parameter must be specified explicitly.

[0137] ordered: Whether there is an order relationship between the elements in the list. This parameter is optional and the default value is false. This parameter should be set to true only when the sequence is ordered, such as video frame sequence and time series.

[0138] For example, for a list whose elements are integers, the list data type can be represented as List[Int]; for a list whose elements are class labels whose class domain is MyClassDom, the list data type can be represented as List[Label[MyClassDom]]

[0139] 3. Unstructured Object Classes

[0140] Unstructured objects, such as images, videos, voice, point clouds, and text, are digital representations of real-world objects. Although the internal structure of these data is rich, they are still treated as a whole in the dataset description file, and their internal structure is not shown. In DSDL, an unstructured object class is an abstraction for all specific types of unstructured objects.

[0141] 3.1 Predefined Unstructured Object Classes

[0142] In the standard library, DSDL provides the following unstructured object classes out of the box:

[0143] Image: Each instance is an image, which can be represented by a matrix of pixels.

[0144] Video: Each instance can be decoded into a sequence of video frames, each of which is an image.

[0145] Text: Each instance is a sequence of words.

[0146] PointCloud: Each instance is a collection of 3D points used to represent the shape of a 3D object.

[0147] LabelMap: Each instance is a matrix of integer labels, each of which corresponds to a category.

[0148] 3.2 Describing an Unstructured Object

[0149] In DSDL, an unstructured object can be declared by an object locator and other parameters belonging to the object.

[0150] Take a picture with the path abc / 0001.jpg as an example. It can be represented in the following way:

[0151] Target locator: Use the target locator abc / 0001.jpg. When a variable or a field has a data type of an unstructured target class and its value is a string, the string will be parsed as a target locator.

[0152] 3.3 Extending Unstructured Object Classes

[0153] DSDL allows users to register extended unstructured object classes by specifying how to load objects from storage. On the client side, you can define a subclass of the abstract base class UnstructuredObject and implement its load method to load objects, thereby extending the unstructured object class. Specifically, in Python, the abstract base class UnstructuredObject is defined in the following way:

[0154]

[0155]

[0156] It should be noted that the load method here needs to receive an already opened file object, not a file path. This design is based on the "separation of concerns" principle: the data system is responsible for parsing the object locator and building a corresponding file reader. Therefore, the subclass of the UnstructuredObject class only needs to consider how to load, parse and verify the object after a file reader is given.

[0157] 4. Structure Class

[0158] Structures are the most common way to represent mixed objects. For example, in a dataset, a standard data sample is composed of multiple components, such as an image and a category label. Therefore, structures are very suitable for representing data samples or their composite components. DSDL allows users to define structure classes to abstract structures of specified types.

[0159] 4.1 Define a structure class

[0160] In DSDL, users can customize structure classes in the $def section of the dataset description file. The structure class is defined by a JSON object, which contains the following properties:

[0161] $def: Its value must be "struct", indicating that the JSON object is defining a structure class.

[0162] $fields: Its value must be a JSON object containing a series of properties, each of which corresponds to a field. Specifically, in each property of $fields, the key will be used as the field name, and the value will be the type specification of the corresponding field.

[0163] The type specification of a field can be specified in two ways:

[0164] Only specify the data type name: only give the data type name of the field (if the data type is a parameter-containing data type, the parameter will use the default value).

[0165] Specifying parameter data types: Users can specify specific parameter data types through JSON objects, where the $type attribute specifies the name of the type and other attributes are used to set the type parameters. You can view the declaration of the label field in the example.

[0166] 4.2 Nested structures

[0167] In DSDL, structures can be nested. For example, a sample of an object detection task may consist of an image and multiple "local objects", each of which can be represented as a structure containing a bounding box and a category label.

[0168] 4.3 Parameter-containing structure class

[0169] A parameter-containing structure class can be considered as a class template, allowing specified parameters to be set when the class is used. By using a parameter-containing structure class, the present invention can define a class in a more general way.

[0170] 5. Category Domain

[0171] The class domain (class_domain) is a detailed definition of the Label (class label data type) field. The instantiation of the Label field requires that the dom parameter therein be assigned a specific class domain (class_domain).

[0172] 5.1 Defining a Category Domain

[0173] In DSDL, users can define custom category domains in the $def section of the dataset description file. The category domain examples are as follows:

[0174]

[0175] A structure class is defined as a YAML object that contains the following properties:

[0176] $def: Its value must be class_domain, indicating that the YAML object is defining a structure class.

[0177] classes: Its value must be a YAML list object containing a series of class names. It is also a string, and each class name corresponds to a field.

[0178] 5.2 References to Category Domains

[0179] The category domain is mainly used to instantiate the Label field in the structure class struct, for example:

[0180]

[0181]

[0182] Of course, you can also use the parameter $xxx as a placeholder at the beginning, for example:

[0183]

[0184] Then instantiate this parameter in the sample-type in the data section (describing the data set), such as:

[0185]

[0186] 5.3 Defining a Hierarchical Category Domain

[0187] In some data sets, categories have a hierarchical relationship. DSDL allows each category to specify its own parent class. The specific operation is:

[0188] Directly define the subclass class domain, and connect the specific category of the parent class of the label in the form of ". " in front of each label:

[0189]

[0190]

[0191] In the category definition, "." is a special character used to separate the upper and lower relationships of categories. If there are multiple words in each label, it is recommended to connect them with "_", for example: baseball_bat. If you want the category label to have special characters such as spaces, it is recommended to store the original label name containing special characters in global-info and the label mapping relationship after conversion stored in class_dom. The specific format is as follows:

[0192]

[0193]

[0194] 6. Data Module

[0195] A dataset description file can be roughly divided into four parts:

[0196] Header: specifies how the current dataset description file needs to be parsed;

[0197] meta section: provides some meta information of the current dataset;

[0198] defs section: provides some global definitions, such as user-defined class domains and structures;

[0199] data section: describes the sample data in the dataset.

[0200] 6.1 Define specific data sections

[0201] The data section is used to define each specific sample. The data section example is as follows:

[0202]

[0203] A data section consists of three modules:

[0204] sample-type: data type definition, which is usually a structure (struct). If the structure contains parameters, the parameters need to be instantiated. In addition, sample-type can also be any other data type (Label, Image, Int, etc.), because the structure (struct) is also a type of data type.

[0205] sample-path: The storage path of samples. If it is actually a path, the contents of samples are read from the file. If it is $local (in this example), it is read directly from the data.samples field of this file.

[0206] Samples: Saves the sample information of the data set. Note that this field will only take effect when sample-path is $local. Otherwise, samples will be read from the path in sample-path first.

[0207] The fields in data.samples correspond one-to-one to the data types given in data.sample-type.

[0208] 6.2Usage of Optional

[0209] If some fields in the data are missing, for example, some of the data in the data.samples section above have missing labels:

[0210]

[0211] Accordingly, when defining a structure, you need to indicate which fields are missing. The method is to put the field names that can have missing values ​​in the $optional part of the struct type. The $optional field is a list with the following format:

[0212]

[0213] Note: When label is omitted in $optional, the corresponding field value can be missing in data.samples, otherwise a warning will be displayed.

[0214] 6.3 Adding global-info field

[0215] For the global information of the data set, you can add a global-info field in the data module to store it. Global information includes but is not limited to:

[0216] Category information supplement (such as synonyms, word definitions, etc.);

[0217] Data information supplement (such as the global vocabulary included in the OCR task, the key point connection method in the key point task, etc.);

[0218] To store global-info information, you need to define its struct and add corresponding fields in the data module. The following is a specific example of global-info usage.

[0219] First, you need to add the following content to the definition file:

[0220]

[0221] The data module looks like this:

[0222]

[0223]

[0224] You can see that a data section has been added with global-info related information, which mainly includes three modules:

[0225] global-info-type: type definition of global information, which is usually a defined structure (struct). If the structure contains parameters, the parameters need to be instantiated;

[0226] global-info-path: The storage location of global information. If it is actually a path, the content of global-info is read from the file. If it is $local (in this example), it is read directly from the data.global-info field of this file.

[0227] global-info: Saves the specific content of global information. Note that this field will only take effect when global-info-path is $local. Otherwise, it will be read from the path in global-info-path first.

[0228] It is worth noting that the fields in data.global-info correspond one-to-one to the data types given in data.global-info-type.

[0229] 6.4 Reading from external files

[0230] When the amount of data is large, it is recommended to extract the data from the data section of the YAML file and put it in a separate JSON file. In the sample-path field of the data section, give the specific path of the JSON file where the data is stored, expressed as a string. The example is as follows:

[0231]

[0232] Store specific data in samples.json. The example is as follows:

[0233]

[0234]

[0235] Specific global information data is stored in global-info.json. The example is as follows:

[0236]

[0237] 7. Link Library

[0238] The present invention introduces the concept of library to further simplify the process of data set description.

[0239] 7.1 Defining and importing a library

[0240] The class ImageClassification defined in the "Dataset Description File" section is relatively general and can be applied to other datasets. Therefore, it can be extracted into a library file so that the dataset description file can directly import it. In general, a library file is a YAML or JSON file that provides some class definitions.

[0241] In the example of "Dataset Description File", you can create a library file named imageclass.yaml with the following content. This library file should be placed in the same directory as the current yaml file. Users can also manually declare the library directory by specifying the -p parameter when executing the dsdl view and dsdl check commands.

[0242]

[0243] The dataset description file can then be simplified by importing the library, as described below:

[0244]

[0245]

[0246] Here, an $import directive is used in the header part. The content of the $import directive should be a list because users can import multiple library files. When multiple library files are imported and these library files contain several definitions with the same name, the definition imported later will overwrite the definition imported earlier. If this happens, the parser will report a warning message.

[0247] 7.2 Using the Library

[0248] Here are some suggestions for defining a DSDL library: define a parameterized class and give specific parameter settings (such as category domain settings) based on the specific application scenario.

[0249] Putting definitions of the same domain in a library file makes distribution and importing easier.

[0250] Documenting the definitions in your library makes them easier for users to understand.

[0251] The following is an example of placing multiple classes related to visual recognition in one library file.

[0252]

[0253]

[0254] By using this library file, the dataset description file can be simplified to:

[0255]

[0256]

[0257] Because MyClassDom is a specific definition and is only used in the current dataset, it is defined in the dataset description file and a more general class is imported from the library file. In addition to DSDL, this embodiment also provides a standard library named cv, which contains a large number of definitions related to computer vision, including various data types and category domains.

[0258] When the scheme of the present invention is actually applied, corresponding task templates can be directly defined for the currently common task types (image classification, target detection, semantic segmentation, instance annotation (polygon annotation), instance annotation (segmentation map annotation), panoramic segmentation, key point detection, target tracking, rotated target detection, optical character recognition (detection), optical character recognition (segmentation), optical character recognition (recognition), optical character recognition (end-to-end), image generation). At the same time, the data set described by DSDL can be visualized through python code and command line tools, and the model can be trained and inferred through mmdetection and Pytorch framework.

[0259] In summary, the present invention proposes a standardized language DSDL framework, which provides a unified description for multi-task and multimodal AI datasets. Concepts such as unstructured object classes, category domains, and link libraries are proposed, allowing users to define different modalities and task types according to unified specifications. Even DSDL descriptions with diverse styles can be accurately parsed by the DSDL parser. The present invention can predefine data templates for mainstream tasks to simplify the use of datasets, and can also be applied to steps such as dataset visualization and model training and reasoning, with strong scalability.

[0260] The DSDL framework proposed in the present invention has the following advantages:

[0261] Universality. The main purpose of DSDL is to provide a unified representation standard that can cover artificial intelligence data in various fields, rather than being designed based on a specific task or field. DSDL can express data of different modalities and structures in a consistent format.

[0262] Portability. Dataset descriptions can be widely distributed and exchanged and can be used in various environments without modification. Achieving this goal is critical to building a thriving development ecosystem. To this end, the present invention needs to carefully examine the implementation details so that they are not aware of the underlying infrastructure or organization, thereby removing unnecessary dependencies based on specific assumptions.

[0263] Extensibility. The boundaries of expression can be expanded without modifying the core standard. For example, in programming languages ​​such as C++ or Python, the application boundaries can be significantly expanded by using libraries or software packages, while the core syntax can remain stable for a long time. Based on libraries and packages, a rich ecosystem can be formed, allowing the corresponding language to remain active and develop for a long time. < / class-domain> < / class-domain> < / class-domain> < / class-domain>

Claims

1. A method for processing an AI data set, characterized in that: The following steps are involved: Design the Dataset Description Language (DSDL) framework, including dataset description files, basic data types, unstructured object classes, structure classes, category domains, data modules, and link libraries; The dataset description language framework is used to describe the dataset of the current task accordingly.

2. The AI ​​data set processing method according to claim 1, characterized in that: The data set description file is used to describe the data structure, including header, mate section, defs section and data section. The header is used to clarify how the current data set description file needs to be parsed; The mate section is used to provide meta information of the current data set; The defs section is used to provide global definitions; The data section is used to describe the sample data in the data set.

3. The AI ​​data set processing method according to any one of claims 1 to 2, characterized in that: The data set description file is defined in the data interaction language YAML or JSON.

4. The AI ​​data set processing method according to claim 2, characterized in that: The basic data types include general basic data types, special basic data types, category annotation data types, object positioning data types, usage type parameters and list data types. The general basic data types include Bool Boolean type, Int integer type, Num general numeric type and Str string type. The special basic data types include: Coord, that is, 2D coordinates, in the format of [x, y]; Coord3D, that is, 3D coordinates, in the format of [x, y, z]; Interval, that is, the sequence interval, the format is [begin, end]; BBox, that is, bounding box, the format is [x, y, w, h]; RotatedBBox, that is, rotated bounding box, the format is [x1,y1,x2,y2,x3,y3,x4,y4] or [x,y,w,h,angle]; Keypoint, that is, key point, the format is [[x1,y1,visable1],[x2,y2,visable2],...]; Polygon, that is, a polygon is represented by multiple 2D coordinates in the format of [[x1,y1],[x2,y2],...]; Date, that is, the date is represented by a string, and the format follows the rules of Python strftime; Time, that is, time is represented by a string, and the format follows the rules of Python strftime; The category annotation data types include: Based on the class name, the format is " <class-domain> :: <class-name> "；< / class-name> < / class-domain> Based on category index, the format is " <class-domain> :: <class-index> "；< / class-index> < / class-domain> Enter the category name directly in the format of <class-name> ；< / class-name> Directly enter the category index in the format <class-index> ；< / class-index> The object location data type is the Loc data type, which is used to reference unstructured objects; The usage type parameters include the Label data type of the usage type parameter, and the Date and Time data types of the usage type parameter; The list data type includes two parameters, etype and ordered, where etype indicates the data type of a single element in the list, and ordered indicates whether there is an order relationship between the elements in the list. The default value is false. When the sequence is ordered, it is set to true.

5. The AI ​​data set processing method according to claim 3, characterized in that: The unstructured object classes include: Image, each instance is an image, represented by a matrix of pixels; Video, each instance is decoded into a sequence of video frames, each of which is an image; Text, each instance is a sequence of words; PointCloud, each instance is a collection of 3D points used to represent the shape of a 3D object; LabelMap, each instance is a matrix of integer labels, each integer label corresponds to a category.

6. The AI ​​data set processing method according to claim 5, characterized in that: The structure class is used to represent mixed objects, including nested structures and structures with parameters.

7. The AI ​​data set processing method according to claim 6, characterized in that: The category domain is used to define the dom parameter in the Label data type field.

8. The AI ​​data set processing method according to claim 7, characterized in that: The data module includes a sample data definition unit, a field missing processing unit, and a global information supplement unit. The sample data definition unit is used to define the data type, set the sample storage path, and save the sample information of the data set; The missing field processing unit is used to store the missing field names in a setting list; The global information supplement unit is used to define the global information type, set the global information storage location and save the specific content of the global information.

9. The AI ​​data set processing method according to claim 8, characterized in that: The global information includes category information and data information.

10. The AI ​​data set processing method according to claim 9, characterized in that: The link library is used to store common classes in the same field, and the dataset description process is simplified by directly importing the link library.

Images