Data processing method and device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment, and a storage medium, wherein the method comprises: a sampling batch data interface calling a sampling single data interface at least twice to form a batch sample; the batch sample comprises at least two samples after a preprocessing operation; the sampling single data interface receives a data set to be sampled and an index sequence of sampling; the sampling single data interface calls a read data sample interface based on the index sequence of sampling to obtain a sample after the preprocessing operation of a sample sampled from the data set to be sampled.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer vision technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] A deep learning framework typically iterates through a neural network in four steps: data processing, forward computation of the neural network, backward computation of the gradients of the network parameters, and updating the network parameters.

[0003] Although different deep learning frameworks follow basically the same steps for iterating a neural network, the specific implementation and interface of each step are quite different, especially the data processing step. This means that the data processing script of one deep learning framework cannot be directly used for another deep learning framework. Summary of the Invention

[0004] In view of this, embodiments of this application provide at least one data processing method, apparatus, device, and storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] On one hand, embodiments of this application provide a data processing method, the method comprising: calling a batch data interface to a single data interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations; the single data interface receiving a dataset to be sampled and a sampling index sequence; the single data interface calling a read data sample interface based on the sampling index sequence to obtain samples after performing the preprocessing operations on samples sampled from the dataset to be sampled.

[0007] On the other hand, embodiments of this application provide a data processing apparatus, the apparatus comprising: a batch data sampling interface, configured to call a single data sampling interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations; the single data sampling interface is configured to receive a dataset to be sampled and a sampling index sequence; the single data sampling interface is further configured to call a data sample reading interface based on the sampling index sequence to obtain samples after performing the preprocessing operations on samples sampled from the dataset to be sampled.

[0008] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0009] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0010] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the above-described method.

[0011] In another aspect, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0014] Figure 1 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0015] Figure 2 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0016] Figure 3 A schematic diagram illustrating the implementation process of a batch sampling method provided in this application embodiment;

[0017] Figure 4 This is a schematic diagram of the structure of a computer vision algorithm interface model provided in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application;

[0019] Figure 6 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to be limiting of this application.

[0024] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0025] Data processing involves reading data from storage devices into memory and processing it into the input format required by neural networks.

[0026] Forward computation in neural networks involves taking the output of data processing as input and calculating the network's output layer by layer.

[0027] The gradients of the network parameters are calculated in reverse. The backpropagation algorithm is used to calculate the gradients of the parameters based on the output of the forward calculation.

[0028] To update network parameters, an optimization algorithm is used to update the neural network parameters based on the gradient of the parameters.

[0029] Computer vision is a functional unit capable of acquiring, processing, and interpreting visual data.

[0030] Deep learning is the process of learning the inherent patterns and representational layers of data.

[0031] Deep Neural Network Learning is a method of creating rich hierarchical representations by training neural networks with many hidden layers.

[0032] Model training is the process of using training data and machine learning algorithms to build or improve the parameters of a machine learning model.

[0033] Training data is a subset of input data samples used to train machine learning models.

[0034] Sampler: Generates a sequence of samples from the dataset based on the dataset's indices. For example, if the dataset has 5 samples, and these 5 samples are numbered [0,1,2,3,4], the sampler might generate a sequence [3,1,2,0,4], and then sample the dataset based on this sequence.

[0035] Objects: In object-oriented programming, the process of creating an object from a class is usually called instantiation. Unlike functions, classes cannot be used directly after they are defined because a class is just an abstract concept. It is necessary to instantiate the class using keywords (such as `new`) to create an object, and then call the object's properties and methods (such as calling an interface) based on the object.

[0036] An interface is a set of predefined functions. An interface is a collection of methods and properties exposed by an object, which exists to implement a specific function.

[0037] Deep learning frameworks: Frameworks used for deep learning, such as Paddle, Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch.

[0038] Although the steps for iterating a neural network are generally consistent across deep learning frameworks, the specific implementations and interfaces of each step differ significantly, especially in data processing. This means that a data processing script from one deep learning framework cannot be directly used with another. If the interfaces for data processing across different deep learning frameworks could be standardized, a universal data processing script could be provided, thereby saving time spent adapting to different frameworks. This application provides a data processing method that can be executed by a processor of a computer device. The computer device can refer to a server, laptop, tablet, desktop computer, mobile device (e.g., mobile phone, portable video player, personal digital assistant, portable gaming device), or any other device with data processing capabilities.

[0039] Figure 1This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps S101 to S103:

[0040] Step S101: The batch data sampling interface calls the single data sampling interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations;

[0041] Step S102: The sampling single data interface receives the dataset to be sampled and the sampling index sequence;

[0042] Step S103: The sampling single data interface calls the data sample reading interface based on the sampling index sequence to obtain the sample after the preprocessing operation on the sampled sample from the dataset to be sampled.

[0043] In some embodiments, the sampling single data interface calls the data sample reading interface based on the sampling index sequence to obtain a sample after performing the preprocessing operation on the sampled sample from the dataset to be sampled, such as... Figure 2 As shown, steps S201 to S203 are included, wherein:

[0044] Step S201: After receiving the current index in the sampled index sequence, the sampling single data interface calls the read data sample interface;

[0045] Step S202: The data sample reading interface samples the dataset to be sampled based on the current index to obtain the current sample;

[0046] Step S203: The data sample reading interface performs preprocessing operations on the current sample.

[0047] In some embodiments, the method further includes: the sampler determining the current index sequentially according to the sampled index sequence, and recording the current index.

[0048] In some embodiments, the method further includes: constructing a sampler interface based on sampler configuration parameters to generate the sampler (also referred to as a sampler object).

[0049] The sampler configuration parameters include at least one of the following: batch size, whether to shuffle the dataset, whether to start again after reading the dataset, and whether to discard the last batch of data after finishing reading (drop_last_batc).

[0050] In some embodiments, the method further includes: the constructing sampler interface generating the sampling index sequence based on the sampler configuration parameters and the dataset to be sampled.

[0051] In some embodiments, the data sample reading interface performs preprocessing operations on the current sample, including at least one of the following: sample loading operation, image preprocessing operation, and data format conversion operation.

[0052] In some embodiments, the samples after the preprocessing operation meet the input format required by the deep learning module.

[0053] In some embodiments, the sampling batch data interface obtains the batch size;

[0054] The sampling batch data interface determines the number of samples obtained after the preprocessing operation.

[0055] If the number of samples after the preprocessing operation is less than the batch size, the sampling batch data interface continues to call the sampling single data interface to obtain the samples after the preprocessing operation until the number of samples after the preprocessing operation is equal to the batch size.

[0056] In some embodiments, the method further includes: a dataset builder interface for constructing the dataset to be sampled based on the received dataset path.

[0057] In some embodiments, the method further includes: the dataset builder interface returning the constructed dataset to be sampled to the constructor sampler interface, the data sample reading interface, the sample single data interface, and the sample batch data interface.

[0058] In some embodiments, the dataset builder interface constructs the dataset to be sampled based on the received dataset path, including: the dataset builder interface constructs the dataset to be sampled (also referred to as a dataset object) based on the dataset path and parameters related to the dataset to be constructed; wherein, the parameters related to the dataset to be constructed include at least one of the following: whether the samples in the dataset to be constructed have been labeled, and the attributes of the dataset to be constructed.

[0059] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0060] The data processing interface should support the conversion of data into formats required by deep learning modules (such as neural network models), such as tensors. The data processing interface should have both outer and inner interfaces. The deep learning framework should be able to prepare continuous training data for algorithm training in an iterative manner. This data processing interface should have the following core functionalities:

[0061] a) Supports the implementation of an iterable sampler, with each iteration returning a batch of data;

[0062] b) Supports sampling data from the dataset according to training requirements;

[0063] c) Supports constructing dataset objects based on dataset paths and dataset-related parameters;

[0064] d) Supports reading part or all of the dataset data from storage devices or services, such as annotation files, data samples, etc.;

[0065] e) Supports preprocessing operations on data, such as image scaling, flipping, and color perturbation.

[0066] The parameters related to the dataset include at least one of the following: whether the samples in the dataset are labeled, and the attributes of the dataset, wherein the attributes of the dataset include whether the dataset is training data or inference data.

[0067] Figure 3 This is a flowchart of the batch sampling process. The batch sampling data interface combines multiple batch samples by calling the single sample sampling data interface, that is, it samples a single sample multiple times. The method for sampling a single sample includes the following steps S311 to S313:

[0068] Step S311: Determine the current index.

[0069] Here, the current index is determined by sampling a single data interface from a sampled index sequence. This sampled index sequence is generated during the sampler generation process.

[0070] Step S312, sample loading operation.

[0071] Here, the data sample reading interface is used to load the sample corresponding to the current index from the dataset stored on the server into memory.

[0072] Step S313, image preprocessing operation.

[0073] Here, the sample corresponding to the current index is preprocessed in memory using the data sample reading interface. In the method of sampling a single sample, firstly, the current index to be read can be obtained using the sample single data interface, then the data sample reading interface is called to read the sample data from the storage device into memory, and finally, the data preprocessing operation set in the data sample reading interface is called to process the data into the format required by the neural network.

[0074] Compared to sampling a single sample (e.g.) Figure 3 (The process section indicated by number 31), for sampling batches of samples (such as...) Figure 3 The method (represented by reference numeral 32) may also include the following step S314:

[0075] Step S314: Determine whether the number of samples after preprocessing is equal to the batch size.

[0076] Here, if the number of samples after preprocessing is less than the batch size, the sampling batch data interface continues to call the sampling single data interface to obtain samples after preprocessing until the number of samples after preprocessing is equal to the batch size.

[0077] This application defines a specification for data processing interfaces (e.g., dataset builder interface, sampler builder interface, data sample reader interface, single data sampler interface, and batch data sampler interface), which is an interface specification for visual training, including data reader and sampler.

[0078] A.1 Data Reading

[0079] Data reading involves retrieving data samples from storage devices into memory and processing them into the input format required by the neural network. Data reading includes two interfaces: a dataset builder interface and a data sample reading interface.

[0080] A.1.1 Dataset Builder Interface

[0081] The dataset builder interface returns a dataset object (i.e., the dataset to be sampled), which can be passed to the constructor sampler interface as input to the constructor sampler interface.

[0082] a) Interface name: construct_dataset(path, other-parameter, dataset).

[0083] b) Interface Function Description:

[0084] A dataset object can be constructed based on the dataset path and related parameters.

[0085] c) Interface parameter list A.1.1:

[0086] Parameter type Keywords Parameter Description Is it optional? enter path Dataset path. Type: string. Required enter Other parameters Other parameters related to the dataset Optional Output dataset Dataset object Required

[0087] d) Interface exception handling:

[0088] No errors: Operation successful.

[0089] Error: File not found: Dataset path does not exist.

[0090] e) Other additional notes: None.

[0091] A.1.2 Data Sample Reading Interface

[0092] The data sample reading interface can be called by the sampler by iterating through individual data interfaces. After obtaining the current index using a single data interface, the data sample reading interface can be called. The data sample reading interface can load samples into memory and perform preprocessing operations on the loaded samples.

[0093] a) Interface name: get_sample(dataset,index,sample).

[0094] b) Interface Function Description:

[0095] Load the sample corresponding to the current index and perform preprocessing operations. This needs to be completed within the data sample reading interface, including but not limited to sample loading, preprocessing, and data format conversion. Sample loading refers to loading the image corresponding to the index. Preprocessing steps involve processing the image, including adjusting its size and brightness. Data format conversion can refer to converting image-type data into tensors.

[0096] c) Interface parameter list:

[0097]

[0098] d) Interface exception handling:

[0099] No errors: Operation successful.

[0100] File not found error: The file corresponding to the sample does not exist.

[0101] e) Other additional notes: None.

[0102] A.2 Sampler

[0103] The sampler defines three interfaces: a sampler construction interface, a sampler traversal interface for a single data set, and a sampler traversal interface for a batch of data sets.

[0104] A.2.1 Constructing the Sampler Interface

[0105] The Sampler constructor interface can receive the output of the dataset constructor as input to generate a sampler object that maintains an index sequence for accessing dataset samples.

[0106] a) Interface name: construct_sampler(sampler_config, dataset, sampler)

[0107] b) Interface Function Description:

[0108] The corresponding sampler is constructed based on sampler_config. The sampler object is constructed based on sampler_config (sampler configuration parameters) and dataset object. The sampler object carries an index sequence and sampling status, which can be understood as the current sampling of a certain sample.

[0109] c) Interface parameter list:

[0110]

[0111] d) Interface exception handling:

[0112] No errors: Operation successful.

[0113] e) Other additional notes: None.

[0114] A.2.2 Sampling a single data interface

[0115] This sampling single data interface first obtains the index of the sample that needs to be iterated, and then calls the data reading sample loading and processing function.

[0116] a) Interface name: iterate_sample(dataset, sampler, sample).

[0117] b) Description of the iterate_sample interface:

[0118] It iterates through the data samples in a predetermined order and returns a single sample. Based on the sampling state of the sampler object, it determines the index of the sample to be iterated over from the index sequence, and inputs this sample index and the dataset object into the data sample reading interface.

[0119] c) Interface parameter list:

[0120]

[0121] d) Interface exception handling:

[0122] No errors: Operation successful.

[0123] e) Other additional notes: None.

[0124] A.2.3 Sampling Batch Data Interface

[0125] This sampler iterates through the batch data interface. The sampler calls the sampler to iterate through a single data interface multiple times and returns a batch of samples for training.

[0126] a) Interface name: itrate_batch(sampler, batch).

[0127] b) Description of the iterate_batch interface:

[0128] Iterate through the data samples in a predetermined order and return a batch of samples.

[0129] Based on the sampling state of the sampler object, multiple indices that need to be iterated over can be determined from the index sequence. Based on these multiple indices, the sampler is called multiple times to traverse a single interface, obtaining batch data. The batch size is one of the parameters specified in the `sampler_config` when the sampler object is generated.

[0130] c) Interface parameter list:

[0131]

[0132] d) Interface exception handling:

[0133] No errors: Operation successful.

[0134] e) Other additional notes: None.

[0135] In this embodiment, the batch sampling data interface includes a single sampling interface, meaning the batch sampling data interface allows for multiple calls to the single sampling interface. For example, each sample corresponds to an index, and a sequence of sampling indices is generated simultaneously with the sampler object. For instance, 3, 2, 8, 4, 6. Therefore, the third data sample is collected first, followed by the second, and then the eighth.

[0136] In this embodiment, the data processing interface may include: a dataset builder interface, a data sample read interface, a sampler builder interface, a sampler traversal interface for a single data sample, and a sampler traversal interface for a batch of data. First, the dataset builder interface and the sampler builder interface can be called. Then, the batch data traversal interface is called, followed by the single data traversal interface. Finally, the data sample read interface is called.

[0137] For example: A dataset object is created using the dataset constructor interface, and a sampler object is created using the sampler constructor interface (when creating a sampler object, an index sequence is generated, and the sampler object records the sampling state. Later, based on the sampling state recorded by the sampler, one or more indices of the current iteration can be read sequentially from this index sequence, and the corresponding data sample can be determined based on this index); the sampler iterates through the batch data interface to determine multiple indices from the index sequence (i.e., the indices corresponding to each batch, such as the first batch including [1.3], the second batch including [2.4], etc.); according to the order of multiple indices, the sampler iterates through the single data interface sequentially and calls the data sample reading interface; (e.g., first sample data sample 1, then sample data sample 3, and combine data samples 1 and 3 into the first batch, etc.) so that the data sample reading interface loads the samples sequentially into memory, processes them, and obtains the processed batch samples. (For example, data sample 1 can be loaded from local storage into pre-allocated memory. In memory, data sample 1 can be resized and converted from vector to tensor to obtain a pre-processed data sample. Then, data sample 3 can be processed. Finally, the two processed data samples can be combined into a batch. This can be used for direct prediction or training in deep learning frameworks, which can promote innovation, improve efficiency, and reduce the adaptation cost for consumers.)

[0138] As can be seen from the above embodiments, the present application implements a data reading specification in the data processing script, thereby allowing the same data reading script to be used for data reading steps in different deep learning frameworks; while in related technologies, different frameworks have different processing for data reading, resulting in the need to develop different data reading scripts for different deep learning frameworks.

[0139] This application implements a sampler specification in the data processing script, which allows the same sampling script to be used for sampling steps in different deep learning frameworks; whereas in related technologies, different frameworks have different processing for data sampling, which results in the need to develop different sampling scripts for different deep learning frameworks.

[0140] As can be seen from this, the unified specifications for data reading and sampling in this application embodiment facilitate the sharing of data processing scripts among different deep learning frameworks.

[0141] For example, a user can directly use a script from one deep learning framework to process a dataset in another deep learning framework; for instance, a script from the PyTorch deep learning framework for processing the ImageNet dataset can be used in the MindSpore deep learning framework. The methods, apparatuses, and other embodiments provided in this application can be used in training platforms that support different deep learning frameworks.

[0142] To better understand the embodiments of this application, the computer vision interface model is briefly described below:

[0143] 1.1 Algorithm Interface Objectives

[0144] By defining the interface adaptation layer, the following goals can be achieved: a) the algorithm implementation can use different deep learning frameworks as backends and can be switched; b) it can run on different hardware, such as servers and distributed clusters; c) the algorithm can read different data through a unified data interface.

[0145] 1.2 Algorithm Interface Model

[0146] The relationship between data, algorithms, and models is described from three levels: system resources, interface adaptation, and algorithm application. For example... Figure 4 The computer vision algorithm interface model shown in the table mainly specifies the technical requirements of the interface adaptation layer: a) System Resource Layer 401: Includes both hardware and software, providing the necessary storage, computation, and inference functions for the computer vision system. The specific implementation of the system resource layer varies among vendors, and this standard does not further describe or specify it; b) Interface Adaptation Layer 402: Provides service interfaces such as data interfaces, optimization interfaces, distributed interfaces, and model interfaces for the upper-layer system processes, ensuring efficient and flexible model training and model migration between different frameworks; c) Algorithm Application Layer 403: Completes the model training and model inference processes, requiring the use of various interfaces defined in this standard during model training.

[0147] Computer vision system interface requirements (requirements include mandatory and optional options; appropriate usage is recommended).

[0148] 2.1 Data and Model Structure

[0149] 2.1.1 Data Structure

[0150] Datasets can be in formats such as images, videos, and binary data. The dataset annotation file is in JSON format and contains annotation information for all samples in the dataset. If the annotation results include other auxiliary files, such as mask layer information, the relative paths of these auxiliary files are stored in the JSON file. Common data types are represented as follows:

[0151] Supports category labels, such as integers, where 0 represents background and positive integers represent foreground; supports bounding boxes, such as using the coordinates of the top-left and bottom-right vertices in the order (top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, bottom-right y-coordinate); supports annotation files being parsed into lists or arrays after passing through the data reading interface, where each element is a dictionary or key-value pair container containing all relevant information for a sample, accessible by the dataset using an index; supports loading specific dataset formats and custom datasets; supports annotation files in mainstream open-source dataset formats, such as COCO and PASCAL VOC.

[0152] 2.1.2 Model Structure

[0153] a) Basic operators should be supported, including but not limited to "+", "-", "*", " / ", convolution operations, etc.; b) The meaning and values ​​of the parameters of the operator should be defined, and the computational logic for obtaining the output from the input during forward operation should be defined; c) Basic operators should support the construction of computational graphs through operator chaining and function nesting; d) The computational graph should support differentiation through backpropagation using the chain rule; e) The computational graph should support construction through conditional judgments or loops; f) User-defined operators should be compiled and allowed to be added to the computational graph; g) Serialization and deserialization of model parameters should be supported.

[0154] 2.2 Training Interface

[0155] 2.2.1 Optimizer Interface

[0156] The optimizer interface should support network model optimization, updating model parameters according to different optimization algorithms such as SGD, Adam, and Momomtum. Specifically, it should implement functions such as gradient calculation and backpropagation, parameter updates, optimization algorithms, and learning rate updates.

[0157] The optimizer interface should implement: a) support implementation as a class whose constructor parameters are the network model or a list of model parameters, as well as other required parameters, such as the learning rate; b) support implementation of the `step()` function for performing a single parameter optimization update. After this function is called, the model parameters should be updated based on the accumulated gradients.

[0158] 2.2.2 Mixed Precision Training Interface

[0159] The mixed-precision training interface provides unified support for mixed-precision training of algorithms, which can reduce memory consumption and improve training speed when the graphics card supports it. This interface should implement the following functions:

[0160] a) Supports precision conversion, converting model parameters to fp16, except for special layers (such as BN layers), while retaining a copy of the fp32 parameters; b) Supports input forward propagation, converting input data to fp16 for forward propagation and loss calculation; c) Supports loss amplification, amplifying the calculated loss with both fixed and dynamic amplification modes; d) Supports gradient calculation, calculating and backpropagating gradients in fp16 mode, then converting them to fp32 and scaling them back to the actual scale proportionally according to the amplification factor in c); e) Supports parameter update, updating parameters in the fp32 parameter copy based on the gradient calculated in d), and then assigning the updated parameters to the fp16 model.

[0161] 2.2.3 Distributed Interface

[0162] Through a distributed interface, the framework can complete the data transfer between multiple processes in a distributed training scenario with multiple machines and multiple GPUs. This interface should have the following core functionalities:

[0163] The distributed interface should cover functions such as `bcast()`, `reduce()`, `scatter()`, `gather()`, `allreduce()`, `allgateher()`, and `sync()`. This set of interfaces should implement the following: support broadcasting data from the main process to each process; support reducing data from each process to the main process; support scattering a set of data from the main process to each process; support collecting data scattered across each process into the main process as a set; support reducing data from each process and then broadcasting it to each process; support collecting scattered data from each process into a set and then broadcasting it to each process; and support ensuring that all previously issued communication commands have been completed.

[0164] 2.2.4 Quantization Training Interface

[0165] The quantization training interface enables algorithms to perceive the information loss caused by model quantization during neural network training. Specifically, during training, quantized weights are approximated using floating-point weights, allowing the quantized model to be simulated during forward propagation. The floating-point error is then calculated and backpropagated to update the weights. Quantization helps accelerate model inference and reduce storage requirements. This interface should implement the following functionalities:

[0166] Supports input quantization, converting input from 32-bit floating-point type to 8-bit or custom-bit fixed-point type; supports quantization and dequantization of convolution and addition operators; supports pseudo-quantization nodes, which should include functions for quantizing and dequantizing floating-point weights; supports error backpropagation, calculating the error through b) during forward propagation, updating the floating-point weights, and then quantizing; supports quantized model output, and the quantized model can be converted into a fixed-point model for storage.

[0167] 2.2.5 Data Processing Interface

[0168] The data interface should support the conversion of data into formats required by the module, such as tensors. The data interface should have both outer and inner interfaces. The framework should be able to iteratively prepare continuous training data for algorithm training. This interface should have the following core functionalities:

[0169] Supports the implementation of an iterable data loader type, with each iteration returning a batch of data;

[0170] It supports sampling data from the dataset according to training requirements; it supports constructing dataset objects based on the dataset path and related parameters; it supports reading part or all of the dataset data from storage devices or services, such as annotation files and data samples; and it supports preprocessing operations on the data, such as image scaling, flipping, and color perturbation.

[0171] 2.2.6 Visual Interface

[0172] The training visualization interface provides visualizations of model structure, parameters, gradients, and features during algorithm training. This interface should support: visualization of model structure diagrams; visualization of feature maps; visualization of weight histograms; visualization of scalar changes; and visualization of convolutional kernels.

[0173] 2.2.7 Distillation Interface

[0174] The distillation interface supports the use of a teacher network to guide the student network during training, thereby improving the training accuracy of the student model: it should support target distillation method; it should support feature distillation method.

[0175] 2.2.8 Graphical-Computational Fusion Interface

[0176] Graph-computation fusion optimizes the overall network execution time by analyzing and optimizing the existing network computation graph logic, reducing overhead during operator execution intervals, and improving the utilization of device computing resources. It supports operations such as splitting, reorganizing, and merging existing computation logic; and allows enabling graph-computation fusion by modifying the context parameter in the training script.

[0177] 2.3 Inference Interface

[0178] 2.3.1 Process Orchestration Interface

[0179] The computer vision system supports workflow orchestration, and its interface meets the following requirements:

[0180] It should be able to combine key processes such as image acquisition, image decoding, image scaling, object detection, image cropping, image classification, and serialization;

[0181] It is advisable to support the plug-in approach for key processes, with configurable properties for each plug-in.

[0182] Users should be able to mount metadata.

[0183] It should support configuration file-based process orchestration management and have management components;

[0184] It should support specifying accelerators for particular processes;

[0185] It should support orchestration of multiple request and multiple output processes;

[0186] The following models should be supported for orchestration: YOLOv3, YOLOv3-tiny, ResNet50, Faster R-CNN, YOLOv4, SSD-VGG16, SSD MobileNet v1 FPN, CRNN, YOLOv5, Faster R-CNN-FPN / Cascade R-CNN-FPN, ResNet-18, DeepLabv3+, CTPN, DeepLabv3, BERT-Base (Uncased), DeepLabv3+, U-Net, Mask R-CNN, FaceNet, SSD MobileNet v1 FPN, OpenPose, Unet++, RetinaNet;

[0187] It should support single-input, single-output, multiple-input, and multiple-output orchestration.

[0188] 2.3.2 Data Processing Interface

[0189] The computer vision system has a data processing interface that meets the following requirements:

[0190] It should support reading data from image files and moving it into a pre-configured cache;

[0191] It should support JPG / JPEG / BMP format image decoding, with a resolution range of (32*32, 8192*8192);

[0192] It should at least support JPG image encoding, with a resolution range of (32*32, 8192*8192);

[0193] Image scaling with specified target width and height should be supported, and image width and height should be scaled and aligned to the step size.

[0194] It should support specifying the expansion ratio in the four directions (top, bottom, left, and right) to expand the area of ​​the target bounding box for cropping.

[0195] It should support H264 / H265 video decoding, with a resolution range of (128*128, 4096*4096).

[0196] Width and height scaling alignment should be supported for step-based scaling.

[0197] It should support resolutions ranging from 128*128 to 1920*1920, and both H264 MP and H265 MP.

[0198] Image normalization, center cropping, affine transformation, and rotation should be supported;

[0199] It should support data transfer between key processes and preferably support the multiple distribution of a single input;

[0200] It should support data transfer between processor memory and main memory;

[0201] It should support frame skipping processing of video data;

[0202] Serialization should be supported.

[0203] 2.3.3 Plug-in Interface

[0204] The computer vision system should support the development and use of visual processing plugins and meet the following requirements: it should support user-developed plugins, registration, and compilation; it should support interfaces for plugin initialization, deinitialization, execution, attribute registration, and retrieval; it should support interfaces for defining variable and immutable ports for plugin input and output; and it should support interfaces for defining and throwing business logic exceptions.

[0205] It should support streaming plugin interfaces to achieve the following functions: sending data of a specified category or channel to different ports; sequentially outputting input data from multiple ports through a single port; supporting multiple instantiation interfaces for plugins of the same type; supporting plugin caching mechanisms and interfaces to enable the transfer of business data (such as decoded video and image data) between plugins; supporting plugin metadata (such as classification information and target information) description interfaces, and implementing the transfer through plugin caching; supporting single-input, single-output, multi-input, and multi-output plugin interfaces; supporting inference plugin interfaces to support target classification, detection, and tensor-based (input) inference; and supporting model post-processing plugin interfaces to interface with models for target detection, classification, semantic segmentation, text generation, text box detection, pose detection, etc.

[0206] The system should support video analytics plugin interfaces to implement the following functions: multi-target (including human, non-human, and facial) path recording; face alignment (correcting detected facial images); video quality diagnosis; and debugging plugin interfaces to enable data export (e.g., JSON format) and data loading and restoration. Screen display plugin interfaces should also be supported to enable drawing basic units on images, such as drawing frames, lines, circles, and writing text.

[0207] 2.3.4 Block Detection Interface

[0208] The computer vision system supports a block detection interface and meets the following requirements:

[0209] It supports filtering duplicate targets in overlapping areas after segmentation; it supports user-defined parameters such as the number / size of segments and overlap, and automatically generates target boxes for image segments; it supports merging images of segmented inference results; and during multi-level inference, it supports filtering post-processing results based on the selection of maximum and minimum area, upper and lower area limits, and confidence thresholds.

[0210] A Module is a basic module in a neural network. Neural network modules build graphs based on this base class.

[0211] The Module provides Function 1: Forward Computation: a) Interface Name: forward; b) Interface Function Description: The module performs a forward computation and returns the computation result. If in training mode, a computation graph is constructed during the forward computation process to calculate the gradient of the module parameters.

[0212] Module provides function 2: Get trainable parameters of the module: a) Interface name: get_parameters; b) Interface function description: Returns the trainable parameters of the module.

[0213] The Module class provides function 3: Retrieving the module and its submodules: a) Interface name: get_modules; b) Interface function description: Optional. This interface returns an iterator that iterates through the module itself and its submodules, returning only one instance of a module.

[0214] Module provides function 4: Get module state: a) Interface name: get_state_dict; b) Interface function description: Returns the module state in key-value pairs, including the module parameters and buffer.

[0215] Module provides function 5: Load module state: a) Interface name: load_state_dict; b) Interface function description: Load the state of the module, including the module's parameters and buffers.

[0216] The Module provides Function 6: Module Backward Computation: a) Interface Name: backward(grad_input, grad_output); b) Interface Function Description: The module performs a backward computation and returns the computation result. If in training mode, the gradients of the module parameters are calculated during the backward computation process. This function is automatically generated by the computation graph and can also be registered later using register_backward_function.

[0217] Module provides function 7: Module forward computation: a) Interface name: register_backward_function; b) Interface function description: The module performs one backward computation and returns the module's computation result. If in training state, the gradient of the module parameters is calculated during the backward computation process.

[0218] Based on the foregoing embodiments, this application provides a data processing device, which includes various units and modules included in each unit. It can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0219] Figure 5 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application, as shown below. Figure 5 As shown, the data processing apparatus 500 includes:

[0220] The batch data sampling interface 510 is used to call the single data sampling interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations.

[0221] The sampling single data interface 520 is used to receive the dataset to be sampled and the sampling index sequence;

[0222] The sampling single data interface is also used to call the read data sample interface based on the sampling index sequence to obtain the sample after the preprocessing operation is performed on the sample sampled from the dataset to be sampled.

[0223] In some embodiments, the sampling single data interface is used to receive the current index in the sampling index sequence and then call the read data sample interface; the read data sample interface is used to sample the dataset to be sampled based on the current index to obtain the current sample; the read data sample interface is used to perform preprocessing operations on the current sample.

[0224] In some embodiments, the apparatus further includes a sampler configured to determine the current index sequentially according to the sampled index sequence and record the current index.

[0225] In some embodiments, the apparatus further includes: constructing a sampler interface for generating the sampler based on sampler configuration parameters, the sampler configuration parameters including at least one of the following: batch size, whether to shuffle the dataset, whether to repeat the start after reading the dataset, and whether to discard the last batch of data after finishing reading (drop_last_batc).

[0226] In some embodiments, the apparatus further includes: the construction sampler interface, for generating the sampling index sequence based on the sampler configuration parameters and the dataset to be sampled.

[0227] In some embodiments, the preprocessing operation on the current sample includes at least one of the following: sample loading operation, image preprocessing operation, and data format conversion operation.

[0228] In some embodiments, the samples after the preprocessing operation meet the input format required by the deep learning module.

[0229] In some embodiments, the sampling batch data interface obtains the batch size;

[0230] The sampling batch data interface is used to determine the number of samples obtained after preprocessing. The sampling batch data interface is used to continue calling the sampling single data interface to obtain samples after preprocessing if the number of samples after preprocessing is less than the batch size, until the number of samples after preprocessing is equal to the batch size.

[0231] In some embodiments, the apparatus further includes a dataset builder interface for constructing the dataset to be sampled based on a received dataset path.

[0232] In some embodiments, the apparatus further includes: the dataset builder interface, configured to return the constructed dataset to be sampled to the constructor sampler interface, the read data sample interface, the sample single data interface, and the sample batch data interface.

[0233] In some embodiments, the dataset builder interface is used to construct the dataset to be sampled based on the received dataset path and parameters related to the dataset to be constructed; wherein the parameters related to the dataset to be constructed include at least one of the following: whether the samples in the dataset to be constructed are labeled, and the attributes of the dataset to be constructed.

[0234] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0235] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0236] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0237] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0238] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0239] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0240] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0241] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0242] It should be noted that, Figure 6 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 6 As shown, the hardware entity of the computer device 600 includes: a processor 601, a communication interface 602, and a memory 603, wherein:

[0243] Processor 601 typically controls the overall operation of computer device 600.

[0244] Communication interface 602 enables computer devices to communicate with other terminals or servers via a network.

[0245] The memory 603 is configured to store instructions and applications executable by the processor 601, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) of the processor 601 and various modules in the computer device 600. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 601, the communication interface 602, and the memory 603 can be performed via bus 604.

[0246] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0247] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0248] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0249] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0250] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0251] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0252] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0253] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applied to a data processing interface, the data processing interface includes: a dataset builder interface, a data sample reading interface, a sampler construction interface, a single data sampling interface, and a batch data sampling interface, the method including: The dataset builder interface constructs the dataset to be sampled based on the received dataset path; The interface for constructing a sampler generates a sampler based on sampler configuration parameters, and generates a sampling index sequence based on the sampler configuration parameters and the dataset to be sampled; The batch sampling data interface calls the single sampling data interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations; the single sampling data interface receives the dataset to be sampled and the sampling index sequence; The sampling single data interface calls the data sample reading interface based on the sampling index sequence to obtain the sample after the preprocessing operation on the sampled sample from the dataset to be sampled.

2. The method according to claim 1, characterized in that, The sampling single data interface calls the data sample reading interface based on the sampling index sequence to obtain a sample after performing the preprocessing operation on the sampled sample from the dataset to be sampled, including: After receiving the current index from the sampled index sequence, the sampling single data interface calls the data sample reading interface; The data sample reading interface samples the dataset to be sampled based on the current index to obtain the current sample; The data sample reading interface performs preprocessing operations on the current sample.

3. The method according to claim 2, characterized in that, The method further includes: The sampler determines the current index sequentially according to the sampled index sequence and records the current index.

4. The method according to claim 1, characterized in that, The method further includes: The sampler configuration parameters include at least one of the following: Batch size (batch_size), whether to shuffle the dataset, whether to start again after reading the dataset, and whether to discard the last batch of data after finishing reading (drop_last_batc).

5. The method according to claim 2 or 3, characterized in that, The data sample reading interface performs preprocessing operations on the current sample, including at least one of the following: Sample loading, image preprocessing, and data format conversion operations.

6. The method according to any one of claims 1 to 4, characterized in that, The samples after preprocessing meet the input format requirements of the deep learning module.

7. The method according to any one of claims 1 to 4, characterized in that, The sampling batch data interface obtains the batch size; The sampling batch data interface determines the number of samples obtained after the preprocessing operation. If the number of samples after the preprocessing operation is less than the batch size, the sampling batch data interface continues to call the sampling single data interface to obtain the samples after the preprocessing operation until the number of samples after the preprocessing operation is equal to the batch size.

8. The method according to claim 7, characterized in that, The method further includes: The dataset builder interface returns the constructed dataset to be sampled to the sampler builder interface, the data sample reading interface, the single data sampling interface, and the batch data sampling interface.

9. The method according to claim 1, characterized in that, The dataset builder interface constructs the dataset to be sampled based on the received dataset path, including: The dataset builder interface constructs the dataset to be sampled based on the dataset path and parameters related to the dataset to be constructed. The parameters related to the dataset to be constructed include at least one of the following: Whether the samples in the dataset to be constructed have been labeled, and the attributes of the dataset to be constructed.

10. A data processing apparatus, characterized in that, The device is applied to a data processing interface, which includes: a dataset builder interface, a data sample reading interface, a sampler construction interface, a single data sampling interface, and a batch data sampling interface. The dataset builder interface is used to construct the dataset to be sampled based on the received dataset path; The sampler construction interface is used to generate a sampler based on sampler configuration parameters, and to generate a sampling index sequence based on the sampler configuration parameters and the dataset to be sampled; The batch data sampling interface is used to call the single data sampling interface at least twice to form a batch sample; the batch sample includes samples after at least two preprocessing operations. The sampling single data interface is used to receive the dataset to be sampled and the sampling index sequence; The sampling single data interface is also used to call the reading data sample interface based on the sampling index sequence to obtain the sample after performing the preprocessing operation on the sample sampled from the dataset to be sampled.

11. The apparatus according to claim 10, characterized in that, The sampling single data interface is used to receive the current index in the sampled index sequence and then call the read data sample interface; The data sample reading interface is used to sample the dataset to be sampled based on the current index to obtain the current sample; The data sample reading interface is used to perform preprocessing operations on the current sample.

12. The apparatus according to claim 11, characterized in that, The device further includes: A sampler is used to determine the current index sequentially according to the sampled index sequence and record the current index.

13. The apparatus according to claim 10, characterized in that, The device further includes: The sampler configuration parameters include at least one of the following: Batch size (batch_size), whether to shuffle the dataset, whether to start again after reading the dataset, and whether to discard the last batch of data after finishing reading (drop_last_batc).

14. The apparatus according to claim 11 or 12, characterized in that, The preprocessing operation on the current sample includes at least one of the following: Sample loading, image preprocessing, and data format conversion operations.

15. The apparatus according to any one of claims 10 to 13, characterized in that, The samples after preprocessing meet the input format requirements of the deep learning module.

16. The apparatus according to any one of claims 10 or 13, characterized in that, The sampling batch data interface obtains the batch size; The sampling batch data interface is used to determine the number of samples obtained after the preprocessing operation. The sampling batch data interface is used to continue calling the sampling single data interface to obtain samples after the preprocessing operation if the number of samples after the preprocessing operation is less than the batch size, until the number of samples after the preprocessing operation is equal to the batch size.

17. The apparatus according to claim 16, characterized in that, The device further includes: The dataset constructor interface is used to return the constructed dataset to be sampled to the constructor sampler interface, the read data sample interface, the sample single data interface, and the sample batch data interface.

18. The apparatus according to claim 10, characterized in that, The dataset builder interface is used to construct the dataset to be sampled based on the received dataset path and parameters related to the dataset to be constructed. The parameters related to the dataset to be constructed include at least one of the following: Whether the samples in the dataset to be constructed have been labeled, and the attributes of the dataset to be constructed.

19. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 9.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images